We Keep Coding

How to resolve 'protocol must not be null' for CopySnapshotRequest across regions

I am attempting to move an EC2 snapshot from one region to another.
When creating a basic Ec2 snapshot copy and copying it to a second Region I get the error protocol must not be null. It appears that the host is also null when you get to the point in the code that will require a host. From my understanding I do not need to provide the protocol (http/https) or the host details when copying a snapshot the is NOT encrypted. It appears that the AWS code should do this behind the scenes.
Off the top of my head this seems like an aws issue but there is not much feedback on the AWS SDK for Java V2 yet as I have checked here as well as github.
Program exception:
java.lang.NullPointerException: protocol must not be null
I have tried adding the destinationRegion to the builder but that also results in the same error.
Execution environment is Amazon Java 11.0.3
I have tried with software.amazon.awssdk versions 2.7.11 all the way through 2.7.29
Basic code snippet
String amazonAccessKeyId = "amazonAccessKeyId";
String amazonSecretKeyId = "amazonSecretKeyId";
String baseRegionName = "baseRegionName"; // Region.AP_NORTHEAST_1.id(); or where ever your snapshot is located
String remoteRegionName = "remoteRegionName "; // Region.AP_NORTHEAST_1.id(); or where ever your snapshot is located
String snapshotId = "snapshotId"; // You will need a snapshot to work with this code
// Setup AWS remote client with credentials
AwsCredentials credentials = AwsBasicCredentials.create(amazonAccessKeyId, amazonSecretKeyId);
Ec2Client amazonEc2RemoteRegionClient = Ec2Client.builder()
.region(Region.of(remoteRegionName))
.credentialsProvider(StaticCredentialsProvider.create(credentials))
.build();
// Setup request
CopySnapshotRequest request = CopySnapshotRequest.builder()
.sourceRegion(Region.of(baseRegionName).id())
.sourceSnapshotId(snapshotId)
.description("Foo Bar Testing...")
.build();
// Invoke copy from remote region to pull in snapshot from source/default region
// This is the line in question
CopySnapshotResponse result = amazonEc2RemoteRegionClient.copySnapshot(request);
Stack trace follows
java.lang.NullPointerException: protocol must not be null.
at software.amazon.awssdk.utils.Validate.paramNotNull(Validate.java:117)
at software.amazon.awssdk.http.DefaultSdkHttpFullRequest.standardizeProtocol(DefaultSdkHttpFullRequest.java:63)
at software.amazon.awssdk.http.DefaultSdkHttpFullRequest.<init>(DefaultSdkHttpFullRequest.java:52)
at software.amazon.awssdk.http.DefaultSdkHttpFullRequest.<init>(DefaultSdkHttpFullRequest.java:41)
at software.amazon.awssdk.http.DefaultSdkHttpFullRequest$Builder.build(DefaultSdkHttpFullRequest.java:331)
at software.amazon.awssdk.http.DefaultSdkHttpFullRequest$Builder.build(DefaultSdkHttpFullRequest.java:170)
at software.amazon.awssdk.services.ec2.transform.internal.GeneratePreSignUrlInterceptor.modifyHttpRequest(GeneratePreSignUrlInterceptor.java:102)
at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.modifyHttpRequestAndHttpContent(ExecutionInterceptorChain.java:99)
at software.amazon.awssdk.core.client.handler.BaseClientHandler.runModifyHttpRequestAndHttpContentInterceptors(BaseClientHandler.java:123)
at software.amazon.awssdk.core.client.handler.BaseClientHandler.finalizeSdkHttpFullRequest(BaseClientHandler.java:68)
at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:106)
at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:73)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:44)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
at software.amazon.awssdk.services.ec2.DefaultEc2Client.copySnapshot(DefaultEc2Client.java:2808)
I expect the snapshot to be copied to the new region specified, without error, and not encrypted.
As a side note I have submitted this to Amazon through my support system and I am awaiting a response from them. I will update this post once they respond to my request.

copySnapshot
sdk error,builder with presignedUrl can skip it

Trouble trying to use Amazon Transcribe with ColdFusion using Java SDK

I'm trying to use the AWS Java SDK to develop a ColdFusion application that uses the Amazon Transcribe service. Unfortunately my knowledge of Java is pitiful (to say nothing of the SDK itself) and I'm having a heck of a time getting anything to happen.
The code below is intended to start a transcription job. It doesn't throw an error, but it also doesn't start the job. I can't even tell if it's sending any information to AWS.
For all I know this code is completely off base, but right now my two biggest questions are:
Am I missing some obvious step to actually send the request to AWS?
How would I access any response I do get back from AWS? A dump of the invokeRequest variable appears to just be the request data.
Thanks in advance for any advice.
(FWIW: CF version is 2016, java version 1.8.0_171, and the AWS SDK is 1.11.331)
<cfscript>
/* Set up credentials */
awsCredentials = createObject('java','com.amazonaws.auth.BasicAWSCredentials').init('#variables.AWSAccessKeyID#', '#variables.AWSSecretKey#');
awsStaticCredentialsProvider = CreateObject('java','com.amazonaws.auth.AWSStaticCredentialsProvider').init(awsCredentials);
/* Create the Transcribe Service Object*/
serviceObject = CreateObject('java', 'com.amazonaws.services.transcribe.AmazonTranscribeAsyncClientBuilder').standard().withCredentials(variables.awsStaticCredentialsProvider).withRegion(#variables.awsRegion#).build();
/* Set up transcription job */
MediaFileUri = CreateObject('java','com.amazonaws.services.transcribe.model.Media').init();
MediaFileUri.setMediaFileUri('#variables.mediafilelocation#');
invokeRequest = CreateObject('java','com.amazonaws.services.transcribe.model.StartTranscriptionJobRequest').init();
invokeRequest.withLanguageCode('en-US');
invokeRequest.withMedia(MediaFileUri);
invokeRequest.withMediaFormat('wav');
invokeRequest.withTranscriptionJobName('#variables.jobname#');
/* Check results of request */
/* Shut down client*/
serviceObject.shutdown();
</cfscript>

Here is how I got this working. I'll start at the beginning, in case anyone reading this is as mystified as I was.
The first step is getting the AWS Java SDK. At some point I was under the impression that there was a separate SDK just for the Amazon Transcribe service, but that's not the case. Once you download the file, place the following jar files in the /{coldfusion}/lib/ folder (I'm not certain all of these are necessary, but this is what worked for me):
aws-java-sdk-xxxx.jar
httpclient-xxxx.jar
httpcore-xxxx.jar
jackson-annotations-xxxx.jar
jackson-core-xxxx.jar
jackson-databind-xxxx.jar
jackson-dataformat-cbor-xxxx.jar
joda-time-xxxx.jar
Restart the ColdFusion service.
The Transcribe service requires that the media file to be transcribed is in S3. I place my files in S3 using ColdFusion's native support into a bucket I've called "transcriptaudio", for example (note the colon separating the key ID from the secret):
<cffile
action = "copy"
source = "c:\temp\myfilename.wav"
destination = "s3://#variables.AWSAccessKeyID#:#variables.AWSSecretKey##transcriptaudio/">
The URL for the media will then be:
https://s3.{awsregion}.amazonaws.com/transcriptaudio/myfilename.wav
Then here is my code to start a transcription job:
<cfscript>
/* Set up credentials */
awsCredentials = CreateObject('java', 'com.amazonaws.auth.BasicAWSCredentials').init('#variables.AWSAccessKeyID#','#variables.AWSSecretKey#');
variables.awsStaticCredentialsProvider = CreateObject('java','com.amazonaws.auth.AWSStaticCredentialsProvider').init(awsCredentials);
/* Create the Transcribe Service Object*/
serviceObject = CreateObject('java', 'com.amazonaws.services.transcribe.AmazonTranscribeAsyncClientBuilder').standard().withCredentials(variables.awsStaticCredentialsProvider).withRegion(#variables.awsRegion#).build();
/* Set up transcription job */
MediaFileUri = CreateObject('java','com.amazonaws.services.transcribe.model.Media').init();
MediaFileUri.setMediaFileUri('#variables.mediafileurlstring#');
requestObject = CreateObject('java','com.amazonaws.services.transcribe.model.StartTranscriptionJobRequest').init();
requestObject.withLanguageCode('en-US');
requestObject.withMedia(MediaFileUri);
requestObject.withMediaFormat('wav');
requestObject.withTranscriptionJobName('#variables.jobName#');
/* Send the request */
sendRequest = serviceObject.startTranscriptionJob(requestObject);
/* Shut down client*/
serviceObject.shutdown();
</cfscript>
As Ageax noted in the comments, the transcription happens asynchronously, so I have a separate CF page to get the transcript after it completes. This code basically assumes the job is complete, but the transcriptionStatus variable will let me do that.
<cfscript>
/* Set up credentials */
awsCredentials = CreateObject('java','com.amazonaws.auth.BasicAWSCredentials').init('#variables.AWSAccessKeyID#','#variables.AWSSecretKey#');
variables.awsStaticCredentialsProvider = CreateObject('java','com.amazonaws.auth.AWSStaticCredentialsProvider').init(awsCredentials);
/* Create the Transcribe Service Object*/
serviceObject = CreateObject('java', 'com.amazonaws.services.transcribe.AmazonTranscribeAsyncClientBuilder').standard().withCredentials(variables.awsStaticCredentialsProvider).withRegion(#variables.awsRegion#).build();
/* Set up results object */
requestResultObject = CreateObject('java', 'com.amazonaws.services.transcribe.model.GetTranscriptionJobRequest').init();
requestResultObject.withTranscriptionJobName('#variables.jobName#');
/* Get the results */
requestResult = serviceObject.GetTranscriptionJob(requestResultObject);
/* parse result object into useful variables */
transcriptionStatus = requestResult.TranscriptionJob.TranscriptionJobStatus.toString();
transcriptURL = requestResult.TranscriptionJob.Transcript.TranscriptFileUri.toString();
</cfscript>
At this point I have the TranscriptURL, which when retrieved with cfhttp returns a vast amount of unnecessary information, at least for my use. Here is my code for getting the actual text of the transcript (the service returns the transcript in an array, so in case there's a possibility that there will be more than one transcript per job for some reason, I loop over the array) (and yes I switch to CF tags here because I'm just more comfortable working in tags):
<CFHTTP url="#variables.transcriptURL#" result="transcriptContentResponse">
<CFSET ResultsStruct = DeserializeJSON(variables.transcriptContentResponse.FileContent)>
<CFSET TranscriptsArray = ResultsStruct.Results.transcripts>
<CFLOOP Array = "#variables.TranscriptsArray#" index="ThisTranscript" >
<cfoutput>
#ThisTranscript['transcript']#
</cfoutput>
</CFLOOP>

Why RDF4J won't work in Bluemix Cloud Foundry

RDF4J is an apache open source graph db that i use often on my PC.
It comes with 2 webApp : RDF4J-server and RDF4J-workbench (a user interface)
On my PC i push the 2 wars in the same Tomcat and everything is ok.
I begin an experiment to push these apps in the Bluemix Cloud (which is a cloud foundry)
The java-tomcat boilerplate wants a single War in order to associate it with the URL of the new container, so i create 2 separate apps in 2 java containers :
1 for RDF4J-Server,
1 for RDF4J-WB.
Both apps are running and i can access to default pages.
In WB, the form 'connect to server' allows you to give the URL of the server you want to work with.
I enter the URL https://rdf4jmyserver.mybluemix.net. WB finds the server but loops on the form, unable to open a db.
I suppose first that a split in 2 containers can be an issue, but i do the following test :
-run RD4J workbench in a local Tomcat on my machine
-connect to the rdf4Jmyserver on the cloud
-> Everything is ok !
So the pb is not to run in 2 separate places.
I investigate a bit more, dowload the source code (thanks open source) and recompile with more and more debug traces .
After a long day, i found the bug in the Workbench code, despite the fact this code is as old as the previous version Sesame: Nobody catch it .
Philosophy for today :
Bluemix works well, but pushing an app in the cloud can reveal old weakness !
I will give the patch in a next post.

Explanation :
When the selected server by workbench is not the default one ( which is supposed to be on the same root url ), the program establish a new cookie to keep trace of the new choice.
The wrong code is here , in CookieHandler.java :
private void initCookie(final Cookie cookie, final HttpServletRequest req) {
final String context = req.getContextPath();
cookie.setPath(null == context ? "/" : context);
}
}
The url is something like https://rdf4j-mywb.mybluemix.net
So, when there is no context for the cookie, the software wants to add a / .
But this code is false.
Going back to the java API, one can see :
public java.lang.String getContextPath()
Returns the portion of the request URI that indicates the context of the request.
The context path always comes first in a request URI.
The path starts with a "/" character but does not end with a "/" character.
For servlets in the default (root) context, this method returns "".
The container does not decode this string.
So this old buried bug can be fixed by :
// cookie.setPath(null == context ? "/" : context);
cookie.setPath(context.isEmpty()? "/" : context);
And now, Workbench works well on Bluemix !
HTH
PS: as rdf4Jserver runs also in a cloud container, datas on disk can disappear when container is reset. Another job has to be done : using an objectstore service in Bluemix. ( another day)

Why is this URL not opened from Play! Framework 1.2.4?

I have a URL in my Play! app that routes to either HTML or XLSX depending on the extension that is passed in the URL, with a routes line like :-
# Calls
GET /calls.{format} Call.index
so calls.html renders the page, calls.xlsx downloads an Excel file (using Play Excel module). All works fine from the browser, a cURL request, etc.
I now want to be able to create an email and have the Excel attached to it, but I cannot pull the attachment. Here's the basic version of what I tried first :-
public static void sendReport(List<Object[]> invoicelines, String emailaddress) throws MalformedURLException, URISyntaxException
{
setFrom("Telco Analysis <test#test.com>");
addRecipient(emailaddress);
setSubject("Telco Analysis report");
EmailAttachment emailAttachment = new EmailAttachment();
URL url = new URL("http://localhost:9001/calls.xlsx");
emailAttachment.setURL(url);
emailAttachment.setName(url.getFile());
emailAttachment.setDescription("Test file");
addAttachment(emailAttachment);
send(invoicelines);
}
but it just doesn't pull the URL content, it just sits there without any error messages, with Chrome's page spinner going and ties up the web server (to the point that requests from another browser/machine don't appear to get serviced). If I send the email without the attachment, all is fine, so it's just the pulling of the file that appears to be the problem.
So far I've tried the above method, I've tried Play's WS webservice library, I've tried manually-crafted HttpRequests, etc. If I specify another URL (such as http://www.google.com) it works just fine.
Anyone able to assist?

I am making an assumption that you are running in Dev mode.
In Dev mode, you will likely have a single request execution pool, but in your controller that send an email, you are sending off a second request, which will block until your previous request has completed (which it won't because it is waiting for the second request to respond)...so....deadlock!
The resaon why external requests work fine, is because you are not causing the deadlock on your Play request pool.
Simple answer to your problem is to increase the value of the play.pool in the application.conf. Make sure that it is uncommented, and choose a value greater than 1!
# Execution pool
# ~~~~~
# Default to 1 thread in DEV mode or (nb processors + 1) threads in PROD mode.
# Try to keep a low as possible. 1 thread will serialize all requests (very useful for debugging purpose)
play.pool=3

Apple push notification services in java, linux development machine

I'm developing a mobile application with a mac, but it connects to a test server I'm running on linux and I'd like to enable APN on this server with a developer certificate, the question is, is it possible to install this certificate on my test server or I'll have to setup the test server on my linux machine?
To make this a little bit clearer:
My development machine: A Mac.
The test server: A linux machine running liferay 6.0.6
I want to install the development certificate on the test server so I can test push notifications.
Thanks a lot!

I know nothing about 'liferay' but this his how I setup a APN connection (on a Rails server) using the certificate. Note that you need to convert the certificate to a .pem file (using 'openssl pkcs12 -in myfile.p12 -out myfile.pem'):
##apn_cert = nil
APN_SSL_KEY_FILE = 'lib/SSLCert_Private_Key.pem'
APN_SSL_HOST = 'gateway.sandbox.push.apple.com'
# APN_SSL_HOST = 'gateway.push.apple.com'
APN_SSL_PORT = 2195
APN_SSL_PASSWORD = '<password>'
def configure_apn_cert
puts "APN Service: Configuring APN cert"
##apn_cert = File.read(File.join(RAILS_ROOT, APN_SSL_KEY_FILE))
##apn_context = OpenSSL::SSL::SSLContext.new
##apn_context.key = OpenSSL::PKey::RSA.new(##apn_cert, APN_SSL_PASSWORD)
##apn_context.cert = OpenSSL::X509::Certificate.new(##apn_cert)
end
def create_and_configure_apn_server
configure_apn_cert if not ##apn_cert
puts "APN Service: Configuring APN SOCKET and SSL connection"
#apn_socket = TCPSocket.new(APN_SSL_HOST, APN_SSL_PORT)
#apn_ssl = OpenSSL::SSL::SSLSocket.new(#apn_socket, ##apn_context)
#apn_ssl.sync = true
#apn_ssl.connect
end
def close_apn_server
#apn_ssl.close
#apn_socket.close
end
def package_build_for_apn( token, content )
"\0\0 #{token}\0#{content.length.chr}#{content}"
end
def package_send_to_apn( package )
puts "APN Service: Sending #{package}"
bytes_written = #apn_ssl.write( package )
if bytes_written != package.length then
puts "APN Service: SSL write failed"
package_to_apn_show_write( bytes_written, package)
end
end
def apn_deliver_payload( token, payload )
# Convert the device string back into a byte string
tokenBinary = Base64.decode64( token )
# Transform the payload into an APN byte string
apn_content = payload.to_hash.to_json
apn_content_len = apn_content.length
# Build the apn_package per APN specification
apn_package = "\0\0 #{tokenBinary}\0#{apn_content_len.chr}#{apn_content}"
# Actually send it.
package_send_to_apn( apn_package )
end
def package_to_apn_show_write( bytes, package)
puts "Wrote: #{bytes_written}/ Tried: #{package.length}"
puts "Package: '#{package}'"
end
def package_to_apn_debug( token, content, package )
puts "Token(#{token.length}): #{token}"
puts "Content(#{content.length}): #{content}"
puts "Package(#{package.length}): #{package}"
end

I'd like to answer my own question because it might help someone, first of all, I had a terrible confusion between the certificates I needed to generate and how to get the device token among other things.
These are the steps I went through to make this work:
Created an application ID.
Created a Certificate Signing Request using "Keychain access" and got the development certificate for push notifications which I installed using again Keychain access.
Exported the certificate as a .p12 file.
Used the notnoop java library (which is on the central maven repository) to send push notifications.
Here is a sample snippet that shows where the certificate fits in:
ApnsService service =
APNS.newService()
.withCert("/path/to/certificate.p12", "MyCertPassword")
.withSandboxDestination()
.build();
The second argument of the method is the password for the p12 file that keychain access makes you to set.
Finally to glue things, I used the [UIApplication sharedApplication] instance to register to receive push notifications and by implementing a method on the AppDelegate I get the token I needed which was in NSData format, so you need to converted into a hex string (there is sample code for that in many sites and questions on this site).
And that's it, that's the process.
Hope this helps!!