We Keep Coding

How to resolve 'protocol must not be null' for CopySnapshotRequest across regions

I am attempting to move an EC2 snapshot from one region to another.
When creating a basic Ec2 snapshot copy and copying it to a second Region I get the error protocol must not be null. It appears that the host is also null when you get to the point in the code that will require a host. From my understanding I do not need to provide the protocol (http/https) or the host details when copying a snapshot the is NOT encrypted. It appears that the AWS code should do this behind the scenes.
Off the top of my head this seems like an aws issue but there is not much feedback on the AWS SDK for Java V2 yet as I have checked here as well as github.
Program exception:
java.lang.NullPointerException: protocol must not be null
I have tried adding the destinationRegion to the builder but that also results in the same error.
Execution environment is Amazon Java 11.0.3
I have tried with software.amazon.awssdk versions 2.7.11 all the way through 2.7.29
Basic code snippet
String amazonAccessKeyId = "amazonAccessKeyId";
String amazonSecretKeyId = "amazonSecretKeyId";
String baseRegionName = "baseRegionName"; // Region.AP_NORTHEAST_1.id(); or where ever your snapshot is located
String remoteRegionName = "remoteRegionName "; // Region.AP_NORTHEAST_1.id(); or where ever your snapshot is located
String snapshotId = "snapshotId"; // You will need a snapshot to work with this code
// Setup AWS remote client with credentials
AwsCredentials credentials = AwsBasicCredentials.create(amazonAccessKeyId, amazonSecretKeyId);
Ec2Client amazonEc2RemoteRegionClient = Ec2Client.builder()
.region(Region.of(remoteRegionName))
.credentialsProvider(StaticCredentialsProvider.create(credentials))
.build();
// Setup request
CopySnapshotRequest request = CopySnapshotRequest.builder()
.sourceRegion(Region.of(baseRegionName).id())
.sourceSnapshotId(snapshotId)
.description("Foo Bar Testing...")
.build();
// Invoke copy from remote region to pull in snapshot from source/default region
// This is the line in question
CopySnapshotResponse result = amazonEc2RemoteRegionClient.copySnapshot(request);
Stack trace follows
java.lang.NullPointerException: protocol must not be null.
at software.amazon.awssdk.utils.Validate.paramNotNull(Validate.java:117)
at software.amazon.awssdk.http.DefaultSdkHttpFullRequest.standardizeProtocol(DefaultSdkHttpFullRequest.java:63)
at software.amazon.awssdk.http.DefaultSdkHttpFullRequest.<init>(DefaultSdkHttpFullRequest.java:52)
at software.amazon.awssdk.http.DefaultSdkHttpFullRequest.<init>(DefaultSdkHttpFullRequest.java:41)
at software.amazon.awssdk.http.DefaultSdkHttpFullRequest$Builder.build(DefaultSdkHttpFullRequest.java:331)
at software.amazon.awssdk.http.DefaultSdkHttpFullRequest$Builder.build(DefaultSdkHttpFullRequest.java:170)
at software.amazon.awssdk.services.ec2.transform.internal.GeneratePreSignUrlInterceptor.modifyHttpRequest(GeneratePreSignUrlInterceptor.java:102)
at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.modifyHttpRequestAndHttpContent(ExecutionInterceptorChain.java:99)
at software.amazon.awssdk.core.client.handler.BaseClientHandler.runModifyHttpRequestAndHttpContentInterceptors(BaseClientHandler.java:123)
at software.amazon.awssdk.core.client.handler.BaseClientHandler.finalizeSdkHttpFullRequest(BaseClientHandler.java:68)
at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:106)
at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:73)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:44)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
at software.amazon.awssdk.services.ec2.DefaultEc2Client.copySnapshot(DefaultEc2Client.java:2808)
I expect the snapshot to be copied to the new region specified, without error, and not encrypted.
As a side note I have submitted this to Amazon through my support system and I am awaiting a response from them. I will update this post once they respond to my request.

copySnapshot
sdk error,builder with presignedUrl can skip it

Memcache delete: Unknown exception deleting key

I am trying to determine if its possible to find out the cause of an exception we occasionally get when using Google App Engine's Memcache.
The documentation states that if the key doesn't exist it returns false, so I believe the error indicates a different issue.
This is the stack trace of the error we are getting:
com.google.appengine.api.memcache.LogAndContinueErrorHandler handleServiceError: Service error in memcache (LogAndContinueErrorHandler.java:49)
com.google.appengine.api.memcache.MemcacheServiceException: Memcache delete: Unknown exception deleting key: ahBzfmlkZXh4LXNlcnZpY2Vzch0LEgdQYXRpZW50GICA6ODn6-ELDAsSA0pvYhgBDA
at com.google.appengine.api.memcache.MemcacheServiceApiHelper$RpcResponseHandler.handleApiProxyException(MemcacheServiceApiHelper.java:69)
at com.google.appengine.api.memcache.MemcacheServiceApiHelper$1.absorbParentException(MemcacheServiceApiHelper.java:111)
at com.google.appengine.api.utils.FutureWrapper.handleParentException(FutureWrapper.java:52)
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:91)
at com.google.appengine.api.memcache.MemcacheServiceImpl.quietGet(MemcacheServiceImpl.java:26)
at com.google.appengine.api.memcache.MemcacheServiceImpl.delete(MemcacheServiceImpl.java:125)
Here is some relevant code:
boolean putValue = memcacheService.put(syncKey, syncKey, Expiration.byDeltaMillis(60000), MemcacheService.SetPolicy.ADD_ONLY_IF_NOT_PRESENT);
memcacheService.delete(syncKey, 100L);
Later requests are unable to put the same key indicating that the object still exists. Given that I am specifying a 60 second expiration, I'd expect that it'd be available after that time, but it's not the case. Right now, the only way we've found to fix this is to clear the entire Memcache.

Verify Azure Table SAS-based credentials

I'm looking for simple way of verifying an arbitrary Azure Table connection string that uses a SAS such as the one below using the Azure Storage's Java SDK:
https://example.table.core.windows.net/example?sig=aaabbbcccdddeeefffggghhh%3D&se=2020-01-01T00%3A00%3A00Z&sv=2015-04-05&tn=example&sp=raud
I tried a bunch of different methods exposed by the CloudTable api, but none of them works.
CloudTable.exists() throws a StorageException, regardless of whether the credentials are valid
getName(), getStorageUri(), getUri(), and other getters - all work locally, regardless of the credentials
getServiceClient().downloadServiceProperties() and getServiceClient().getServiceStats() also throw various exceptions, while getServiceClient().getEndpoint() and getServiceClient().getCredentials() and others always work locally.
Why don't I just query the Table for a row or two? Well, in many cases I need to verify a SAS that gives only write or update premissions (without delete or read permissions), and I do not want to execute a statement that changes something in the table just to check the credentials.

To answer your questions:
CloudTable.exists() throws a StorageException, regardless of whether
the credentials are valid
I believe there's a bug with the SDK when using this method with SAS Token. I remember running into the same issue some time back.
getName(), getStorageUri(), getUri(), and other getters - all work
locally, regardless of the credentials
These will work as they don't make network call. They simply use the data available to them in the different instance variables and return the data.
getServiceClient().downloadServiceProperties() and
getServiceClient().getServiceStats() also throw various exceptions,
while getServiceClient().getEndpoint() and
getServiceClient().getCredentials() and others always work locally.
In order for getServiceClient().someMethod() to work using SAS, you would need Account SAS instead of Service SAS (which you're using right now).
Why don't I just query the Table for a row or two? Well, in many cases
I need to verify a SAS that gives only write or update premissions
(without delete or read permissions), and I do not want to execute a
statement that changes something in the table just to check the
credentials.
One possible way to check the validity of a SAS Token for write operation is to perform a write operation which you know will fail with an error. For example, you can try to insert an entity which is already there. In this case, you should get a Conflict (409) error. Other thing you could try to do is perform an optimistic write by specifying some random Etag value and check for Precondition Failed (412) error. If you get a 403 error or 404 error, that would indicate there's something wrong with your SAS token.

Couchbase failure to find bucket local instance

Following the implementation guide from Spring on creating CouchbaseConfig by extending AbstractCouchbaseConfiguration.
Problem #1
My local Couch instance exists on "http://localhost:5984/" At application runtime the default port always results in the value "11210" which ultimatly results in connectivity failure.
Solution
To resolve this issue I am overriding the implementation which creates CouchbaseClient within my CouchbaseConfig component (Code snippet below)
public CouchbaseClient couchbaseClient() throws Exception {
CouchbaseClient client = null;
URI local=new URI("http://127.0.0.1:5984");
List<URI> baseURIs=new ArrayList<URI>();
baseURIs.add(local);
client = new CouchbaseClient(baseURIs, bucketValue, "");
......
I am still not able to locate the bucket I have setup in my local couch instance
http-bio-8080-exec-25] INFO 18:18:14,172 SLF4JLogger.log(87) - Could not fetch config from http seed nodes.
com.couchbase.client.vbucket.ConfigurationException: Configuration for bucket "bucketValue" was not found in server list ([http://127.0.0.1:5984]).
at com.couchbase.client.vbucket.ConfigurationProviderHTTP.readPools(ConfigurationProviderHTTP.java:271)
at com.couchbase.client.vbucket.ConfigurationProviderHTTP.getBucketConfiguration(ConfigurationProviderHTTP.java:143)
at com.couchbase.client.vbucket.provider.BucketConfigurationProvider.bootstrapHttp(BucketConfigurationProvider.java:351)
Any information on how to resolve this would be greatly appreciated.

So, I think what you have done here is confused Couchbase and CouchDb - they are entirely different platforms, even though they have confusingly-similar names.
You need to install a local instance of Couchbase if that is what you are trying to do, and get rid of Couchdb.

AWS Error Message: InvalidInstanceID.NotFound

I'm trying to start a Amazon EC2 cloud machine with [startInstance][2] method using aws-sdk in Java. My code is as follows.
public String startInstance(String instanceId) throws Exception {
List<String> instanceIds = new ArrayList<String>();
instanceIds.add(instanceId);
StartInstancesRequest startRequest = new StartInstancesRequest(
instanceIds);
startRequest.setRequestCredentials(getCredentials());
StartInstancesResult startResult = ec2.startInstances(startRequest);
List<InstanceStateChange> stateChangeList = startResult
.getStartingInstances();
log.trace("Starting instance '{}':", instanceId);
// Wait for the instance to be started
return waitForTransitionCompletion(stateChangeList, "running",
instanceId);
}
When I run the above code, i'm getting the following AWS error:
Status Code: 400, AWS Request ID: e1bd4795-a609-44d1-9e80-43611e80006b, AWS Erro
r Code: InvalidInstanceID.NotFound, AWS Error Message: The instance ID 'i-2b97ac
2f' does not exist
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpCli
ent.java:538)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.ja
va:283)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:168
)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.jav
a:5208)
at com.amazonaws.services.ec2.AmazonEC2Client.startInstances(AmazonEC2Cl
ient.java:2426)

AWS Error Message: The instance ID 'i-2b97ac2f' does not exist
You'll have to take the AWS response for granted here, i.e. the instance does not exist ;)
But seriously: Presumably you have already verified that you are actually running an instance with this ID in your account? Then this is most likely caused by targeting the wrong API endpoint, insofar an instance ID is only valid within a specific region (if not specified, the region defaults to 'us-east-1', see below).
In this case you need to specify the actual instance region via the setEndpoint() method of the AmazonEC2Client object within the apparently global ec2 variable before calling startInstances().
There are some examples regarding Using Regions with the AWS SDKs and all currently available AWS regional endpoint URLs are listed in Regions and Endpoints, specifically the Amazon Elastic Compute Cloud (EC2) defaults to 'us-east-1':
If you just specify the general endpoint (ec2.amazonaws.com), Amazon
EC2 directs your request to the us-east-1 endpoint.

We run a service (Qubole) that frequently spawns and then tags (and in some cases terminates) AWS instances immediately.
We have found that Amazon will, every once in a while, claim an instanceid as invalid - even though it has just created it. Retrying a few times with some sleep time thrown in usually solves the problem. Even a total retry interval of 15s proved insufficient in rare cases.
This experience comes from the useast region. We do not make api calls to different regions - so that is not an explanation. More likely - this is the infamous eventual consistency at work - where AWS is unable to provide read-after-write consistency for these api calls.

I am using the AWS ruby api and I noticed the same issue when creating an AMI image and its status is pending when I look in the AWS console but after a while the image is available for use.
Here is my script
image = ec2.images.create(:name => image_name, :instance_id => ami_id, :description => desc)
sleep 5 while image.state != :available
I sleep for about 5 sec for image to be in available but I get the error saying that the "AWS Error Message: InvalidInstanceID.NotFound". During my testing this is fine but most of the time this seems to be failing during continuous integration builds.

InvalidInstanceID.NotFound means the specified instance does not exist.
Ensure that you have indicated the region in which the instance is located, if it's not in the default region.
This error may occur because the ID of a recently created instance has not propagated through the system. For more information, see Eventual Consistency.