I am trying to determine if its possible to find out the cause of an exception we occasionally get when using Google App Engine's Memcache.
The documentation states that if the key doesn't exist it returns false, so I believe the error indicates a different issue.
This is the stack trace of the error we are getting:
com.google.appengine.api.memcache.LogAndContinueErrorHandler handleServiceError: Service error in memcache (LogAndContinueErrorHandler.java:49)
com.google.appengine.api.memcache.MemcacheServiceException: Memcache delete: Unknown exception deleting key: ahBzfmlkZXh4LXNlcnZpY2Vzch0LEgdQYXRpZW50GICA6ODn6-ELDAsSA0pvYhgBDA
at com.google.appengine.api.memcache.MemcacheServiceApiHelper$RpcResponseHandler.handleApiProxyException(MemcacheServiceApiHelper.java:69)
at com.google.appengine.api.memcache.MemcacheServiceApiHelper$1.absorbParentException(MemcacheServiceApiHelper.java:111)
at com.google.appengine.api.utils.FutureWrapper.handleParentException(FutureWrapper.java:52)
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:91)
at com.google.appengine.api.memcache.MemcacheServiceImpl.quietGet(MemcacheServiceImpl.java:26)
at com.google.appengine.api.memcache.MemcacheServiceImpl.delete(MemcacheServiceImpl.java:125)
Here is some relevant code:
boolean putValue = memcacheService.put(syncKey, syncKey, Expiration.byDeltaMillis(60000), MemcacheService.SetPolicy.ADD_ONLY_IF_NOT_PRESENT);
memcacheService.delete(syncKey, 100L);
Later requests are unable to put the same key indicating that the object still exists. Given that I am specifying a 60 second expiration, I'd expect that it'd be available after that time, but it's not the case. Right now, the only way we've found to fix this is to clear the entire Memcache.
Related
I'm trying to create an object in Minio. Sometimes it works and sometimes no.
When I execute :
minioClient.makeBucket(bucketName); // bucketName es a random unique string
I donĀ“t get error, but if If ask Minio if bucket exists:
minioClient.bucketExists(bucketName)
it returns false
After that, If I try to launch once again minioClient.makeBucket(bucketName) because minioClient.bucketExists(bucketName) is returning false, then I get this error (simplified):
BucketAlreadyOwnedByYou
So that, when I get previous error, I try to create object in Minio :
minioClient.putObject(bucketName, objectName, is,null,customMetaData,null, null ); // is is an InputStream of the file I want to store in Minio
But I get this error, saying bucket does not exist;
ErrorResponse(code=NoSuchBucket, message=The specified bucket does not exist, bucketName=888bf891-064f-4270-b0f0-85514c0eae02, objectName=L7aEmF8Ppk, resource=/888bf891-064f-4270-b0f0-85514c0eae02/L7aEmF8Ppk, requestId=16733C7840E6854D, hostId=null)
Any help will be appreciated.
I'm using version 6.0.13 of Minio Java API
This not how MinIO should behave. I have only seen this error where it was either a programming error or a setup error.
When it is a setup error, the root problem is that instead of creating 1 server pool with 4 distributed nodes, people have created 4 server pools each with a single node. This means that when your application gets load balanced, it will sometimes hit the server pool where you created the bucket and sometimes not.
Another cause can be an non-consistent filesystem used as backend. For example NFS with certain settings can have behavior like this. The recommended filesystem is XFS.
Without knowing more about your setup I cannot tell you which one it is.
We have a java application that has multiple threads and that connects to an active directory application to read the data. But most of time, some thread (not all) fails with exception:
"javax.naming.OperationNotSupportedException: [LDAP: error code 12 - 00000057: LdapErr: DSID-0C090753, comment: Error processing control, data 0, v1db1 ];" when getting next page of result (pagesize = 1000)
I googled and found that we should use only one context through the search. We have done the same thing, we are using the same context to get next page, but failing with the exception. Our domain controller is windows server 2008 r2. Is it related to domain controller or are we missing something here?
I got the solution as we are passing control for sorting of data on attribute CN. The sorting control while searching on DC is not supported once we disabled sorting it worked.
Is it possible to auto evict the record with value Optional.absent()? In some applications the Optional.absent() may not be the value for some keys. For example, if an application contains http calls where the key can be some string and the value is response returned from the http call, the http call can return some invalid values (for example null) because of network issues or authentication failures, then the invalid can be saved as Optional.absent() with the key in cache. At a later point, if the network and authentication problems are fixed, the key->invalidValue still remains in the cache. What is the best to fix this problem?
For example, if an application contains http calls where the key can be some string and the value is response returned from the http call, the http call can return some invalid values (for example null) because of network issues or authentication failures
If possible, I would change this behavior to throw an exception when the request fails or the response is invalid - that's what exceptions are for. See Effective Java: Item 57 for more.
then the invalid can be saved as Optional.absent() with the key in cache. At a later point, if the network and authentication problems are fixed, the key->invalidValue still remains in the cache. What is the best to fix this problem?
Is there a reason you need to save the invalid result in the cache? If you don't care about the absent case and simply want it excluded from the cache the easiest option would be to just not cache it in the first place. Throwing an exception on bad results would make that easy.
If you do need to keep the invalid results in the cache temporarily you can clear them once you're ready with a simple for loop:
ConcurrentMap<K, V> cacheAsMap = cache.asMap();
for (Entry<K, V> e : cacheAsMap .entrySet()) {
if (!e.getValue().isPresent()) {
cacheAsMap.remove(e.getKey(), e.getValue());
}
}
By using ConcurrentMap.remove() you avoid a possible race condition where the entry is updated after e.getValue().isPresent() is called but before the entry is actually invalidated.
For a long time, ever since a recent change that allowed our users to keep their lost work in case of a system failure for our contact log form, we have had reports of a problem with one of our input forms. When the user tries to enter a very basic contact log (No files, one option, a date to mark it, and the log itself), they are bounced out of our app. Immediately afterwards, if they log in and try again, they can successfully submit the form.
We have been so far unable to duplicate this error on our test server, but we have narrowed down the error thrown when it happens. In our log, it looks like this:
[5/23/13 13:18:47:837 EDT] 46b24806 PropertyUtils E org.apache.commons.beanutils.PropertyUtils Method invocation failed.
[5/23/13 13:18:47:853 EDT] 46b24806 PropertyUtils E org.apache.commons.beanutils.PropertyUtils TRAS0014I: The following exception was logged java.lang.IllegalArgumentException: argument type mismatch
at java.lang.reflect.Method.invoke(Native Method)
at org.apache.commons.beanutils.PropertyUtilsBean.invokeMethod(PropertyUtilsBean.java(Inlined Compiled Code))
It goes on for several lines, but the jist of the errors seem to indicate a Struts problem. The method being utilized is a do method that writes to an SQL database, though the database itself doesn't seem to be the source of this problem.
Any sort of help or guidance would be appreciated. We have thrown multiple theories across the table, but without being able to duplicate the problem, it's fairly difficult to find a solution. Thank you in advance.
ONE ADDITIONAL NOTE: We are using Struts v. 1.2
I'm trying to start a Amazon EC2 cloud machine with [startInstance][2] method using aws-sdk in Java. My code is as follows.
public String startInstance(String instanceId) throws Exception {
List<String> instanceIds = new ArrayList<String>();
instanceIds.add(instanceId);
StartInstancesRequest startRequest = new StartInstancesRequest(
instanceIds);
startRequest.setRequestCredentials(getCredentials());
StartInstancesResult startResult = ec2.startInstances(startRequest);
List<InstanceStateChange> stateChangeList = startResult
.getStartingInstances();
log.trace("Starting instance '{}':", instanceId);
// Wait for the instance to be started
return waitForTransitionCompletion(stateChangeList, "running",
instanceId);
}
When I run the above code, i'm getting the following AWS error:
Status Code: 400, AWS Request ID: e1bd4795-a609-44d1-9e80-43611e80006b, AWS Erro
r Code: InvalidInstanceID.NotFound, AWS Error Message: The instance ID 'i-2b97ac
2f' does not exist
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpCli
ent.java:538)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.ja
va:283)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:168
)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.jav
a:5208)
at com.amazonaws.services.ec2.AmazonEC2Client.startInstances(AmazonEC2Cl
ient.java:2426)
AWS Error Message: The instance ID 'i-2b97ac2f' does not exist
You'll have to take the AWS response for granted here, i.e. the instance does not exist ;)
But seriously: Presumably you have already verified that you are actually running an instance with this ID in your account? Then this is most likely caused by targeting the wrong API endpoint, insofar an instance ID is only valid within a specific region (if not specified, the region defaults to 'us-east-1', see below).
In this case you need to specify the actual instance region via the setEndpoint() method of the AmazonEC2Client object within the apparently global ec2 variable before calling startInstances().
There are some examples regarding Using Regions with the AWS SDKs and all currently available AWS regional endpoint URLs are listed in Regions and Endpoints, specifically the Amazon Elastic Compute Cloud (EC2) defaults to 'us-east-1':
If you just specify the general endpoint (ec2.amazonaws.com), Amazon
EC2 directs your request to the us-east-1 endpoint.
We run a service (Qubole) that frequently spawns and then tags (and in some cases terminates) AWS instances immediately.
We have found that Amazon will, every once in a while, claim an instanceid as invalid - even though it has just created it. Retrying a few times with some sleep time thrown in usually solves the problem. Even a total retry interval of 15s proved insufficient in rare cases.
This experience comes from the useast region. We do not make api calls to different regions - so that is not an explanation. More likely - this is the infamous eventual consistency at work - where AWS is unable to provide read-after-write consistency for these api calls.
I am using the AWS ruby api and I noticed the same issue when creating an AMI image and its status is pending when I look in the AWS console but after a while the image is available for use.
Here is my script
image = ec2.images.create(:name => image_name, :instance_id => ami_id, :description => desc)
sleep 5 while image.state != :available
I sleep for about 5 sec for image to be in available but I get the error saying that the "AWS Error Message: InvalidInstanceID.NotFound". During my testing this is fine but most of the time this seems to be failing during continuous integration builds.
InvalidInstanceID.NotFound means the specified instance does not exist.
Ensure that you have indicated the region in which the instance is located, if it's not in the default region.
This error may occur because the ID of a recently created instance has not propagated through the system. For more information, see Eventual Consistency.