I am facing cluster block exception due to lack of space on elastic search node. But after sufficient space is available I need to manually set the "index.blocks.read_only_allow_delete": null value using java high level rest client. Same I am able to do with Kibana but need some java equivalent of that.
Kibana request :
PUT /_settings
{
"index.blocks.read_only_allow_delete": null
}
Elastic version : 6.8.4
Java : 8
As the High Level Rest Client has no Support for his feature, you need to use the Low Level Rest Client or issue the http request by yourself
Related
I'm upgrading my Elasticsearch cluster from 6.8.X to 7.5.X. I have this breaking change; basically in the REST response, there is a {hits: {total: { value: 123 } } } object instead of a {hits: { total: 123 } } int value.
Elasticsearch provides an parameter that you can add to your searches rest_total_hits_as_int; however, I cannot figure out how to set this on queries being formatted by their java RestHighLevelClient or in their SearchRequest object. (I'm currently using the ES 6.8.0 libraries.)
When I send a request from the 6.8 client to a 7.5 cluster, I always get 0 total hits. I get an error when I send a request from the 7.5 client to the 6.8 cluster. (Not backwards compatible.)
Is there a migration path that I'm missing?
Just add this:
searchSourceBuilder.trackTotalHits(true);
We're using AWS Lambda (Java) and the elasticsearch client to connect to a hosted elasticsearch instance on AWS. I encounter a long wait on the first request of about 2.5 seconds (on top of a cold start). After that it is very quick. I can't really figure out where this delay is coming from and I'm trying to optimize it.
private void testPerformanceElasticSearch() throws Exception {
log.info("1. Before testing elasticsearch client");
AWS4Signer signer = new AWS4Signer();
signer.setServiceName("es");
signer.setRegionName("eu-west-1");
HttpRequestInterceptor interceptor = new AWSRequestSigningApacheInterceptor("es", signer, new DefaultAWSCredentialsProviderChain());
String endpoint = "https://" + Utils.getEnvironmentVariable("ELASTIC_SEARCH_ENDPOINT");
RestHighLevelClient restHighLevelClient = new RestHighLevelClient(RestClient.builder(HttpHost.create(endpoint)).setHttpClientConfigCallback(hacb -> hacb.addInterceptorLast(interceptor)));
log.info("2. After getting elasticsearch client");
log.info("3. Before doing a elasticsearch query");
log.info("4");
BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
log.info("5");
TermsQueryBuilder termsQueryBuilder = QueryBuilders.termsQuery("userId", "abc");
log.info("6");
boolQueryBuilder.must(termsQueryBuilder);
log.info("7");
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
log.info("8");
searchSourceBuilder.query(boolQueryBuilder);
log.info("9");
SearchRequest searchRequest = new SearchRequest("users");
log.info("10");
searchRequest.source(searchSourceBuilder);
log.info("11");
restHighLevelClient.search(searchRequest);
log.info("12");
log.info("13. After testing elasticsearch");
}
And then I get logging like this; you can see between '5' and '6' there is more than a 2 second delay which I can't really place:
17:16:06.871INFO[PlacesPerformance] 1. Before testing elasticsearch client
17:16:06.932INFO[PlacesPerformance] 2. After getting elasticsearch client
17:16:06.933INFO[PlacesPerformance] 3. Before doing a elasticsearch query
17:16:06.935INFO[PlacesPerformance] 4
17:16:06.942INFO[PlacesPerformance] 5
17:16:09.179INFO[PlacesPerformance] 6
17:16:09.179INFO[PlacesPerformance] 7
17:16:09.181INFO[PlacesPerformance] 8
17:16:09.181INFO[PlacesPerformance] 9
17:16:09.183INFO[PlacesPerformance] 10
17:16:09.183INFO[PlacesPerformance] 11
17:16:09.362INFO[PlacesPerformance] 12
17:16:09.362INFO[PlacesPerformance] 13. After testing elasticsearch
Any suggestions on how to improve this?
UPDATE:
Strange. Whenever I run the code in a lambda, I experience the 2.5 second delay when constructing the request (not even executing it). Locally, it works fine though. I tried the following:
1. Local against local elasticsearch. No delay.
2. Local against AWS elasticsearch. No delay.
3. Lambda with signing request. DELAY.
4. Lambda without signing request. DELAY.
5. Lambda with a 'match all' query. DELAY
6. Lambda with a http address. DELAY.
7. Lambda with a custom runtime. DELAY.
8. Lambda with a custom runtime. DELAY.
9. Lambda with standard Java 8 runtime. DELAY.
The problem is that at the first request (real request, not warmup request as warmup requests don't go through your application code, it doesn't trigger loading classes which are used in actual request path) JVM loads (read, parse, verify, etc ...) related classes, initializes security components (ciphers, etc ...) and TLS handshake is done (requires multiple RTT, with Java 9 and TLS 1.3 this should be reduced).
The similar long duration behaviour is also seen for first AWS service calls (DynamoDB, SQS, etc ...)
As I am the author Thundra warmup plugin, I am thinking of introducing hook points for warmup messages as custom action will be able to executed like initializing security components, loading classes, etc ...
Lambda functions inside VPCs have a great impact on the startup time. You said your ES is a hosted instance, so I assume it's backed by a VPC.
Even if it's not in a VPC, Java cold starts are usually, by nature, longer than runtimes like Node or Python, because the JVM needs to be started up first. This is mainly where your 2.5 seconds come from.
OK. How to fix the issue?
It depends on how many concurrent connections you need to ElasticSearch. If one function is able to handle all the incoming requests, you can then limit the concurrent execution of your Lambda function to 1, so you make sure you are always hitting the same container (as long as these requests are made in a ±5 min time frame).
Now, if you don't know upfront how many concurrent Lambda functions will execute, you kind of have no way out. You could try warming up your Lambda functions beforehand, but then you'd need to fire like 100 requests at the same time to warm up 100 different containers.
Please check this answer as I go through the concurrent model of Lambda functions and how the cold/warm starts work.
I am happy to edit my answer if you have more info to share or if I wasn't clear enough.
I'm attampting to use a bulk HTTP api in Java on AWS ElasticSearch 2.3.
When I use a rest client for teh bulk load, I get the following error:
504 GATEWAY_TIMEOUT
When I run it as Lambda in Java, for HTTP Posts, I get:
{
"errorMessage": "2017-01-09T19:05:32.925Z 8e8164a7-d69e-11e6-8954-f3ac8e70b5be Task timed out after 15.00 seconds"
}
Through testing I noticed the bulk API doesn't work these with these settings:
"number_of_shards" : 5,
"number_of_replicas" : 5
When shards and replicas are set to 1, I can do a bulk load no problem.
I have tried using this setting to allow for my bulk load as well:
"refresh_interval" : -1
but so far it made no impact at all. In Java Lambda, I load my data as an InputStream from S3 location.
What are my options at this point for Java HTTP?
Is there anything else in index settings I could try?
Is there anything else in AWS access policy I could try?
Thank you for your time.
1Edit:
I also have tried these params: _bulk?action.write_consistency=one&refresh But makes no difference so far.
2Edit:
here is what made my bulk load work - set consistency param (I did NOT need to set refresh_interval):
URIBuilder uriBuilder = new URIBuilder(myuri);
uriBuilder = uriBuilder.addParameter("consistency", "one");
HttpPost post = new HttpPost(uriBuilder.build());
HttpEntity entity = new InputStreamEntity(myInputStream);
post.setEntity(entity);
From my experience, the issue can occur when your index replication settings can not be satisfied by your cluster. This happens either during a network partition, or if you simply set a replication requirement that can not be satisfied by your physical cluster.
In my case, this happens when I apply my production settings (number_of_replicas : 3) to my development cluster (which is single node cluster).
Your two solutions (setting the replica's to 1 Or setting your consistency to 1) resolve this issue because they allow Elastic to continue the bulk index w/o waiting for additional replica's to come online.
Elastic Search probably could have a more intuitive message on failure, maybe they do in Elastic 5.
Setting your cluster to a single
I am trying to retrieve a value from a memcached Couchbase bucket using the Java SDK without success. The value was serialized from .NET Is it possible to do this using the REST API?
In couchbase buckets, you just need to send a get request like this:
SERVER_IP:8091/couchBase/COUCHBASE_BUCKET_NAME/KEY
and the value is returned.
I have a memcached value, and even though I have the keys, I cannot extract any values.
When in "COUCHBASE_BUCKET" I put the name of the memcached bucket i get 404 not found.
I tried a lot of variations like /pools/buckets, or pools/default/bucket before memcached bucket name, but with no success.
Any help would be deeply appreciated.
In a comment, user scalabilitysolved suggested to use the Java api, which was my first choice
client = new CouchbaseClient(nodes, "Bucket", "password");
Object getObject = client.get("1key");
// Print the value from synchronous get
if (getObject != null) {
System.out.println("Synchronous Get Suceeded: ");;
} else {
System.err.println("Synchronous Get failed");
}
System.out.println(getObject);
the response in console is this:
2014-01-17 09:17:40.421 INFO com.couchbase.client.CouchbaseMemcachedConnection: Added {QA sa=xxxxxxxxx:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2014-01-17 09:17:40.430 INFO com.couchbase.client.CouchbaseClient: CouchbaseConnectionFactory{, bucket='Bucket', nodes=[http://xxxxxxx:8091/pools], order=RANDOM, opTimeout=2500, opQueue=16384, opQueueBlockTime=10000, obsPollInt=10, obsPollMax=500, obsTimeout=5000, viewConns=10, viewTimeout=75000, viewWorkers=1, configCheck=10, reconnectInt=1100, failureMode=Redistribute, hashAlgo=NATIVE_HASH}
2014-01-17 09:17:40.431 INFO com.couchbase.client.CouchbaseClient: viewmode property isn't defined. Setting viewmode to production mode
2014-01-17 09:17:40.431 INFO com.couchbase.client.CouchbaseMemcachedConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl#127fe8ac
2014-01-17 09:17:40.489 INFO net.spy.memcached.auth.AuthThread: Authenticated to /xxxxxx:11210
Synchronous Get failed
null
The reason that your probably having trouble is because the Couchbase Java SDK serializes data before storing it into Couchbase (and memcached). In order to know how the data was serialized the SDK uses the flags field in the message to indicate how the data is serialized.
It sounds like you have some weird value in your flags field which is causing the Java SDK to not know what to do with the data since it sees an unexpected value in the flags field.
My guess is that you set the data into Couchbase using something other the the Java SDK. The first thing I would do is to only use the Java SDK to make sure you can set and retrieve data.
I'm trying to start a Amazon EC2 cloud machine with [startInstance][2] method using aws-sdk in Java. My code is as follows.
public String startInstance(String instanceId) throws Exception {
List<String> instanceIds = new ArrayList<String>();
instanceIds.add(instanceId);
StartInstancesRequest startRequest = new StartInstancesRequest(
instanceIds);
startRequest.setRequestCredentials(getCredentials());
StartInstancesResult startResult = ec2.startInstances(startRequest);
List<InstanceStateChange> stateChangeList = startResult
.getStartingInstances();
log.trace("Starting instance '{}':", instanceId);
// Wait for the instance to be started
return waitForTransitionCompletion(stateChangeList, "running",
instanceId);
}
When I run the above code, i'm getting the following AWS error:
Status Code: 400, AWS Request ID: e1bd4795-a609-44d1-9e80-43611e80006b, AWS Erro
r Code: InvalidInstanceID.NotFound, AWS Error Message: The instance ID 'i-2b97ac
2f' does not exist
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpCli
ent.java:538)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.ja
va:283)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:168
)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.jav
a:5208)
at com.amazonaws.services.ec2.AmazonEC2Client.startInstances(AmazonEC2Cl
ient.java:2426)
AWS Error Message: The instance ID 'i-2b97ac2f' does not exist
You'll have to take the AWS response for granted here, i.e. the instance does not exist ;)
But seriously: Presumably you have already verified that you are actually running an instance with this ID in your account? Then this is most likely caused by targeting the wrong API endpoint, insofar an instance ID is only valid within a specific region (if not specified, the region defaults to 'us-east-1', see below).
In this case you need to specify the actual instance region via the setEndpoint() method of the AmazonEC2Client object within the apparently global ec2 variable before calling startInstances().
There are some examples regarding Using Regions with the AWS SDKs and all currently available AWS regional endpoint URLs are listed in Regions and Endpoints, specifically the Amazon Elastic Compute Cloud (EC2) defaults to 'us-east-1':
If you just specify the general endpoint (ec2.amazonaws.com), Amazon
EC2 directs your request to the us-east-1 endpoint.
We run a service (Qubole) that frequently spawns and then tags (and in some cases terminates) AWS instances immediately.
We have found that Amazon will, every once in a while, claim an instanceid as invalid - even though it has just created it. Retrying a few times with some sleep time thrown in usually solves the problem. Even a total retry interval of 15s proved insufficient in rare cases.
This experience comes from the useast region. We do not make api calls to different regions - so that is not an explanation. More likely - this is the infamous eventual consistency at work - where AWS is unable to provide read-after-write consistency for these api calls.
I am using the AWS ruby api and I noticed the same issue when creating an AMI image and its status is pending when I look in the AWS console but after a while the image is available for use.
Here is my script
image = ec2.images.create(:name => image_name, :instance_id => ami_id, :description => desc)
sleep 5 while image.state != :available
I sleep for about 5 sec for image to be in available but I get the error saying that the "AWS Error Message: InvalidInstanceID.NotFound". During my testing this is fine but most of the time this seems to be failing during continuous integration builds.
InvalidInstanceID.NotFound means the specified instance does not exist.
Ensure that you have indicated the region in which the instance is located, if it's not in the default region.
This error may occur because the ID of a recently created instance has not propagated through the system. For more information, see Eventual Consistency.