I'm coding a console app to Insert data from JSOIN files into Elasticsearch 7.5.1. The _id field should be an int and act like auto-increment. To make that happen, before inserting I get the last ID inserted and increment 1.
My problem is that I'm having trouble getting the last ID, because the ordering is happening on a string. This is what I mean: if you have 10 items and the last ID is 10, when querying and sorting it will return 9.
This is my query when using Postman:
GET my_index/_search
{
"size": 1,
"query": {
"match_all": {}
},
"sort": [{
"_id": {
"order": "desc"
}
}
]
}
And my Java function using their client:
private static String getLastElasticSearchId(String index)
{
RestHighLevelClient client = getElasticSearchClient();
SearchRequest searchRequest = new SearchRequest(index);
SearchSourceBuilder b = new SearchSourceBuilder();
b.query(QueryBuilders.matchAllQuery());
b.sort(new FieldSortBuilder("_id").order(SortOrder.DESC));
b.from(0);
b.size(1);
searchRequest.source(b);
try {
SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
SearchHits hits = searchResponse.getHits();
if(hits.getTotalHits().value > 0){
SearchHit[] searchHits = hits.getHits();
for (SearchHit hit : searchHits) {
return hit.getId();
}
}
else {
return "0";
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
client.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return null;
}
Both return 9 as the last ID, even having another item in the index with 10 as ID.
I cannot apply string padding in the IDs as answered here.
How can I achieve what I need?
Try this:
GET my_index/_search
{
"size": 1,
"query": {
"match_all": {}
},
"sort" : {
"_script" : {
"type" : "number",
"script" : {
"lang": "painless",
"source": "Integer.parseInt(doc['_id'].value)"
},
"order" : "desc"
}
}
}
Hope this helps
I'm new to development part using API and JSON file, so be patient with me. I'm trying to implement a Rest API in java where I need to send a Post request to an URL to process but when I tried it in my localhost, the result is that what I receive is just empty Json file.
I was using the dependency from org.json.simple.JSONObject, but I need to change the dependency to org.json.JSONObject. I know that they are two different library and that's why I am a bit stuck. I looked on the forum and on the internet but I didn't find a solution for my own problem. If it is possible I want to also ask if there is a way to convert a String to a JSON.
Here is the main class.
public class DataService {
public static JSONObject processData(JSONObject jsonObject) {
System.out.println(jsonObject);
System.out.println(jsonObject.toString());
ObjectMapper mapper = new ObjectMapper();
mapper.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES);
Data data = new Data();
try {
data = mapper.readValue(jsonObject.toString(), Data.class);
} catch (IOException e) {
e.printStackTrace();
}
List<DataJson> timeserie = data.getData();
List<Double> values = new ArrayList<Double>();
DataJson inter;
for (int i = 0; i<timeserie.size(); i++){
inter = timeserie.get(i);
values.add(inter.getValue());
}
int EmbeddingDimension;
EmbeddingDimension = data.getEmbeddingDimension();
data.setResult(DynamicProperties.PermEn(values, EmbeddingDimension));
String url = "http://localhost:8080/crosscpp/toolbox/test";
OkHttpClient client = new OkHttpClient();
ObjectMapper objectMapper = new ObjectMapper();
RequestBody body = null;
try {
body = RequestBody.create(
MediaType.parse("application/json; charset=utf-8"), objectMapper.writeValueAsString(data));
} catch (JsonProcessingException e) {
e.printStackTrace();
}
Request request = new Request.Builder()
.url(url)
.post(body)
.build();
Call call = client.newCall(request);
try {
Response response = call.execute();
String result = response.body().string();
JSONObject json = null;
/*JSONParser parser = new JSONParser();
JSONObject json = null;
try {
json = (JSONObject) parser.parse(result);
} catch (ParseException e) {
e.printStackTrace();
}*/ //Look for another solution.
return json;
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
The Json file I will send.
{
"inputAeonSubscriptionUrl": "xxxx",
"outputAeonPublicationUrl": "xxxx",
"EmbeddingDimension": 3,
"offerId": "xxxxxx",
"measurement-channel-id": "1",
"id": "xxxxxx",
"submissionDate": {
"min": "2019-04-09",
"max": "2019-05-07"
},
"travelDates": {
"min": "2019-05-13",
"max": "2019-05-17"
},
"travelledDuration": {
"min": 1,
"max": 2
},
"geoBoundingBox": {
"latitude-max": 51.507561,
"latitude-min": 51.497715,
"longitude-min": 7.482349,
"longitude-max": 7.500885
},
"data": [
{
"value": 1,
"timestamp": "2019-04-09"
},
{
"value": 3,
"timestamp": "2019-04-09"
},
{
"value": 2,
"timestamp": "2019-04-09"
},
{
"value": 1,
"timestamp": "2019-04-09"
},
{
"value": 2,
"timestamp": "2019-04-10"
},
{
"value": 3,
"timestamp": "2019-04-10"
},
{
"value": 2,
"timestamp": "2019-04-10"
}
]
}
I am expecting a result where it send back the received JSON file with an added attribute where it shows the process done on the values.
I am new to elastic search , i am reading the doc so far its good but i am not able to write a method to get by range, Below is a method that gets by ID which works perfectly but how can i do this to get a list of data that match the price range.
public Map<String, Object> getTourById(String id){
GetRequest getRequest = new GetRequest(INDEXTOUR, TYPETOUR, id);
GetResponse getResponse = null;
try {
getResponse = restHighLevelClient.get(getRequest);
} catch (java.io.IOException e){
e.printStackTrace();
e.getLocalizedMessage();
}
Map<String, Object> sourceAsMap = getResponse.getSourceAsMap();
return sourceAsMap;
}
Above method works fine now below is the method to get by range and return datas that matches the price
public Map<String, Object> getTourByPriceRange(int minPrice, int maxPrice) {
GetRequest getRequest = new GetRequest(INDEXTOUR, TYPETOUR, "requires an ID");
QueryBuilder qb = QueryBuilders
.rangeQuery("price")
.from(minPrice)
.to(maxPrice)
.includeLower(true)
.includeUpper(true);
GetResponse getResponse = null;
try {
getResponse = restHighLevelClient.get(getRequest);
} catch (java.io.IOException e) {
e.printStackTrace();
e.getLocalizedMessage();
}
Map<String, Object> sourceAsMap = getResponse.getSourceAsMap();
return sourceAsMap;
}
Only the query builder is accurate in the above method, the result i want to get is a bunch of data that price fall in the range given.
{
"departure": {
"city": "\u0438\u043d\u0441\u043a",
"date": "2018-08-10"
},
"extras": [],
"hotel": {
"beach": {
"distance": 0,
"type": "\u041f\u0435\u0447\u0430\u043d\u044b\u0439"
},
"country": "\u0413\u0440\u0446\u0438\u044f",
"distanceToAirport": 0,
"facilities": [
"Standard Without Balcony"
],
"food": "\u0422\u043e\u043b\u044c\u043a\u043e \u0437\u0430\u0432\u0442\u0440\u0430\u043a\u0438",
"photoUrl": "https://s1.tez-tour.com/hotel/7021893.jpg",
"regionName": "\u0425\u0430\u043b\u043a\u0438\u0434\u0438\u043a\u0438 - \u041a\u0430\u0441\u0441\u0430\u043d\u0434\u0440\u0430",
"stars": 4,
"title": "KALLIKRATEIA"
},
"nights": 8,
"people": 1,
"price": 2595 // i want to use price as parameters for the search method
}
Good job so far!! In order to issue a search request you need to use SearchRequest and not GetRequest which is meant to retrieve a single document by ID.
QueryBuilder qb = QueryBuilders
.rangeQuery("price")
.from(minPrice)
.to(maxPrice)
.includeLower(true)
.includeUpper(true);
SearchRequest searchRequest = new SearchRequest(INDEXTOUR);
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.query(qb);
searchRequest.types(TYPETOUR);
searchRequest.source(searchSourceBuilder);
SearchResponse searchResponse = null;
try {
searchResponse = restHighLevelClient.search(searchRequest);
} catch (java.io.IOException e) {
e.printStackTrace();
e.getLocalizedMessage();
}
Am writing Spring Boot rest service to get products from elastic search based on _id ( from elastic search ). Am able to print the response from the service, means service able to fetch but while converting to JSONObject to return back to UI its throwing below error
No converter found for return value of type: class org.json.JSONObject
Please find my code below.
public JSONObject getProductById(String id){
String[] includes = new String[]{id};
String[] excludes = Strings.EMPTY_ARRAY;
GetRequest getRequest = new GetRequest(INDEX, TYPE, SOURCE);
getRequest.routing(id);
GetResponse getResponse = null;
try {
getResponse = restHighLevelClient.get(getRequest);
} catch (java.io.IOException e){
e.getLocalizedMessage();
}
//GetResponse getResponse = null;
// create the search request
SearchRequest searchRequest = new SearchRequest(INDEX);
searchRequest.types(TYPE);
// create the match query on the author field
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
MatchQueryBuilder matchQueryBuilder = new MatchQueryBuilder("_id", id);
searchSourceBuilder.query(matchQueryBuilder);
searchRequest.source(searchSourceBuilder);
// send the request
SearchResponse searchResponse = null;
try {
searchResponse = restHighLevelClient.search(searchRequest);
logger.info("response ---->"+searchResponse);
} catch (IOException e) {
e.getLocalizedMessage();
}
// read the response
String productName = null;
Product product = null;
SearchHit[] searchHits = searchResponse.getHits().getHits();
for (SearchHit hit : searchHits) {
// get each hit as a Map
Map<String, Object> sourceAsMap = hit.getSourceAsMap();
product=new Product();
product.setName(sourceAsMap.get("name").toString());
/*productName = (String) sourceAsMap.get("name");*/
}
Gson gson=new Gson();
JSONObject productJSON = null;
String prodStr=gson.toJson(product);
try {
productJSON=new JSONObject(prodStr);
} catch (JSONException e) {
e.printStackTrace();
}
return productJSON;
}
You do not need to convert the product to JSONObject as Spring Boot does the serialization automatically. Just change method return type to Product and directly return product.
Example & Guide: https://spring.io/guides/gs/rest-service/
This is what I have, but the number of children never prints out. I'm getting the raw JSON, then making a JSONArray, accessing the second member's children. What am I missing here? I have similar code that works perfectly, only difference is in the JSON, it does not start with an array
JSON Input:
[
{
"kind":"Listing",
"data":{
"modhash":"",
"children":[
{
"kind":"t3",
"data":{
"domain":"",
"banned_by":null,
"media_embed":{
},
"subreddit":"",
"selftext_html":"",
"selftext":"",
"likes":null,
"secure_media":null,
"link_flair_text":null,
"id":"1zeek5",
"secure_media_embed":{},
"clicked":false,
"stickied":false,
"author":"xVictoryy",
"media":null,
"score":1,
"approved_by":null,
"over_18":false,
"hidden":false,
"thumbnail":"",
"subreddit_id":"t5_2sdpm",
"edited":false,
"link_flair_css_class":null,
"author_flair_css_class":null,
"downs":0,
"saved":false,
"is_self":true,
"permalink":"",
"name":"t3_1zeek5",
"created":1393843471.0,
"url":"",
"author_flair_text":null,
"title":"Seeking advice.",
"created_utc":1393814671.0,
"ups":1,
"num_comments":3,
"visited":false,
"num_reports":null,
"distinguished":null
}
}
],
"after":null,
"before":null
}
},
{
"kind":"Listing",
"data":{
"modhash":"",
"children":[
{
"kind":"t1",
"data":{
"subreddit_id":"t5_2sdpm",
"banned_by":null,
"subreddit":"",
"likes":null,
"replies":{
"kind":"Listing",
"data":{
"modhash":"",
"children":[
{
"kind":"t1",
"data":{
"subreddit_id":"t5_2sdpm",
"banned_by":null,
"subreddit":"cscareerquestions",
"likes":null,
"replies":"",
"saved":false,
"id":"cfsxjqn",
"gilded":0,
"author":"xVictoryy",
"parent_id":"t1_cfsx26m",
"approved_by":null,
"body":"",
"edited":false,
"author_flair_css_class":null,
"downs":0,
"body_html":"",
"link_id":"t3_1zeek5",
"score_hidden":false,
"name":"t1_cfsxjqn",
"created":1393845230.0,
"author_flair_text":null,
"created_utc":1393816430.0,
"distinguished":null,
"num_reports":null,
"ups":1
}
}
],
"after":null,
"before":null
}
},
"saved":false,
"id":"cfsx26m",
"gilded":0,
"author":"dauphic",
"parent_id":"t3_1zeek5",
"approved_by":null,
"body":"A lot of schools don't expect high school Calculus.",
"edited":false,
"author_flair_css_class":"",
"downs":0,
"body_html":"",
"link_id":"t3_1zeek5",
"score_hidden":false,
"name":"t1_cfsx26m",
"created":1393844079.0,
"author_flair_text":"Software Engineer",
"created_utc":1393815279.0,
"distinguished":null,
"num_reports":null,
"ups":1
}
},
{
"kind":"t1",
"data":{
"subreddit_id":"t5_2sdpm",
"banned_by":null,
"subreddit":"cscareerquestions",
"likes":null,
"replies":"",
"saved":false,
"id":"cft3lbj",
"gilded":0,
"author":"I_EAT_GUSHERS",
"parent_id":"t3_1zeek5",
"approved_by":null,
"body":"",
"edited":false,
"author_flair_css_class":"",
"downs":0,
"body_html":"",
"link_id":"t3_1zeek5",
"score_hidden":false,
"name":"t1_cft3lbj",
"created":1393864015.0,
"author_flair_text":"Looking for internship",
"created_utc":1393835215.0,
"distinguished":null,
"num_reports":null,
"ups":1
}
}
],
"after":null,
"before":null
}
}
]
My code:
List<Comment> fetchComments() {
Log.d("running", "attempting fetch...");
String raw = RemoteData.readContents(url);
List<Comment> list = new ArrayList<Comment>();
try {
JSONObject data = new JSONArray(raw).getJSONObject(1);
JSONArray children = data.getJSONArray("children");
Log.d("running", "comments: " + children.length());
}
} catch (Exception e) {
Log.e("fetchComments()", e.toString());
}
return list;
}
public static String readContents(String url){
HttpURLConnection hcon=getConnection(url);
if(hcon==null) return null;
try{
StringBuffer sb=new StringBuffer(8192);
String tmp="";
BufferedReader br=new BufferedReader(
new InputStreamReader(
hcon.getInputStream()
)
);
while((tmp=br.readLine())!=null)
sb.append(tmp).append("\n");
br.close();
return sb.toString();
}catch(IOException e){
Log.d("READ FAILED", e.toString());
return null;
}
}
You didn't get into data object... You only have "kind" and "data" tags in your list items, so first get into "data" tag then get "children". Try like this:
List<Comment> fetchComments() {
Log.d("running", "attempting fetch...");
String raw = RemoteData.readContents(url);
List<Comment> list = new ArrayList<Comment>();
try {
JSONObject data = new JSONArray(raw).getJSONObject(1);
JSONArray children = data.getJSONObject("data").getJSONArray("children");
Log.d("running", "comments: " + children.length());
}
} catch (Exception e) {
Log.e("fetchComments()", e.toString());
}
return list;
}
Your JSON array contains objects that have a field "data" that contains an object that contains a field "children".
You're doing:
JSONObject data = new JSONArray(raw).getJSONObject(1);
JSONArray children = data.getJSONArray("children");
You missed the data field.
JSONObject obj = new JSONArray(raw).getJSONObject(1);
JSONArray children = obj.getJSONObject("data").getJSONArray("children");