We Keep Coding

Spring data elasticsearch to create index dynamically based on request parameter, percolator support and create index via Elasticsearch Operations

I read through https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/#reference to begin with
My requirements
I want to use percolator. Is there any support for it in spring data elasticsearch? I don't see any in the above link although I understand that percolating is same as indexing (technically from using spring data elasticsearch's perspective). So I can use the indexing part of spring data elasticsearch but just checking if there are any that are specific to percolator.
I want to create an index dynamically. I do understand I can achieve that using SpEL template expression as mentioned in https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/#elasticsearch.mapping.meta-model.annotations but my case is slightly different, I will get the index name via the RequestParam as part of the API call. So this means as of my knowledge I cannot use SpEL or try something like https://stackoverflow.com/a/33520421/4068218
I see I can use ElasticsearchOperations or ElasticsearchRepository to create Index. Because of #2 (i.e index name via request parameter) I think ElasticsearchOperations better suits but I see IndexOperations facilitating createMapping, createSettings but not both together. I see putMapping too but I dont see anything that says both mapping and settings. The reason I want both is I want to create something like below to begin with
"settings" : {
"index" : {
"number_of_shards" : 1,
"number_of_replicas" : 0
}
},
"mappings": {
"properties": {
"message": {
"type": "text"
},
"query": {
"type": "percolator"
}
}
}
Bottom line :- How do I create an index (name of the index will be dynamic via request param) with mappings, settings using ElasticsearchOperations?
Any lead/help is much appreciated

First of all thank you very much #P.J.Meisch. Upvoted both your comments as a token of gratitude.
Below worked for me. Below might help others in future
Document mapping = Document.create().fromJson("{\n" +
"\n" +
" \"properties\": {\n" +
" \"message\": {\n" +
" \"type\": \"text\"\n" +
" },\n" +
" \"query\": {\n" +
" \"type\": \"percolator\"\n" +
" }\n" +
" }\n" +
"\n" +
"}");
Map<String, Object> settings = ImmutableMap.of( "number_of_shards" ,2,"number_of_replicas",1);
elasticsearchOperations.indexOps(IndexCoordinates.of("whatever-indexname-you-need")).create(settings,mapping);

How to upload a file in restassured with java

We have an API which take a file from system and shows on the application, for which I am trying to automate with rest assured and Java
I have tried, changing the image to binary code and then adding it as parameter that does not work.
Map<String, String> paramSample = new HashMap<>();
paramSample.put("api_key", "A813302*************");
paramSample.put("method", "my");
paramSample.put("body", "{\n" +
" \"to\":\"91xxxxxxxx\",\n" +
" \"type\": \"image\", \"image\" : {\"caption\" : \"{{caption}}\"},\n" +
"\"callback\":\"{{callback}}\"\n" +
"}");
paramSample.put("from", "91xxxxxxx");
paramSample.put("file","C:\\Users\\sobhit.s\\Pictures\\SMS-2047.png");
RequestSpecification request = given();
Response responseSample = request.params(paramSample).get(ExecutionConfig.BASE_URL).then().extract().response();
String res=responseSample.prettyPrint();
Response is-
{
"status": "xxxx",
"message": "Invalid file format. Upload valid file."
}

First if you are unsure, do this in Postman and then recreate the same in code.
This way you will have a post man to demonstrate your coding problem.
Use .queryParam() only for params and not for the body content. Body content should be under .body()
Use .multiPart() to upload the file as a multi part quest. Hope this helps.
given().queryParam(
"api_key", "A813302*************",
"method", "my",
"from", "91xxxxxxx")
.body("{\n" +
" \"to\":\"91xxxxxxxx\",\n" +
" \"type\": \"image\", \"image\" : {\"caption\" : \"{{caption}}\"},\n" +
"\"callback\":\"{{callback}}\"\n" +
"}")
.multiPart(new File("C:/Users/sobhit.s/Pictures/SMS-2047.png"))
.when()
.get(ExecutionConfig.BASE_URL)
.prettyPrint();

How to get a file from a curl get request in Java?

I'm trying to use an API to download some XBRL files. In order to do that I need to do a curl request, like this:
curl -XGET http://distribution.virk.dk/offentliggoerelser --data-binary #query_regnskaber.json
The idea is, as I understand it, that "#query_regnskaber.json" is a json file / json query that I need to send with my request and in return I get a XBRL file(s) / some data. I'm using Java with the play framework (not specifically using play framework for the curl request though, but maybe someone know some play features to do curl requests).
This is my current code:
String jsonStr =
"{" +
"\"query\": {" +
"\"bool\": {" +
"\"must\": [" +
"{" +
"\"term\": {" +
"\"offentliggoerelse.dokumenter.dokumentMimeType\": \"application\"" +
"}" +
"}," +
"{" +
"\"term\": {" +
"\"offentliggoerelse.dokumenter.dokumentMimeType\": \"xml\"" +
"}" +
"}," +
"{" +
"\"range\": {" +
"\"offentliggoerelse.offentliggoerelsesTidspunkt\": {" +
"\"from\": \"2016-12-01\"" +
"}" +
"}" +
"}" +
"]," +
"\"must_not\": []," +
"\"should\": []" +
"}" +
"}," +
"\"size\": 1000" +
"}";
String urlStr = "http://distribution.virk.dk/offentliggoerelser";
JSONObject jsonObj = new JSONObject(jsonStr);
URL myURL = new URL(urlStr);
HttpURLConnection urlCon = (HttpURLConnection)myURL.openConnection();
urlCon.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
urlCon.setRequestMethod("GET");
urlCon.setDoInput(true);
urlCon.setDoOutput(true);
urlCon.connect();
OutputStream os = urlCon.getOutputStream();
os.write(jsonObj.toString().getBytes("UTF-8"));
os.close();
BufferedReader br = new BufferedReader(new InputStreamReader((urlCon.getInputStream())));
String output;
System.out.println("Output from Server .... \n");
while ((output = br.readLine()) != null) {
System.out.println(output);
}
urlCon.disconnect();
Something goes wrong and I'm not sure whether it's because of some missing settings, my code or both. I get the 403 error on the "urlCon.getInputStream()" call.
The only documentation I can find for the API is in Danish. It also mentions that it uses ElasticSearch, which I assume is used to find specific XBRL files that can be found on "http://distribution.virk.dk/offentliggoerelser/_search". Finding specific XBRL files is something I want to be able to do to. Just in case, here is a link to the API documentation.
I'm using the example json query that can be found in the documentation, in my code.
Thank you for your help.
My json test query:
{
"query": {
"bool": {
"must": [
{
"term": {
"offentliggoerelse.dokumenter.dokumentMimeType": "application"
}
},
{
"term": {
"offentliggoerelse.dokumenter.dokumentMimeType": "xml"
}
},
{
"range": {
"offentliggoerelse.offentliggoerelsesTidspunkt": {
"from": "2014-10-01"
}
}
}
],
"must_not": [],
"should": []
}
},
"size": 1000
}

it seems its a ElasticSearch at the backend and not much has changed,
to send query to http://distribution.virk.dk/offentliggoerelser is forbidden as is in ElasticSearch. (you can't query index directly)
But should work if you send POST (GET seems to work too) query to http://distribution.virk.dk/offentliggoerelser/_search (NOTE /_search)
so change
String urlStr = "http://distribution.virk.dk/offentliggoerelser";
to
String urlStr = "http://distribution.virk.dk/offentliggoerelser/_search";
Optionally change
urlCon.setRequestMethod("GET");
to
urlCon.setRequestMethod("POST");
NOTE:
in case you are wondering why your CURL works,
well it doesn't, because you use XGET instead of XPOST it simply ignores the query file you are sending thus spits out some information that don't correspond to your query
which is clearly wrong.

(Posted solution on behalf of the OP).
I added "/_search" to the site and changed the request method to POST. Explanation in nafas' answer.

Setting doOutput to TRUE turns it into a POST. Remove, along with whatever output you're sending. Request parameters should be in the URL for GET requests.

RallyRestAPI querying ScopedAttributeDefinition types

Using the RallyRestAPI, is there a way to query ScopedAttributeDefinition types? This appears to define custom datatypes in Rally. I am trying to build a data dictionary of the custom types we use in Rally and associate these custom types to the object they are attributes of. (In case that doesn't make sense, here's an example: We have a custom field called Enabler Lead on Rally PortfolioItems. I'd like to query Rally for all custom fields for PortfolioItem and get the Enabler Field, and its metadata, from the Rally REST API.)
I'm using the Java client.

I filed a github issue to add support for the ScopedAttributeDefinition: https://github.com/RallyTools/RallyRestToolkitForJava/issues/19
In the meantime though you can work around it by using the underlying http client directly. This code queries for all projects in your workspace and then finds the type definition for Portfolio Item and then for each project grabs all the custom attribute defs via the scopedattributedefinition endpoint and prints out their hidden/required status.
QueryRequest projectQuery = new QueryRequest("project");
projectQuery.setLimit(Integer.MAX_VALUE);
QueryResponse projectResponse = restApi.query(projectQuery);
QueryRequest typeDefQuery = new QueryRequest("typedefinition");
typeDefQuery.setQueryFilter(new QueryFilter("Name", "=", "Portfolio Item"));
QueryResponse typeDefResponse = restApi.query(typeDefQuery);
JsonObject piTypeDef = typeDefResponse.getResults().get(0).getAsJsonObject();
for(JsonElement projectResult : projectResponse.getResults()) {
JsonObject project = projectResult.getAsJsonObject();
System.out.println("Project: " + project.get("Name").getAsString());
//Begin hackery (note we're not handling multiple pages-
// if you have more than 200 custom attributes you'll have to page
String scopedAttributeDefUrl = "/project/" + project.get("ObjectID").getAsLong() +
"/typedefinition/" + piTypeDef.get("ObjectID").getAsLong() + "/scopedattributedefinition" +
"?fetch=Hidden,Required,Name&query=" + URLEncoder.encode("(Custom = true)", "utf-8");
String attributes = restApi.getClient().doGet(scopedAttributeDefUrl);
QueryResponse attributeResponse = new QueryResponse(attributes);
//End hackery
for(JsonElement customAttributeResult : attributeResponse.getResults()) {
JsonObject customAttribute = customAttributeResult.getAsJsonObject();
System.out.print("\tAttribute: " + customAttribute.get("Name").getAsString());
System.out.print(", Hidden: " + customAttribute.get("Hidden").getAsBoolean());
System.out.println(", Required: " + customAttribute.get("Required").getAsBoolean());
}
System.out.println();
}
Any other info you'd like for each of those custom fields should be accessible just by querying the Attributes collection from the piTypeDef.

How to return JsonArray in a Java Response object

I'm trying to implement java based web-service server which returns to Json and java script based web-service client. Here is my java part :
#Path("/myapp")
#Consumes(MediaType.APPLICATION_JSON)
#Produces(MediaType.APPLICATION_JSON)
public class MyRecommender {
#POST
#Path("/recs")
public Response getRecommendations() {
//Assume recommendation List contains some Recommendation objects
//I removed it for simplicity.
List<Recommendation> recommendation;
JsonArrayBuilder builder = Json.createArrayBuilder();
for (Recommendation rec : recommendations) {
builder.add(Json.createObjectBuilder().add("firstPersonName", "\"" + rec.getFirstPerson().getName() + "\"")
.add("firsPersonURL", "\"" + rec.getFirstPerson().getURL() + "\"")
.add("secondPersonName", "\"" + rec.getSecondPerson().getName() + "\"")
.add("secondPersonURL", "\"" + rec.getSecondPerson().getURL() + "\"")
.add("score", "\"" + rec.getSimilarity() + "\""));
}
JsonArray jsonData = builder.build();
return Response.ok(jsonData, MediaType.APPLICATION_JSON).header("Access-Control-Allow-Origin", "*")
.header("Access-Control-Allow-Methods", "POST").allow("OPTIONS").build();
}
}
}
Now, when I call this function from my js client, I got :
POST http://localhost:8080/myapp/recs 500 (Request failed.)
But when I replace the for loop and return with the following code snipped I got response in my js correctly.
Changed part :
// remove for loop and change jsonData type.
String jsonData = "{\"name\":\"John\"}";
return Response.ok(jsonData, MediaType.APPLICATION_JSON).header("Access-Control-Allow-Origin", "*")
.header("Access-Control-Allow-Methods", "POST").allow("OPTIONS").build();
So, I wonder what might be problem ? Since its my first time with web-services, I have some difficulty to debug my code.
EDIT
By the way, I get also another error when I try to first version of getRecommendations() functions(with loop)
XMLHttpRequest cannot load http://localhost:8080/myapp/recs.
No 'Access-Control-Allow-Origin' header is present on the requested resource.
Origin 'http://localhost:3000' is therefore not allowed access.
The response had HTTP status code 500.
But as I said, When I remove the loop and put second code snipped into getRecommendations() functions, both of the errors are gone and I get the response in my website.
EDIT2
When I changed the loop and return statement of getRecommendations() function with the below I again get the same errors
JsonObject value = Json.createObjectBuilder()
.add("name", "John").build();
return Response.ok(value, MediaType.APPLICATION_JSON).header("Access-Control-Allow-Origin", "*")
.header("Access-Control-Allow-Methods", "POST").allow("OPTIONS").build();
EDIT 3
As far as I understood, createObjectBuilder().build() or JsonArrayBuilder.build() return an JSON object and below of this build statement in my getRecommendations() function is not even run. So, I think my problem how could I give Access-Control-Allow-Origin permission to this object?

If it is ok to use a third Party Library, using Google's Gson could be an efficient solution in this case. When converting Models to JSON it is quite handy.
What you can do is something like this.
ArrayList<Recommendation> recommendationList = new ArrayList<Recommendation>();
Gson gson = new Gson();
String jsonData = gson.toJson(reccomendationList);
To use it as a dependency in your POM file you could do this.
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.3.1</version>
<scope>compile</scope>
</dependency>

You should try simplest approach:
return Response.ok().entity(recommendation).header("Access-Control-Allow-Origin", "*")
.header("Access-Control-Allow-Methods", "POST").allow("OPTIONS").build();