I'm trying to use an API to download some XBRL files. In order to do that I need to do a curl request, like this:
curl -XGET http://distribution.virk.dk/offentliggoerelser --data-binary #query_regnskaber.json
The idea is, as I understand it, that "#query_regnskaber.json" is a json file / json query that I need to send with my request and in return I get a XBRL file(s) / some data. I'm using Java with the play framework (not specifically using play framework for the curl request though, but maybe someone know some play features to do curl requests).
This is my current code:
String jsonStr =
"{" +
"\"query\": {" +
"\"bool\": {" +
"\"must\": [" +
"{" +
"\"term\": {" +
"\"offentliggoerelse.dokumenter.dokumentMimeType\": \"application\"" +
"}" +
"}," +
"{" +
"\"term\": {" +
"\"offentliggoerelse.dokumenter.dokumentMimeType\": \"xml\"" +
"}" +
"}," +
"{" +
"\"range\": {" +
"\"offentliggoerelse.offentliggoerelsesTidspunkt\": {" +
"\"from\": \"2016-12-01\"" +
"}" +
"}" +
"}" +
"]," +
"\"must_not\": []," +
"\"should\": []" +
"}" +
"}," +
"\"size\": 1000" +
"}";
String urlStr = "http://distribution.virk.dk/offentliggoerelser";
JSONObject jsonObj = new JSONObject(jsonStr);
URL myURL = new URL(urlStr);
HttpURLConnection urlCon = (HttpURLConnection)myURL.openConnection();
urlCon.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
urlCon.setRequestMethod("GET");
urlCon.setDoInput(true);
urlCon.setDoOutput(true);
urlCon.connect();
OutputStream os = urlCon.getOutputStream();
os.write(jsonObj.toString().getBytes("UTF-8"));
os.close();
BufferedReader br = new BufferedReader(new InputStreamReader((urlCon.getInputStream())));
String output;
System.out.println("Output from Server .... \n");
while ((output = br.readLine()) != null) {
System.out.println(output);
}
urlCon.disconnect();
Something goes wrong and I'm not sure whether it's because of some missing settings, my code or both. I get the 403 error on the "urlCon.getInputStream()" call.
The only documentation I can find for the API is in Danish. It also mentions that it uses ElasticSearch, which I assume is used to find specific XBRL files that can be found on "http://distribution.virk.dk/offentliggoerelser/_search". Finding specific XBRL files is something I want to be able to do to. Just in case, here is a link to the API documentation.
I'm using the example json query that can be found in the documentation, in my code.
Thank you for your help.
My json test query:
{
"query": {
"bool": {
"must": [
{
"term": {
"offentliggoerelse.dokumenter.dokumentMimeType": "application"
}
},
{
"term": {
"offentliggoerelse.dokumenter.dokumentMimeType": "xml"
}
},
{
"range": {
"offentliggoerelse.offentliggoerelsesTidspunkt": {
"from": "2014-10-01"
}
}
}
],
"must_not": [],
"should": []
}
},
"size": 1000
}
it seems its a ElasticSearch at the backend and not much has changed,
to send query to http://distribution.virk.dk/offentliggoerelser is forbidden as is in ElasticSearch. (you can't query index directly)
But should work if you send POST (GET seems to work too) query to http://distribution.virk.dk/offentliggoerelser/_search (NOTE /_search)
so change
String urlStr = "http://distribution.virk.dk/offentliggoerelser";
to
String urlStr = "http://distribution.virk.dk/offentliggoerelser/_search";
Optionally change
urlCon.setRequestMethod("GET");
to
urlCon.setRequestMethod("POST");
NOTE:
in case you are wondering why your CURL works,
well it doesn't, because you use XGET instead of XPOST it simply ignores the query file you are sending thus spits out some information that don't correspond to your query
which is clearly wrong.
(Posted solution on behalf of the OP).
I added "/_search" to the site and changed the request method to POST. Explanation in nafas' answer.
Setting doOutput to TRUE turns it into a POST. Remove, along with whatever output you're sending. Request parameters should be in the URL for GET requests.
Related
I read through https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/#reference to begin with
My requirements
I want to use percolator. Is there any support for it in spring data elasticsearch? I don't see any in the above link although I understand that percolating is same as indexing (technically from using spring data elasticsearch's perspective). So I can use the indexing part of spring data elasticsearch but just checking if there are any that are specific to percolator.
I want to create an index dynamically. I do understand I can achieve that using SpEL template expression as mentioned in https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/#elasticsearch.mapping.meta-model.annotations but my case is slightly different, I will get the index name via the RequestParam as part of the API call. So this means as of my knowledge I cannot use SpEL or try something like https://stackoverflow.com/a/33520421/4068218
I see I can use ElasticsearchOperations or ElasticsearchRepository to create Index. Because of #2 (i.e index name via request parameter) I think ElasticsearchOperations better suits but I see IndexOperations facilitating createMapping, createSettings but not both together. I see putMapping too but I dont see anything that says both mapping and settings. The reason I want both is I want to create something like below to begin with
"settings" : {
"index" : {
"number_of_shards" : 1,
"number_of_replicas" : 0
}
},
"mappings": {
"properties": {
"message": {
"type": "text"
},
"query": {
"type": "percolator"
}
}
}
Bottom line :- How do I create an index (name of the index will be dynamic via request param) with mappings, settings using ElasticsearchOperations?
Any lead/help is much appreciated
First of all thank you very much #P.J.Meisch. Upvoted both your comments as a token of gratitude.
Below worked for me. Below might help others in future
Document mapping = Document.create().fromJson("{\n" +
"\n" +
" \"properties\": {\n" +
" \"message\": {\n" +
" \"type\": \"text\"\n" +
" },\n" +
" \"query\": {\n" +
" \"type\": \"percolator\"\n" +
" }\n" +
" }\n" +
"\n" +
"}");
Map<String, Object> settings = ImmutableMap.of( "number_of_shards" ,2,"number_of_replicas",1);
elasticsearchOperations.indexOps(IndexCoordinates.of("whatever-indexname-you-need")).create(settings,mapping);
We have an API which take a file from system and shows on the application, for which I am trying to automate with rest assured and Java
I have tried, changing the image to binary code and then adding it as parameter that does not work.
Map<String, String> paramSample = new HashMap<>();
paramSample.put("api_key", "A813302*************");
paramSample.put("method", "my");
paramSample.put("body", "{\n" +
" \"to\":\"91xxxxxxxx\",\n" +
" \"type\": \"image\", \"image\" : {\"caption\" : \"{{caption}}\"},\n" +
"\"callback\":\"{{callback}}\"\n" +
"}");
paramSample.put("from", "91xxxxxxx");
paramSample.put("file","C:\\Users\\sobhit.s\\Pictures\\SMS-2047.png");
RequestSpecification request = given();
Response responseSample = request.params(paramSample).get(ExecutionConfig.BASE_URL).then().extract().response();
String res=responseSample.prettyPrint();
Response is-
{
"status": "xxxx",
"message": "Invalid file format. Upload valid file."
}
First if you are unsure, do this in Postman and then recreate the same in code.
This way you will have a post man to demonstrate your coding problem.
Use .queryParam() only for params and not for the body content. Body content should be under .body()
Use .multiPart() to upload the file as a multi part quest. Hope this helps.
given().queryParam(
"api_key", "A813302*************",
"method", "my",
"from", "91xxxxxxx")
.body("{\n" +
" \"to\":\"91xxxxxxxx\",\n" +
" \"type\": \"image\", \"image\" : {\"caption\" : \"{{caption}}\"},\n" +
"\"callback\":\"{{callback}}\"\n" +
"}")
.multiPart(new File("C:/Users/sobhit.s/Pictures/SMS-2047.png"))
.when()
.get(ExecutionConfig.BASE_URL)
.prettyPrint();
I am using java low level REST client api.
I am fetching particular fields on matching or related datas from elasticsearch by using wildcards.
I am struck with it getting data as well as searching data with wildcard.
How to fetch result in java list .
Here is my code:
HttpEntity entity1 = new NStringEntity( "{\n" + " \"query\" : {\n" + " \"wildcard\": { \"Content\":\"java *\"} \n" + "} \n"+ "}",ContentType.APPLICATION_JSON);
Response response = restClient.performRequest("GET", "/test/_search",Collections.singletonMap("pretty", "true"), entity1);
System.out.println(EntityUtils.toString(response.getEntity()));
Could you please help me out? Thank you!
I am having some ajax difficulties (and am newish to ajax, but have done a fair bit of searching) when calling a java rest server that I have written. I have a test html page running within the Rest server (using Jetty) and I use the following call -- which works successfully using a .get:
$('#getRVLakeForm').submit(function(e) {
var handle = $('#insertRVLakeHandle').val();
var lakekey = $('#RVLakeKey').val();
var temp = $('#RVLakeTemp').val();
var speed = $('#RVLakeWindspeed').val();
var degrees = $('#RVLakeWindDegrees').val();;
$.get('${pageContext.request.contextPath}/api/recent/lake/' + handle + "/" + temp + "/" + speed + "/" + degrees + "/" + lakekey, function(data) {
alert(data.lake[0].oid);
$('#RMLakeResponse').text("back");
});
Here is the JSON the Rest server returns:
{"user":[],"lake":[{"oid":"519b9a1a3004f8fa1287502a","type":"Lake","handle":"nightstalker","viewedhandle":"","viewedlaketag":"TXFORK","viewedusername":"","timestamp":1369152026668}]}
This call executes the Rest Api and gets JSON back as expected... When I attempt to call the same Rest Api from HTML/PHP application, running under MAMP -- the ajax call works on the out bound call -- I can debug the Java Rest server and see the call come in, and it executes as designed creating JSON to send out. The problem is that I never get a call to the success function in the ajax call (I am not seeing the error function called either).
urlrecent = "http://localhost:8080/server/api/recent/lake/" + handle + "/" + temp + "/" + speed + "/" + degrees + "/" + lakekey;
$.ajax({
type: "GET",
contentType: "application/json",
dataType: "jsonp",
url: urlrecent,
success: function (data) {
alert("hello there!");
alert(data)
},
error: function () {
alert('failed');
}
});
I have even tried a getJSON instead..no luck. In the mean time I will continue search Stack and the web for something that will work. I have tried a dataType of just "json" as well.
Thanks in advance.
=====
Yes, munged the contentType, but that didn't matter.
Try it in IE8 and set dataType: "html"
urlrecent = "http://localhost:8080/server/api/recent/lake/" + handle + "/" + temp + "/" + speed + "/" + degrees + "/" + lakekey;
$.ajax({
type: "GET",
contentType: "application/json",
dataType: "html",
url: urlrecent,
success: function (data) {
alert("hello there!");
data = JSON.parse(data);
alert(data)
},
error: function () {
alert('failed');
}
});
if you get syntex error at data = JSON.parse(data) or "hello there" in alert , it is encoding problem. In this case you should use response.setCharacterEncoding("UTF8")
I am working json with struts2. When i send response to my jsp i am getting extra space in front of my message not creating problem for , How to truncate that space. My code as follows
HttpServletResponse response = ServletActionContext.getResponse();
response.setContentType("text/plain");
response.getWriter().write(sStatusMessage.trim());
response.flushBuffer();
response.reset();
Above code is my action code , when i reponse to jsp it taking extra space . How to remove that space
My json like this
fnOnDeleting: function (tr, id, fnDeleteRow) {
var oSettings = oTable.fnSettings();
jConfirm("Please confirm that you want to delete row " + oSettings.aoColumns[1].sTitle + ' with "' + $("td:first",tr).text() + '"', "Confirm Delete", function (r) {
if (r)
{
fnDeleteRow(id);
}
});
return false;
},
sAddDeleteToolbarSelector: ".dataTables_length"
});
I would try making the type JSON for starters:
http://en.wikipedia.org/wiki/JSON