We Keep Coding

How to get a file from a curl get request in Java?

I'm trying to use an API to download some XBRL files. In order to do that I need to do a curl request, like this:
curl -XGET http://distribution.virk.dk/offentliggoerelser --data-binary #query_regnskaber.json
The idea is, as I understand it, that "#query_regnskaber.json" is a json file / json query that I need to send with my request and in return I get a XBRL file(s) / some data. I'm using Java with the play framework (not specifically using play framework for the curl request though, but maybe someone know some play features to do curl requests).
This is my current code:
String jsonStr =
"{" +
"\"query\": {" +
"\"bool\": {" +
"\"must\": [" +
"{" +
"\"term\": {" +
"\"offentliggoerelse.dokumenter.dokumentMimeType\": \"application\"" +
"}" +
"}," +
"{" +
"\"term\": {" +
"\"offentliggoerelse.dokumenter.dokumentMimeType\": \"xml\"" +
"}" +
"}," +
"{" +
"\"range\": {" +
"\"offentliggoerelse.offentliggoerelsesTidspunkt\": {" +
"\"from\": \"2016-12-01\"" +
"}" +
"}" +
"}" +
"]," +
"\"must_not\": []," +
"\"should\": []" +
"}" +
"}," +
"\"size\": 1000" +
"}";
String urlStr = "http://distribution.virk.dk/offentliggoerelser";
JSONObject jsonObj = new JSONObject(jsonStr);
URL myURL = new URL(urlStr);
HttpURLConnection urlCon = (HttpURLConnection)myURL.openConnection();
urlCon.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
urlCon.setRequestMethod("GET");
urlCon.setDoInput(true);
urlCon.setDoOutput(true);
urlCon.connect();
OutputStream os = urlCon.getOutputStream();
os.write(jsonObj.toString().getBytes("UTF-8"));
os.close();
BufferedReader br = new BufferedReader(new InputStreamReader((urlCon.getInputStream())));
String output;
System.out.println("Output from Server .... \n");
while ((output = br.readLine()) != null) {
System.out.println(output);
}
urlCon.disconnect();
Something goes wrong and I'm not sure whether it's because of some missing settings, my code or both. I get the 403 error on the "urlCon.getInputStream()" call.
The only documentation I can find for the API is in Danish. It also mentions that it uses ElasticSearch, which I assume is used to find specific XBRL files that can be found on "http://distribution.virk.dk/offentliggoerelser/_search". Finding specific XBRL files is something I want to be able to do to. Just in case, here is a link to the API documentation.
I'm using the example json query that can be found in the documentation, in my code.
Thank you for your help.
My json test query:
{
"query": {
"bool": {
"must": [
{
"term": {
"offentliggoerelse.dokumenter.dokumentMimeType": "application"
}
},
{
"term": {
"offentliggoerelse.dokumenter.dokumentMimeType": "xml"
}
},
{
"range": {
"offentliggoerelse.offentliggoerelsesTidspunkt": {
"from": "2014-10-01"
}
}
}
],
"must_not": [],
"should": []
}
},
"size": 1000
}

it seems its a ElasticSearch at the backend and not much has changed,
to send query to http://distribution.virk.dk/offentliggoerelser is forbidden as is in ElasticSearch. (you can't query index directly)
But should work if you send POST (GET seems to work too) query to http://distribution.virk.dk/offentliggoerelser/_search (NOTE /_search)
so change
String urlStr = "http://distribution.virk.dk/offentliggoerelser";
to
String urlStr = "http://distribution.virk.dk/offentliggoerelser/_search";
Optionally change
urlCon.setRequestMethod("GET");
to
urlCon.setRequestMethod("POST");
NOTE:
in case you are wondering why your CURL works,
well it doesn't, because you use XGET instead of XPOST it simply ignores the query file you are sending thus spits out some information that don't correspond to your query
which is clearly wrong.

(Posted solution on behalf of the OP).
I added "/_search" to the site and changed the request method to POST. Explanation in nafas' answer.

Setting doOutput to TRUE turns it into a POST. Remove, along with whatever output you're sending. Request parameters should be in the URL for GET requests.

Cannot get angular $http to work

I'm trying to get a simple angularjs login screen to work.
The angular app sends the login details via http get method to a java servlet and anticipates a json response with success/fail. The java servlet is running on Tomcat 8.0.
Unfortunately, the angular app doesn't seem to be able to receive data from the servlet (it does send the data to the servlet) - the errorCallback method of "then" is called everytime.
Also, accessing the servlet's url directly from the browser works fine (the browser shows the response string).
Can you help me to find the problem?
This is the div element in the html page:
<div ng-controller = "loginCtrl">
<input type="text" ng-model = "userName" placeholder="Username"></input><br>
<input type = "text" ng-model = "userPass" placeholder="Password"></input><br>
<button type = "button" ng-click = "login()">Login</button><br>
{{message}}
</div>
This is the js code:
var expenseApp = angular.module("expenseApp",[]);
expenseApp.controller('loginCtrl',['$scope','$http',function($scope,$http) {
$scope.userName = "";
$scope.userPass = "";
$scope.message = "type in your credentials";
$scope.login = function() {
var address = "http://localhost:8080/ExpenseSystemServer/LoginServlet?userName=" + $scope.userName + "&userPass=" + $scope.userPass;
$http({method:'get',
url:address})
.then(function(data, status, headers, config) {
$scope.message = "http success";
},
function(data, status, headers, config) {
$scope.message = "http error";
});
};
}]);
And this is the servlet doGet method in java (the servlet's class name is LoginServlet):
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
response.setContentType("application/json");
response.setCharacterEncoding("UTF-8");
String userName = request.getParameter("userName");
String userPass = request.getParameter("userPass");
System.out.print("Login attempt with " + userName + "; " + userPass + ": ");
if (userName == null || userPass == null || !userName.equals("admin") || !userPass.equals("pass")){
response.getWriter().write("{'success':false,'message':'wrong details'}");
System.out.println("failed.");
}
else {
response.getWriter().write("{'success':true, 'user':{'name':'admin'},'message':'Hi admin!'}");
System.out.println("succeeded.");
}
}
Can you help me here?
Thanks.

You're sending invalid JSON from the servlet. JSON keys and string values must be in double quotes, not single quotes. Use JSONLint to validate your JSON. Or better, create a Java object, and a marshaller like Jackson to transform the object to valid JSON.
Also, instead of sending back a successful response (with code 200), with an object property "success" set to false, you should return an error response (400 if the required credentials are not there at all, 401 if they're not valid). Doing that not only shows respect for the HTTP protocol, but allows using http promises as intended:
http(...).then(successCallback, errorCallback)
instead of
http(...).then(successButActuallyMaybeErrorCallback, anotherErrorCallback)

Your JSON is invalid using single quotes that should be double
It's a really bad practice to create your own when most languages can encode it for you from native arrays and objects. I don't know best way using java but it needs to look like:
response.getWriter().write('{"success":false,"message":"wrong details"}');
note switch of single and double quotes

Java : How to handle POST request without form?

I'm sending a http post request from javascript, with some json data.
Javascript
var data = {text : "I neeed to store this string in database"}
var xhr= new XMLHttpRequest();
xhr.open("POST","http://localhost:9000/postJson" , true);
xhr.setRequestHeader("Content-Type", "application/x-www-form-urlencoded; charset=UTF-8");
xhr.send(data);
xhr.setRequestHeader("Connection", "close");
//Also, I've tried a jquery POST
//$.post('postJson', {'data=' : JSON.stringify(data)});
//But this doesn't make a request at all. What am I messing up here?
Route
POST /postJson controllers.Application.postJson()
Controller
public static Result postJson(){
//What should I write here to get the data
//I've tried the below but values is showing null
RequestBody rb=request().body();
final Map<String,String[]> values=rb.asFormUrlEncoded();
}
What is the way to parse the POST request body?
Much thanks!

Retreive the request body directly as JSON... no need to complicate your life.
public static Result postJson() {
JsonNode rb = request().body().asJson();
//manipulate the result
String textForDBInsertion = rb.get("text").asText(); //retreives the value for the text key as String
Logger.debug("text for insertion: " + textForDBInsertion
+ "JSON from request: " + rb);
return ok(rb);
}
Also, I recommend you use the AdvancedRestClient Chrome plugin for testing. This way you can eliminate from the equation client-side code errors.
Cheers!

REST disregards any part that is not English characters , when receiving UTF-8

Sending ajax request in UTF8 to the server that uses REST , disregards any part that is not English characters
I'm using JAVA with REST on the server side , and the client sends ajax requests in UTF8 , that includes Hebrew words .
The AJAX request :
var clientNumber = '12344432432';
var userID = '321321321';
var someHebrewWord = ...;
var someEnglishWord = ....;
var servletUrl = '/Management/services/manager/' + clientNumber + '/' + userID + '/' + someEnglishWord + '/' someHebrewWord;
alert('Sending this :' + servletUrl);
$.ajax({
url: servletUrl,
type: 'POST',
cache: false,
data: { },
success: function(data){
alert('Return value is:' + data);
window.location = "./replyPage.html";
}
, error: function(jqXHR, textStatus, err){
alert('text status '+textStatus+', err '+err + " " + JSON.stringify(jqXHR));
}
});
On the server side , I use REST :
#Produces({ "application/json" })
#Path("/manager")
public class Management {
#POST
#Path("{clientNumber }/{userID }/{someEnglishWord}/{someHebrewWord}")
#Produces("application/json")
public boolean insert(#PathParam("clientNumber") String clientNumber, #PathParam("userID") String userID,
#PathParam("someEnglishWord") String someEnglishWord, #PathParam("someHebrewWord") String someHebrewWord)
{
// do some stuff
}
#POST
#Path("{clientNumber }/{userID }/{someEnglishWord}")
#Produces("application/json")
public boolean updateSomething(#PathParam("clientNumber") String clientNumber, #PathParam("userID") String userID , #PathParam("someEnglishWord") String someEnglishWord)
{
// do other stuff
}
// more code
}
So , when the AJAX request is sent , the updateSomething() is invoked instead of insert() ,
even though I'm sending 4 fields , and not 3 !
What causes this , and how can I fix it ?
Much appreciated

Allowed characters in a URL is restricted. You have to encode the URL with encodeURIComponent.
A better option might be posting those parameters in a data -variable and using #FormParam instead of #PathParam.

How I can do facebook batch fql in java

in facebook fql theres this code
https://developers.facebook.com/docs/reference/api/batch/
curl \
-F 'access_token=…' \
-F 'batch=[ \
{"method": "GET", "relative_url": "me"}, \
{"method": "GET", "relative_url": "me/friends?limit=50"} \
]'\
https://graph.facebook.com
it suppose to to be sent with json
but I really dont understand how to do this
any help ?
thanks

You can simple use the BatchFB api its very powerful and easy , you dont have to deal will all of these stuff and it use the fql
for example to get all your friends
Later<ArrayNode> friendsArrayList = this.Batcher.query("SELECT uid FROM user WHERE uid IN (SELECT uid2 FROM friend WHERE uid1 = me())");
for (JsonNode friend : friendsArrayList.get()) {
.......
}
and its batched

I believe your question is how to execute a batch request using Facebook Graph API. For this you have to issue a POST request to
"https://graph.facebook.com"
and the post data to be sent should be
"batch=[{'method': 'GET', 'relative_url': 'me'}, {'method': 'GET', 'relative_url': 'me/friends?limit=50'}]&access_token=#accesstoken"
in your case [#accesstoken must be replaced with your access token value].
This request will return the details of the owner of the access token(normally the current logged in user) and a list of 50 facebook friends(contains id and name fields) of the user along with page headers(can be omitted).
I am not sure whether you meant java or Javascript. Please be specific on it.
I am a C# programmer basically. Will provide you a code to execute the above request in C# here.
WebRequest webRequest = WebRequest.Create("https://graph.facebook.com");
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-UrlEncoded";
byte[] buffer = Encoding.UTF8.GetBytes("batch=[{'method': 'GET', 'relative_url': 'me'}, {'method': 'GET', 'relative_url': 'me/friends?limit=50'}]&access_token=#ACCESSTOKEN");
webRequest.ContentLength = buffer.Length;
using (Stream stream = webRequest.GetRequestStream())
{
stream.Write(buffer, 0, buffer.Length);
using (WebResponse webResponse = webRequest.GetResponse())
{
if (webResponse != null)
{
using (StreamReader streamReader = new StreamReader(webResponse.GetResponseStream(), Encoding.UTF8))
{
string data = streamReader.ReadToEnd();
}
}
}
}
Here the variable data will contain the result.

Salah, here is the example i use as reference, i am sorry though i do not remember where i found.
FB.api("/", "POST", {
access_token:"MY_APPLICATION_ACCESS_TOKEN",
batch:[
{
"method":"GET",
"name":"get-photos",
"omit_response_on_success": true,
"relative_url":"MY_ALBUM_ID/photos"
},
{
"method": "GET",
"depends_on":"get-photos",
"relative_url":"{result=get-photos:$.data[0].id}/likes"
}
]
}, function(response) {
if (!response || response.error) {
console.log(response.error_description);
} else {
/* Iterate through each Response */
for(var i=0,l=response.length; i<l; i++) {
/* If we have set 'omit_response_on_success' to true in the Request, the Response value will be null, so continue to the next iteration */
if(response[i] === null) continue;
/* Else we are expecting a Response Body Object in JSON, so decode this */
var responseBody = JSON.parse(response[i].body);
/* If the Response Body includes an Error Object, handle the Error */
if(responseBody.error) {
// do something useful here
console.log(responseBody.error.message);
}
/* Else handle the data Object */
else {
// do something useful here
console.log(responseBody.data);
}
}
}
});