I have a problem with getting XML from this webpage. In the browser it shows correctly and there is no issue, but when it comes to Java, it is different.
I've tried two methods which both of them resulted in exception.
// Method 1 - Using Java's URL
URL url = new URL(/* mentioned link */);
String rawXML = new String(url.openStream().readAllBytes(), StandardCharsets.UTF_8); // java.io.IOException: Invalid Http response
// Method 2 - Using Apache's HTTP client
HttpGet httpGet = new HttpGet(/* mentioned link */);
String rawXML = EntityUtils.toString(HttpClients.createDefault().execute(httpGet).getEntity()); // org.apache.http.ProtocolException: The server failed to respond with a valid HTTP response
Downloading this webpage with wget and using argument --content-on-error works but it is unreliable since wget is not always available on all systems like Windows.
The response does not contains headers so java rejects it
wget "https://www.strava.cz/foxisapi/foxisapi.dll/istravne.istravne.process?xmljidelnickyA&zarizeni=3148" -O so-69226464.html
--2021-09-17 13:44:29-- https://www.strava.cz/foxisapi/foxisapi.dll/istravne.istravne.process?xmljidelnickyA&zarizeni=3148
Resolving www.strava.cz (www.strava.cz)... 82.99.180.77
Connecting to www.strava.cz (www.strava.cz)|82.99.180.77|:443... connected.
HTTP request sent, awaiting response... 200 No headers, assuming HTTP/0.9
Length: unspecified
This java class making a raw HTTP GET request is able to get the contents. Based on this page.
The request sent is
GET /foxisapi/foxisapi.dll/istravne.istravne.process?xmljidelnickyA&zarizeni=3148 HTTP/1.1\r\n
User-Agent: RawHttpGet\r\n
Host: www.strava.cz\r\n
Accept: */*\r\n
Java code:
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.net.Socket;
import java.nio.charset.StandardCharsets;
import javax.net.ssl.SSLSocketFactory;
public class RawHttpGet {
private static String hostname = "www.strava.cz";
public static void main(String[] args) throws IOException {
Socket socket = SSLSocketFactory.getDefault().createSocket(hostname, 443);
// UTF-8 encdoding
//BufferedWriter out = new BufferedWriter(new OutputStreamWriter(socket.getOutputStream(), StandardCharsets.UTF_8));
// Encoding for this request
BufferedWriter out = new BufferedWriter(new OutputStreamWriter(socket.getOutputStream(), "Cp1250"));
BufferedReader in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
StringBuffer buff = new StringBuffer("GET /foxisapi/foxisapi.dll/istravne.istravne.process?xmljidelnickyA&zarizeni=3148 HTTP/1.1\r\n");
buff.append("User-Agent: RawHttpGet\r\n");
buff.append("Accept: */*\r\n");
buff.append("Host: " + hostname + "\r\n");
buff.append("\r\n");
System.out.println(" * Request");
System.out.println(buff.toString());
// send message
out.write(buff.toString());
out.flush();
// read response
System.out.println(" * Response");
// Default system encoding
//System.out.println(new String(socket.getInputStream().readAllBytes()));
// Encoding for this request
System.out.println(new String(socket.getInputStream().readAllBytes(), "Cp1250"));
out.close();
in.close();
}
}
Related
I'm using Positionstack API to build my APP with a location function on Android. The API works well when I test it on java in the local environment. However, it keeps returning a syntax error message and an Error 400 code when I send the request on Android Studio through an activity.
The error message
I/System.out: 400
I/System.out: {"error":{"code":"bad_request","message":"Could not decode value from JSON format. Error was: \u0022Syntax error\u0022."}}
The class of sending requests. It works well in the local environment but fails on the emulator. It establishes a HttpUrlConnection and uses the GET method to retrieve the result from API. It returns a 400 status code on the Android Studio emulator.
import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
public class NetTest {
public static String sendRequest(String urlParam, String coordinate){
HttpURLConnection con = null;
BufferedReader buffer = null;
StringBuffer resultBuffer = null;
InputStream is;
try{
// prepare the params and send request
URL url = new URL(urlParam);
con = (HttpURLConnection) url.openConnection();
con.setConnectTimeout(5000);
con.setReadTimeout(5000);
con.setRequestMethod("POST");
con.setRequestProperty("Content-Type","application/json;charset=UTF-8");
con.setDoOutput(true);
con.set
// DataOutputStream wr = new DataOutputStream(con.getOutputStream());
// wr.writeBytes("access_key=xxx");
// wr.writeBytes("query=-33.7,127");
// wr.flush();
// wr.close();
System.out.println("message out");
// receive response
int responseCode = con.getResponseCode();
System.out.println(responseCode);
if (responseCode == 200) {
is = con.getInputStream();
}else {
is = con.getErrorStream();
}
resultBuffer = new StringBuffer();
String line;
buffer = new BufferedReader(new InputStreamReader(is,"UTF-8"));
while ((line = buffer.readLine()) != null){
resultBuffer.append(line);
}
return resultBuffer.toString();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return "";
}
public static void main(String[] args) {
String coordinate = "-33.7,127";
String url = "http://api.positionstack.com/v1/reverse";
System.out.println(sendRequest(url,coordinate));
}
}
The manifest
<application
android:usesCleartextTraffic="true">
...
</application>
<uses-permission android:name="android.permission.INTERNET" />
Thanks a lot!
The problem may be the encoding method of Android, but I don't know how to change it, or even see it.
Use this codeblock -> first create jsonObject and add your key values and then set in output Stream. you can use this link for reference -> https://www.baeldung.com/httpurlconnection-post
JSONObject jsonObject = new JSONObject();
jsonObject.put("access_key", "Enter your access key here");
jsonObject.put("query", "Enter your query string here");
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
String jsonString = jsonObject.toString();
byte[] input = jsonString.getBytes(StandardCharsets.UTF_8);
wr.write(input, 0, input.length);
wr.flush();
wr.close();
Also check the logger for what is sent in request body, use okhttp as search string in logger
Thanks to #abhishekrajak. I think his answer can solve most questions in this case.
In my case, however, I received an Error 403 status code from the API that said that my subscription couldn't use this service. I think the problem is that using OutputStream of HttpUrlConnection will automatically transfer your request method from "GET" to "POST", and my subscription is only able to receive the "GET" service.
Anyway, no doubt that #abhishekrajak 's answer can successfully pass the parameter in the correct encoding.
I solved this problem by using an alternative API. The original API may not be a good choice in the first place.
i want to send a request includes bytearray as form-data. Everyone using "File" but i have just the "bytearray" and i don't want to use any path. My request in curl:
curl -F file=#file server
In java what i tried:
byte[] fileByte = Base64.decodeBase64(parameter);
ByteArrayInputStream myFile = new ByteArrayInputStream(fileByte);
MultipartEntityBuilder multipartEntityBuilder = MultipartEntityBuilder.create();
CloseableHttpClient client = HttpClients.createDefault();
HttpPost httpPost = new HttpPost("server");
multipartEntityBuilder.addBinaryBody("file", myFile, ContentType.APPLICATION_OCTET_STREAM, "filename");
HttpEntity multipart = multipartEntityBuilder.build();
httpPost.setEntity(multipart);
and i tried
multipartEntityBuilder.addBinaryBody("file", fileByte, ContentType.DEFAULT_BINARY, "filename");
//OR FileBody, ByteArrayEntity, InputStream or like thats
So, that methods not worked for me. How i can send the request successfully ?
EDIT: i used the ptsv2 with postman and result ->
when i send the same request and file with java the result ->
i think issue is certainly related to the ByteArray or InputStream. I must find the another Type for the my byte[] or right method for post in java with using the File type but without path.
I don't know what "server" is, but maybe the problem is with the server you are sending to. I ran your code, writing to a public HTTP POST test server available on the internet, and it works fine for me. Here's the code I ran:
import org.apache.http.HttpEntity;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.mime.MultipartEntityBuilder;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
class ListsTest {
public static void main(String[] args) {
byte[] fileByte = "Sample string data".getBytes(StandardCharsets.UTF_8);
MultipartEntityBuilder multipartEntityBuilder = MultipartEntityBuilder.create();
CloseableHttpClient client = HttpClients.createDefault();
HttpPost httpPost = new HttpPost("https://ptsv2.com/t/r7ypk-1613291354/post");
multipartEntityBuilder.addBinaryBody("file", fileByte, ContentType.DEFAULT_BINARY, "filename");
HttpEntity multipart = multipartEntityBuilder.build();
httpPost.setEntity(multipart);
try {
client.execute(httpPost);
} catch (IOException e) {
e.printStackTrace();
}
}
}
If you go to the target test server bucket:
https://ptsv2.com/t/r7ypk-1613291354
you can see what the full request looks like as received by the server, and see that the payload makes it over just fine as a file with the appropriate filename and contents. The file content portion of the request display looks like this:
Your alternate method worked equally well for me. I chose to post this version as it is simpler, not requiring you to wrap your byte array in an InputStream object.
I have hired a web service that sends sms messages to mobile phone. I have tried to make the post request but I amb getting the error:
Exception in thread "main" javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
The webservice company gave me all the information needed for making the the url: the user, the password, the params needed, the response etc
The gsm param is the mobile phone number.
msg is the message to send.
import java.net.HttpURLConnection;
import javax.net.ssl.HttpsURLConnection;
import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.InputStreamReader;
import java.net.URL;
public class Test3 {
public static void main(String[] args) throws Exception {
Test3 obj = new Test3();
System.out.println("Testing 1 - Send Http POST request");
obj.sendPost();
}
private void sendPost() throws Exception {
//
String url = "https://...url/send";//(that url its just an example, its not the real)
HttpsURLConnection httpClient = (HttpsURLConnection) new URL(url).openConnection();
//add reuqest header
httpClient.setRequestMethod("POST");
httpClient.setRequestProperty("User-Agent", "Mozilla/5.0");
httpClient.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
String urlParameters = "?user=TEST_USER&company=XXX&passwd=TEST_PWD&gsm=666000222&type=plus&msg=hello&sender=me";
// Send post request
httpClient.setDoOutput(true);
try (DataOutputStream wr = new DataOutputStream(httpClient.getOutputStream())) {
wr.writeBytes(urlParameters);
wr.flush();
}
int responseCode = httpClient.getResponseCode();
System.out.println("\nSending 'POST' request to URL : " + url);
System.out.println("Post parameters : " + urlParameters);
System.out.println("Response Code : " + responseCode);
try (BufferedReader in = new BufferedReader(
new InputStreamReader(httpClient.getInputStream()))) {
String line;
StringBuilder response = new StringBuilder();
while ((line = in.readLine()) != null) {
response.append(line);
}
System.out.println(response.toString());
}
}
}
Also If I tried to acces the the webservice call by writting the url to the navbar of Mozilla Firefox I got that error:
Secure Connection Failed
An error occurred during a connection to 'url' SSL peer was unable to negotiate an acceptable set of security parameters.
Error code: SSL_ERROR_HANDSHAKE_FAILURE_ALERT
The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
Given that the domain in question (push.tempos21.com) works for me with the browser but does not work for you neither with app nor browser, it is likely that it is not a problem of app or browser but of your connection to the target. I suspect that there is some deep inspection firewall or IPS somewhere in the path which allows the initial TCP connect but then blocks the traffic once the target is known from the ClientHello (start of TLS handshake).
Here is my Java code to get some information from stackexchange.
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.InputStreamReader;
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
public class StackExchangeConnect {
private static final String URL = "http://api.stackexchange.com/2.2/answers?key=U4DMV*8nvpm3EOpvf69Rxw((&site=stackoverflow&fromdate=1396310400&todate=1398729600&order=desc&sort=activity&filter=default";
public static void main(String[] args) throws Exception {
StackExchangeConnect http = new StackExchangeConnect();
System.out.println("Testing 1 - Send Http GET request");
http.sendGet();
}
// HTTP GET request
private void sendGet() throws Exception {
HttpClient client = new DefaultHttpClient();
HttpGet request = new HttpGet(URL);
// add request header
request.addHeader("Accept", "application/json");
HttpResponse response = client.execute(request);
System.out.println("Response Code : "
+ response.getStatusLine().getStatusCode());
BufferedReader rd = new BufferedReader(new InputStreamReader(response
.getEntity().getContent()));
StringBuffer result = new StringBuffer();
String line = "";
while ((line = rd.readLine()) != null) {
result.append(line);
}
System.out.println(result.toString());
}
}
I am getting following in my console.
Testing 1 - Send Http GET request
Response Code : 200
�
I tried writing the contents to file, and it's like
�\00\00\00\00\00\00�Z�n�H�Cy�;_�`df1��&���.F�ٔK���|I���eƒ�M�,GO�Ev��:U�_gE����_g���U����ʭ7�i�r5���t��]uUd�ʤZW����]���n\���l]�y�pW���[�uӬ�$���;�W��4�:��2��&��:�6�L�N8�2���)�,18?ԯ1Q���E�VMa�ի����_�1�-+���<\�̲}��sS��E��^�co�[W��{y;�:f�n�NgE}e�u��r��E�Ng�-+x8����{S���Uf���V�c!�`wY�xl�r݂n�����әY�w��J�.��quw�h��v\00�0!�`"�k𘀩��U����q�i�:��L�$C4�y���Se�6��Y�hȆ�%u�5��F�0b�̓�P�0���b�*�x������b�����+λ�;/��M���?�����z>#2�1g^���S���ӏ9y��.�ͪ]�v!��F��h/��&�/�I^D3ċ!��6dI�0\00l�a� �S���B��8�q�P���h� �H<�ݼ��g�^|.����y���G/���c�\00u2a���LQ�����_ņ=ů7w�A?�Ĵ>�$%X� �͇f��Ib�%AA�_8���T����(C�����S�,�Y��A�nYu<K��$��1y��(\00Rg�E�6��Lx�B#���d&x#�->����(?^�pu�^M�粹6����-0�X#�q'���l9���'��~`�&J�}`\'"E,�ڦFJJ�^�>�H*r�q��ܦVZ�X�[ vtzKAn��g�nG ���!�z��:HJ:�oPH�oiFH�m11rt���L#i\00$όu� 'J��+�,KU4fs[5e������vh�� h��K+��4�T>�!*,���JH2.��>$�rC�)�9�!�3k��eee�O��2�`�N�ݯ]���~^ɦ}#��g��>rM��LT�Ѡ\00�m�8�Z�&�z����L�]�j.Ք�8Z�g�3(!�U�a��D\�#��Y�G�m�^lV��&�o��G�cGu���e|���8��(:\9��D&,�T������O���8y�����OQ��|�L�{\00PR�l>\000�d�"����!J�Pa%A�vs����!��j�2�p�3ąS�aĴ��M���&�B�A���9,Dx���b�,�)�&��mI�2]��g9M2F K&s�5��+����5���++xc4\�Ԓy��|ĘV�l�2b�3���D%��m!`㇇7�f}}���K��[(�4���d]�fS���^�~��ֿ}����=��ɶ��kY(�"��|]p��m�2�4DA�����3$��tѩ�j*�T�Y��� Џ.齦W�lzٹ%���Y �(d�#���P���m���{�-"���T)�(IS�n��)}��!I����bq#���O��1a*��0ؼ�("��C}9�<LNۖ�B0�ڮС�qŹ�0�e�N%#.�X ���F�+�kS�h�3K�i��$bd"��#H�H5�)F=�DL�r�r���E�������<ٶ^e�{8�x���(oب�F��� g���]�B�l�6/��Q�gR`O�g��p%1s�I[�G���Ooq�J8�*(m榮��ӏ9�TP�������ᬳ�Z�w[1{��P���`�s�Kn���HG�3�p��Kᣦ����$2�9v1f5U����Ɲ�x�!~K��[����c�Li�˽ͧҰ�|�G(x�J��)��vk��N{s��?~{�F�N>�|�/�G�q������\uF�OƵ���<}��؇q�٭�p�K�8�RN��I�\00�b�" W,�q$�z��[v��ɶ�1hu�\00Z���"�G�c�����C"���> ��#�(ÿ��4��$���e$%qb)C9�^������r��h��i%�����`�!��Y�b�;<B���f;�P�����쮎7��r�{A�Ru��ܬ6l&t�M��ۤ^�ӊ�0���$B���C�)Ti{��)9�� 刋�U�ζ��,�Tk����'������o�u�L6��Y=x?_�I.C���\ ����S�"0�����Fi�d�f{V�R��V���j#e�s-�Ny�s�3���.���4��X�7:�&u����H%oy��km�(U�O����\B�?N<x�)���j_L'���[%�5ÔZ��J�4��9�hR�\����0�d����M��櫐8�T�P��}�aE[%N<�"v>a��������[�T�ɫ�nV��L|nw��=�,��f�����X��:�]�!�,�� hƩ<t�`�CT��r =O�CiNS��pim�����-��T7����#'�֩��({=>�K���C1<���Ngצ�Zv#꿡?7ec����#�o�Tni�U���.�C���Ý�=1\00\00
When I try the same URL in browser it's getting the JSON. If I try other URL in program, it's getting the JSON properly. Why is that?
And also, is there any Java API so that the JSON String I can convert to some Object?
NOTE:
Apache HTTP Client - 4.3.3, Apache HTTP Core - 4.3.2, Commons logging - 1.1.3
UPDATE 1
I am printing response. System.out.println(response.toString()); And I am getting the following output.
HTTP/1.1 200 OK [Cache-Control: private, Content-Type: application/json; charset=utf-8, Content-Encoding: gzip, Access-Control-Allow-Origin: *, Access-Control-Allow-Methods: GET, POST, Access-Control-Allow-Credentials: false, Date: Tue, 29 Apr 2014 06:31:45 GMT, Content-Length: 2551]
It looks like it's compressed, w hich should be handled by HTTPClient; try using the builder and disabling compression.
The content being returned is gzip compressed. you can see this in your output of the response:
... Content-Encoding: gzip ...
Given that you're using the 4.3.3 version of the Apache HTTP client library, you simply need to do the following:
HttpClient client = HttpClientBuilder.create().build();
This will construct a HttpClient that has no problem decompressing the content, and you will get the JSON you're expecting ( The DefaultHttpClient does not support compression.)
The content is gzipped. You can verify that using
System.out.println(connection.getContentEncoding());
To get the content, use something like this.(This is java.net.URl version. Please change accordingly for httpclient)
GZIPInputStream gs = new GZIPInputStream(connection.getInputStream());
BufferedReader in = new BufferedReader(new InputStreamReader(gs));
String inputLine;
while ((inputLine = in.readLine()) != null)
System.out.println(inputLine);
in.close();
You should render the response as UTF-8.
And your second question is the Jackson library. It can conver json objects into Java objects.
I am trying to write a small class using the Apache HttpClient library that would do an HTTPS post to a specified URL sending some XML. When I run my code, the HTTP status line I receive back is "403 Service Error". Here's the complete error HTML returned:
$errorDump java.net.SocketTimeoutException:Read timed out
$errorInfo
$errorDump java.net.SocketTimeoutException:Read timed out
$error Read timed out
$localizedError Read timed out
$errorType java.net.SocketTimeoutException
$user
$time 2011-10-25 09:39:29 EDT
$error Read timed out
$errorType java.net.SocketTimeoutException
This is the code I am using:
import java.io.ByteArrayInputStream;
import java.io.InputStreamReader;
import java.io.BufferedReader;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.InputStreamEntity;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.util.EntityUtils;
public class HttpXmlPost {
public static void main(String[] args)
{
String url = "https://someurlhere.com";
String xmlStr = "<?xml version=\"1.0\" encoding=\"UTF-8\" ?><xmlTag></xmlTag>";
String content = request(xmlStr, url);
System.out.println(content);
}
private static String request(String xmlStr, String url) {
boolean success = false;
String content = "";
HttpClient httpclient = new DefaultHttpClient();
try {
HttpPost httpPost = new HttpPost(url.trim());
InputStreamEntity reqEntity = new InputStreamEntity(new ByteArrayInputStream(xmlStr.getBytes() ), -1);
reqEntity.setContentType("application/xml");
reqEntity.setChunked(true);
httpPost.setEntity(reqEntity);
System.out.println("Executing request " + httpPost.getRequestLine());
HttpResponse response = httpclient.execute(httpPost);
HttpEntity resEntity = response.getEntity();
System.out.println("----------------------------------------");
System.out.println(response.getStatusLine());
if(response.getStatusLine().getStatusCode() == 200){
success = true;
}
if (resEntity != null) {
System.out.println("Response content length: " + resEntity.getContentLength());
System.out.println("Chunked?: " + resEntity.isChunked());
}
BufferedReader reader = new BufferedReader(new InputStreamReader(resEntity.getContent()));
StringBuilder buf = new StringBuilder();
char[] cbuf = new char[ 2048 ];
int num;
while ( -1 != (num=reader.read( cbuf ))) {
buf.append( cbuf, 0, num );
}
content = buf.toString();
EntityUtils.consume(resEntity);
}
catch (Exception e) {
System.out.println(e);
}
finally {
httpclient.getConnectionManager().shutdown();
}
return content;
}
}
Whatever XML I pass in doesn't seem to matter, it gives the same error no matter what. Note that this actually works with some URLs. For example, if I put https://www.facebook.com, it goes through. However, it doesn't work for my specified URL. I thought it might be a certificate issue, tried to add some code to trust any certificate, didn't seem to work either, though I may have done it wrong. Any help is appreciated.
Based on the SocketTimeoutException in the first line of the response HTML, I'm guessing that the component which implements the handler for the URL to which you are posting is having some connection problems to a source system it needs to generate the response data.
Basically, it looks like the problem is on the server, not your client.