I am trying to write a simple HttpClient program.
This is the first time I am working with HttpClient, I am quite confused which jars to include.
I have included the apache-httpcomponents-httpclient.jar and org.apache.commons.httpclient.jar with these ones when I create a HttpClient object I see different methods in the client object
package com.comverse.rht;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.ArrayList;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpException;
import org.apache.commons.httpclient.HttpStatus;
import org.apache.commons.httpclient.NameValuePair;
import org.apache.commons.httpclient.URI;
import org.apache.commons.httpclient.URIException;
import org.apache.commons.httpclient.methods.GetMethod;
import org.apache.commons.httpclient.methods.PostMethod;
import org.apache.http.HttpResponse;
import org.apache.http.client.methods.HttpGet;
public class HttpClientTest {
public static void main(String[] args) throws URIException {
URI url = new URI("http://www.google.com/search?q=httpClient");
HttpClient client = new HttpClient();
GetMethod get = new GetMethod();
PostMethod post = new PostMethod();
String responseString;
StringBuilder sb = new StringBuilder();
String line;
// add request header
get.setURI(url);
get.addRequestHeader("User-Agent", "shaiksha429");
try {
int respCode = client.executeMethod(get);
System.out.println("Response Code:" +respCode);
System.out.println(
"PCRF HTTP Status" + HttpStatus.getStatusText(respCode)
);
responseString = get.getResponseBodyAsString();
BufferedReader rd = null;
rd = new BufferedReader(
new InputStreamReader(get.getResponseBodyAsStream())
);
while ((line = rd.readLine()) != null) {
sb.append(line + '\n');
}
System.out.println(sb);
} catch (HttpException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
But when I google I see a different example as below. What is the difference between the two? Why one HttpClient has "execute" and the other has "executeMethod". Which one I need to use?
String url = "http://www.google.com/search?q=httpClient";
HttpClient client = HttpClientBuilder.create().build();
HttpGet request = new HttpGet(url);
// add request header
request.addHeader("User-Agent", USER_AGENT);
HttpResponse response = client.execute(request);
System.out.println("Response Code : " + response.getStatusLine().getStatusCode());
BufferedReader rd = new BufferedReader(
new InputStreamReader(response.getEntity().getContent())
);
StringBuffer result = new StringBuffer();
String line = "";
while ((line = rd.readLine()) != null) {
result.append(line);
}
There were a lot of changes from HttpClient version 3 to version 4. The second example is definitely from HttpClient 4, so the first example is probably from the previous version.
Here is code that will do your google search, and read the result into a string
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
connectionManager.setMaxTotal(60);
connectionManager.setDefaultMaxPerRoute(6);
try (CloseableHttpClient client = HttpClients.custom().setConnectionManager(connectionManager).build()) {
HttpGet request = new HttpGet("http://www.google.com/search?q=httpClient");
request.setHeader("User-Agent", "HttpClient");
try (CloseableHttpResponse response = client.execute(request)) {
MediaType mediaType = MediaType.parseMediaType(response.getFirstHeader("Content-Type").getValue());
Charset charSet = mediaType.getCharSet();
HttpEntity entity = response.getEntity();
InputStream is = entity.getContent();
String body = CharStreams.toString(new InputStreamReader(is, charSet));
System.out.println("body = " + body);
EntityUtils.consume(entity);
}
}
First, you probably want to create a connection pool, so you can reuse the connection if you send multiple requests to the same server. The pool is typically created during application initialisation , for instance as a Spring singleton bean.
Here I used the ClosableHttpClient because it works with resource-try syntax, and you need to close both the httpClient, the response and the inputStream when you are done reading. The HttpClient is actually a lightweight object, the state like socket connection and cookies are stored elsewhere.
I use Spring's MediaType.parseMediaType() to get the char encoding, and Guavas CharStreams to convert the inputStream to a String. In my case google encoded the content using latin-1, because "search" is "søgning" in Danish.
The last step is to use EntityUtils.consume(entity), to ensure that all entity data has been read. If you use connection pooling this is important, because unread data will cause the connection to be thrown away, instead of being reused by the connection manager (this is extremely important if you are using https).
You're using a library whose interface has changed across its major versions. You can't casually copy jars and copy/paste examples without understanding which release you're using and which release an example or snippet was from.
Look at the examples that accompany the latest release and take anything old with a grain of salt.
Apache seems to move especially fast.
Related
I'm using Positionstack API to build my APP with a location function on Android. The API works well when I test it on java in the local environment. However, it keeps returning a syntax error message and an Error 400 code when I send the request on Android Studio through an activity.
The error message
I/System.out: 400
I/System.out: {"error":{"code":"bad_request","message":"Could not decode value from JSON format. Error was: \u0022Syntax error\u0022."}}
The class of sending requests. It works well in the local environment but fails on the emulator. It establishes a HttpUrlConnection and uses the GET method to retrieve the result from API. It returns a 400 status code on the Android Studio emulator.
import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
public class NetTest {
public static String sendRequest(String urlParam, String coordinate){
HttpURLConnection con = null;
BufferedReader buffer = null;
StringBuffer resultBuffer = null;
InputStream is;
try{
// prepare the params and send request
URL url = new URL(urlParam);
con = (HttpURLConnection) url.openConnection();
con.setConnectTimeout(5000);
con.setReadTimeout(5000);
con.setRequestMethod("POST");
con.setRequestProperty("Content-Type","application/json;charset=UTF-8");
con.setDoOutput(true);
con.set
// DataOutputStream wr = new DataOutputStream(con.getOutputStream());
// wr.writeBytes("access_key=xxx");
// wr.writeBytes("query=-33.7,127");
// wr.flush();
// wr.close();
System.out.println("message out");
// receive response
int responseCode = con.getResponseCode();
System.out.println(responseCode);
if (responseCode == 200) {
is = con.getInputStream();
}else {
is = con.getErrorStream();
}
resultBuffer = new StringBuffer();
String line;
buffer = new BufferedReader(new InputStreamReader(is,"UTF-8"));
while ((line = buffer.readLine()) != null){
resultBuffer.append(line);
}
return resultBuffer.toString();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return "";
}
public static void main(String[] args) {
String coordinate = "-33.7,127";
String url = "http://api.positionstack.com/v1/reverse";
System.out.println(sendRequest(url,coordinate));
}
}
The manifest
<application
android:usesCleartextTraffic="true">
...
</application>
<uses-permission android:name="android.permission.INTERNET" />
Thanks a lot!
The problem may be the encoding method of Android, but I don't know how to change it, or even see it.
Use this codeblock -> first create jsonObject and add your key values and then set in output Stream. you can use this link for reference -> https://www.baeldung.com/httpurlconnection-post
JSONObject jsonObject = new JSONObject();
jsonObject.put("access_key", "Enter your access key here");
jsonObject.put("query", "Enter your query string here");
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
String jsonString = jsonObject.toString();
byte[] input = jsonString.getBytes(StandardCharsets.UTF_8);
wr.write(input, 0, input.length);
wr.flush();
wr.close();
Also check the logger for what is sent in request body, use okhttp as search string in logger
Thanks to #abhishekrajak. I think his answer can solve most questions in this case.
In my case, however, I received an Error 403 status code from the API that said that my subscription couldn't use this service. I think the problem is that using OutputStream of HttpUrlConnection will automatically transfer your request method from "GET" to "POST", and my subscription is only able to receive the "GET" service.
Anyway, no doubt that #abhishekrajak 's answer can successfully pass the parameter in the correct encoding.
I solved this problem by using an alternative API. The original API may not be a good choice in the first place.
I am trying to use some code that I got from a website that has sports data served publically via an API (http://developer.fantasydata.com).
The site provide some sample JAVA code to make the http request. For some reason the setEntity method for the declared request (request) is showing a "cannot find symbol error.
package epl.fixtures.test.app;
import java.net.URI;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.utils.URIBuilder;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;
public class EPLFixturesTestApp {
/**
* #param args the command line arguments
*/
public static void main(String[] args) {
// TODO code application logic here
HttpClient httpclient = HttpClients.createDefault();
try
{
URIBuilder builder = new URIBuilder("https://api.fantasydata.net/soccer/v2/json/CompetitionDetails/EPL");
URI uri = builder.build();
HttpGet request = new HttpGet(uri);
request.setHeader("Ocp-Apim-Subscription-Key", "****************");
// Request body
StringEntity reqEntity = new StringEntity("{body}");
request.setEntity(reqEntity);
HttpResponse response = httpclient.execute(request);
HttpEntity entity = response.getEntity();
if (entity != null)
{
System.out.println(EntityUtils.toString(entity));
}
}
catch (Exception e)
{
System.out.println(e.getMessage());
}
}
}
The line causing the issue is the request.setEntity(reqEntity); line
Can anyone explain this to me please? I have all the relevant jar files from apache added to the project libraries directory.
Thanks
HttpGet does not have a setEntity method.
This makes sense, since the request body has no meaning in GET requests.
Only classes implementing HttpEntityEnclosingRequest have this method.
I don't know why the documentation uses it, but it seems to work when omitting those two lines (which look meaningless anyway). Code:
URIBuilder builder = new URIBuilder("https://api.fantasydata.net/soccer/v2/json/CompetitionDetails/EPL");
URI uri = builder.build();
HttpGet request = new HttpGet(uri);
request.setHeader("Ocp-Apim-Subscription-Key", "****************");
HttpResponse response = httpclient.execute(request);
HttpEntity entity = response.getEntity();
if (entity != null)
{
System.out.println(EntityUtils.toString(entity));
}
I'd like to use http://www.imdbapi.com/ in Java, but I don't know I can access the http response. I tried the following:
public Map<String, String> get(String title)
{
URL url = new URL("http://www.imdbapi.com/?t=" + title);
URLConnection conn = url.openConnection();
conn.getContent();
}
You can use URLConnection#getInputStream():
InputStream input = conn.getInputStream();
// ...
Or just the shorthand URL#openStream() directly:
InputStream input = url.openStream();
// ...
Once having it, just send it to a JSON parser of your choice, such as for example Gson:
InputStream input = new URL("http://www.imdbapi.com/?t=" + URLEncoder.encode(title, "UTF-8")).openStream();
Map<String, String> map = new Gson().fromJson(new InputStreamReader(input, "UTF-8"), new TypeToken<Map<String, String>>(){}.getType());
// ...
(note that I fixed your query string to be properly URL encoded)
See also:
Using java.net.URLConnection to fire and handle HTTP requests
Converting JSON to Java
When you go the website and type in the sample movie (i did True Grit ) you are actually able to see the response you would be getting. It looks something like this:
{"Title":"True Grit","Year":"2010","Rated":"PG-13","Released":"22 Dec 2010","Genre":"Adventure, Drama, Western","Director":"Ethan Coen, Joel Coen","Writer":"Joel Coen, Ethan Coen","Actors":"Jeff Bridges, Matt Damon, Hailee Steinfeld, Josh Brolin","Plot":"A tough U.S. Marshal helps a stubborn young woman track down her father's murderer.","Poster":"http://ia.media-imdb.com/images/M/MV5BMjIxNjAzODQ0N15BMl5BanBnXkFtZTcwODY2MjMyNA##._V1._SX320.jpg","Runtime":"1 hr 50 mins","Rating":"8.0","Votes":"51631","ID":"tt1403865","Response":"True"}
After knowing this info, you can easily parse your InputStream, which you obtain from your connection.
Good luck!
The below code should get you started. You need to add URL encoding if you are going to send special characters. In-order to parse JSON response you could probably use parser available in java at [link] http://www.JSON.org/
package problem;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.net.URL;
import java.net.URLConnection;
public class Test {
public static void main(String args[])
{
BufferedReader rd;
OutputStreamWriter wr;
try
{
URL url = new URL("http://www.imdbapi.com/?i=&t=dexter");
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
wr = new OutputStreamWriter(conn.getOutputStream());
wr.flush();
// Get the response
rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line;
while ((line = rd.readLine()) != null) {
System.out.println(line);
}
}
catch (Exception e) {
System.out.println(e.toString());
}
}
}
I recomend use http-request built on apache http api.
private static final HttpRequest<Map<String, String>> HTTP_REQUEST =
HttpRequestBuilder.createGet("http://www.imdbapi.com/",
new TypeReference<Map<String, String>>{}
).build();
public Map<String, String> get(String title) {
ResponseHandler<Map<String, String>> responseHandler = HTTP_REQUEST.execute("t", title);
return responseHandler.orElse(Collections.emptyMap()); //returns response parsed as map or empty map when response body is empty
}
I am trying to write a small class using the Apache HttpClient library that would do an HTTPS post to a specified URL sending some XML. When I run my code, the HTTP status line I receive back is "403 Service Error". Here's the complete error HTML returned:
$errorDump java.net.SocketTimeoutException:Read timed out
$errorInfo
$errorDump java.net.SocketTimeoutException:Read timed out
$error Read timed out
$localizedError Read timed out
$errorType java.net.SocketTimeoutException
$user
$time 2011-10-25 09:39:29 EDT
$error Read timed out
$errorType java.net.SocketTimeoutException
This is the code I am using:
import java.io.ByteArrayInputStream;
import java.io.InputStreamReader;
import java.io.BufferedReader;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.InputStreamEntity;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.util.EntityUtils;
public class HttpXmlPost {
public static void main(String[] args)
{
String url = "https://someurlhere.com";
String xmlStr = "<?xml version=\"1.0\" encoding=\"UTF-8\" ?><xmlTag></xmlTag>";
String content = request(xmlStr, url);
System.out.println(content);
}
private static String request(String xmlStr, String url) {
boolean success = false;
String content = "";
HttpClient httpclient = new DefaultHttpClient();
try {
HttpPost httpPost = new HttpPost(url.trim());
InputStreamEntity reqEntity = new InputStreamEntity(new ByteArrayInputStream(xmlStr.getBytes() ), -1);
reqEntity.setContentType("application/xml");
reqEntity.setChunked(true);
httpPost.setEntity(reqEntity);
System.out.println("Executing request " + httpPost.getRequestLine());
HttpResponse response = httpclient.execute(httpPost);
HttpEntity resEntity = response.getEntity();
System.out.println("----------------------------------------");
System.out.println(response.getStatusLine());
if(response.getStatusLine().getStatusCode() == 200){
success = true;
}
if (resEntity != null) {
System.out.println("Response content length: " + resEntity.getContentLength());
System.out.println("Chunked?: " + resEntity.isChunked());
}
BufferedReader reader = new BufferedReader(new InputStreamReader(resEntity.getContent()));
StringBuilder buf = new StringBuilder();
char[] cbuf = new char[ 2048 ];
int num;
while ( -1 != (num=reader.read( cbuf ))) {
buf.append( cbuf, 0, num );
}
content = buf.toString();
EntityUtils.consume(resEntity);
}
catch (Exception e) {
System.out.println(e);
}
finally {
httpclient.getConnectionManager().shutdown();
}
return content;
}
}
Whatever XML I pass in doesn't seem to matter, it gives the same error no matter what. Note that this actually works with some URLs. For example, if I put https://www.facebook.com, it goes through. However, it doesn't work for my specified URL. I thought it might be a certificate issue, tried to add some code to trust any certificate, didn't seem to work either, though I may have done it wrong. Any help is appreciated.
Based on the SocketTimeoutException in the first line of the response HTML, I'm guessing that the component which implements the handler for the URL to which you are posting is having some connection problems to a source system it needs to generate the response data.
Basically, it looks like the problem is on the server, not your client.
I know there used to be a way to get it with Apache Commons as documented here:
http://hc.apache.org/httpclient-legacy/apidocs/org/apache/commons/httpclient/HttpMethod.html
...and an example here:
http://www.kodejava.org/examples/416.html
...but I believe this is deprecated.
Is there any other way to make an http get request in Java and get the response body as a string and not a stream?
Here are two examples from my working project.
Using EntityUtils and HttpEntity
HttpResponse response = httpClient.execute(new HttpGet(URL));
HttpEntity entity = response.getEntity();
String responseString = EntityUtils.toString(entity, "UTF-8");
System.out.println(responseString);
Using BasicResponseHandler
HttpResponse response = httpClient.execute(new HttpGet(URL));
String responseString = new BasicResponseHandler().handleResponse(response);
System.out.println(responseString);
Every library I can think of returns a stream. You could use IOUtils.toString() from Apache Commons IO to read an InputStream into a String in one method call. E.g.:
URL url = new URL("http://www.example.com/");
URLConnection con = url.openConnection();
InputStream in = con.getInputStream();
String encoding = con.getContentEncoding();
encoding = encoding == null ? "UTF-8" : encoding;
String body = IOUtils.toString(in, encoding);
System.out.println(body);
Update: I changed the example above to use the content encoding from the response if available. Otherwise it'll default to UTF-8 as a best guess, instead of using the local system default.
Here's an example from another simple project I was working on using the httpclient library from Apache:
String response = new String();
List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>(1);
nameValuePairs.add(new BasicNameValuePair("j", request));
HttpEntity requestEntity = new UrlEncodedFormEntity(nameValuePairs);
HttpPost httpPost = new HttpPost(mURI);
httpPost.setEntity(requestEntity);
HttpResponse httpResponse = mHttpClient.execute(httpPost);
HttpEntity responseEntity = httpResponse.getEntity();
if(responseEntity!=null) {
response = EntityUtils.toString(responseEntity);
}
just use EntityUtils to grab the response body as a String. very simple.
This is relatively simple in the specific case, but quite tricky in the general case.
HttpClient httpclient = new DefaultHttpClient();
HttpGet httpget = new HttpGet("http://stackoverflow.com/");
HttpResponse response = httpclient.execute(httpget);
HttpEntity entity = response.getEntity();
System.out.println(EntityUtils.getContentMimeType(entity));
System.out.println(EntityUtils.getContentCharSet(entity));
The answer depends on the Content-Type HTTP response header.
This header contains information about the payload and might define the encoding of textual data. Even if you assume text types, you may need to inspect the content itself in order to determine the correct character encoding. E.g. see the HTML 4 spec for details on how to do that for that particular format.
Once the encoding is known, an InputStreamReader can be used to decode the data.
This answer depends on the server doing the right thing - if you want to handle cases where the response headers don't match the document, or the document declarations don't match the encoding used, that's another kettle of fish.
Below is a simple way of accessing the response as a String using Apache HTTP Client library.
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.ResponseHandler;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.BasicResponseHandler;
//...
HttpGet get;
HttpClient httpClient;
// initialize variables above
ResponseHandler<String> responseHandler = new BasicResponseHandler();
String responseBody = httpClient.execute(get, responseHandler);
The Answer by McDowell is correct one. However if you try other suggestion in few of the posts above.
HttpEntity responseEntity = httpResponse.getEntity();
if(responseEntity!=null) {
response = EntityUtils.toString(responseEntity);
S.O.P (response);
}
Then it will give you illegalStateException stating that content is already consumed.
How about just this?
org.apache.commons.io.IOUtils.toString(new URL("http://www.someurl.com/"));
We can use the below code also to get the HTML Response in java
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.HttpResponse;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import org.apache.log4j.Logger;
public static void main(String[] args) throws Exception {
HttpClient client = new DefaultHttpClient();
// args[0] :- http://hostname:8080/abc/xyz/CheckResponse
HttpGet request1 = new HttpGet(args[0]);
HttpResponse response1 = client.execute(request1);
int code = response1.getStatusLine().getStatusCode();
try (BufferedReader br = new BufferedReader(new InputStreamReader((response1.getEntity().getContent())));) {
// Read in all of the post results into a String.
String output = "";
Boolean keepGoing = true;
while (keepGoing) {
String currentLine = br.readLine();
if (currentLine == null) {
keepGoing = false;
} else {
output += currentLine;
}
}
System.out.println("Response-->" + output);
} catch (Exception e) {
System.out.println("Exception" + e);
}
}
Here's a lightweight way to do so:
String responseString = "";
for (int i = 0; i < response.getEntity().getContentLength(); i++) {
responseString +=
Character.toString((char)response.getEntity().getContent().read());
}
With of course responseString containing website's response and response being type of HttpResponse, returned by HttpClient.execute(request)
Following is the code snippet which shows better way to handle the response body as a String whether it's a valid response or error response for the HTTP POST request:
BufferedReader reader = null;
OutputStream os = null;
String payload = "";
try {
URL url1 = new URL("YOUR_URL");
HttpURLConnection postConnection = (HttpURLConnection) url1.openConnection();
postConnection.setRequestMethod("POST");
postConnection.setRequestProperty("Content-Type", "application/json");
postConnection.setDoOutput(true);
os = postConnection.getOutputStream();
os.write(eventContext.getMessage().getPayloadAsString().getBytes());
os.flush();
String line;
try{
reader = new BufferedReader(new InputStreamReader(postConnection.getInputStream()));
}
catch(IOException e){
if(reader == null)
reader = new BufferedReader(new InputStreamReader(postConnection.getErrorStream()));
}
while ((line = reader.readLine()) != null)
payload += line.toString();
}
catch (Exception ex) {
log.error("Post request Failed with message: " + ex.getMessage(), ex);
} finally {
try {
reader.close();
os.close();
} catch (IOException e) {
log.error(e.getMessage(), e);
return null;
}
}
Here is a vanilla Java answer:
import java.net.http.HttpClient;
import java.net.http.HttpResponse;
import java.net.http.HttpRequest;
import java.net.http.HttpRequest.BodyPublishers;
...
HttpClient client = HttpClient.newHttpClient();
HttpRequest request = HttpRequest.newBuilder()
.uri(targetUrl)
.header("Content-Type", "application/json")
.POST(BodyPublishers.ofString(requestBody))
.build();
HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString());
String responseString = (String) response.body();
If you are using Jackson to deserialize the response body, one very simple solution is to use request.getResponseBodyAsStream() instead of request.getResponseBodyAsString()
Using Apache commons Fluent API, it can be done as mentioned below,
String response = Request.Post("http://www.example.com/")
.body(new StringEntity(strbody))
.addHeader("Accept","application/json")
.addHeader("Content-Type","application/json")
.execute().returnContent().asString();