I have a simple http client that passes every request to ExecutorService and applies them with a delay.
protected static final int RETRY_ATTEMPTS = 5;
private static final int GROUP_REQUEST_DELAY_MS = 55;
private static final ScheduledExecutorService REQUEST_FROM_USER_EXECUTOR =
Executors.newSingleThreadScheduledExecutor();
public RequestResponse post(String url) throws IOException {
HttpPost httpPost = new HttpPost(url);
HttpResponse httpResponse = call(httpPost);
return new RequestResponse(
httpResponse.getStatusLine().getStatusCode(),
EntityUtils.toString(httpResponse.getEntity()),
headers(httpResponse.getAllHeaders())
);
}
private HttpResponse call(HttpRequestBase request) throws IOException {
int attempts = 0;
HttpResponse httpResponse = null;
SocketException socketException = null;
do {
try {
httpResponse = client.execute(request);
} catch(SocketException e) {
socketException = e;
}
if(httpResponse != null)
break;
attempts++;
log.debug("Attempt: {}, SocEx: {}", attempts, socketException != null);
}while(attempts < RETRY_ATTEMPTS);
if(httpResponse == null)
// TODO
if(socketException != null) {
log.error("Network problem");
logRequest(request, httpResponse);
throw socketException;
}
return httpResponse;
}
public synchronized Future<RequestResponse> sendAsGroup(String url) {
return REQUEST_FROM_GROUP_EXECUTOR.schedule(() -> post(url), GROUP_REQUEST_DELAY_MS, TimeUnit.MILLISECONDS);
}
Sometimes the server throws an http 504 error or so. I want to process this error and resubmit this request. How can I do this correctly without exceeding the limit for server requests?
You should be using HttpRequestRetryHandler to recover from transport level (TCP) errors and ServiceUnavailableRetryStrategy to retry the request execution in case of a protocol level (HTTP) error.
Related
I have the following java code with try with resources:
public static CloseableHttpResponse getHttpResponse()
{
try (CloseableHttpClient client = HttpClientBuilder.create().build()) {
try (CloseableHttpResponse response = client.execute(request)) {
return response;
}
}
}
in another method will use the response returned by getHttpResponse:
public void test() {
CloseableHttpResponse response = getHttpResponse();
if (response) {
int statusCode = response.getStatusLine().getStatusCode();
if (statusCode == HttpStatus.SC_OK) {
// do something
}
}
}
Looks like after CloseableHttpResponse response = getHttpResponse();, the client and response already closed, and I can not put this two methods into one, are there any ways that still use the try with resources in another method?
The best approach is the Execute Around idiom. Instead of getHttpResponse returning a CloseableHttpResponse pass in a lambda (typically) to be executed. The resource can then be closed in a try-with-resource statement cleanly.
/*** NICE ***/
// Function instead of Consumer would allow the method to return a value.
private void httpResponse(
Consumer<CloseableHttpResponse> op
) /* throws IOException */ {
try (CloseableHttpClient client = HttpClientBuilder.create().build()) {
try (CloseableHttpResponse response = client.execute(request)) {
if (response != null) { // Really?
op.accept(response);
}
}
}
}
Used as:
httpResponse(response -> {
int statusCode = response.getStatusLine().getStatusCode();
if (statusCode == HttpStatus.SC_OK) {
// do something
}
});
The hacky alternative is to include a try statement within getHttpResponse that only closed in error conditions.
/*** HACKY ***/
private CloseableHttpResponse httpResponse() /* throws IOException */ {
boolean success = false;
CloseableHttpClient client = HttpClientBuilder.create().build();
try {
CloseableHttpResponse response = client.execute(request);
try {
success = true;
return response;
} finally {
if (!success) {
response.close();
}
}
} finally {
if (!success) {
client.close();
}
}
}
client will be closed as soon as the program leaves the scope of the try-with-resources. Can you try building the try with resources around the getHttpResponse method? For example:
public static CloseableHttpResponse getHttpResponse() {
try (CloseableHttpClient client = HttpClientBuilder.create().build()) {
CloseableHttpResponse response = client.execute(request)
return response;
}
}
And then you can rewrite your test method() like this:
public void test() {
try(CloseableHttpResponse response = getHttpResponse()) {
if (response) {
int statusCode = response.getStatusLine().getStatusCode();
if (statusCode == HttpStatus.SC_OK) {
// do something
}
}
}
}
I am using Http Apache Client for Java and am making a request against the SteamWEB API to this steam link.
That causes an error in the Http Apache Client:
org.apache.http.client.ClientProtocolException
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at pl.edu.tirex.helper.http.Http.handleConnection(Http.java:77)
at pl.edu.tirex.helper.http.Http.get(Http.java:60)
at pl.edu.tirex.helper.http.Http.get(Http.java:64)
at pl.edu.tirex.helper.http.Http.get(Http.java:68)
at pl.edu.tirex.steamapi.steamguard.SteamGuard.fetchConfirmations(SteamGuard.java:42)
at pl.edu.tirex.steambot.SteamBOT.<init>(SteamBOT.java:93)
at pl.edu.tirex.steambot.SteamBOT.main(SteamBOT.java:56)
Caused by: org.apache.http.HttpException: steammobile protocol is not supported
at org.apache.http.impl.conn.DefaultRoutePlanner.determineRoute(DefaultRoutePlanner.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:157)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
... 8 more
My code for connection to send result:
private static final PoolingHttpClientConnectionManager CONNECTION_MANAGER = new PoolingHttpClientConnectionManager();
static
{
CONNECTION_MANAGER.setDefaultMaxPerRoute(2);
CONNECTION_MANAGER.setConnectionConfig(new HttpHost("https://steamcommunity.com/", 443, "steammobile"), ConnectionConfig.DEFAULT);
CONNECTION_MANAGER.setMaxTotal(4);
}
public static void main(String[] args)
{
HttpGet httpget = new HttpGet("url");
CloseableHttpClient httpclient = HttpClients.custom().setDefaultCookieStore(cookieStore).setConnectionManager(CONNECTION_MANAGER).build();
try (CloseableHttpResponse response = httpclient.execute(httpget, this.context))
{
HttpEntity entity = response.getEntity();
try (InputStream instream = entity.getContent())
{
if (handle != null)
{
if (response.getStatusLine().getStatusCode() >= 400)
{
handle.handleError(instream);
}
else
{
handle.handle(instream);
}
}
}
}
catch (HttpHostConnectException | InterruptedIOException | ClientProtocolException ignored)
{
}
}
How do I cope with this error?
These solution works properly:
private static final PoolingHttpClientConnectionManager CONNECTION_MANAGER = new PoolingHttpClientConnectionManager();
static
{
CONNECTION_MANAGER.setDefaultMaxPerRoute(2);
CONNECTION_MANAGER.setMaxTotal(4);
}
public static void main(String[] args)
{
HttpGet httpget = new HttpGet("url");
httpget.addHeader("X-Requested-With", "com.valvesoftware.android.steam.community");
CloseableHttpClient httpclient = HttpClients.custom().setDefaultCookieStore(cookieStore).setConnectionManager(CONNECTION_MANAGER).build();
try (CloseableHttpResponse response = httpclient.execute(httpget, this.context))
{
HttpEntity entity = response.getEntity();
try (InputStream instream = entity.getContent())
{
if (handle != null)
{
if (response.getStatusLine().getStatusCode() >= 400)
{
handle.handleError(instream);
}
else
{
handle.handle(instream);
}
}
}
}
catch (HttpHostConnectException | InterruptedIOException | ClientProtocolException ignored)
{
}
}
I have this code:
private CompleteRoutingResponseWrapper sendRoutingRequestString(String routingRequestUrl) {
routingRequestUrl = urlUtils.getHttpUrl(routingRequestUrl);
CompleteRoutingResponseWrapper answer = new CompleteRoutingResponseWrapper();
CloseableHttpClient client = HttpClientBuilder.create().build();
CloseableHttpResponse response = null;
try {
Stopwatch stopWatch = Stopwatch.createStarted();
response = client.execute(new HttpGet(routingRequestUrl));
stopWatch.stop();
int statusCode = response.getStatusLine().getStatusCode();
if (statusCode != 200) {
answer.errorMsg = handleError(routingRequestUrl, response, statusCode);
answer.completeRoutingResponse = new com.waze.routing.automation.dataModel.serverResponse.dto
.CompleteRoutingResponse();
} else {
String bodyAsString = EntityUtils.toString(response.getEntity());
answer.completeRoutingResponse = handleSuccess(bodyAsString);
answer.latency = stopWatch.elapsed(TimeUnit.MILLISECONDS);
handleErrorInBody(routingRequestUrl, answer, bodyAsString);
}
} catch (Exception e) {
e.printStackTrace();
}
return answer;
}
I call this method with hundreds of urls.
for some of them I get sporadically 404.
however when I to to call these urls from the browser itself, I get OK response.
what can cause this? how can I stabilize my code?
It happens sporadically and not for all urls.
I am new to HttpClient. I am using DefaultHttpClient (which is said to be thread-safe).
In my app I have created two threads which have been scheduled to execute every 10 minutes simultaneously. Sometimes I found that the DefaultClientConnection automatically gets shutdown.
What could be the possible reasons?
Though DefaultHttpClient is thread-safe, shall I need to use PoolingClientConnectionManager in this scenario ?
You should use PoolingClientConnectionManager.
Also you must use IdleConnectionMonitorThread for monitor idle connection.
some code from my source :
private final PoolingClientConnectionManager connectionManager;
private final IdleConnectionMonitorThread connectionMonitorThread = null;
private final DefaultHttpClient httpclient;
initialization:
final HttpParams params = new BasicHttpParams();
final HttpProtocolParamBean paramsBean = new HttpProtocolParamBean(params);
paramsBean.setVersion(HttpVersion.HTTP_1_1);
paramsBean.setContentCharset("UTF-8");
paramsBean.setUseExpectContinue(false);
params.setBooleanParameter(HttpConnectionParams.STALE_CONNECTION_CHECK, false);
params.setIntParameter("http.socket.timeout", 20000);
params.setIntParameter("http.connection.timeout", 30000);
params.setBooleanParameter("http.protocol.handle-redirects", true);
params.setBooleanParameter(HttpConnectionParams.TCP_NODELAY, true);
params.setIntParameter(HttpConnectionParams.SOCKET_BUFFER_SIZE, 32 * 1024);
params.setParameter("http.protocol.cookie-policy", CookiePolicy.BROWSER_COMPATIBILITY);
// params.setParameter("http.useragent", "Crawler Airupt(http://www.airupt.com/)");
params.setParameter("http.useragent",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.79 Safari/537.1");
params.setParameter("http.language.Accept-Language", "en-us");
params.setParameter("http.protocol.content-charset", "UTF-8");
params.setParameter("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
params.setParameter("Cache-Control", "max-age=0");
final SSLSocketFactory socketFactory = SSLSocketFactory.getSocketFactory();
final SchemeRegistry schemeRegistry = new SchemeRegistry();
schemeRegistry.register(new Scheme("http", 80, PlainSocketFactory.getSocketFactory()));
schemeRegistry.register(new Scheme("https", 443, socketFactory/* SSLSocketFactory.getSocketFactory() */));
connectionManager = new PoolingClientConnectionManager(schemeRegistry);
connectionManager.setDefaultMaxPerRoute(500000);
connectionManager.setMaxTotal(2000000);
httpclient = new DefaultHttpClient(connectionManager, params);
httpclient.setKeepAliveStrategy(new ConnectionKeepAliveStrategy() {
#Override
public long getKeepAliveDuration(final HttpResponse response, final HttpContext context) {
final HeaderElementIterator it = new BasicHeaderElementIterator(response
.headerIterator(HTTP.CONN_KEEP_ALIVE));
while (it.hasNext()) {
final HeaderElement he = it.nextElement();
final String param = he.getName();
final String value = he.getValue();
if (value != null && param.equalsIgnoreCase("timeout")) {
try {
return Long.parseLong(value) * 1000;
} catch (final NumberFormatException ignore) {
}
}
}
return 30 * 1000;
}
});
httpclient.setRedirectStrategy(new DefaultRedirectStrategy());
httpclient.addRequestInterceptor(new HttpRequestInterceptor() {
#Override
public void process(final HttpRequest request, final HttpContext context) throws HttpException, IOException {
if (!request.containsHeader("Accept-Encoding")) {
request.addHeader("Accept-Encoding", "gzip");
}
}
});
httpclient.addResponseInterceptor(new HttpResponseInterceptor() {
#Override
public void process(final HttpResponse response, final HttpContext context) throws HttpException, IOException {
response.setEntity(new BufferedHttpEntity(response.getEntity()));
final HttpEntity entity = response.getEntity();
final Header ceheader = entity.getContentEncoding();
if (ceheader != null) {
final HeaderElement[] codecs = ceheader.getElements();
for (int i = 0; i < codecs.length; i++) {
if (codecs[i].getName().equalsIgnoreCase("gzip")) {
response.setEntity(new GzipDecompressingEntity(response.getEntity()));
return;
}
}
}
}
});
startConnectionMonitorThread();
few addition methods:
private synchronized void startConnectionMonitorThread() {
if (connectionMonitorThread == null) {
connectionMonitorThread = new IdleConnectionMonitorThread(connectionManager);
}
connectionMonitorThread.start();
}
private synchronized void stopConnectionMonitorThread() {
if (connectionMonitorThread != null) {
connectionMonitorThread.shutdown();
connectionManager.shutdown();
}
}
public void shutdown() {
stopConnectionMonitorThread();
final ClientConnectionManager cm = httpclient.getConnectionManager();
if (cm != null) {
httpclient.getConnectionManager().shutdown();
}
}
Using:
final HttpGet httpGet = new HttpGet(url);
final HttpResponse response = httpclient.execute(httpGet);
final StatusLine statusLine = response.getStatusLine();
final int responseCode = statusLine.getStatusCode();
if (responseCode >= 300) {
logger.error(" {}. Received statusCode {}", url, responseCode);
httpGet.abort();
//throw some exception;
}
final HttpEntity entity = response.getEntity();
if (entity == null) {
//throw some exception or ignore;
}
responseBody = EntityUtils.toString(entity);
This code/params optimized for Crawler. to fast receive a lot of pages. with using gzip(if possible) and https if require without using cookies. For addition cookies you need to add cookieStore like httpclient.setCookieStore();
In my app, I need to send all sorts of POST requests to a server. some of those requests have responses and others don't.
this is the code I'm using to send the requests:
private static final String TAG = "Server";
private static final String PATH = "http://10.0.0.2:8001/data_connection";
private static HttpResponse response = null;
private static StringEntity se = null;
private static HttpClient client;
private static HttpPost post = null;
public static String actionKey = null;
public static JSONObject sendRequest(JSONObject req) {
try {
client = new DefaultHttpClient();
actionKey = req.getString("actionKey");
se = new StringEntity(req.toString());
se.setContentEncoding(new BasicHeader(HTTP.CONTENT_ENCODING, "application/json"));
se.setContentType(new BasicHeader(HTTP.CONTENT_TYPE, "application/json"));
post = new HttpPost(PATH);
post.setEntity(se);
Log.d(TAG, "http request is being sent");
response = client.execute(post);
Log.d(TAG, "http request was sent");
if (response != null) {
InputStream in = response.getEntity().getContent();
String a = convertFromInputStream(in);
in.close();
return new JSONObject(a);
}
} catch (UnsupportedEncodingException e) {
Log.d(TAG, "encoding request to String entity faild!");
e.printStackTrace();
} catch (ClientProtocolException e) {
Log.d(TAG, "executing the http POST didn't work");
e.printStackTrace();
} catch (IOException e) {
Log.d(TAG, "executing the http POST didn't work");
e.printStackTrace();
} catch (JSONException e) {
Log.d(TAG, "no ActionKey");
e.printStackTrace();
}
return null;
}
private static String convertFromInputStream(InputStream in)
throws IOException {
BufferedReader br = new BufferedReader(new InputStreamReader(in));
StringBuilder sb = new StringBuilder();
String line = null;
while ((line = br.readLine()) != null) {
sb.append(line);
}
return (sb.toString());
}
This is the code for the AsyncTask class that sends the request:
class ServerRequest extends AsyncTask<JSONObject, Void, JSONObject> {
#Override
protected JSONObject doInBackground(JSONObject... params) {
JSONObject req = params[0];
JSONObject response = Server.sendRequest(req);
return response;
}
#Override
protected void onPostExecute(JSONObject result) {
// HANDLE RESULT
super.onPostExecute(result);
}
}
my problem starts when the server doesn't return a response. the AsyncTask thread stays open even after the work is done because the HTTPClient never closes the connection.
Is there a way to not wait for a response? this is something that will definitely add a lot of overhead to the server since all the Android apps trying to connect to it will keep the connection alive, and will probably cause many problems on the app itself.
Basically, what I'm looking for is a method that will allow me to send to POST message and kill the connection right after the sending of the request since there is no response coming my way.
Just, Set ConnectionTimeOut with HttpClient Object, (Code is for your understanding in your case it may be different)
int TIMEOUT_MILLISEC = 30000;
HttpParams httpParams = new BasicHttpParams();
HttpConnectionParams.setConnectionTimeout(httpParams, TIMEOUT_MILLISEC);
HttpConnectionParams.setSoTimeout(httpParams, TIMEOUT_MILLISEC);
HttpClient httpclient = new DefaultHttpClient(httpParams);
HttpPost httppost = new HttpPost(url);
httppost.addHeader("Content-Type", "application/json");
Now, It will terminate the Connection after TimeoOut you defined. But be sure this will throw TimeOutException so You have to handle this exception in your HttpRequest.. (Use Try -catch)
EDIT: Or you can use HttpRequestExecutor class.
From class HttpRequestExecutor of package org.apache.http.protocol
protected boolean canResponseHaveBody (HttpRequest request, HttpResponse response)
Decide whether a response comes with an entity. The implementation in this class is based on RFC 2616. Unknown methods and response codes are supposed to indicate responses with an entity.
Derived executors can override this method to handle methods and response codes not specified in RFC 2616.