We Keep Coding

How to correctly handling HTTP request errors?

I have a simple http client that passes every request to ExecutorService and applies them with a delay.
protected static final int RETRY_ATTEMPTS = 5;
private static final int GROUP_REQUEST_DELAY_MS = 55;
private static final ScheduledExecutorService REQUEST_FROM_USER_EXECUTOR =
Executors.newSingleThreadScheduledExecutor();
public RequestResponse post(String url) throws IOException {
HttpPost httpPost = new HttpPost(url);
HttpResponse httpResponse = call(httpPost);
return new RequestResponse(
httpResponse.getStatusLine().getStatusCode(),
EntityUtils.toString(httpResponse.getEntity()),
headers(httpResponse.getAllHeaders())
);
}
private HttpResponse call(HttpRequestBase request) throws IOException {
int attempts = 0;
HttpResponse httpResponse = null;
SocketException socketException = null;
do {
try {
httpResponse = client.execute(request);
} catch(SocketException e) {
socketException = e;
}
if(httpResponse != null)
break;
attempts++;
log.debug("Attempt: {}, SocEx: {}", attempts, socketException != null);
}while(attempts < RETRY_ATTEMPTS);
if(httpResponse == null)
// TODO
if(socketException != null) {
log.error("Network problem");
logRequest(request, httpResponse);
throw socketException;
}
return httpResponse;
}
public synchronized Future<RequestResponse> sendAsGroup(String url) {
return REQUEST_FROM_GROUP_EXECUTOR.schedule(() -> post(url), GROUP_REQUEST_DELAY_MS, TimeUnit.MILLISECONDS);
}
Sometimes the server throws an http 504 error or so. I want to process this error and resubmit this request. How can I do this correctly without exceeding the limit for server requests?

You should be using HttpRequestRetryHandler to recover from transport level (TCP) errors and ServiceUnavailableRetryStrategy to retry the request execution in case of a protocol level (HTTP) error.

Steammobile protocol is not supported

I am using Http Apache Client for Java and am making a request against the SteamWEB API to this steam link.
That causes an error in the Http Apache Client:
org.apache.http.client.ClientProtocolException
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at pl.edu.tirex.helper.http.Http.handleConnection(Http.java:77)
at pl.edu.tirex.helper.http.Http.get(Http.java:60)
at pl.edu.tirex.helper.http.Http.get(Http.java:64)
at pl.edu.tirex.helper.http.Http.get(Http.java:68)
at pl.edu.tirex.steamapi.steamguard.SteamGuard.fetchConfirmations(SteamGuard.java:42)
at pl.edu.tirex.steambot.SteamBOT.<init>(SteamBOT.java:93)
at pl.edu.tirex.steambot.SteamBOT.main(SteamBOT.java:56)
Caused by: org.apache.http.HttpException: steammobile protocol is not supported
at org.apache.http.impl.conn.DefaultRoutePlanner.determineRoute(DefaultRoutePlanner.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:157)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
... 8 more
My code for connection to send result:
private static final PoolingHttpClientConnectionManager CONNECTION_MANAGER = new PoolingHttpClientConnectionManager();
static
{
CONNECTION_MANAGER.setDefaultMaxPerRoute(2);
CONNECTION_MANAGER.setConnectionConfig(new HttpHost("https://steamcommunity.com/", 443, "steammobile"), ConnectionConfig.DEFAULT);
CONNECTION_MANAGER.setMaxTotal(4);
}
public static void main(String[] args)
{
HttpGet httpget = new HttpGet("url");
CloseableHttpClient httpclient = HttpClients.custom().setDefaultCookieStore(cookieStore).setConnectionManager(CONNECTION_MANAGER).build();
try (CloseableHttpResponse response = httpclient.execute(httpget, this.context))
{
HttpEntity entity = response.getEntity();
try (InputStream instream = entity.getContent())
{
if (handle != null)
{
if (response.getStatusLine().getStatusCode() >= 400)
{
handle.handleError(instream);
}
else
{
handle.handle(instream);
}
}
}
}
catch (HttpHostConnectException | InterruptedIOException | ClientProtocolException ignored)
{
}
}
How do I cope with this error?

These solution works properly:
private static final PoolingHttpClientConnectionManager CONNECTION_MANAGER = new PoolingHttpClientConnectionManager();
static
{
CONNECTION_MANAGER.setDefaultMaxPerRoute(2);
CONNECTION_MANAGER.setMaxTotal(4);
}
public static void main(String[] args)
{
HttpGet httpget = new HttpGet("url");
httpget.addHeader("X-Requested-With", "com.valvesoftware.android.steam.community");
CloseableHttpClient httpclient = HttpClients.custom().setDefaultCookieStore(cookieStore).setConnectionManager(CONNECTION_MANAGER).build();
try (CloseableHttpResponse response = httpclient.execute(httpget, this.context))
{
HttpEntity entity = response.getEntity();
try (InputStream instream = entity.getContent())
{
if (handle != null)
{
if (response.getStatusLine().getStatusCode() >= 400)
{
handle.handleError(instream);
}
else
{
handle.handle(instream);
}
}
}
}
catch (HttpHostConnectException | InterruptedIOException | ClientProtocolException ignored)
{
}
}

com.sun.net.httpserver doesn't work with org.apache.http.client

I make two requests from one client to the server and client hangs on second execute of method httpclient.execute(httpPost); if I create new client for each request everything is OK. If I make POST to google(for example) everything is OK. What's the problem?
Server:
public static void main(String[] args) throws IOException {
HttpServer server = HttpServer.create(new InetSocketAddress("localhost", 8080),2);
server.setExecutor(Executors.newFixedThreadPool(5));
HttpContext context = server.createContext("/", new GlobalHandler());
context.getFilters().add(new GlobalFilter());
server.createContext("/setup.exe", new GetFileHandler());
server.start();
}
private static class GlobalHandler implements HttpHandler {
#Override
public void handle(HttpExchange t) throws IOException {
Headers headers = t.getRequestHeaders();
System.out.println(headers.entrySet());
System.out.println(t.getHttpContext());
System.out.println(t.getRequestURI());
System.out.println(t.getProtocol());
System.out.println(t.getRequestMethod());
String resp = "Hello";
if (Objects.equals(t.getRequestMethod(), "POST")) {
System.out.println("POST");
Map params = (Map) t.getAttribute("parameters");
resp = params.toString();
System.out.println(params);
}
byte[] bytesToWrite = resp.getBytes("UTF-8");
t.sendResponseHeaders(200, bytesToWrite.length);
OutputStream os = t.getResponseBody();
os.write(bytesToWrite);
}
}
Client:
RequestConfig requestConfig = RequestConfig.custom().setConnectTimeout(1 * 1000).build();
httpclient = HttpClientBuilder.create().setDefaultRequestConfig(requestConfig).build();
for(int i= 0;i<2;i++) {
HttpPost httpPost = new HttpPost("http://google.com");
httpPost.addHeader("Connection","keep-alive");
List<NameValuePair> nvps = new ArrayList<NameValuePair>();
nvps.add(new BasicNameValuePair("username", "вг"));
nvps.add(new BasicNameValuePair("password", "аб"));
httpPost.setEntity(new UrlEncodedFormEntity(nvps, "UTF-8"));
CloseableHttpResponse response2 = httpclient.execute(httpPost);
try {
System.out.println(response2.getStatusLine());
HttpEntity entity2 = response2.getEntity();
if(entity2.getContentLength() >0) {
byte[] content = new byte[(int) entity2.getContentLength()];
entity2.getContent().read(content);
System.out.println(new String(content, "UTF-8"));
}
EntityUtils.consume(entity2);
} finally {
response2.close();
}
}
httpclient.close();

Storing session cookie to maintain log in session

I've been trying to get this work for a while now. Im working on an app where the user signs in with a username and password which uses a httppost request to post to the server. i get the correct response, and during the post i store the session cookie that the server gives me. (I store it in a cookie store)
But when i try to click a link on the menu ( which does a second http post) after i logged in, the servers gives me a message saying that i am not logged in. But i send the cookie that i recieved in the first post to the server in the second post, yet the server does not recognize that i am logged in. To test this more easily i used the chrome plug in "Postman" which lets you post to websites easily. The only time it worked was when i log in to the website using chrome then use Postman to do the second post, which successfully gives me a response. however, when i use Postman to log in, then also use postman to attempt the second post , again, "Not logged in". Im guessing that the cookie is not being stored properly in the app. How could i go about fixing this? I read some stuff about storing the cookies in something called "Shared Preferences", is that possibly a fix? If so, what is it and how could i store the cookies there?
public static String executeHttpPost(String url, ArrayList<NameValuePair> postParameters) throws Exception {
BufferedReader in = null;
try {
LoginLayout.httpClient = new DefaultHttpClient();
HttpPost request = new HttpPost(url);
UrlEncodedFormEntity formEntity = new UrlEncodedFormEntity(postParameters);
request.setEntity(formEntity);
CookieStore cookiestore = LoginLayout.httpClient.getCookieStore();
HttpResponse response = LoginLayout.httpClient.execute(request);
List<Cookie> cookies = LoginLayout.httpClient.getCookieStore().getCookies();
cookiestore.addCookie(cookie);
cookie = cookies.get(0);
cookieValue = "ASPSESSIONIDCQTCRACT=" + cookiestore.getCookies();
System.out.println("The cookie" + cookieValue);
List<Cookie> cookiess = cookiestore.getCookies();
cookiee = cookies.get(0);
Header[] headers = response.getAllHeaders();
System.out.println("length" + headers.length);
for (int i=0; i < headers.length; i++) {
Header h = headers[i];
System.out.println( "Header names: "+h.getName());
System.out.println( "Header Value: "+h.getValue());
}
in = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
StringBuffer sb = new StringBuffer("");
String line = "";
String NL = System.getProperty("line.separator");
while ((line = in.readLine()) != null) {
sb.append(line + NL);
}
in.close();
// System.out.println( mCookie);
String result = sb.toString();
return result;
}
finally {
if (in != null) {
try {
in.close();
} catch (IOException e) {
e.printStackTrace();
}
}
} }
Here is the getter so i can access the cookie from the cookie store in the next activity
public static String getCookie(){
return cookiee.getName() +"="+cookiee.getValue();
}
Here is the second post where i try to retrieve the stored cookie, which it seems to do sucessfully, however the server doesnt recognize i am logged in
public static String executeHttpPost(String url, ArrayList<NameValuePair> postParameters) throws Exception {
BufferedReader in = null;
try {
HttpPost request = new HttpPost(url);
request.setHeader("Cookie", LoginLayout.getCookie());
UrlEncodedFormEntity formEntity = new UrlEncodedFormEntity(postParameters);
request.setEntity(formEntity);
HttpResponse response = LoginLayout.httpClient.execute(request);
Header[] headers = response.getAllHeaders();
System.out.println("length" + headers.length);
for (int i=0; i < headers.length; i++) {
Header h = headers[i];
System.out.println( "Header names: "+h.getName());
System.out.println( "Header Value: "+h.getValue());
}
in = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
StringBuffer sb = new StringBuffer("");
String line = "";
String NL = System.getProperty("line.separator");
while ((line = in.readLine()) != null) {
sb.append(line + NL);
}
in.close();
//System.out.println( mCookie);
String result = sb.toString();
return result;
}
finally {
if (in != null) {
try {
in.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}

You have to make sure that your HttpClient is using the same HttpContext on each request.
The CookieStore is associated with the HttpContext so create a new instance of HttpContext will create a new CookieStore as well.
The best way I found is to create a static instance of HttpContext and use it on every request.
Below I added an part of a class I'm using in my apps:
public class ApiClient {
// Constants
private final static String TAG = "ApiClient";
private final static String API_URL = "your-url";
// Data
private static ApiClient mInstance;
private HttpClient mHttpClient;
private ThreadSafeClientConnManager mConnectionManager;
private HttpPost mPost;
/*
* we need it static because otherwise it will be recreated and the session
* will be lost
*/
private static HttpContext mHttpContext;
private HttpParams mParams;
private Context mContext;
public ApiClient(Context pContext) {
mParams = new BasicHttpParams();
mContext = pContext;
if (null == mHttpContext) {
CookieStore cookieStore = new BasicCookieStore();
mHttpContext = new BasicHttpContext();
mHttpContext.setAttribute(ClientContext.COOKIE_STORE, cookieStore);
}
ConnManagerParams.setMaxTotalConnections(mParams, 300);
HttpProtocolParams.setVersion(mParams, HttpVersion.HTTP_1_1);
SchemeRegistry schemeRegistry = new SchemeRegistry();
schemeRegistry.register(new Scheme("http", PlainSocketFactory.getSocketFactory(), 80));
mConnectionManager = new ThreadSafeClientConnManager(mParams, schemeRegistry);
mHttpClient = new DefaultHttpClient(mConnectionManager, mParams);
}
public static ApiClient getInstance(Context pContext) {
if (null == mInstance) {
return (mInstance = new ApiClient(pContext));
} else {
return mInstance;
}
}
public void testPOST() {
List<NameValuePair> requestParams = new ArrayList<NameValuePair>();
requestParams.add(new BasicNameValuePair("param1", "value1"));
requestParams.add(new BasicNameValuePair("param2", "value2"));
mPost = new HttpPost(API_URL);
try {
mPost.setEntity(new UrlEncodedFormEntity(requestParams, HTTP.UTF_8));
HttpResponse responsePOST = mHttpClient.execute(mPost, mHttpContext);
HttpEntity resEntity = responsePOST.getEntity();
String result = EntityUtils.toString(resEntity);
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
} catch (ClientProtocolException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}
To test it try setting the correct API_URL and
ApiClient api = ApiClient.getInstance(somContext);
api.testPOST();

Httpclient 4, error 302. How to redirect?

I want to access one site that first requires an (tomcat server) authentication and then log in with a POST request and keep that user to see the site's pages. I use Httpclient 4.0.1
The first authentication works fine but not the logon that always complains about this error: "302 Moved Temporarily"
I keep cookies & I keep a context and yet nothing. Actually, it seems that the logon works, because if I write incorrect parameters or user||password, I see the login page. So I guess what doesn't work is the automatic redirection.
Following my code, which always throws the IOException, 302:
DefaultHttpClient httpclient = new DefaultHttpClient();
CookieStore cookieStore = new BasicCookieStore();
httpclient.getParams().setParameter(
ClientPNames.COOKIE_POLICY, CookiePolicy.BROWSER_COMPATIBILITY);
HttpContext context = new BasicHttpContext();
context.setAttribute(ClientContext.COOKIE_STORE, cookieStore);
//ResponseHandler<String> responseHandler = new BasicResponseHandler();
Credentials testsystemCreds = new UsernamePasswordCredentials(TESTSYSTEM_USER, TESTSYSTEM_PASS);
httpclient.getCredentialsProvider().setCredentials(
new AuthScope(AuthScope.ANY_HOST, AuthScope.ANY_PORT),
testsystemCreds);
HttpPost postRequest = new HttpPost(cms + "/login");
List<NameValuePair> formparams = new ArrayList<NameValuePair>();
formparams.add(new BasicNameValuePair("pUserId", user));
formparams.add(new BasicNameValuePair("pPassword", pass));
postRequest.setEntity(new UrlEncodedFormEntity(formparams, "UTF-8"));
HttpResponse response = httpclient.execute(postRequest, context);
System.out.println(response);
if (response.getStatusLine().getStatusCode() != HttpStatus.SC_OK)
throw new IOException(response.getStatusLine().toString());
HttpUriRequest currentReq = (HttpUriRequest) context.getAttribute(
ExecutionContext.HTTP_REQUEST);
HttpHost currentHost = (HttpHost) context.getAttribute(
ExecutionContext.HTTP_TARGET_HOST);
String currentUrl = currentHost.toURI() + currentReq.getURI();
System.out.println(currentUrl);
HttpEntity entity = response.getEntity();
if (entity != null) {
long len = entity.getContentLength();
if (len != -1 && len < 2048) {
System.out.println(EntityUtils.toString(entity));
} else {
// Stream content out
}
}

For 4.1 version:
DefaultHttpClient httpclient = new DefaultHttpClient();
httpclient.setRedirectStrategy(new DefaultRedirectStrategy() {
public boolean isRedirected(HttpRequest request, HttpResponse response, HttpContext context) {
boolean isRedirect=false;
try {
isRedirect = super.isRedirected(request, response, context);
} catch (ProtocolException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
if (!isRedirect) {
int responseCode = response.getStatusLine().getStatusCode();
if (responseCode == 301 || responseCode == 302) {
return true;
}
}
return isRedirect;
}
});

For HttpClient 4.3.x :
HttpClient httpClient = HttpClientBuilder.create().setRedirectStrategy(new LaxRedirectStrategy()).build();

In later versions of HttpCLient (4.1+), you can just do this:
DefaultHttpClient client = new DefaultHttpClient()
client.setRedirectStrategy(new LaxRedirectStrategy())
LaxRedirectStrategy will automatically redirect HEAD, GET, and POST requests. For a stricter implementation, use DefaultRedirectStrategy.

You have to implement custom redirection handler that will indicate that response to POST is a redirection. This can be done by overriding isRedirectRequested() method as shown below.
DefaultHttpClient client = new DefaultHttpClient();
client.setRedirectHandler(new DefaultRedirectHandler() {
#Override
public boolean isRedirectRequested(HttpResponse response, HttpContext context) {
boolean isRedirect = super.isRedirectRequested(response, context);
if (!isRedirect) {
int responseCode = response.getStatusLine().getStatusCode();
if (responseCode == 301 || responseCode == 302) {
return true;
}
}
return isRedirect;
}
});
In later version of HttpClient, the class name is DefaultRedirectStrategy, but similar solution can be used there.

httpclient.setRedirectHandler(new DefaultRedirectHandler());
See HttpClient Javadoc

Redirects are not handled automatically by HttpClient 4.1 for other methods than GET and PUT.

Extend the DefaultRedirectStrategy class and override the methods.
#Override
protected URI createLocationURI(String arg0) throws ProtocolException {
// TODO Auto-generated method stub
return super.createLocationURI(arg0);
}
#Override
protected boolean isRedirectable(String arg0) {
// TODO Auto-generated method stub
return true;
}
#Override
public URI getLocationURI(HttpRequest arg0, HttpResponse arg1,
HttpContext arg2) throws ProtocolException {
// TODO Auto-generated method stub
return super.getLocationURI(arg0, arg1, arg2);
}
#Override
public HttpUriRequest getRedirect(HttpRequest request,
HttpResponse response, HttpContext context)
throws ProtocolException {
URI uri = getLocationURI(request, response, context);
String method = request.getRequestLine().getMethod();
if (method.equalsIgnoreCase(HttpHead.METHOD_NAME)) {
return new HttpHead(uri);
} else {
return new HttpPost(uri);
}
}
#Override
public boolean isRedirected(HttpRequest request, HttpResponse response,
HttpContext context) throws ProtocolException {
// TODO Auto-generated method stub
return super.isRedirected(request, response, context);
}
in this case isRedirectable method will always return true and getRedirect method will return post request in place of get request.