Send HTTP GET request with header - java

From my Android app I want to request a URL with GET parameters and read the response.
In the request I must add a x-zip header.
The URL is something like
http://example.com/getmethod.aspx?id=111&method=Test
Can some one provide me code for that?
Two things are important: that it is a GET request and contains the x-zip header .
EDIT:
try {
HttpClient client = new DefaultHttpClient();
String getURL = "http://example.com/getmethod.aspx?id=111&method=Test";
HttpGet get = new HttpGet(getURL);
get.setHeader("Content-Type", "application/x-zip");
HttpResponse responseGet = client.execute(get);
HttpEntity resEntityGet = responseGet.getEntity();
if (resEntityGet != null) {
//do something with the response
Log.i("GET ",EntityUtils.toString(resEntityGet));
}
} catch (Exception e) {
e.printStackTrace();
}
I try with this code but I get code with .net error: Object reference not set to an instance of an object...
I think but I'm not sure this if for x-zip header, is header in my code ok?

Here's a code excerpt we're using in our app to set request headers. You'll note we set the CONTENT_TYPE header only on a POST or PUT, but the general method of adding headers (via a request interceptor) is used for GET as well.
/**
* HTTP request types
*/
public static final int POST_TYPE = 1;
public static final int GET_TYPE = 2;
public static final int PUT_TYPE = 3;
public static final int DELETE_TYPE = 4;
/**
* HTTP request header constants
*/
public static final String CONTENT_TYPE = "Content-Type";
public static final String ACCEPT_ENCODING = "Accept-Encoding";
public static final String CONTENT_ENCODING = "Content-Encoding";
public static final String ENCODING_GZIP = "gzip";
public static final String MIME_FORM_ENCODED = "application/x-www-form-urlencoded";
public static final String MIME_TEXT_PLAIN = "text/plain";
private InputStream performRequest(final String contentType, final String url, final String user, final String pass,
final Map<String, String> headers, final Map<String, String> params, final int requestType)
throws IOException {
DefaultHttpClient client = HTTPClientFactory.newClient();
client.getParams().setParameter(HttpProtocolParams.USER_AGENT, mUserAgent);
// add user and pass to client credentials if present
if ((user != null) && (pass != null)) {
client.getCredentialsProvider().setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, pass));
}
// process headers using request interceptor
final Map<String, String> sendHeaders = new HashMap<String, String>();
if ((headers != null) && (headers.size() > 0)) {
sendHeaders.putAll(headers);
}
if (requestType == HTTPRequestHelper.POST_TYPE || requestType == HTTPRequestHelper.PUT_TYPE ) {
sendHeaders.put(HTTPRequestHelper.CONTENT_TYPE, contentType);
}
// request gzip encoding for response
sendHeaders.put(HTTPRequestHelper.ACCEPT_ENCODING, HTTPRequestHelper.ENCODING_GZIP);
if (sendHeaders.size() > 0) {
client.addRequestInterceptor(new HttpRequestInterceptor() {
public void process(final HttpRequest request, final HttpContext context) throws HttpException,
IOException {
for (String key : sendHeaders.keySet()) {
if (!request.containsHeader(key)) {
request.addHeader(key, sendHeaders.get(key));
}
}
}
});
}
//.... code omitted ....//
}

You do it exactly as you showed with this line:
get.setHeader("Content-Type", "application/x-zip");
So your header is fine and the problem is some other input to the web service. You'll want to debug that on the server side.

Related

How to reset/remove/clear the request body in rest-assured

I have to pass form-parameter as the body to my request. When I try
Response post = request.urlEncodingEnabled(true).log().all().config(RestAssured.config()
.encoderConfig(EncoderConfig.encoderConfig()
.encodeContentTypeAs("x-www-form-urlencoded", ContentType.URLENC)))
I am getting the error message as "You can either send form parameters OR body content in POST, not both!"
When I checked the log, previous api's response passed as body to this request. How to remove/reset/clear the body and pass only the form-parameter.
You should always use a new RequestSpecification Instance for each request.
Before each new request call a function like:
public void beforeNewRequest(){
restUtils.resetRestAssured(); //reset existing instance
restUtils = RestUtils.getInstance(); //get new instance
}
RestUtil.java class
public class RestUtils {
private static RestUtils apiUtilsInstance = null;
private RequestSpecification httpRequest;
private RestUtils() {
httpRequest = RestAssured.given();
}
public static RestUtils getInstance() {
if (apiUtilsInstance == null)
apiUtilsInstance = new RestUtils();
return apiUtilsInstance;
}
public RequestSpecification getRequestSpecification() {
return httpRequest;
}
public void resetRestAssured() {
apiUtilsInstance = null;
}
}

Java HttpURLConnection status code 302

I'm trying to get this code block to run but I keep getting a 302. I've tried to show the flow of the code. I just don't know what's wrong.
import java.net.HttpURLConnection;
import java.util.HashMap;
import java.util.Map;
import java.util.Base64;
public class AuthenticateLoginLogoutExample {
public static void main(String[] args) throws Exception {
new AuthenticateLoginLogoutExample().authenticateLoginLogoutExample(
"http://" + Constants.HOST + "/qcbin",
Constants.DOMAIN,
Constants.PROJECT,
Constants.USERNAME,
Constants.PASSWORD);
}
public void authenticateLoginLogoutExample(final String serverUrl,
final String domain, final String project, String username,
String password) throws Exception {
RestConnector con =
RestConnector.getInstance().init(
new HashMap<String, String>(),
serverUrl,
domain,
project);
AuthenticateLoginLogoutExample example =
new AuthenticateLoginLogoutExample();
//if we're authenticated we'll get a null, otherwise a URL where we should login at (we're not logged in, so we'll get a URL).
It's this next line when it starts on the isAuthenticated() method.
String authenticationPoint = example.isAuthenticated();
Assert.assertTrue("response from isAuthenticated means we're authenticated. that can't be.", authenticationPoint != null);
//do a bunch of other stuff
}
So we go into the isAuthenticated method:
public String isAuthenticated() throws Exception {
String isAuthenticateUrl = con.buildUrl("rest/is-authenticated");
String ret;
Then here on this next line trying to get the response. con.httpGet
Response response = con.httpGet(isAuthenticateUrl, null, null);
int responseCode = response.getStatusCode();
//if already authenticated
if (responseCode == HttpURLConnection.HTTP_OK) {
ret = null;
}
//if not authenticated - get the address where to authenticate
// via WWW-Authenticate
else if (responseCode == HttpURLConnection.HTTP_UNAUTHORIZED) {
Iterable<String> authenticationHeader =
response.getResponseHeaders().get("WWW-Authenticate");
String newUrl =
authenticationHeader.iterator().next().split("=")[1];
newUrl = newUrl.replace("\"", "");
newUrl += "/authenticate";
ret = newUrl;
}
//Not ok, not unauthorized. An error, such as 404, or 500
else {
throw response.getFailure();
}
return ret;
}
That jumps us to another class and into this method:
public Response httpGet(String url, String queryString, Map<String,
String> headers)throws Exception {
return doHttp("GET", url, queryString, null, headers, cookies);
}
The doHttp takes us here. type = "GET", url = "http://SERVER/qcbin/rest/is-authenticated", the rest are all empty.
private Response doHttp(
String type,
String url,
String queryString,
byte[] data,
Map<String, String> headers,
Map<String, String> cookies) throws Exception {
if ((queryString != null) && !queryString.isEmpty()) {
url += "?" + queryString;
}
HttpURLConnection con = (HttpURLConnection) new URL(url).openConnection();
con.setRequestMethod(type);
String cookieString = getCookieString();
prepareHttpRequest(con, headers, data, cookieString);
This con.connect() on the next line never connects.
con.connect();
Response ret = retrieveHtmlResponse(con);
updateCookies(ret);
return ret;
}
The prepareHttpRequest code:
private void prepareHttpRequest(
HttpURLConnection con,
Map<String, String> headers,
byte[] bytes,
String cookieString) throws IOException {
String contentType = null;
//attach cookie information if such exists
if ((cookieString != null) && !cookieString.isEmpty()) {
con.setRequestProperty("Cookie", cookieString);
}
//send data from headers
if (headers != null) {
//Skip the content-type header - should only be sent
//if you actually have any content to send. see below.
contentType = headers.remove("Content-Type");
Iterator<Entry<String, String>>
headersIterator = headers.entrySet().iterator();
while (headersIterator.hasNext()) {
Entry<String, String> header = headersIterator.next();
con.setRequestProperty(header.getKey(), header.getValue());
}
}
// If there's data to attach to the request, it's handled here.
// Note that if data exists, we take into account previously removed
// content-type.
if ((bytes != null) && (bytes.length > 0)) {
con.setDoOutput(true);
//warning: if you add content-type header then you MUST send
// information or receive error.
//so only do so if you're writing information...
if (contentType != null) {
con.setRequestProperty("Content-Type", contentType);
}
OutputStream out = con.getOutputStream();
out.write(bytes);
out.flush();
out.close();
}
}
And the getCookieString method:
public String getCookieString() {
StringBuilder sb = new StringBuilder();
if (!cookies.isEmpty()) {
Set<Entry<String, String>> cookieEntries =
cookies.entrySet();
for (Entry<String, String> entry : cookieEntries) {
sb.append(entry.getKey()).append("=").append(entry.getValue()).append(";");
}
}
String ret = sb.toString();
return ret;
}
Does anyone have any idea what went wrong? I don't know why it keeps returning a 302.
EDIT: Added chrome developer image as requested.
I haven't followed your entire code, but http 302 means a redirection
https://en.wikipedia.org/wiki/HTTP_302
Depending on the kind of redirection, that could work smoothly or not. For instance the other day I faced a http to https redirection and I have to solve it checking the location header manually.
What I would do is to check first the headers in the browser, in Chrome go to Developer Tools, Network and check the Response Headers (screenshot). You should see there for a 302 response a Location Header, with the new URL you should follow.
302 means there's a page there, but you really want a different page (or you want this page and then that other page). If you look at the headers you get back from the server when it gives you a 302, you'll probably find a "Location:" header telling you where to query next, and you'll have to write yet another transaction.
Browsers interpret the 302 response and automatically redirect to the URL specified in the "Location:" header.

Requests and response from netty server hangs

I have the following code to create a netty web server based on http server created in the netty's example. My buisness logic is the following.
public class HttpServerHandler extends SimpleChannelInboundHandler<Object> {
private final static Logger LOG = LogManager
.getLogger(HttpServerHandler.class);
private WorkflowService workflowService;
private HttpRequest request;
private final StringBuffer buff = new StringBuffer();
private API avalancheApi;
public HttpServerHandler(WorkflowService workflowService) {
this.workflowService = workflowService;
this.avalancheApi = new API(this.workflowService);
}
#Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
LOG.debug("channelActive");
LOG.debug(ctx.toString());
};
#Override
public void channelReadComplete(ChannelHandlerContext ctx) {
ctx.flush();
}
#Override
public void channelRead0(ChannelHandlerContext ctx, Object msg)
throws IOException {
avalancheApi.setContext(ctx);
if (msg instanceof HttpRequest) {
HttpRequest request = this.request = (HttpRequest) msg;
if (HttpHeaders.is100ContinueExpected(request)) {
send100Continue(ctx);
}
String command = getCommand(request);
LOG.debug(command);
Map<String, List<String>> parameters = getParameters(request);
LOG.debug(parameters);
switch (command) {
case "/login":
ctx = avalancheApi.login(parameters);
break;
case "/test":
ctx = avalancheApi.test();
break;
default:
break;
}
}
if (msg instanceof LastHttpContent) {
LOG.debug("msg is of LastHttpContent");
}
if (!HttpHeaders.isKeepAlive(request)) {
// If keep-alive is off, close the connection once the content is
// fully written.
ctx.writeAndFlush(Unpooled.EMPTY_BUFFER).addListener(
ChannelFutureListener.CLOSE);
}
}
public class API {
private static final Logger LOG = LogManager.getLogger(API.class);
private ChannelHandlerContext ctx;
private HttpResponse response;
private WorkflowService workflowService;
public API(WorkflowService workflowService) {
this.workflowService = workflowService;
this.ctx = null;
}
public void setContext(ChannelHandlerContext ctx) {
this.ctx = ctx;
}
public ChannelHandlerContext login(Map<String, List<String>> parameters) {
boolean success;
String username = getUsername(parameters);
String password = getPassword(parameters);
User user = null;
user = workflowService.login(username, password);
success = validateLogin(user);
this.response = writeLoginResponse(success);
this.ctx.write(this.response);
writeLoginContext(success, response);
return this.ctx;
}
private void writeLoginContext(boolean success, HttpResponse response) {
JsonObject jsonResponseMessage = new JsonObject();
jsonResponseMessage.addProperty("result", success);
LOG.debug(jsonResponseMessage.toString());
this.ctx.write(Unpooled.copiedBuffer(jsonResponseMessage.toString(),
CharsetUtil.UTF_8));
this.response.headers().set(HttpHeaders.Names.CONTENT_LENGTH,
jsonResponseMessage.toString().length());
}
private HttpResponse writeLoginResponse(boolean success) {
if (success)
return createSuccessfullLoginResponse();
else
return createLoginFailureResponse();
}
private HttpResponse createLoginFailureResponse() {
return Response.loginFailureResponse();
}
private HttpResponse createSuccessfullLoginResponse() {
return Response.loginSuccessResponse();
}
}
Response class is only creating the response and the content_type which is of application/json. Content Length is set in the API class. Using python client with requests, results in the request made in http://localhost/login?username=name&password=pass works only once. The second time everything works, but it doesn't finish processing the request and send the response object. Api calls get executed normally, and I also get the message of LastHttpContext message getting print. The problem sometimes happens with browser too. Am I missing something? Maybe the content data and the content length doesn't match? Could it be that when making requests from python client, the content of the previous context isn't flushed and the content_length value of the header and content length of the context doesn't match?
Just wild guess
this.response.headers().set(HttpHeaders.Names.CONTENT_LENGTH,
jsonResponseMessage.toString().length());
Instead, shouldn't you be doing jsonResponseMessage.toString().getBytes().length ?? Sometimes, one character is not just one byte.
My guess is that you have overwritten the context in your API class, and as a result, are writing the response to the wrong context. Is your HttpServerHandler marked with #Shareable?

How to send requests and receive responses to/from an Apache CXF-based service?

I have several servlets, which
take JSON-encoded requests as inputs,
process them and
return responses to the client as JSON-encoded objects.
Up to now I used Android as client (sample Android code see below).
Now I want to write a plain old Java program, which would send requests and receive the responses (do the same as the Android code). For this purpose I wrote a Java test (code see below, section Java code) and ran it.
At the client side I get this error:
21:43:38.930 [main] ERROR r.a.c.t.TestAcceptanceProcedure1 -
java.io.IOException: Server returned HTTP response code: 405 for URL: http://myserver/myapp/rest/GetUserIdService
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441) ~[na:1.6.0_23]
at testclient.TestAcceptanceProcedure1.test(TestAcceptanceProcedure1.java:47) ~[test-classes/:na]
In the server log, I see this message:
WARNING: No operation matching request path "/myapp/rest/GetUserIdService" is found, Relative Path: /, HTTP Method: GET, ContentType: */*, Accept: text/html,image/gif,image/jpeg,*/*,*/*;q=.2,. Please enable FINE/TRACE log level for more details
Question: How should I change my Java test to fix this error?
Note that the server is up and running (when I execute the Android code, it works).
Android code:
Sending the request and receiving the response:
final GetSimulationStatusRequest request = new GetSimulationStatusRequest();
final String json = Utils.convertToJson(request, getClass());
final String serverUrl = Utils.getServerUrl(this, "GetSimulationStatusService");
final IGetSimulationStatusAsyncTask getSimulationStatusTask =
asyncTaskFactory.createGetSimulationStatusAsyncTask(getWebServiceHelper());
Utils.setRequestAndServerUrl(json, serverUrl, getSimulationStatusTask);
final GetSimulationStatusResponse simulationStatusReponse =
getSimulationStatusTask.get();
Utils.convertToJson:
public static String convertToJson(final Object aRequest, Class<? extends Activity> aClass) {
final ObjectMapper mapper = new ObjectMapper();
String json = null;
try {
json = mapper.writeValueAsString(aRequest);
} catch (final JsonProcessingException exception) {
Log.e(aClass.getSimpleName(), exception.getLocalizedMessage(),
exception);
}
return json;
}
Utils.setRequestAndServerUrl:
public static void setRequestAndServerUrl(final String aJson,
final String aServerUrl, final IAsyncTask aTask) {
aTask.addNameValuePair("request", aJson);
aTask.sendRequest(new String[] { aServerUrl });
}
GetSimulationStatusAsyncTask:
public class GetSimulationStatusAsyncTask
extends AsyncTask<String, String, GetSimulationStatusResponse>
implements IGetSimulationStatusAsyncTask {
private static final String TAG = GetSimulationStatusAsyncTask.class.getSimpleName();
private IWebServiceTaskHelper helper;
private ICcpResponseParser<GetSimulationStatusResponse> responseParser =
new CcpResponseParser<GetSimulationStatusResponse>();
public GetSimulationStatusAsyncTask(final IWebServiceTaskHelper aHelper) {
helper = aHelper;
}
#Override
public void addNameValuePair(final String aName, final String aValue) {
helper.addNameValuePair(aName, aValue);
}
#Override
protected GetSimulationStatusResponse doInBackground(String... aArgs) {
return (GetSimulationStatusResponse)Utils.processResponse(this.helper, TAG, responseParser,
GetSimulationStatusResponse.class, aArgs);
}
#Override
public void sendRequest(final String[] aArgs) {
execute(aArgs);
}
}
Java code:
#Test
public void test() throws JsonProcessingException, MalformedURLException {
final GetUserIdRequest request = new GetUserIdRequest();
request.setDeviceId("PC1");
final String requestAsString = convertToJson(request);
final String serverUrl = getServerUrl("GetUserIdService");
final URL url = new URL(serverUrl);
HttpURLConnection connection = null;
InputStream inputStream = null;
try {
connection = (HttpURLConnection) url.openConnection();
connection.addRequestProperty("request", requestAsString);
connection.connect();
inputStream = connection.getInputStream();
final String responseAsString = IOUtils.toString(inputStream);
LOGGER.debug("responseAsString: " + responseAsString);
} catch (final IOException exception) {
LOGGER.error("", exception);
}
finally
{
IOUtils.closeQuietly(inputStream);
}
}
private String convertToJson(final GetUserIdRequest aRequest) throws JsonProcessingException {
final ObjectMapper mapper = new ObjectMapper();
return mapper.writeValueAsString(aRequest);
}
private String getServerUrl(final String aServiceName)
{
return "http://myserver.com/myapp/rest/" + aServiceName;
}
Update 1 (09.10.2013 12:23 MSK):
#Path("/GetSimulationStatusService")
public class GetSimulationStatusService extends BaseCcpService {
private GetSimulationStatusRequestParser requestParser =
new GetSimulationStatusRequestParser();
#POST
#Produces("text/plain")
public String getSimulationStatus(#FormParam("request") final String aRequestJson)
throws JsonProcessingException
{
final GetSimulationStatusRequest request = requestParser.convert(aRequestJson);
final GetSimulationStatusResponse response = new GetSimulationStatusResponse();
response.setRequestId(request.getId());
response.setCycle(getPersistence().getCurrentCycle(request.getUserId()));
response.setLabourForce(getPersistence().getLabourForceSimulationParameter(
request.getUserId()));
return getObjectMapper().writeValueAsString(response);
}
}
Update 2 (09.10.2013 20:48 MSK): When I change the code like shown below, I get 500 HTTP response. At the server side, the aRequest argument of the method GetUserIdService.getUserId is null.
connection = (HttpURLConnection) url.openConnection();
connection.addRequestProperty("request", requestAsString);
connection.setRequestMethod("POST"); // Added this line
connection.connect();
Update 3 (09.10.2013 23:15): This one works:
#Test
public void test() throws JsonProcessingException, MalformedURLException
{
final GetUserIdRequest request = new GetUserIdRequest();
request.setDeviceId("PC1");
final String requestAsString = convertToJson(request);
final String serverUrl = getServerUrl("GetUserIdService");
final URL url = new URL(serverUrl);
HttpURLConnection connection = null;
InputStream inputStream = null;
OutputStream outputStream = null;
try {
connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("POST");
connection.setDoOutput(true);
connection.connect();
outputStream = connection.getOutputStream();
IOUtils.write("request=" + requestAsString, outputStream);
inputStream = connection.getInputStream();
final String responseAsString = IOUtils.toString(inputStream);
LOGGER.debug("responseAsString: " + responseAsString);
} catch (final IOException exception) {
LOGGER.error("", exception);
}
finally
{
IOUtils.closeQuietly(inputStream);
IOUtils.closeQuietly(outputStream);
}
}
The 405 HTTP error code means that given method (GET) is not supported by the endpoint. Probably instead of GET request you want to send POST. I don't know what kind of request is sent by Android client.
Do you have endpoint specification/documentation?
Here you'll find information how to invoke POST using plain Java API. If you can use external libraries in your test then it can be achieved a lot easier using RESTEasy.

Servlet Filter is Returning "Proxy Error" on AWS

I have set up a Filter to add crawler support for my GWT web application. The idea is to catch all requests that contain "_escaped_fragment_=" and supply a snapshot for the crawler.
I have set up the Filter using Guice as follows:
filter("/*").through(CrawlerFilter.class);
The following is the code for the CrawlerFilter class (many thanks to Patrick):
#Singleton
public class CrawlerFilter implements Filter {
private static final Logger logger = Logger.getLogger(CrawlerFilter.class.getName());
/**
* Special URL token that gets passed from the crawler to the servlet
* filter. This token is used in case there are already existing query
* parameters.
*/
private static final String ESCAPED_FRAGMENT_FORMAT1 = "_escaped_fragment_=";
private static final int ESCAPED_FRAGMENT_LENGTH1 = ESCAPED_FRAGMENT_FORMAT1.length();
/**
* Special URL token that gets passed from the crawler to the servlet
* filter. This token is used in case there are not already existing query
* parameters.
*/
private static final String ESCAPED_FRAGMENT_FORMAT2 = "&" + ESCAPED_FRAGMENT_FORMAT1;
private static final int ESCAPED_FRAGMENT_LENGTH2 = ESCAPED_FRAGMENT_FORMAT2.length();
private class SyncAllAjaxController extends NicelyResynchronizingAjaxController {
private static final long serialVersionUID = 1L;
#Override
public boolean processSynchron(HtmlPage page, WebRequest request, boolean async) {
return true;
}
}
private WebClient webClient = null;
private static final long _pumpEventLoopTimeoutMillis = 30000;
private static final long _jsTimeoutMillis = 1000;
private static final long _pageWaitMillis = 200;
final int _maxLoopChecks = 2;
public void doFilter(ServletRequest request, ServletResponse response, FilterChain filterChain) throws IOException,
ServletException {
// Grab the request uri and query strings.
final HttpServletRequest httpRequest = (HttpServletRequest) request;
final String requestURI = httpRequest.getRequestURI();
final String queryString = httpRequest.getQueryString();
final HttpServletResponse httpResponse = (HttpServletResponse) response;
if ((queryString != null) && (queryString.contains(ESCAPED_FRAGMENT_FORMAT1))) {
// This is a Googlebot crawler request, let's return a static
// indexable html page post javascript execution, as rendered in the browser.
final String domain = httpRequest.getServerName();
final int port = httpRequest.getServerPort();
// Rewrite the URL back to the original #! version
// -- basically remove _escaped_fragment_ from the query.
// Unescape any %XX characters as need be.
final String urlStringWithHashFragment = requestURI + rewriteQueryString(queryString);
final String scheme = httpRequest.getScheme();
final URL urlWithHashFragment = new URL(scheme, "127.0.0.1", port, urlStringWithHashFragment); // get from localhost
final WebRequest webRequest = new WebRequest(urlWithHashFragment);
// Use the headless browser to obtain an HTML snapshot.
webClient = new WebClient(BrowserVersion.FIREFOX_3_6);
webClient.getCache().clear();
webClient.setJavaScriptEnabled(true);
webClient.setThrowExceptionOnScriptError(false);
webClient.setRedirectEnabled(false);
webClient.setAjaxController(new SyncAllAjaxController());
webClient.setCssErrorHandler(new SilentCssErrorHandler());
if (logger.getLevel() == Level.FINEST)
logger.log(Level.FINEST, "HtmlUnit starting webClient.getPage(webRequest) where webRequest = "
+ webRequest.toString());
final HtmlPage page = webClient.getPage(webRequest);
// Important! Give the headless browser enough time to execute
// JavaScript
// The exact time to wait may depend on your application.
webClient.getJavaScriptEngine().pumpEventLoop(_pumpEventLoopTimeoutMillis);
int waitForBackgroundJavaScript = webClient.waitForBackgroundJavaScript(_jsTimeoutMillis);
int loopCount = 0;
while (waitForBackgroundJavaScript > 0 && loopCount < _maxLoopChecks) {
++loopCount;
waitForBackgroundJavaScript = webClient.waitForBackgroundJavaScript(_jsTimeoutMillis);
if (waitForBackgroundJavaScript == 0) {
if (logger.getLevel() == Level.FINEST)
logger.log(Level.FINEST, "HtmlUnit exits background javascript at loop counter " + loopCount);
break;
}
synchronized (page) {
if (logger.getLevel() == Level.FINEST)
logger.log(Level.FINEST, "HtmlUnit waits for background javascript at loop counter "
+ loopCount);
try {
page.wait(_pageWaitMillis);
}
catch (InterruptedException e) {
logger.log(Level.SEVERE, "HtmlUnit ERROR on page.wait at loop counter " + loopCount);
e.printStackTrace();
}
}
}
webClient.getAjaxController().processSynchron(page, webRequest, false);
if (webClient.getJavaScriptEngine().isScriptRunning()) {
logger.log(Level.WARNING, "HtmlUnit webClient.getJavaScriptEngine().shutdownJavaScriptExecutor()");
webClient.getJavaScriptEngine().shutdownJavaScriptExecutor();
}
// Return the static snapshot.
final String staticSnapshotHtml = page.asXml();
httpResponse.setContentType("text/html;charset=UTF-8");
final PrintWriter out = httpResponse.getWriter();
out.println("<hr />");
out.println("<center><h3>This is a non-interactive snapshot for crawlers. Follow <a href=\"");
out.println(urlWithHashFragment + "\">this link</a> for the interactive application.<br></h3></center>");
out.println("<hr />");
out.println(staticSnapshotHtml);
// Close web client.
webClient.closeAllWindows();
out.println("");
out.flush();
out.close();
if (logger.getLevel() == Level.FINEST)
logger.log(Level.FINEST, "HtmlUnit completed webClient.getPage(webRequest) where webRequest = "
+ webRequest.toString());
}
else {
if (requestURI.contains(".nocache.")) {
// Ensure the gwt nocache bootstrapping file is never cached.
// References:
// https://stackoverflow.com/questions/4274053/how-to-clear-cache-in-gwt
// http://seewah.blogspot.com/2009/02/gwt-tips-2-nocachejs-getting-cached-in.html
//
final Date now = new Date();
httpResponse.setDateHeader("Date", now.getTime());
httpResponse.setDateHeader("Expires", now.getTime() - 86400000L); // One day old.
httpResponse.setHeader("Pragma", "no-cache");
httpResponse.setHeader("Cache-control", "no-cache, no-store, must-revalidate");
}
filterChain.doFilter(request, response);
}
}
/**
* Maps from the query string that contains _escaped_fragment_ to one that
* doesn't, but is instead followed by a hash fragment. It also unescapes
* any characters that were escaped by the crawler. If the query string does
* not contain _escaped_fragment_, it is not modified.
*
* #param queryString
* #return A modified query string followed by a hash fragment if
* applicable. The non-modified query string otherwise.
* #throws UnsupportedEncodingException
*/
private static String rewriteQueryString(String queryString) throws UnsupportedEncodingException {
// Seek the escaped fragment.
int index = queryString.indexOf(ESCAPED_FRAGMENT_FORMAT2);
int length = ESCAPED_FRAGMENT_LENGTH2;
if (index == -1) {
index = queryString.indexOf(ESCAPED_FRAGMENT_FORMAT1);
length = ESCAPED_FRAGMENT_LENGTH1;
}
if (index != -1) {
// Found the escaped fragment, so build back the original decoded
// one.
final StringBuilder queryStringSb = new StringBuilder();
// Add url parameters if any.
if (index > 0) {
queryStringSb.append("?");
queryStringSb.append(queryString.substring(0, index));
}
// Add the hash fragment as a replacement for the escaped fragment.
queryStringSb.append("#!");
// Add the decoded token.
final String token2Decode = queryString.substring(index + length, queryString.length());
final String tokenDecoded = URLDecoder.decode(token2Decode, "UTF-8");
queryStringSb.append(tokenDecoded);
return queryStringSb.toString();
}
return queryString;
}
#Override
public void destroy() {
if (webClient != null)
webClient.closeAllWindows();
}
#Override
public void init(FilterConfig config) throws ServletException {
}
}
It uses HtmlUnit to create the snapshot.
However; the error occurs when I try to access the snapshot using a regular browser. The URL that I enter is of the form:
http://www.myapp.com/?_escaped_fragment_=myobject%3Bid%3D507ac730e4b0e2b7a73b1b81
But the processing by the Filter results in the following error:
Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /.
Reason: Error reading from remote server
Apache/2.2.22 (Amazon) Server at www.myapp.com Port 80
Any help would be appreciated.

Categories