For some queries I receive this ServiceException after 10 seconds of
waiting for the response.
I also tried with direct http get request and the same result.
Ex: Contract%20Colectiv%20de%20Munc%C4%83
1. My code:
URL feedUrl = new URL("https://docs.google.com/feeds/default/private/ full/folder%3A" + folderId + "/contents/-/pdf");
DocumentQuery query = new DocumentQuery(feedUrl);
query.setFullTextQuery(searchText);
client.setConnectTimeout(0);// with or without this line I receive the same result (I also put 30000 value - same result)
client.setReadTimeout(0);// with or without this line I receive the same result
DocumentListFeed feed = client.getFeed(query, DocumentListFeed.class);
2. This is the stacktrace for the exception that I receive with documentlist api query:
com.google.gdata.util.ServiceException: An unknown error has occurred.
<errors xmlns='http://schemas.google.com/g/2005'>
<error><domain>GData</domain><code>ServiceException</code>
<internalReason>An unknown error has occurred</internalReason>
</error></errors>
at com.google.gdata.client.http.HttpGDataRequest.handleErrorResponse(HttpGDataRequest.java:624)
at com.google.gdata.client.http.GoogleGDataRequest.handleErrorResponse(GoogleGDataRequest.java:563)
at com.google.gdata.client.http.HttpGDataRequest.checkResponse(HttpGDataRequest.java: 552)
at com.google.gdata.client.http.HttpGDataRequest.execute(HttpGDataRequest.java: 530)
at com.google.gdata.client.http.GoogleGDataRequest.execute(GoogleGDataRequest.java:535)
at com.google.gdata.client.Service.getFeed(Service.java:1135)
...
3. This is the exception I receive with direct http get request:
java.io.IOException: Server returned HTTP response code: 500 for URL: https://docs.google.com/feeds/default/private/full/folder%3[my_folder_doc-id]/contents/-/pdf?q="[query_text]"&max-results=25
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java: 1436)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java: 379)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java: 318)
at GoogleDocsManager.googleSearch(GoogleDocsManager.java:281)
Additional information:
1. My folder contains almost 300k files. Could this be the problem?
2. In ~85% of searches I get the correct response ()
3. In browser the same interogation returns "The server encountered an error. Please try again later", but after refresh works fine.
Can anyone help me to get a "workaround" for this problem? Or how to
avoid it?
I posted this issue some months ago on documentlist api group but since this group is read-only I can't get any information about this issue.
This is the 500 response I get with direct http request (after ~10 seconds):
<errors xmlns='http://schemas.google.com/g/2005'>
<error>
<domain>GData</domain>
<code>ServiceException</code>
<internalReason>An unknown error has occurred.</internalReason>
</error>
</errors>
This is the code:
URL feedUrl = new URL("https://docs.google.com/feeds/default/private/full/folder%3A" + folderId + "/contents/-/pdf?max-results=25&q=" + searchText);
HttpURLConnection copyHttpUrlConn = (HttpURLConnection) feedUrl.openConnection();
copyHttpUrlConn.setDoOutput(true);
copyHttpUrlConn.setRequestMethod("GET");
copyHttpUrlConn.setRequestProperty("GData-Version", "3.0");
copyHttpUrlConn.setRequestProperty("Authorization", "GoogleLogin auth=" + authToken);
int respCode = copyHttpUrlConn.getResponseCode();
System.out.println("Response code: " + respCode);
InputStreamReader isr = null;
if(respCode != 200){
isr = new InputStreamReader(copyHttpUrlConn.getErrorStream());
}
else{
isr = new InputStreamReader(copyHttpUrlConn.getInputStream());
}
BufferedReader br = new BufferedReader(isr);
String line = null;
while((line = br.readLine()) != null){
System.out.println(line);
}
Other recent queries with problems:
title:2012-05 "exceptii de neconstitutionalitate penal"
"litigii de munca"
"moş crăciun srl"
"bil terenuri sa bucuresti"
"ordonanta de plata"
As a workaround you can consider fetching the large document list offline using a task queue, and serializing the information you want to display in the datastore.
Using task queue will give you give you automatic retry if there is an error, and up to 10 minutes deadline for the urlfetch and processing deadline.
Related
I am trying to reproduce the example from the Wiki tutorial for Project Wonder REST:
community.org/display/WEB/Your+First+Rest+Project#YourFirstRestProject-Addingpostsandauthorswithcurl
I am the point where you add entries in the DB with curl (I couldn't do it, I added them via SQL).
I am trying to run the curl command to retrieve entries and get an error "Empry reply from server". The console reports the following:
Request start for URI /cgi-bin/WebObjects/BlogTutorial.woa/ra/blogEntries.json
Headers{accept = ("*/*"); host = ("127.0.0.1:45743"); user-agent = ("curl/7.38.0"); }
[2015-8-14 17:20:19 CEST] <WorkerThread14> <er.rest.routes.ERXRouteRequestHandler>: Exception while handling action named "index" on action class "your.app.rest.controllers.BlogEntryController" :com.webobjects.foundation.NSForwardException [java.lang.reflect.InvocationTargetException] null:java.lang.reflect.InvocationTargetException
_ignoredPackages:: ("com.webobjects", "java.applet", "java.awt", "java.awt.datatransfer", "java.awt.event", "java.awt.image", "java.beans", "java.io", "java.lang", "java.lang.reflect", "java.math", "java.net", "java.rmi", "java.rmi.dgc", "java.rmi.registry", "java.rmi.server", "java.security", "java.security.acl", "java.security.interfaces", "java.sql", "java.text", "java.util", "java.util.zip")
Headers{cache-control = ("private", "no-cache", "no-store", "must-revalidate", "max-age=0"); expires = ("Fri, 14-Aug-2015 15:20:19 GMT"); content-type = ("text/html"); content-length = ("9296"); pragma = ("no-cache"); x-webobjects-loadaverage = ("1"); date = ("Fri, 14-Aug-2015 15:20:19 GMT"); set-cookie = (); }
The request start and both Headers messages are mine, through an override of dispatchRequest.
Any ideas?
I'm experiencing java.net.ConnectException in random ways.
My servlet runs in Tomcat 6.0 (JDK 1.6).
The servlet periodically fetches data from 4-5 third-party web servers.
The servlet uses a ScheduledExecutorService to fetch the data.
Run locally, all is fine and dandy. Run on my prod server, I see semi-random failures to fetch data from 1 of the third parties (Canadian weather data).
These are the URLs that are failing (plain RSS feeds):
http://weather.gc.ca/rss/city/pe-1_e.xml
http://weather.gc.ca/rss/city/pe-2_e.xml
http://weather.gc.ca/rss/city/pe-3_e.xml
http://weather.gc.ca/rss/city/pe-4_e.xml
http://weather.gc.ca/rss/city/pe-5_e.xml
http://weather.gc.ca/rss/city/pe-6_e.xml
http://meteo.gc.ca/rss/city/pe-1_f.xml
http://meteo.gc.ca/rss/city/pe-2_f.xml
http://meteo.gc.ca/rss/city/pe-3_f.xml
http://meteo.gc.ca/rss/city/pe-4_f.xml
http://meteo.gc.ca/rss/city/pe-5_f.xml
http://meteo.gc.ca/rss/city/pe-6_f.xml
Strange: each cycle, when I periodically fetch this data, the success/fail is all over the map: some succeed, some fail, but it never seems to be the same twice. So, I'm not completely blocked, just randomly blocked.
I slowed down my fetches, by introducing a 61s pause between each one. That had no effect.
The guts of the code that does the actual fetch:
private static final int TIMEOUT = 60*1000; //msecs
public String fetch(String aURL, String aEncoding /*UTF-8*/) {
String result = "";
long start = System.currentTimeMillis();
Scanner scanner = null;
URLConnection connection = null;
try {
URL url = new URL(aURL);
connection = url.openConnection(); //this doesn't talk to the network yet
connection.setConnectTimeout(TIMEOUT);
connection.setReadTimeout(TIMEOUT);
connection.connect(); //actually connects; this shouldn't be needed here
scanner = new Scanner(connection.getInputStream(), aEncoding);
scanner.useDelimiter(END_OF_INPUT);
result = scanner.next();
}
catch (IOException ex) {
long end = System.currentTimeMillis();
long time = end - start;
fLogger.severe(
"Problem connecting to " + aURL + " Encoding:" + aEncoding +
". Exception: " + ex.getMessage() + " " + ex.toString() + " Cause:" + ex.getCause() +
" Connection Timeout: " + connection.getConnectTimeout() + "msecs. Read timeout:" +
connection.getReadTimeout() + "msecs."
+ " Time taken to fail: " + time + " msecs."
);
}
finally {
if (scanner != null) scanner.close();
}
return result;
}
Example log entry showing a failure:
SEVERE: Problem connecting to http://weather.gc.ca/rss/city/pe-5_e.xml Encoding:UTF-8.
Exception: Connection timed out java.net.ConnectException: Connection timed out
Cause:null
Connection Timeout: 60000msecs.
Read timeout:60000msecs.
Time taken to fail: 15028 msecs.
Note that the time to fail is always 15s + a tiny amount.
Also note that it fails to reach the configured 60s timeout for the connection.
The host-server admins (Environment Canada) state that they don't have any kind of a blacklist for the IP address of misbehaving clients.
Also important: the code had been running for several months without this happening.
Someone suggested that instead I should use curl, a bash script, and cron. I implemented that, and it works fine.
I'm not able to solve this problem using Java.
first time i'm using aws api in java to get the cloud watch statistics for my ec2-instance. i googled about this and i found some code snippet. here it is
AmazonCloudWatchClient cloudWatch = new AmazonCloudWatchClient(
new BasicAWSCredentials(AccessKey, SecretKey));
cloudWatch.setEndpoint("ec2-<my-static-ip>.compute-1.amazonaws.com");
long offsetInMilliseconds = 1000 * 60 * 60 * 24;
Dimension instanceDimension = new Dimension();
instanceDimension.setName("Instanceid");
instanceDimension.setValue(InstanceId);
GetMetricStatisticsRequest request = new GetMetricStatisticsRequest()
.withStartTime(
new Date(new Date().getTime()
- offsetInMilliseconds))
.withNamespace("AWS/EC2")
.withPeriod(60 * 60)
.withDimensions(
new Dimension().withName("InstanceId").withValue(
InstanceId))
.withMetricName("CPUUtilization")
.withStatistics("Average", "Maximum")
.withEndTime(new Date());
GetMetricStatisticsResult getMetricStatisticsResult = cloudWatch
.getMetricStatistics(request);
double avgCPUUtilization = 0;
List dataPoint = getMetricStatisticsResult.getDatapoints();
for (Object aDataPoint : dataPoint) {
Datapoint dp = (Datapoint) aDataPoint;
avgCPUUtilization = dp.getAverage();
System.out.println(InstanceId
+ " instance's average CPU utilization : "
+ dp.getAverage());
}
} catch (AmazonServiceException ase) {
System.out
.println("Caught an AmazonServiceException, which means the request was made "
+ "to Amazon EC2, but was rejected with an error response for some reason.");
System.out.println("Error Message: " + ase.getMessage());
System.out.println("HTTP Status Code: " + ase.getStatusCode());
System.out.println("AWS Error Code: " + ase.getErrorCode());
System.out.println("Error Type: " + ase.getErrorType());
System.out.println("Request ID: " + ase.getRequestId());
}
so, using this code i tried to get statistics, but first time it throws error saying
com.amazonaws.AmazonClientException: Unable to execute HTTP request:Connection to https://ec2-<my-static-ip>.compute-1.amazonaws.com refused
then i thought it was sending https requests. so i enabled ssl on my instance and tried, then i'm getting below exception.
com.amazonaws.AmazonClientException: Unable to execute HTTP request: peer not authenticated
i was using OpenJDK in my instance, so i thought that may causing the problem. then i removed openjdk and installed Oracle JDK 1.7. but still same problem.
My questions are,
1) how can i send only HTTP (instead of HTTPS) requests to get statistics?
2)how to get rid of this problem, so that i can get my results?
But please don't ask me to read any docs, because i messed up by searching in net, blogs,forums, docs... etc. then i end up here. so, please just provide me solution or tell me where i'm going wrong.
Can anybody please help me out this issue.
thank you in Advance.
Got Solution.
1) removed setting end point for AmazonCloudWatchClient.
2) problem with the AWS credentials (Access key ID, Secret key).So, i created another set of credentials and gave CloudWatchFullAccess policy for the user.
Now it is working like Charm... :-)
Thanks.
I'm trying to make use of Windows Search by using java Jacob library. but I'm having troubles to specify the maxRecords option to limit the number of rows get back.
I'm trying to do it by using this line:
Dispatch.put(connection, "MaxRecords", new Variant(10));
After setting up the connection:
connection = new Dispatch("ADODB.Connection");
Dispatch.call(connection, "Open", "Provider=Search.CollatorDSO;Extended Properties='Application=Windows';");
//-------> error in the following line <-------
Dispatch.put(connection, "MaxRecords", new Variant(10));
results = Dispatch.call(connection, "Execute",
"SELECT System.ItemName, System.DateModified " +
"FROM SystemIndex " +
"WHERE Directory='file:C:/my/folder/path' AND Contains('a')").toDispatch();
while (!Dispatch.get(results, "EOF").getBoolean()) {
Dispatch fields = Dispatch.get(results, "Fields").toDispatch();
String filename = Dispatch.get(Dispatch.call(fields, "Item", new Integer(0)).toDispatch(), "Value").toString();
String filedate = Dispatch.get(Dispatch.call(fields, "Item", new Integer(1)).toDispatch(), "Value").toString();
list.put(filename, filedate);
Dispatch.call(results, "MoveNext");
}
What am I doing wrong?
There's no error on compilation but on executing I get this message:
com.jacob.com.ComFailException: A COM exception has been encountered:
At Invoke of: MaxRecords
Description: 80020007 / No named arguments.
...
Internal Server Error (500) - The server encountered an unexpected condition which prevented it from fulfilling the request
And this one when accessing through my restful by URL:
Internal Server Error
The server encountered an unexpected condition which prevented it from fulfilling the request
You can get technical details here.
Please continue your visit at our home page.
Everything works fine without that line.
According to the docs, the Connection object does not have a MaxRecords property. I think you'd want to set MaxRecords on a RecordSet object.
EDIT:
I haven't tried these, but would try along the following lines:
connection = new Dispatch("ADODB.Connection");
Dispatch.call(connection, "Open", "Provider=Search.CollatorDSO;Extended Properties='Application=Windows';");
String sql = "SELECT System.ItemName, System.DateModified " +
"FROM SystemIndex " +
"WHERE Directory='file:C:/my/folder/path' AND Contains('a')"
recordSet = new Dispatch("ADODB.Recordset");
Dispatch.put(recordSet, "MaxRecords", new Variant(10));
Dispatch.call(recordSet, "Open", sql, connection);
while (!Dispatch.get(recordSet, "EOF").getBoolean()) {
...
}
I want to setup Janrain authentication to my Play! project which is hosted on GAE and uses GAE module. But I get the following error while I try to login:
RuntimeException occured : Cannot parse JSON (check logs)
And Play highlighs the following line as error:
JsonElement rpxJson = rpxRequest.get().getJson();
Here is method that I use for token callback:
public static void tokenCallback(String token) {
Properties p = Play.configuration;
// Try the driver
String rpxApi = p.getProperty("login.rpx.apiKey");
WSRequest rpxRequest = WS.url("http://rpxnow.com/api/v2/auth_info");
// get RPX
rpxRequest.setParameter("token", token);
rpxRequest.setParameter("apiKey", rpxApi);
JsonElement rpxJson = rpxRequest.get().getJson();
JsonElement profile = rpxJson.getAsJsonObject().get("profile");
String identifier = profile.getAsJsonObject().getAsJsonPrimitive("identifier").getAsString();
welcome(identifier);
}
And here is the error that I get from terminal:
Internal Server Error (500) for request POST /login/tokencallback
Execution exception (In /app/controllers/Login.java around line 27)
RuntimeException occured : Cannot parse JSON (check logs)
play.exceptions.JavaExecutionException: Cannot parse JSON (check logs)
at play.mvc.ActionInvoker.invoke(ActionInvoker.java:237)
at Invocation.HTTP Request(Play!)
Caused by: java.lang.RuntimeException: Cannot parse JSON (check logs)
at play.libs.WS$HttpResponse.getJson(WS.java:668)
at controllers.Login.tokenCallback(Login.java:27)
at play.mvc.ActionInvoker.invokeWithContinuation(ActionInvoker.java:557)
at play.mvc.ActionInvoker.invoke(ActionInvoker.java:508)
at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:484)
at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:479)
at play.mvc.ActionInvoker.invoke(ActionInvoker.java:161)
... 1 more
Caused by: com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Expected EOF at line 2 column 1
at com.google.gson.JsonParser.parse(JsonParser.java:65)
at com.google.gson.JsonParser.parse(JsonParser.java:45)
at play.libs.WS$HttpResponse.getJson(WS.java:665)
... 7 more
Caused by: com.google.gson.stream.MalformedJsonException: Expected EOF at line 2 column 1
at com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1310)
at com.google.gson.stream.JsonReader.peek(JsonReader.java:390)
at com.google.gson.JsonParser.parse(JsonParser.java:60)
... 9 more
What can I do? Please, help me to solve this problem.
Thanks in advance.
OK, Here is my first suggestion. Try using the HTTPS connection for the URL. I ran into some problems with the HTTP connection. Here is how I do the Janrain connection:
WSRequest rpxRequest = WS.url("https://rpxnow.com/api/v2/auth_info");
// get RPX
rpxRequest.setParameter("token", token);
rpxRequest.setParameter("apiKey", rpxApi);
HttpResponse res = null;
try {
res = rpxRequest.post();
} catch (JavaExecutionException ex) {
Log.error("unknown error ", ex);
Validation.addError("", "Unknown Error: please try again");
Validation.keep();
Secure.login();
} catch (Exception ex) {
Log.error("Most likely SSL error", ex);
Validation.addError("", "SSL Error: please try again");
Validation.keep();
Secure.login();
}
if (res.getStatus() != 200) {
Log.error("status 200 error");
Validation.addError("", "Status 200 error: please try again");
Validation.keep();
Secure.login();
}
JsonElement rpxJson = res.getJson();
JsonElement profile = rpxJson.getAsJsonObject().get("profile");
JsonObject profileJson = profile.getAsJsonObject();
Having called the URL http://rpxnow.com/api/v2/auth_info , it immediately redirects to https://rpxnow.com/api/v2/auth_info (http s ). I suspect you don't get the JSON answer, but a http redirect code in your call to the web service.
Two possibilites:
1) Change the web service call to https://rpxnow.com/api/v2/auth_info , this probably solves your problem, failing that;
2) Change the line JsonElement rpxJson = rpxRequest.get().getJson(); into something like
HttpResponse httpResponse = rpxRequest.get();
Logger.log ( httpResponse.getString() );
if ( httpResponse.success() ) {
JsonElement rpxJson = httpResponse.getJson();
} else {
// fail gracefully
}
and report back on the contents of the answer which gets logged in the second line.