jclouds IOExpection: Error writing request body to server - java

We are using jclouds with Rackspace and when uploading lots of files via cloudfile api (multi threaded)
Once in while we are getting an exception on objectApi.put line (see example code at bottom)
Exception
16-Jul-2015 11:58:00.811 SEVERE [threadsPool-1] org.jclouds.logging.jdk.JDKLogger.logError error after writing 8192/streaming bytes to https://*****/****.jpg
java.io.IOException: Error writing request body to server
at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3478)
at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3461)
at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:53)
at com.google.common.io.ByteStreams.copy(ByteStreams.java:74)
at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:297)
at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:160)
at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:64)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:91)
at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
at com.sun.proxy.$Proxy176.put(Unknown Source)
at
Similar issue with S3
can be found here
Example Code
ObjectApi objectApi = cloudFiles.getObjectApi(REGION, container);
ByteSource byteSource = Files.asByteSource(file);
Payload payload = Payloads.newByteSourcePayload(byteSource);
objectApi.put(hashedName, payload);
The question:
Any one has experience some behavior like that? maybe someone has workaround for that kind of issue?
Thanks
Alon

Networks are unreliable, so expect some exceptions when using cloud services, especially when dealing with many files. Specifically for jclouds uploads, we have some example code here:
https://github.com/jclouds/jclouds-examples/tree/master/blobstore-uploader
Edit: I have also added a JIRA issue to make sure we add a test specifically for this situation in swift:
https://issues.apache.org/jira/browse/JCLOUDS-965

Related

Java http post: Difference in reading bytes through curl and in java

I am stuck in a strange issue, I am posting some image data to a server, now I created the requested using curl and then traced back it.
Next was to create similar request in java.
The code posts raw binary image data to server, but when I compare the binary data in java to that posted by curl, there is a minute difference due to which I am getting 400 response code from server.
The difference I think is in few dots.
Below is the request generate by curl (linux).
Generate by curl
Now here is the request generate by Java, when I read bytes.
Click here
Java code looks something like this:
PrintWriter out = new PrintWriter(os);
out.println("POST /1izb0rx1 HTTP/1.1");
out.println("User-Agent: curl/7.35.0");
out.println("Host: requestb.in");
out.println("Accept: */*");
out.println("Content-Disposition:inline; filename=test.png");
out.println("Authorization: Basic YW5kcm9pZDpUZXN0dGVzdDExISE=");
out.println("Content-Length: "+"24143");
out.println("Content-Type: application/x-www-form-urlencoded");
out.println();
out.println("."+imgBytes);
Any idea what can be causing this issue ?
Thanks
So,
I got it working, the problem was that certain classes on Android are broken and not behaving as the way they behave on core Java.
The same code that was working on Java, wasn't working here, reason being, a change in header occurring here (On Android).
This issue is very well mentioned here also:
https://github.com/WP-API/WP-API/issues/1862
Thus I was facing a similar issue, and adding external updated jars were conflicting with the ones on Android.
Finally I used a small HTTP Request library: https://github.com/kevinsawicki/http-request
The code is below:
HttpRequest request = HttpRequest.post(url);
request.authorization("Basic "+ah);
request.part("file", fName+".png", "image/png", new File(file));
request.part("title", "test");
if(request.code()==201) {
StringWriter sw = new StringWriter();
request.receive(sw);
onMedia(Media.parse(new JsonParser().parse(sw.toString()).getAsJsonObject()));
}
Thanks
Do not use PrintWriter to send raw bytes.

Play 2 GET method maximum length

I am using Play framework (Java) to create REST apis for my application.
I am facing problem with length of REST API with GET.
I need below web method to be used:
public static Result getregistereduser(String userlist)
{
//userlist is string of user numbers seperated by "$" ,
.....
....
}
in routes I have :
GET /getregistereduser controllers.Application.getregistereduser(userlist)
Problem I am facing is when userlist is large it only takes partial string.
I'm not sure what version of Play you're using, but Play shouldn't truncate long GET requests. You should get an error response if the GET request is too large for it to handle.
Up until recently that error would be a 500 Internal Server Error caused by an org.jboss.netty.handler.codec.frame.TooLongFrameException. With Play 2.3.0 (not yet released) you'll get a 414 Request URI too long response instead.
Play inherits its HTTP header parsing from Netty. The first line of the header (GET /path HTTP/1.1) can be up to 4096 bytes by default. This can be configured by with the http.netty.maxInitialLineLength system property (see this mailing list thread).
I agree that you're most likely seeing problem with your HTTP client. I'd recommend using a debugging proxy like Charles (or a free alternative) to look at exactly what HTTP requests your client is sending.

Adding soap header authentication to wsdl2java generated code

I'm in the process of creating a Java web services client from a wsdl. I used Eclipses's Dynamic Web Project and new Web Services Client to generate the code with wsdl2java with Apache Axis 1.4. I need to add SOAP authentication to this code in order for it to work with the service. I couldn't find a place to do that in the generated code. After copious research I found this, which I've used as the backbone for my code so far.
Adding ws-security to wsdl2java generated classes
Before I was getting a "Error occurred while processing security for the message" or something along those lines. Now I am getting
"Exception: Did not understand "MustUnderstand" header(s):{http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd}Security Message: null"
I've tried many things to get past this exception. This is the code I've arrived at now.
javax.xml.namespace.QName headerName = new javax.xml.namespace.QName(
"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd", "Security");
org.apache.axis.message.SOAPHeaderElement header = new org.apache.axis.message.SOAPHeaderElement(headerName);
header.setActor(null);
header.setMustUnderstand(true);
// Add the UsernameToken element to the WS-Security header
javax.xml.soap.SOAPElement utElem = header.addChildElement("UsernameToken");
utElem.setAttribute("Id", "uuid-3453f017-d595-4a5b-bc16-da53e5831cd1-1");
javax.xml.soap.SOAPElement userNameElem = utElem.addChildElement("Username");
userNameElem.setValue("username");
javax.xml.soap.SOAPElement passwordElem = utElem.addChildElement("Password");
passwordElem.setAttribute("Type", "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText");
passwordElem.setValue("password");
header.setProcessed(true);
// Finally, attach the header to the binding.
setHeader(header)
This code is located in my Binding_ServiceStub class (in its' createCall method).
We have created clients in both C# and VB with this wsdl, and there it's as easy as just changing the ClientCredentials variable which is an extension of the proxy class generated. I was hoping for something similar here.
Here's the security policy from the wsdl code as well.
<wsp:Policy><sp:UsernameToken sp:IncludeToken="http://schemas.xmlsoap.org/ws/2005/07/securitypolicy/IncludeToken/AlwaysToRecipient"><wsp:Policy><sp:WssUsernameToken10/></wsp:Policy></sp:UsernameToken></wsp:Policy>
Does anyone know what else I can do here? Why this exception is happening? I've tried many different combinations of prefixes and setProcesses and setMustUnderstand values all in vain (and based on my research of this exception).
And if anyone knows a way in which to add Soap header authentication to wsdl2java code I would take that too. Just need this to work and you would think something like this would be a little more straightforward or at least have more examples out there.
Update-
Confirmed that the same header passed using SOAPUI works fine. Must be something with the framework? I created a custom handler to process the SOAP Message but that didn't help. Is Axis 1.4 and JAX-RPC the problem? (I know they're outdated but still...)
Cool. I decided to just use Apache CXF as my framework and using this it's as easy as adding
javax.xml.ws.BindingProvider bp = (javax.xml.ws.BindingProvider) port;
bp.getRequestContext().put("ws-security.username", username);
bp.getRequestContext().put("ws-security.password", password);
Wow that's much better. Don't use Axis 1.4 lesson learned.

Error while using part="...,processingDetails" in a youtube.videos.insert() request

We're developing in java 7 and YouTube Data API v3.
When we are calling the upload service with "processingDetails" part, we are getting a 503 Service Unavailable Error. What are we doing wrong? If we remove it, we can upload the video just fine. But we want/need to know about processingDetails.
Here is the code:
YouTube.Videos.Insert videoInsert = this.youtube.videos().insert("snippet,statistics,status,processingDetails", videoObjectDefiningMetadata, mediaContent);
everything else it's the same as the java client provided in google API.
Here is the request:
https://www.googleapis.com/upload/youtube/v3/videos?part=snippet,statistics,status,processingDetails
And here is the exception stack trace:
2013-06-14 17:50:40,375 ERROR [] [main] (...)despegar.client.impl.YouTubeMediaClient (YouTubeMediaClient.java:264) : Error
com.google.api.client.googleapis.json.GoogleJsonResponseException: 503 Service Unavailable
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:111)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:38)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:423)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:343)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:460)
at com.despegar.client.impl.YouTubeMediaClient.uploadVideoWithFailureRecovery(YouTubeMediaClient.java:285)
at com.despegar.client.impl.YouTubeMediaClient.uploadAttempt(YouTubeMediaClient.java:260)
Thanks!
Two things:
The part parameter values you include in your youtube.videos.insert() request should correspond to the parts you're specifying in the upload metadata. Normally this would be snippet (since that's the part containing the required title and description) and sometimes status (if you're also setting the public/private status of the video). You can't set statistics or processingDetails during an upload, so you shouldn't include them in the list of parts. You can still request the statistics or processingDetails parts after the upload completes, via a youtube.videos.list() call.
The API should return a meaningful error message explaining this to you instead of returning a generic 503. I'll file a bug with the engineering team asking them to take care of that.

Enabling versioning in Amazon S3 bucket

I am trying to enable versioning for Amazon S3 bucket using Java. But, I am not able to do. I get an exception
"Exception Status Code: 400, AWS Request ID: DC53C8220CEC7D4C, AWS
Error Code: MalformedXML, AWS Error Message: The XML you provided was not well-formed or did not validate against our published schema, S3 Extended Request ID: qAdibjSkoFltjoYTFZSdTOnh8JXwZrxkjgrTcgaXqZYGIgVdbRxr8VXzwkO4ilaG"
Can somebody please point out the error in the code. I am attaching the portion of the code responsible for enabling bucket versioning.
public void enableVersioning(String bucketName) {
SetBucketVersioningConfigurationRequest request =
new SetBucketVersioningConfigurationRequest(bucketName,
new BucketVersioningConfiguration("ENABLED"));
AmazonS3 s3 = new AmazonS3Client(credentials); // I have the credentials
s3.setBucketVersioningConfiguration(request);
}
Thanks in advance.
They should be the same, but I would use BucketVersioningConfiguration.ENABLED instead of the String literal if I were you. Do an import static if you think it clutters up the code too much. (Who knows, it might even mysteriously fix your problem)
Just did pretty much exactly this myself and it worked, this was the only difference I could find.
OhHiThere is correct - you should be using the LITERAL:
SetBucketVersioningConfigurationRequest request =
new SetBucketVersioningConfigurationRequest(bucketName,
new BucketVersioningConfiguration(BucketVersioningConfiguration.ENABLED));
The error is almost certainly because "ENABLED" is not the same as BucketVersioningConfiguration.ENABLED (which is defined as "Enabled").
I have also seen this error:
The XML you provided was not well-formed
message trying to turn versioning off when it has been on (only a suspend is allowed in that case).

Categories