We Keep Coding

REST Service - Multipart Post (JSON and File) as stream on https using Jersey

I have a REST Service on https connection that accepts file upload as multipart (i.e. metadata of the file and file itself)
How can I use Jersey (for websphere) or HttpClient to call REST service and send file as multipart. I want send file as multiple streams of different sizes because we can have file more than 1GB. Moreover, the REST service is using Windows NT authentication for authorization and is on https.
Can anyone give example how I can achieve this? I have used multipart httpClient. Sending it as a stream does not work. Below is my code using httpClient 4.5.2
====================================
InputStream stream = new FileInputStream("test.doc");
MultipartEntityBuilder entityBuilder = MultipartEntityBuilder.create();
entityBuilder.setStrictMode();
InputStreamBody streamBody = new InputStreamBody(stream, docHandler.getFilename());
FormBodyPart filePart = FormBodyPartBuilder.create()
.setName("Binary")
.addField(Constants.RETRIEVAL_NAME_FIELD, "test.doc")
.addField("Content-Type",docHandler.getContentType())
.setBody(streamBody)
.build();
entityBuilder.addPart(filePart);
HttpPost httpPostRequest = new HttpPost();
httpPostRequest.setEntity(entityBuilder.build());
httpClient.execute(httpPostRequest);
====================================
But when I execute this code, I am getting following error
org.apache.http.client.NonRepeatableRequestException: Cannot retry request with a non-repeatable request entity
Any idea why I am getting this error. If I convert stream to byte array and use ByteArrayBody instead, then it works fine but I noticed in Fiddler that three request calls are being made to the server and in every call the entire content of the file is being copied. So, if my file is of 1GB then entire content will be sent to the server three times.
Firstly, how can I achieve sending large file in chunks or multiple streams so that entire file is not sent in one call. Secondly, is there a way to avoid having 3 calls to the server for NTLM authentication?
Any pointers?
Cheers!

The Exception occurs because InputStreamBody is not repeatable (especially for large streams). A general InputStream can be read from only once.
FileBody is repeatable, as a File can be read from multiple times.
One reason for the repeatable reads may be the (NTLM)authorisation(did not check this).
You may avoid this by doing the first two NTML authorisation steps before the actual request and setting/sending the
Authorization: NTLM <base64-encoded type-3-message> header, but that does not solve the problem, because the network may not be reliable enough and you have to retry anyway.
You basically have two options:
use repeatable ContentBody implementations only like FileBody or own repeatable ContentBody instance.
make sure the request does not need to be retried.
Please note the latter is not always possible. As mentioned before request retries due to authentication failures can be avoided, but those due to I/O errors cannot.

Process HTTP MIME REST message in java

I am debugging an application which gets a bad Request, Response from JAX-RS. I have monitored the HTTP message using TCPMon and figured out that The MIME boundary in Header does not match the Boundary in message parts.
What I am trying to do is, Writing a simple java program that changes that boundary of the message part according to the header MIME boundary.
but When I edit the String and sent it back to the JAX-RS service I get errors like Unexpected EndOfLine,Uxexpected EOF.
What I did was , opened a server socket and read the message from incoming connection from my application and edited the MIME boundary of Message parts
String mimeBoundary = message.toString().split("\r\n")[5].split(";")[1].split("=")[1];
message.replace(message.indexOf("MIMEBoundary"), message.indexOf("MIMEBoundary")+61,mimeBoundary);
message.replace(message.indexOf("MIMEBoundary"),message.indexOf("MIMEBoundary")+61,mimeBoundary);
And send the Message(String) to the JAX-RS service.
What am I doing wrong here? someone please help.
Thank You.

Streaming an upload with HttpClient/MultipartEntity

I've got a Tomcat instance right now that takes uploads and does some processing work on the data.
I want to replace this with a new servlet that conforms to a similar API. At first, I want this new servlet to just proxy all of the requests to the old one. They're running on separate JVMs, but on the same host.
I've been trying to use the HttpClient to proxy the upload, but it seems that the client waits for the stream to finish before it proxies the request. For large files, this causes the servlet to crash (I think it's buffering everything in memory).
Here's the code I'm currently using:
HttpPost httpPost = new HttpPost("http://localhost:8081/servlet");
String filePartName = request.getHeader("file_part_name");
_logger.info("Attaching file " + filePartName);
try {
Part filePart = request.getPart(filePartName);
MultipartEntity mpe = new MultipartEntity();
mpe.addPart(
filePartName,
new InputStreamBody(filePart.getInputStream(), filePartName)
);
httpPost.setEntity(mpe);
} catch (ServletException | IOException e) {
_logger.error("Caught exception trying to cross the streams, thanks Ghostbusters.", e);
throw new IllegalStateException("Could not proxy the request", e);
}
HttpResponse postResponse;
try {
postResponse = HTTP_CLIENT.execute(httpPost);
} catch (IOException e) {
_logger.error("Caught exception trying to cross the streams, thanks Ghostbusters.", e);
throw new IllegalStateException("Could not proxy the request", e);
}
I can't seem to figure out how to get HttpClient/HttpPost to stream the data as it comes in, instead of blocking until the first upload completes. Has anyone done something similar before? Is there an easier solution?
Thanks!

The issue lies in the way your request is processed by the Mime/Multiplart framework (the one you use to process your HTTPServletRequest, and access file parts).
The nature of a MIME/Multipart request is simple (at a high level), instead of having a traditionnal key=value content, those requests have much more complex syntax, that allows them to carry arbitrary, unstructured data (files to upload).
It basically looks like (taken from wikipedia):
Content-type: multipart/mixed; boundary="'''frontier'''"
This is a multi-part message in MIME format.
--'''frontier'''
Content-type: text/plain
This is the body of the message.
--'''frontier'''
Content-type: application/octet-stream
Content-Disposition: form-data; name="image1"
Content-transfer-encoding: base64
PGh0bWw+CiAgPGhlYWQ+CiAgPC9oZWFkPgogIDxib2R5PgogICAgPHA+VGhpcyBpcyB0aGUg
Ym9keSBvZiB0aGUgbWVzc2FnZS48L3A+CiAgPC9ib2R5Pgo8L2h0bWw+Cg==
--'''frontier'''--
The important part to note is that parts (that are separated by the boundary '''frontier''' here) have "names" (through the Content Disposition header), then follows the content. One such request can have any number of parts.
Now of course, the most simple, straightforward way to implement the parsing of such a request is to process it till the end, detect the boundary, and create a temporary file (or in-memory cache) to hold each part, identified by name.
Seeing the framework can not know what part you will need first (you may need the second part in your servlet call before the first), it parses the whole stream, and then, gives you back the control.
Therefore your call is blocked at this line
Part filePart = request.getPart(filePartName);
Here, the framework has to wait to parse the whole MIME part, before letting you use the result (even a rethorical, super optimised parser could not both parse lazily the stream, and allow you random access to any parts of the message, you'd have to choose between the two options).
So there's not much you can do...
Except, not use the Multipart parser. I wouldn't recommend this if you're not familiar with MIME (and/or MIME libraries such as Apache James), nor confident that you are in control of your request's structure.
But if you are, then you may bypass the framework processing, and access the raw stream of the request. You'd parse the MIME structure by hand, and stop when you hit the start of the request's body, and start building your HTTP Post at this point, being carefull to actually take care of MIME level technicalities (de-base64 ? de-gzip ?, ...).
Alternatively, if you think your server crashes because of an out of memory, it may very well be possible that your framework is configured to cache contents of the multpart in memory. But if there is a way to configure it to cache to disk, then this is a possible workaround.

Yahoo YQL bad request

I'm trying to use Yahoo Content Analysis using a file containing text as input. So every character and length is possible.
This code works with a simple text String (no special characters, short text) however when I use longer texts or special characters I get a Bad Request error (HTTP 400) sometimes with an error message like "no viable alternative at character '['" or without an error message.
I encode every request and HTTP Post shouldn't have any limit as to the length.
Does the Yahoo service place a limit on the length of the request and/or are there any characters that it can't handle?
Any help to help this work is appreciated!
Here's my code (using commons-httpclient):
String fileInput = FileUtils.readFileToString(f);
StringBuilder builder = new StringBuilder();
builder.append("http://query.yahooapis.com/v1/public/yql?");
System.out.println(fileInput);
builder.append("q=")
.append(URLEncoder.encode("select * from contentanalysis.analyze where text='"+ fileInput +"'" , "UTF-8"))
.append("&format=json");
final String postUrl = builder.toString();
HttpClient client = new HttpClient();
PostMethod method = new PostMethod(postUrl);
// Send POST request
int statusCode = client.executeMethod(method);

I think the problem is that while you are sending the request as an HTTP POST, the YQL query and text are all included in the URL. YQL does not really have a way for you to make HTTP POST requests directly, so I can think of a couple options:
Directly use the Content Analysis web service with an HTTP POST (docs)
Create a custom YQL data table which uses the <execute> tag to run custom JavaScript which could do the POST (example)
Of these options I think the former would be easier.

How do I get Rest Assured to return the text (non-encrypted or streamed) value in my REST response?

I recently moved over to Java and am attempting to write some REST tests against the netflix REST service.
I'm having an issue in that my response using rest assured either wants to send a gzip encoded response or "InputStream", neither of which provide the actual XML text in the content of the response. I discovered the "Accept-Encoding" header yet making that blank doesn't seem to be the solution. With .Net I never had to mess with this and I can't seem to find the proper means of returning a human readable response.
My code:
RestAssured.baseURI = "http://api-public.netflix.com";
RestAssured.port = 80;
Response myResponse = given().header("Accept-Encoding", "").given().auth().oauth(consumerKey, consumerSecret, accessToken, secretToken).param("term", "star wars").get("/catalog/titles/autocomplete");
My response object has a "content" value with nothing but references to buffers, wrapped streams etc. Trying to get a ToString() of the response doesn't work. None of the examples I've seen seem to work in my case.
Any suggestions on what I'm doing wrong here?

This has worked for me:
given().config(RestAssured.config().decoderConfig(DecoderConfig.decoderConfig().noContentDecoders())).get(url)

I guess in Java land everything is returned as an input stream. Using a stream reader grabbed me the data I needed.

Until its version 1.9.0, Rest-assured has been providing by default in the requests the header "Accept-Encoding:gzip,deflate" with no way of changing it.
See
https://code.google.com/p/rest-assured/issues/detail?id=154

It works for me:
String responseJson = get("/languages/").asString();