I am creating a java process to download WebEx recordings using their NBR API's NBRFileOpenService call. It returns a multipart response with the recording file contents attached. I have it somewhat working with the code below. However, when the recording file is large enough, I get OutOfMemoryError exception.
It is quite common for the recordings to be large and if the API only returned the file alone, I could just stream the download, however I'm not so sure how I can safely handle the multipart response. So I'm wondering if there is any way to read the file metadata as well as save the binary content to a file without holding the entire response in memory.
API Response Format:
------=_Part_674_458057647.1593732813745
Content-Type: text/xml; charset=UTF-8
Content-Transfer-Encoding: binary
Content-Id: <AD79B5747EFC01CDDA9A281BA8CDEF0C>
[SOAP RESPONSE]
------=_Part_674_458057647.1593732813745
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
Content-Id: <C498AB4664B57130F869695A1C5B584E>
[FILE METADATA]
------=_Part_674_458057647.1593732813745
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
Content-Id: <003D9EBA1E491CE2E9E5903C996EFD4C>
[BINARY FILE CONTENT]
------=_Part_674_458057647.1593732813745--
My Code:
public void retrieveRecordingFile(String uri, String recordId, String serviceType) throws Exception {
HttpClient httpClient = generateHttpClient();
HttpPost httpPost = new HttpPost(uri);
httpPost.addHeader("Content-Type", ContentType.APPLICATION_XML.getMimeType());
httpPost.addHeader("SOAPAction", "NBRFileOpenService");
String requestXml = buildNBRDownloadFileXml(recordId, serviceType);
HttpEntity httpEntity = new ByteArrayEntity(requestXml.getBytes(Charset.forName("UTF-8")));
httpPost.setEntity(httpEntity);
HttpResponse httpResponse = httpClient.execute(httpPost);
if (httpResponse.getStatusLine().getStatusCode() == 200) {
MimeMultipart mimeMultipart = new MimeMultipart(new ByteArrayDataSource(httpResponse.getEntity().getContent(), "multipart/form-data"));
String filename = null;
File targetFile = null;
for (int i = 0; i < mimeMultipart.getCount(); i++) {
if (i == 1) {
filename = retrieveFileName(mimeMultipart.getBodyPart(i).getInputStream());
} else if (i == 2) {
targetFile = new File(DOWNLOAD_DIR + filename);
FileUtils.copyInputStreamToFile(mimeMultipart.getBodyPart(i).getInputStream(), targetFile);
}
}
}
}
Any help is truly appreciated.
Related
The server I am sending a POST request to requires extra parameters in the Content-Disposition field that are easily added in C# code, but I am struggling to replicate this functionality in Java.
The working C# code:
using (var content = new MultipartFormDataContent()) {
var fileContent = new ByteArrayContent(System.IO.File.ReadAllBytes("filepath"));
fileContent.Headers.ContentDisposition = new ContentDispositionHeaderValue("form-data")
{
Name = "file",
FileName = "file.zip.encrypted",
};
fileContent.Headers.ContentDisposition.Parameters.Add(new NameValueHeaderValue("Type", "CSV"));
fileContent.Headers.ContentDisposition.Parameters.Add(new NameValueHeaderValue("Token", jwt));
content.Add(fileContent);
var requestUri = "url";
var result = client.PostAsync(requestUri, content).Result;
When I print the above request headers the Content-Disposition header looks like:
Content-Disposition: form-data; name=file; filename=file.zip.encrypted; Type=CSV; Token=jwt
Attempting to replicate this POST request in Java Apache Http:
File file = new File("filepath");
String headerValue = "form-data; name=file; filename=\"file.zip.encrypted\"; Type=\"CSV\"; Token=\""+jwtToken+"\"";
try (CloseableHttpClient client2 = HttpClients.createDefault()) {
HttpPost post2 = new HttpPost(url);
HttpEntity entity = MultipartEntityBuilder.create().addPart("file", new FileBody(file)).build();
post2.setHeader("Content-Disposition", headerValue);
post2.setEntity(entity);
try (CloseableHttpResponse response2 = client2.execute(post2)) {
System.out.println(response2.toString());
}
}
However, when I print the Headers in this request, only the name and filename fields are captured, and not the other parameters required in the Content-Disposition header. This is leading to Internal Server Error responses, as the Header does not contain the required parameters. (tried with and without the added quotes around field values)
Content-Disposition: form-data; name="file"; filename="file.zip.encrypted"
Any help getting the C# POST request behavior replicated in Java would be greatly appreciated, thanks!
I using Okhttp3 for download file from server in android application. my link is http://www.webweb.infinityfreeapp.com/lichi/download.php?path=Add.jpg it download file in firefox, chorme smoothly, while in okhttp3 response string shows
<html><body><script>document.cookie="_test=9e105a99e90025d241c180c29fad3231 ; expires=Thu, 31-Dec-37 23:55:55 GMT; path=/" ;document.location.href="http://www.webweb.infinityfreeapp.com/lichi/download.php?path=Add.jpg&i=1";</script></body></html>
but i feel response string has Add.jpg file data. so, what can i change in okhttp3 code or php code that i gather App.jpg data in response string of okhttp3
Php Code
if(isset($_GET['path']))
{
$url = $_GET['path'];
$type = "application/pdf";
$completePath = "http://www.webweb.infinityfreeapp.com/lichi/";
$visibleName = "$url";
$completePath .= $url;
// Force download
header("Content-disposition: attachment; filename=$visibleName");
header("Content-Type: application/force-download");
header("Content-Transfer-Encoding: $type\n");
// header("Content-Length: ".filesize($completePath));
header("Pragma: no-cache");
header("Cache-Control: must-revalidate, post-check=0, pre-check=0, public");
header("Expires: 0");
readfile($completePath);
die();
}
I comment Content-Length because it crash system in uncomment
Java code
OkHttpClient client = new OkHttpClient();
String url = "http://www.webweb.infinityfreeapp.com/lichi/download.php?path=Add.jpg";
Call call = client.newCall(new Request.Builder().url(url).get().build());
Response response = call.execute();
if (response.code() == 200 || response.code() == 201) {
Headers responseHeaders = response.headers();
for (int i = 0; i < responseHeaders.size(); i++)
Log.d(LOG_TAG, responseHeaders.name(i) + ": " + responseHeaders.value(i));
String str = response.body().string();
}
Here str contain above html file information instead Add.jpg file data. so please give answer
good question
Autually if we send a get request to
http://www.webweb.infinityfreeapp.com/lichi/download.php?path=Add.jpg
we get the right resutl just like
<html><body><script>document.cookie="_test=9e105a99e90025d241c180c29fad3231 ; expires=Thu, 31-D...";</script></body></html> .
we can get a file in browser, because browser can parse html ，
when browser get the string result which is a html page, it create another request with a new Header （Cookie=_test=9e105a99e90025d241c180c29fad3231), and with the Cookie, we get an image file from server.
Thanks for quick and good solution.
i just add header as:-
.header("Cookie", "_test=9e105a99e90025d241c180c29fad3231")
and send again with above code, actual result come
I am facing an issue with Windows Computer Vision API. If I send a request with contentType = application/json and image URL in JSON request body things work fine but on sending a binary image(base 64 encoded) with contentType = application/octet-stream it gives me ImageFormatInvalid in the response.
Any help is much appreciated. Thank you!
Code:
final String binaryData = "data:image/jpeg;base64, /9............
ByteArrayEntity requestEntity = new ByteArrayEntity(binaryData.getBytes(),
ContentType.APPLICATION_OCTET_STREAM);
request.setEntity(requestEntity);
HttpResponse response = httpClient.execute(request);
HttpEntity entity = response.getEntity();
Response: InvalidImageFormat
It seems I was generating the binary incorrectly.
Worked when made changes as below:
byte[] decodedString = Base64.decodeBase64(binaryData);
ByteArrayEntity requestEntity = new
ByteArrayEntity(decodedString,ContentType.APPLICATION_OCTET_STREAM);
request.setEntity(requestEntity);
How to write java code for egnyte chunked upload and send to rest service of egnyte api.
https://developers.egnyte.com/docs/read/File_System_Management_API_Documentation#Chunked-Upload
long size = f.getTotalSpace();
int sizeOfFiles = 1024 * 1024;// 1MB
byte[] buffer = new byte[sizeOfFiles];
ResponseEntity<String> responseEntity = null;
String fileName = f.getName();
String url = DOWNLOAD_OR_UPLOAD + "-chunked" + egnyteSourcePath + f.getName();
HttpHeaders headers = buildEgnyteEntity();
HttpEntity entity = new HttpEntity<>(headers);
//try-with-resources to ensure closing stream
try (FileInputStream fis = new FileInputStream(f);
BufferedInputStream bis = new BufferedInputStream(fis)) {
int bytesAmount = 0;
while ((bytesAmount = bis.read(buffer)) > 0) {
//write each chunk of data into separate file with different number in name
String filePartName = String.format("%s.%03d", fileName, partCounter++);
File newFile = new File(f.getParent(), filePartName);
responseEntity = restTemplate.exchange(url, HttpMethod.POST, entity, String.class);
}
}
return responseEntity;
I think there's a couple of things missing in your code.
First thing is that you don't specify required headers. You should provide X-Egnyte-Chunk-Num with int value with number of your chunk, starting from 1. In X-Egnyte-Chunk-Sha512-Checksum header you should provide SHA512 checksum.
Second thing is that first request will give you an UploadId in response header in X-Egnyte-Upload-Id. You need to specify that as a header in your second and following requests.
Third thing is that I don't see you use your bytesAmount in the request. I'm not sure you're providing the data.
I'm not a Java guy, more of a C# one, but I've written a post how to upload and download big files with Egnyte API on my blog: http://www.michalbialecki.com/2018/02/11/sending-receiving-big-files-using-egnyte-api-nuget-package/. This can give you an idea how sending loop can be structured.
Hope that helps.
With a client like this
StringBody body = new StringBody("form_username", Charset.forName("UTF-8"));
multipart.addPart("username", body);
ByteArrayBody bBody = new ByteArrayBody(bs, "form_command.dat");
multipart.addPart("data", bBody);
httppost.setEntity(multipart);
How are the values supposed to be retrieved in the netty server. I already have a HttpRequestDecoder added to the pipeline. And the messageReceived handled thus
HttpRequest request = (HttpRequest) e.getMessage();
this.mRequest = request;
if (is100ContinueExpected(request)) {
send100Continue(e);
}
ChannelBuffer content = request.getContent();
if (content.readable()) {
System.out.println("Content()\n" + content.toString(CharsetUtil.UTF_8) + "\r\n");
}
Print outputs .
Content()
--Xdq2t6unVsUp191MKhpR6BXz5P7Eoo
Content-Disposition: form-data; name="username"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
form_username
--Xdq2t6unVsUp191MKhpR6BXz5P7Eoo
Content-Disposition: form-data; name="data"; filename="form_command.dat"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
--Xdq2t6unVsUp191MKhpR6BXz5P7Eoo--
End of contents
You need to use the new HttpPostRequestDecoder. This is only available in the upcoming Netty 3.5 (3 branch) and Netty 4 (master branch).
Here an example usage.
If you need to use it now, you can just copy the files mentioned in this pull request into your project namespace and use it.
Hope this helps.