I have an app to upload files to a cloud service using HttpURLConnection:
HttpURLConnection httpCon = (HttpURLConnection) url.openConnection();
httpCon.setDoOutput(true);
httpCon.setRequestMethod("PUT");
OutputStreamWriter out = new OutputStreamWriter(
httpCon.getOutputStream());
FileInputStream fis = new FileInputStream("C:/Users/me/Pictures/_MG_9324.JPG");
IOUtils.copy(fis,httpCon.getOutputStream());
out.close();
InputStream is= httpCon.getInputStream();
IOUtils.copy(is,System.out);
The program didn't block at the first IOUtils.copy, instead, it blocks at the second IOUtils.copy. My understanding is that the program blocks since the upload didn't complete. So my questions during the upload, how do I monitor how much data is uploaded from client side?
The output is written to a ByteArrayOutputStream until you call getInputStream() or getResponseCode(), so that the Content-length header can be set correctly. So the second copy operation includes the upload. To avoid that, use chunked transfer mode.
Don't wrap your OutputStream in an OutputStreamWriter, instead wrap your OutputStream in the below class. You can pass in a callback function that'll listen for the bytes being written.
public class CallbackEnabledOutputStream extends OutputStream {
private OutputStreamListener outputStreamListener;
private OutputStream realOutputStream;
private int byteWriteThreshold;
private int bytesWritten = 0;
public CallbackEnabledOutputStream(OutputStreamListener outputStreamListener, OutputStream realOutputStream, int byteWriteThreshold) {
super();
this.outputStreamListener = outputStreamListener;
this.realOutputStream = realOutputStream;
this.byteWriteThreshold = byteWriteThreshold;
}
#Override
public void write(int b) throws IOException {
realOutputStream.write(b);
bytesWritten++;
if(bytesWritten > byteWriteThreshold) {
outputStreamListener.bytesWritten(bytesWritten);
bytesWritten = 0;
}
}
#Override
public void flush() throws IOException {
realOutputStream.flush();
}
#Override
public void close() throws IOException {
realOutputStream.flush();
}
}
OutputStreamListener
public class OutputStreamListener {
private int byteCount = 0;
public void bytesWritten(int byteCount) {
this.byteCount += byteCount;
}
public int getByteCount() {
return byteCount;
}
}
Related
I want to compress multiples files into a zip files, I'm dealing with big files, and then download them into the client, for the moment I'm using this:
#RequestMapping(value = "/download", method = RequestMethod.GET, produces = "application/zip")
public ResponseEntity <StreamingResponseBody> getFile() throws Exception {
File zippedFile = new File("test.zip");
FileOutputStream fos = new FileOutputStream(zippedFile);
ZipOutputStream zos = new ZipOutputStream(fos);
InputStream[] streams = getStreamsFromAzure();
for (InputStream stream: streams) {
addToZipFile(zos, stream);
}
final InputStream fecFile = new FileInputStream(zippedFile);
Long fileLength = zippedFile.length();
StreamingResponseBody stream = outputStream - >
readAndWrite(fecFile, outputStream);
return ResponseEntity.ok()
.header(HttpHeaders.ACCESS_CONTROL_EXPOSE_HEADERS, HttpHeaders.CONTENT_DISPOSITION)
.header(HttpHeaders.CONTENT_DISPOSITION, "attachment;filename=" + "download.zip")
.contentLength(fileLength)
.contentType(MediaType.parseMediaType("application/zip"))
.body(stream);
}
private void addToZipFile(ZipOutputStream zos, InputStream fis) throws IOException {
ZipEntry zipEntry = new ZipEntry(generateFileName());
zos.putNextEntry(zipEntry);
byte[] bytes = new byte[1024];
int length;
while ((length = fis.read(bytes)) >= 0) {
zos.write(bytes, 0, length);
}
zos.closeEntry();
fis.close();
}
This take a lot of time before all files are zipped and then the downloading start, and for large files this kan take a lot of time, this is the line responsible for the delay:
while ((length = fis.read(bytes)) >= 0) {
zos.write(bytes, 0, length);
}
So is there a way to download files immediately while their being zipped ?
Try this instead. Rather than using the ZipOutputStream to wrap a FileOutputStream, writing your zip to a file, then copying it to the client output stream, instead just use the ZipOutputStream to wrap the client output stream so that when you add zip entries and data it goes directly to the client. If you want to also store it to a file on the server then you can make your ZipOutputStream write to a split output stream, to write both locations at once.
#RequestMapping(value = "/download", method = RequestMethod.GET, produces = "application/zip")
public ResponseEntity<StreamingResponseBody> getFile() throws Exception {
InputStream[] streamsToZip = getStreamsFromAzure();
// You could cache already created zip files, maybe something like this:
// String[] pathsOfResourcesToZip = getPathsFromAzure();
// String zipId = getZipId(pathsOfResourcesToZip);
// if(isZipExist(zipId))
// // return that zip file
// else do the following
StreamingResponseBody streamResponse = clientOut -> {
FileOutputStream zipFileOut = new FileOutputStream("test.zip");
ZipOutputStream zos = new ZipOutputStream(new SplitOutputStream(clientOut, zipFileOut));
for (InputStream in : streamsToZip) {
addToZipFile(zos, in);
}
};
return ResponseEntity.ok()
.header(HttpHeaders.ACCESS_CONTROL_EXPOSE_HEADERS, HttpHeaders.CONTENT_DISPOSITION)
.header(HttpHeaders.CONTENT_DISPOSITION, "attachment;filename=" + "download.zip")
.contentType(MediaType.parseMediaType("application/zip")).body(streamResponse);
}
private void addToZipFile(ZipOutputStream zos, InputStream fis) throws IOException {
ZipEntry zipEntry = new ZipEntry(generateFileName());
zos.putNextEntry(zipEntry);
byte[] bytes = new byte[1024];
int length;
while ((length = fis.read(bytes)) >= 0) {
zos.write(bytes, 0, length);
}
zos.closeEntry();
fis.close();
}
public static class SplitOutputStream extends OutputStream {
private final OutputStream out1;
private final OutputStream out2;
public SplitOutputStream(OutputStream out1, OutputStream out2) {
this.out1 = out1;
this.out2 = out2;
}
#Override public void write(int b) throws IOException {
out1.write(b);
out2.write(b);
}
#Override public void write(byte b[]) throws IOException {
out1.write(b);
out2.write(b);
}
#Override public void write(byte b[], int off, int len) throws IOException {
out1.write(b, off, len);
out2.write(b, off, len);
}
#Override public void flush() throws IOException {
out1.flush();
out2.flush();
}
/** Closes all the streams. If there was an IOException this throws the first one. */
#Override public void close() throws IOException {
IOException ioException = null;
for (OutputStream o : new OutputStream[] {
out1,
out2 }) {
try {
o.close();
} catch (IOException e) {
if (ioException == null) {
ioException = e;
}
}
}
if (ioException != null) {
throw ioException;
}
}
}
For the first request for a set of resources to be zipped you wont know the size that the resulting zip file will be so you can't send the length along with the response since you are streaming the file as it is zipped.
But if you expect there to be repeated requests for the same set of resources to be zipped, then you can cache your zip files and simply return them on any subsequent requests; You will also know the length of the cached zip file so you can send that in the response as well.
If you want to do this then you will have to be able to consistently create the same identifier for each combination of the resources to be zipped, so that you can check if those resources were already zipped and return the cached file if they were. You might be able to could sort the ids (maybe full paths) of the resources that will be zipped and concatenate them to create an id for the zip file.
I am trying to write JUnit using MockitoJUnitRunner.
I am passing file id to my function which is downloading file from cloud and returning zip file as download.
here is my code
public void getLogFile(HttpServletResponse response, String id) throws IOException {
response.setContentType("Content-type: application/zip");
response.setHeader("Content-Disposition", "attachment; filename=LogFiles.zip");
ServletOutputStream out = response.getOutputStream();
ZipOutputStream zos = new ZipOutputStream(new BufferedOutputStream(out));
zos.putNextEntry(new ZipEntry(id));
InputStream inputStream = someDao.getFile(id);
BufferedInputStream fif = new BufferedInputStream(inputStream);
int data = 0;
while ((data = fif.read()) != -1) {
zos.write(data);
}
fif.close();
zos.closeEntry();
zos.close();
}
And my JUnit function is
#Mock
private MockHttpServletResponse mockHttpServletResponse;
anyInputStream = new ByteArrayInputStream("test data".getBytes());
#Test
public void shouldDownloadFile() throws IOException {
ServletOutputStream outputStream = mock(ServletOutputStream.class);
when(mockHttpServletResponse.getOutputStream()).thenReturn(outputStream);
=> when(someDao.download(anyString())).thenReturn(anyInputStream);
controller.getLogFile(mockHttpServletResponse, id);
verify(mockHttpServletResponse).setContentType("Content-type: application/zip");
verify(mockHttpServletResponse).setHeader("Content-Disposition","attachment; filename=LogFiles.zip");
verify(atmosdao).download(atmosFilePath);
}
This unit test is passing but I want to verify what is written on outputStream, how can I do it ? as I am writing "test data" to mocked outputStream like
anyInputStream = new ByteArrayInputStream("test data".getBytes());
when(someDao.download(anyString())).thenReturn(anyInputStream);
mockHttpServletResponse.getContentAsString() is giving me null !
Is it possible to assert MockHttpServletResponse which is written using zipoutputStream ? if yes then how can i do it ?
Thanks.
Instead of mocking your OutputStream, you could create a custom one:
public class CustomOutputStream extends ServletOutputStream {
private ByteArrayOutputStream out = new ByteArrayOutputStream();
private String content;
#Override
public void write(int b) throws IOException {
out.write(b);
}
#Override
public void close() throws IOException {
content = new String(out.toByteArray());
out.close();
super.close();
}
public String getContentAsString() {
return this.content;
}
}
This class will store all the bytes written to it and keep them in the content field.
Then you replace this:
ServletOutputStream outputStream = mock(ServletOutputStream.class);
by this:
CustomOutputStream outputStream = new CustomOutputStream();
When your servlet calls getOutputStream(), it will use the custom one, and in the end getContentAsString() will return you the output that was written to your servlet.
Note: the output is zipped, so the String will contain strange characters. If you want the original string, you'll have to unzip it (and in this case I'd use the byte array returned by out.toByteArray() instead of the String, because when you create a String this way you can have encoding problems when calling string.getBytes())
I got what I was looking for...
This is what I did to assert data written to zipoutputStream using powerMockito.
#Test
public void ShouldAttemptToWriteDownloadedFileToZipOutputStream() throws Exception {
InputStream anyInputStream = new ByteArrayInputStream("test data".getBytes());
ServletOutputStream outputStream = mock(ServletOutputStream.class);
BufferedOutputStream bufferedOutputStream = Mockito.mock(BufferedOutputStream.class);
PowerMockito.whenNew(BufferedOutputStream.class).withArguments(outputStream).thenReturn(bufferedOutputStream);
ZipOutputStream zipOutputStream = Mockito.mock(ZipOutputStream.class);
PowerMockito.whenNew(ZipOutputStream.class).withArguments(bufferedOutputStream).thenReturn(zipOutputStream);
BufferedInputStream bufferedInputStream = new BufferedInputStream(anyInputStream);
PowerMockito.whenNew(BufferedInputStream.class).withArguments(anyInputStream).thenReturn(bufferedInputStream);
subjectUnderTest.getLogFile(mockHttpServletResponse, "12345");
int data = 0;
while ((data = bufferedInputStream.read()) != -1) {
verify(zipOutputStream).write(data);
}
}
Thanks Hugo for your help !
I've written a basic REST Server using Jersey2 on top of Jetty, to test out HTTP Chunked Transfer-Encoding, and gzip Content-Encoding. However I've found that the recommended method of implementing a WriterInceptor to apply a GZIPOutputStream for gzip encoding results in the server blocking instead of sending through a gzip'd chunk.
I believe it is the GZIPOutputStream waiting for it's own buffer to fill up, so I tried overriding write() method in the WriterInterceptor to force a flush() after every write (as my server always writes one chunk at a time) but that made no difference. Is there a way of forcing the flush to occur whenever a write occurs?
App.java
public class App
{
public static int lineCount=0;
public static void main( String[] args ) {
System.out.println( "Hello World!" );
ResourceConfig config = new ResourceConfig();
config.packages("com.example.mockAPIjava");
ServletHolder servlet = new ServletHolder(new ServletContainer(config));
EncodingFilter.enableFor(config, GZipEncoder.class);
Server server = new Server(2222);
ServletContextHandler context = new ServletContextHandler(server, "/*");
context.addServlet(servlet, "/*");
try {
server.start();
server.join();
} catch (Exception e) {
throw new RuntimeException(e);
} finally {
server.destroy();
}
}
}
GZIPWriterInterceptor.java
#Provider
#Compress
public class GZIPWriterInterceptor implements WriterInterceptor {
#Override
public void aroundWriteTo(WriterInterceptorContext context)
throws IOException, WebApplicationException {
MultivaluedMap<String,Object> headers = context.getHeaders();
headers.add("Content-Encoding", "gzip");
final OutputStream outputStream = context.getOutputStream();
context.setOutputStream(new GZIPOutputStream(outputStream) {
#Override
public void write(final int b) throws IOException {
out.write(b);
out.flush();
}
#Override
public void write(final byte[] b) throws IOException {
out.write(b);
out.flush();
}
#Override
public void write(final byte[] b, final int off, final int len) throws IOException {
out.write(b, off, len);
out.flush();
}
});
context.proceed();
}
}
Resource.java
#Path("stream")
public class Resource {
#GET
#Path("test")
#Compress
#Produces(MediaType.APPLICATION_JSON)
public ChunkedOutput<String> helloWorld(#Context HttpHeaders header, #Context HttpServletResponse response) {
final ChunkedOutput<String> output = new ChunkedOutput<String>(String.class, "\r\n");
new Thread() {
public void run() {
BufferedReader br = null;
try {
String chunk;
// open file for reading
File file = new File("/tmp/stream.txt");
FileReader fr = new FileReader(file);
br = new BufferedReader(fr);
while ((chunk = getNextString(br)) != null) {
// write a chunk every second
output.write(chunk);
try {
Thread.sleep(1 * 1000);
} catch(InterruptedException ex) {
Thread.currentThread().interrupt();
}
}
} catch (Exception e) {
// IOException thrown when writing the
// chunks of response: should be handled
e.printStackTrace();
} finally {
try {
output.close();
// simplified: IOException thrown from
// this close() should be handled here...
if (br!=null) { br.close(); }
} catch (IOException e1){
e1.printStackTrace();
}
}
}
}.start();
// the output will be probably returned even before
// a first chunk is written by the new thread
return output;
}
private String getNextString(BufferedReader br) throws IOException, ParseException {
App.lineCount++;
return br.readLine();;
}
}
Compress.java
//#Compress annotation is the name binding annotation for the GZIPWriterInterceptor
#NameBinding
#Retention(RetentionPolicy.RUNTIME)
public #interface Compress {}
By overriding the write methods of GZIPOutputStream, you have just stopped it from gzipping!
public void write(final int b) throws IOException {
out.write(b);
out.flush();
}
Because you've overridden it to not invoke super.write (which you should have done), but rather out.write, you're sending directly to the context OutputStream, uncompressed.
Presumably, the receiving side is expecting gzip data and not receiving it, which may lead to all kinds of wrong behaviour.
Change the code to invoke super.write and flush:
public void write(final int b) throws IOException {
super.write(b);
flush();
}
etc.
I have a problem sending images to my server using httpurlconnection. i have read Android documentation and another HttpUrlconnection implementation but i don't know where am doing it the wrong way since am getting a HTTP_BAD_REQUEST error code(400). what important thing am i missing in my code below?
My response code always return 400 but my link is ok since am able to achieve this using httpclient
link = "my link.com";
try {
URL url = new URL(link);
connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("POST");
connection.setDoOutput(true);
connection.setUseCaches(false);
connection.setChunkedStreamingMode(0);
connection.setRequestProperty("Content-Type", "image/jpeg");
BufferedOutputStream outputStream = new BufferedOutputStream(connection.getOutputStream());
FileInputStream stream = new FileInputStream(file);
byte[] buffer = new byte[1024];
int bytesRead;
while ((bytesRead =stream.read(buffer ,0 ,buffer.length)) != -1){
outputStream.write(buffer);
outputStream.flush();
}
outputStream.flush();
responseCode = connection.getResponseCode();
I think the issue is how the image is being added to the output stream. All of the connection configuration steps look good.
I tried this method recently and it worked well:
https://vikaskanani.wordpress.com/2011/01/11/android-upload-image-or-file-using-http-post-multi-part/
It is also good practice to wrap in an AsyncTask. I have noticed that MultipartEntity is now deprecated, but you can replace with MultipartEntityBuilder.
update
To listen to file upload events and update your progressbar, You can override the writeTo method of any HttpEntity implementation and count bytes as they get written to the output stream.
DefaultHttpClient httpclient = new DefaultHttpClient();
try {
HttpPost httppost = new HttpPost("http://www.google.com/sorry");
MultipartEntity outentity = new MultipartEntity() {
#Override
public void writeTo(final OutputStream outstream) throws IOException {
super.writeTo(new CoutingOutputStream(outstream));
}
};
outentity.addPart("stuff", new StringBody("Stuff"));
httppost.setEntity(outentity);
HttpResponse rsp = httpclient.execute(httppost);
HttpEntity inentity = rsp.getEntity();
EntityUtils.consume(inentity);
} finally {
httpclient.getConnectionManager().shutdown();
}
static class CoutingOutputStream extends FilterOutputStream {
CoutingOutputStream(final OutputStream out) {
super(out);
}
#Override
public void write(int b) throws IOException {
out.write(b);
System.out.println("Written 1 byte");
}
#Override
public void write(byte[] b) throws IOException {
out.write(b);
System.out.println("Written " + b.length + " bytes");
}
#Override
public void write(byte[] b, int off, int len) throws IOException {
out.write(b, off, len);
System.out.println("Written " + len + " bytes");
}
}
update
If you would like to update a progress bar based on the http progress this link provides a great example
Link
I'm trying to post an InputStream to a RESTful service. For normal files, this is fine.
In another place I'm trying to write a number of files to a piped zip stream on the fly. To do this I have a class that extends InputStream and when read() is called it writes the next file to the pipe. When the first file has been written I call ZipOutputStream.closeEntry() but it hangs. Why??
When I test this class in a unit test it works fine. When I try to post this object it hangs. The debugger is telling me its waiting for lock on SocketWrapper.
Note, I also tried setting media type to application/octet-stream. Also, the RESTful method is never called.
The stream class...
static class MultiStreamZipInputStream extends InputStream {
private final Iterator<InputStream> streams;
private final byte[] buffer = new byte[4096];
private InputStream inputStream;
private ZipOutputStream zipOutputStream;
private InputStream currentStream;
private int counter = 0;
public MultiStreamZipInputStream(List<InputStream> streamList) {
streams = streamList.iterator();
currentStream = streams.next();
try {
PipedOutputStream out = new PipedOutputStream();
inputStream = new PipedInputStream(out);
zipOutputStream = new ZipOutputStream(out);
ZipEntry entry = new ZipEntry(String.valueOf(counter++)); // Use counter for random name
zipOutputStream.putNextEntry(entry);
} catch (IOException ex) {
ex.printStackTrace();
}
}
#Override
public int read()
throws IOException {
if (inputStream.available() != 0)
return inputStream.read();
if (currentStream == null)
return -1;
int bytesRead = currentStream.read(buffer);
if (bytesRead >= 0) {
zipOutputStream.write(buffer, 0, bytesRead);
zipOutputStream.flush();
} else {
currentStream.close();
zipOutputStream.closeEntry();
if (!streams.hasNext()) {
currentStream = null;
return -1;
}
currentStream = streams.next();
zipOutputStream.putNextEntry(new ZipEntry(String.valueOf(counter++)));
}
return read();
}
}
The posting code...
MultiStreamZipInputStream myStream = ...
Client client = Client.create();
Builder webResource = client.resource("some URL").type("application/x-zip-compressed");
webResource.post(ClientResponse.class, myStream);
The REST method on the other end...
#POST
#Path("/somemethod")
#Produces(MediaType.APPLICATION_JSON)
#Consumes({"application/x-zip-compressed"})
public Response someMethod(InputStream data) {...