I'm trying to post an InputStream to a RESTful service. For normal files, this is fine.
In another place I'm trying to write a number of files to a piped zip stream on the fly. To do this I have a class that extends InputStream and when read() is called it writes the next file to the pipe. When the first file has been written I call ZipOutputStream.closeEntry() but it hangs. Why??
When I test this class in a unit test it works fine. When I try to post this object it hangs. The debugger is telling me its waiting for lock on SocketWrapper.
Note, I also tried setting media type to application/octet-stream. Also, the RESTful method is never called.
The stream class...
static class MultiStreamZipInputStream extends InputStream {
private final Iterator<InputStream> streams;
private final byte[] buffer = new byte[4096];
private InputStream inputStream;
private ZipOutputStream zipOutputStream;
private InputStream currentStream;
private int counter = 0;
public MultiStreamZipInputStream(List<InputStream> streamList) {
streams = streamList.iterator();
currentStream = streams.next();
try {
PipedOutputStream out = new PipedOutputStream();
inputStream = new PipedInputStream(out);
zipOutputStream = new ZipOutputStream(out);
ZipEntry entry = new ZipEntry(String.valueOf(counter++)); // Use counter for random name
zipOutputStream.putNextEntry(entry);
} catch (IOException ex) {
ex.printStackTrace();
}
}
#Override
public int read()
throws IOException {
if (inputStream.available() != 0)
return inputStream.read();
if (currentStream == null)
return -1;
int bytesRead = currentStream.read(buffer);
if (bytesRead >= 0) {
zipOutputStream.write(buffer, 0, bytesRead);
zipOutputStream.flush();
} else {
currentStream.close();
zipOutputStream.closeEntry();
if (!streams.hasNext()) {
currentStream = null;
return -1;
}
currentStream = streams.next();
zipOutputStream.putNextEntry(new ZipEntry(String.valueOf(counter++)));
}
return read();
}
}
The posting code...
MultiStreamZipInputStream myStream = ...
Client client = Client.create();
Builder webResource = client.resource("some URL").type("application/x-zip-compressed");
webResource.post(ClientResponse.class, myStream);
The REST method on the other end...
#POST
#Path("/somemethod")
#Produces(MediaType.APPLICATION_JSON)
#Consumes({"application/x-zip-compressed"})
public Response someMethod(InputStream data) {...
Related
static void copyStream(final InputStream inputStream, final OutputStream outputStream, final Client client) throws IOException {
if (client.getReqType() == ReqType.STREAMING_PARTIAL || client.getRange() != null) {
copyPartialStream(inputStream, outputStream, client);
return;
}
Site site = client.getSite();
Qos qos = new Qos(site);
String siteId = site.getSiteId();
byte[] buff = new byte[BUFF_SIZE]; // 1024*32
int readBytes = 0;
try (BufferedInputStream is = new BufferedInputStream(inputStream, BUFF_SIZE)) {
try (BufferedOutputStream out = new BufferedOutputStream(outputStream, BUFF_SIZE)) {
while ((readBytes = is.read(buff)) > -1) {
long startChunk = System.currentTimeMillis();
out.write(buff, 0, readBytes);
client.addSendSize(readBytes);
KHttp.increaseTrafficLog(siteId, readBytes);
long elapsedChunk = System.currentTimeMillis() - startChunk;
client.addSendMils(elapsedChunk);
if (qos.pause(client)) {
long sleepMillis = qos.getSleepMillis(client, readBytes, elapsedChunk);
if (sleepMillis > 0) {
qos.sleep(sleepMillis);
client.addSendMils(sleepMillis);
}
}
}
}
} catch (IOException e) {
client.setAbort(true);
}
}
I am looking at code that provides a link to receive a file using Tomcat. However, if the extension is attached, an error occurs. Not all extensions, but only when the mp3 extension is attached, it is downloaded normally, and when other extensions are attached, an error occurs. It works even if there is no extension at all. I think there is a problem with this code. I hope you can tell me if there is any code with errors. :(
I have code which Zip and Unzip Text. I'm facing weird behavior - only when I close the GZIPOutputStream in certain place, the code works but when I try to place the GZIPOutputStream close in the finally block, it breaks and does not work. If you place the : gzipoutputstream.close() in the czip function where comment is placed, it will work. However if you keep it only in finally block, it will break. Why?
the ByteStreams.copy function is from com.google.common.io guava
public class Main {
public static byte[] dzip(byte[] s) throws IOException {
ByteArrayInputStream sStream = null;
ByteArrayOutputStream oStream = null;
InputStream gzipoutputStream = null;
ByteBuffer arrReturn = null;
try {
sStream = new ByteArrayInputStream(s);
oStream = new ByteArrayOutputStream();
gzipoutputStream = new GZIPInputStream(sStream, 1024);
ByteStreams.copy(gzipoutputStream, oStream);
arrReturn = ByteBuffer.wrap(oStream.toByteArray());
}
catch (Exception e) {
return null;
} finally {
if (gzipoutputStream != null) {
gzipoutputStream.close();
}
if (oStream != null) {
oStream.close();
}
if (sStream != null) {
sStream.close();
}
}
return arrReturn.array();
}
public static byte[] czip(byte[] s) throws IOException {
ByteArrayInputStream sStream =null;
ByteArrayOutputStream oStream =null;
OutputStream gzipoutputstream = null;
byte[] returnValue = null;
try {
sStream = new ByteArrayInputStream(s);
oStream = new ByteArrayOutputStream(s.length / 2);
gzipoutputstream = new GZIPOutputStream(oStream, 1024);
ByteStreams.copy(sStream, gzipoutputstream);
//////////// ------------------------------------ \\\\\\\\\\\\
//gzipoutputstream.close(); // < --- Works only when placed here
//////////// ------------------------------------ \\\\\\\\\\\\
returnValue = oStream.toByteArray(); // Here the zip is 000
}catch(Exception e) {
return null;
} finally {
if (gzipoutputstream != null) {
gzipoutputstream.close();
}
if (oStream != null) {
oStream.close();
}
if (sStream != null) {
sStream.close();
}
}
return returnValue;
}
public static void main(String[] args) throws IOException {
String s = "12313dsfafafafaf";
byte[] source = (byte[]) s.getBytes();
byte[] returnZipByteArr = czip(source);
byte[] dd = dzip(returnZipByteArr);
String ss = new String(dd);
System.out.println(ss);
}
}
That's expected. The javadoc of close() says:
Writes remaining compressed data to the output stream and closes the underlying stream.
So if you try to access the bytes in the ByteArrayOutputStream() before close() has been called, the gzip compression hasn't been finished yet: the GZIP stream needs to know that nothing will never be written again to properly write the remaining data.
You could call finish() to have the same effect but without closing the stream (and thus still close it in the finally block).
I want to compress multiples files into a zip files, I'm dealing with big files, and then download them into the client, for the moment I'm using this:
#RequestMapping(value = "/download", method = RequestMethod.GET, produces = "application/zip")
public ResponseEntity <StreamingResponseBody> getFile() throws Exception {
File zippedFile = new File("test.zip");
FileOutputStream fos = new FileOutputStream(zippedFile);
ZipOutputStream zos = new ZipOutputStream(fos);
InputStream[] streams = getStreamsFromAzure();
for (InputStream stream: streams) {
addToZipFile(zos, stream);
}
final InputStream fecFile = new FileInputStream(zippedFile);
Long fileLength = zippedFile.length();
StreamingResponseBody stream = outputStream - >
readAndWrite(fecFile, outputStream);
return ResponseEntity.ok()
.header(HttpHeaders.ACCESS_CONTROL_EXPOSE_HEADERS, HttpHeaders.CONTENT_DISPOSITION)
.header(HttpHeaders.CONTENT_DISPOSITION, "attachment;filename=" + "download.zip")
.contentLength(fileLength)
.contentType(MediaType.parseMediaType("application/zip"))
.body(stream);
}
private void addToZipFile(ZipOutputStream zos, InputStream fis) throws IOException {
ZipEntry zipEntry = new ZipEntry(generateFileName());
zos.putNextEntry(zipEntry);
byte[] bytes = new byte[1024];
int length;
while ((length = fis.read(bytes)) >= 0) {
zos.write(bytes, 0, length);
}
zos.closeEntry();
fis.close();
}
This take a lot of time before all files are zipped and then the downloading start, and for large files this kan take a lot of time, this is the line responsible for the delay:
while ((length = fis.read(bytes)) >= 0) {
zos.write(bytes, 0, length);
}
So is there a way to download files immediately while their being zipped ?
Try this instead. Rather than using the ZipOutputStream to wrap a FileOutputStream, writing your zip to a file, then copying it to the client output stream, instead just use the ZipOutputStream to wrap the client output stream so that when you add zip entries and data it goes directly to the client. If you want to also store it to a file on the server then you can make your ZipOutputStream write to a split output stream, to write both locations at once.
#RequestMapping(value = "/download", method = RequestMethod.GET, produces = "application/zip")
public ResponseEntity<StreamingResponseBody> getFile() throws Exception {
InputStream[] streamsToZip = getStreamsFromAzure();
// You could cache already created zip files, maybe something like this:
// String[] pathsOfResourcesToZip = getPathsFromAzure();
// String zipId = getZipId(pathsOfResourcesToZip);
// if(isZipExist(zipId))
// // return that zip file
// else do the following
StreamingResponseBody streamResponse = clientOut -> {
FileOutputStream zipFileOut = new FileOutputStream("test.zip");
ZipOutputStream zos = new ZipOutputStream(new SplitOutputStream(clientOut, zipFileOut));
for (InputStream in : streamsToZip) {
addToZipFile(zos, in);
}
};
return ResponseEntity.ok()
.header(HttpHeaders.ACCESS_CONTROL_EXPOSE_HEADERS, HttpHeaders.CONTENT_DISPOSITION)
.header(HttpHeaders.CONTENT_DISPOSITION, "attachment;filename=" + "download.zip")
.contentType(MediaType.parseMediaType("application/zip")).body(streamResponse);
}
private void addToZipFile(ZipOutputStream zos, InputStream fis) throws IOException {
ZipEntry zipEntry = new ZipEntry(generateFileName());
zos.putNextEntry(zipEntry);
byte[] bytes = new byte[1024];
int length;
while ((length = fis.read(bytes)) >= 0) {
zos.write(bytes, 0, length);
}
zos.closeEntry();
fis.close();
}
public static class SplitOutputStream extends OutputStream {
private final OutputStream out1;
private final OutputStream out2;
public SplitOutputStream(OutputStream out1, OutputStream out2) {
this.out1 = out1;
this.out2 = out2;
}
#Override public void write(int b) throws IOException {
out1.write(b);
out2.write(b);
}
#Override public void write(byte b[]) throws IOException {
out1.write(b);
out2.write(b);
}
#Override public void write(byte b[], int off, int len) throws IOException {
out1.write(b, off, len);
out2.write(b, off, len);
}
#Override public void flush() throws IOException {
out1.flush();
out2.flush();
}
/** Closes all the streams. If there was an IOException this throws the first one. */
#Override public void close() throws IOException {
IOException ioException = null;
for (OutputStream o : new OutputStream[] {
out1,
out2 }) {
try {
o.close();
} catch (IOException e) {
if (ioException == null) {
ioException = e;
}
}
}
if (ioException != null) {
throw ioException;
}
}
}
For the first request for a set of resources to be zipped you wont know the size that the resulting zip file will be so you can't send the length along with the response since you are streaming the file as it is zipped.
But if you expect there to be repeated requests for the same set of resources to be zipped, then you can cache your zip files and simply return them on any subsequent requests; You will also know the length of the cached zip file so you can send that in the response as well.
If you want to do this then you will have to be able to consistently create the same identifier for each combination of the resources to be zipped, so that you can check if those resources were already zipped and return the cached file if they were. You might be able to could sort the ids (maybe full paths) of the resources that will be zipped and concatenate them to create an id for the zip file.
I am starting with a response from a HTTP request:
InputStream responseInputStream = response.getEntityInputStream()
I need to gzip that response so I can upload it to s3 and save it compressed:
this.s3.putObject(new PutObjectRequest(bucketName, key, gzippedResponseInputStream, meta));
I am aware that I can get the byte[] array out of responseInputStream and then gzip them into a new InputStream. However, that can be very inefficient with a large amount of data.
I know that there have been similar questions asked on SO, but I have not found anything that seems to address the specific need of starting with an InputStream and finishing with a gzipped InputStream.
Thanks for any help!
I think you're looking for a PipedInputStream
Here's how it can be done.
public InputStrema getGZipStream() {
final PipedOutputStream pos = new PipedOutputStream();
PipedInputStream pis = new PipedInputStream();
try (final InputStream responseInputStream = response.getEntityInputStream();
){
pis.connect(pos);
Thread thread = new Thread() {
public void run () {
startWriting(pos, responseInputStream);
}
};
thread.start();
} catch(Exception e) {
e.printStackTrace();
}
return pis;
}
public void startWriting(OutputStream out, InputStream in) {
try (GZIPOutputStream gOut = GZIPOutputStream(out);) {
byte[] buffer = new byte[10240];
int len = -1;
while ((len = in.read(buffer)) != -1) {
gOut.write(buffer, 0, len);
}
} catch (Exception ex) {
ex.printStackTrace();
} finally {
try {
out.close();
} catch( Exception e) {
e.printStackTrace();
}
}
}
I haven't tested this code, please let me know if this works.
public final class Example {
public static void main(String[] args) throws IOException, InterruptedException {
final PipedInputStream inputStream = new PipedInputStream();
final PipedOutputStream outputStream = new PipedOutputStream(inputStream);
Thread compressorThread = new Thread() {
#Override
public void run() {
try (FileInputStream dataSource = new FileInputStream(args[0])) {
try (GZIPOutputStream sink = new GZIPOutputStream(outputStream)) {
final byte[] buffer = new byte[8 * 1024];
for (int bytesRead = dataSource.read(buffer); bytesRead >= 0; bytesRead = dataSource.read(buffer)) {
sink.write(buffer, 0, bytesRead);
}
}
} catch (IOException ex) {
//TODO handle exception -> maybe use callable + executor
}
}
};
compressorThread.start();
try (FileOutputStream destination = new FileOutputStream(args[1])) {
final byte[] buffer = new byte[8 * 1024];
for (int bytesRead = inputStream.read(buffer); bytesRead >= 0; bytesRead = inputStream.read(buffer)) {
destination.write(buffer, 0, bytesRead);
}
}
compressorThread.join();
}
}
You are right, my previous example was wrong. You can use piped streams. The catch here is that you cannot use the input and output stream from the same thread. Also don't forget to join() on the writing thread. You can test my example by supplyng two parameters:
args[0] -> the source file
args[1] -> the destination to write the compressed content
PS: #11thdimension was a few minutes faster with his piped stream solutions, so if you find this helpful please accept his answer
I would like to send file from client to server and be able do it again in the future.
So my client connect to server and upload file, ok - it works but it hangs at the end..
so here is my code in client, the server side is quite similar.
private void SenderFile(File file) {
try {
FileInputStream fis = new FileInputStream(file);
OutputStream os = socket.getOutputStream();
IoUtil.copy(fis, os);
} catch (Exception ex) {
ex.printStackTrace();
}
}
IoUtils found on Stack :)
public static class IoUtil {
private final static int bufferSize = 8192;
public static void copy(InputStream in, OutputStream out)
throws IOException {
byte[] buffer = new byte[bufferSize];
int read;
while ((read = in.read(buffer, 0, bufferSize)) != -1) {
out.write(buffer, 0, read);
}
out.flush();
}
}
Explanation: my client has a socket connected to server, and I send any file to him.
My server download it but hangs at the end because he is listening for more infromation.
If I choose another file, my server will download new data to the existing one.
How could I upload any file to server, make my server work on and be able download another one file properly?
ps. If I add to ioutil.copy at the end of function out.close my server will work on but the connection will be lost. I do not know what to do :{
After update:
Client side:
private void SenderFile(File file) {
try {
FileInputStream fis = new FileInputStream(file);
OutputStream os = socket.getOutputStream();
DataOutputStream wrapper = new DataOutputStream(os);
wrapper.writeLong(file.length());
IoUtil.copy(fis, wrapper);
} catch (Exception ex) {
ex.printStackTrace();
}
}
Server side (thread listening for any message from client):
public void run() {
String msg;
File newfile;
try {
//Nothing special code here
while ((msg = reader.readLine()) != null) {
String[] message = msg.split("\\|");
if (message[0].equals("file")) {//file|filename|size
String filename = message[1];
//int filesize = Integer.parseInt(message[2]);
newfile = new File("server" + filename);
InputStream is = socket.getInputStream();
OutputStream os = new FileOutputStream(newfile);
DataInputStream wrapper = new DataInputStream(is);
long fileSize = wrapper.readLong();
byte[] fileData = new byte[(int) fileSize];
is.read(fileData, 0, (int) fileSize);
os.write(fileData, 0, (int) fileSize);
System.out.println("Downloaded file");
} else
//Nothing special here too
}
} catch (Exception ex) {
ex.printStackTrace();
}
}
Ok, now I can download file - still once, another one is downloaded but unable to read. For example, second time I want send by client a file.png. I got it on server, but this file is not possible to view.
Thanks in advance :)
You need to make your server able to differentiate files. The easiest way is to tell in advance how many bytes the receiving end should expect for a single file; this way, it knows when to stop reading and wait for another one.
This is what the SenderFile method could look like:
private void SenderFile(File file)
{
try
{
FileInputStream fis = new FileInputStream(file);
OutputStream os = socket.getOutputStream();
DataOutputStream wrapper = new DataOutputStream(os);
wrapper.writeLong(file.length());
IoUtil.copy(fis, wrapper);
}
catch (Exception ex)
{
ex.printStackTrace();
}
}
And this is what the ReceiveFile method could look like:
// the signature of the method is complete speculation, adapt it to your needs
private void ReceiveFile(File file)
{
FileOutputStream fos = new File(file);
InputStream is = socket.getInputStream();
DataInputStream wrapper = new DataInputStream(is);
// will not work for very big files, adapt to your needs too
long fileSize = wrapper.readLong();
byte[] fileData = new byte[fileSize];
is.read(fileData, 0, fileSize);
fos.write(fileData, 0, fileSize);
}
Then don't close the socket.