We Keep Coding

Unexpected end of ZLIB stream large csv file

I am trying to decompress a csv file that is in the form name.csv.gz and its I think its something like 600M compressed and we'll say something in the ballpark of 7Gb when decompressed
byte[] buffer = new byte[4096];
try {
GZIPInputStream gzis = new GZIPInputStream(new FileInputStream("/run/media/justin/DATA/2000000033673205_53848.TEST_SCHEDULE_GCO.20180706.090850.2000000033673205.x04q13.csv.gz"));
FileOutputStream out = new FileOutputStream("/run/media/justin/DATA/unzipped.txt");
int len;
while((len = gzis.read(buffer)) > 0) {
out.write(buffer,0,len);
}
gzis.close();
out.close();
System.out.println("DONE!!");
} catch(IOException e) {e.printStackTrace();}
this is the code I am using to decompress it, and at the end, I get the error Unexpected end of ZLIB stream and I am missing several million lines at the end of the file. I haven't found anything on google that has led me in any prosperous directions so any help is greatly appreciated!
Edit: I forgot a line of code at the top (*facepalm) also, I have increased the buffer size, from 2048 to 4096, and I am getting more lines after decompression, so would I be correct in assuming that I just didn't allocate a large enough buffer? (or is this a naive assumption?)

I have increased the buffer size, from 2048 to 4096, and I am getting more lines after decompression, so would I be correct in assuming that I just didn't allocate a large enough buffer? (or is this a naive assumption?)
This is no problem of your buffer size, it's more a problem with the GZIPInputStream.read() methode. The buffer size only declares how "often" the while-loop should read and write, cause a bigger buffer => higher transfer rate => less loops
Your problem is inside of the GZIPInputStream class or has something to do with the used files, maybe try a smaller file first.

Missing one byte when transferring an image over Socket Java

I have a problem transferring a file over socket.
I Wrote a simple client / server app and the client takes a screenshot and send it to server.
The problem is the file is not completed whatever i do, It's always missing the first byte from the array which makes the photo damaged.
When I open the photo in any hex editor and compare the original photo with the one that the client sent, I can see the missing byte, as if I add it, the photo opens without the problem. The size of the sent file missing just one byte !
Here is a photo for the problem :
Original photo
sent photo
Here is the code :
Server ( Receiver ) :
byte[] buf;
InputStream inp;
try (BufferedOutputStream out1 = new BufferedOutputStream(new FileOutputStream(new File("final.jpeg")))) {
buf = new byte[s.getReceiveBufferSize()];
inp = new DataInputStream(s.getInputStream());
Thread.sleep(200);
int len = 0;
while ((len = inp.read(buf)) >0){
out1.write(buf,0,len);
}
out1.flush();
inp.close();
out1.close();
}
Client ( Sender ):
BufferedImage screenshot = new Robot().createScreenCapture(new Rectangle(Toolkit.getDefaultToolkit().getScreenSize()));
ByteArrayOutputStream os = new ByteArrayOutputStream();
ImageIO.write(screenshot, "jpeg", os);
ImageIO.write(screenshot, "jpeg", new File("test.jpeg"));
OutputStream out = new BufferedOutputStream( connection.getOutputStream());
out.write(os.toByteArray());
out.close();
I have tried to send the array with the same way I receive it but no lock. I have tried with, and without buffered, I have tried flush in both sides, I tried to turn off Nod antivirus, Tried a sleep when sending length,
I almost tried everything without success .
I have tried on both, My pc and a virtual machine windows 7 !
Any help will be appreciated.
Edit :
First 10 bytes from the original file :
first 10 bytes from the sent file :

The code you posted does not lose data. Somewhere prior to executing the server code you posted, you have executed a single InputStream.read() of one byte, possibly in a misguided attempt to test for end of stream.
The sleep is just literally a waste of time. Remove it. You don't need the DataInput/OutputStreams either.

Please keep in mind that DataInputStream signals end of stream by returning value -1 from read() therefore your server reading loop should look like this:
while ((len = inp.read(buf)) != -1){
out1.write(buf,0,len);
}
Perhaps this helps.

The client code looks fine. Must be the server. You only posted the part when "some" input stream is written to a file. What happens before? Anyone doing a read() on the input stream?
Sorry for writing this in the "answer" section. Apparently, I cannot comment yet.

Ok it was my fault ! I was looking for something wrong in server side but the fault was in client side ! I opened a DataInputStream to read the order coming from server without closing it and that was the problem.

How to get file size of http file path

I'm using the below code to download files from a remote location via http. Some assets are not fully downloading and appear to be corrupt. It looks to be about 5% of the time. I'm thinking it'd be good to ensure I've downloaded the full file by getting the file size in advance and comparing it to what I've downloaded to be sure nothing was missed.
Through some google searches and looking at the objects I'm already working with I don't see an obvious way to obtain this file size. Can someone point me in the right direction?
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setInstanceFollowRedirects(true);
InputStream is = con.getInputStream();
file = new File(destinationPath+"."+remoteFile.getExtension());
BufferedInputStream bis = new BufferedInputStream(is);
BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(file.getAbsolutePath()));
while ((i = bis.read()) != -1) {
bos.write(i);
}
bos.flush();
bis.close();

con.getContentLength() may give you what you want, but only if the server provided it as a response header. If the server used "chunked" encoding instead of providing a Content-Length header then the total length is not available up-front.

Check out the getContentLength() method here HttpURLConnection that inherits from URLConnection.

You can use InputStream#available() method.
It returns an estimate of the number of bytes that can be read (or skipped over) from this input stream without blocking by the next invocation of a method for this input stream.
The next invocation might be the same thread or another thread. A single read or skip of this many bytes will not block, but may read or skip fewer bytes.
FileInputStream fis = new FileInputStream(destinationPath+"."+remoteFile.getExtension());
System.out.println("Total file size to read (in bytes) : "+ fis.available());

Obtaining the wav file from ServletInputStream

Our current project requires us to send an audio file to the server and then use the audio file for further computation.
Using the Java sound api, I was able to capture the recording and save it as a wav file in my system. Then in order to pass the audio wav to the server, I am using Apache Commons HttpClient to post a request to the server. (I am using InputstreamEntity provided by apache and sending the data as a chunk).
The problem appears when i am trying to recreate/retrieve the wav file on the server. I understand that I would have to use the AudioSystem.write API to create the wav file (exactly as what was done on my system). However what I observe is that althought the file gets created , it does not play (I am using vlc media player to test it FYI). I have searched in Google for sample codes and have tried to implement it, but is unable to play it once the file gets created.
The sample code snippets indicates the approaches i have tried:
//******************************************************************
try {
InputStream is = request.getInputStream();
FileOutputStream fs = new FileOutputStream("output123.wav");
byte[] tempbuffer = new byte[4096];
int bytesRead;
while((bytesRead=is.read(tempbuffer))!=-1)
{
fs.write(tempbuffer, 0,bytesRead);
}
is.close();
fs.close();
AudioInputStream inputStream =AudioSystem.getAudioInputStream(newFile("output123.wav"));
int numofbytes = inputStream.available();
byte[] buffer = new byte[numofbytes];
inputStream.read(buffer);
int bytesWritten = AudioSystem.write(inputStream, AudioFileFormat.Type.WAVE,new File("outputtest.wav"));
System.out.println("written"+bytesWritten);
Approach 2
InputStream is = request.getInputStream();
System.out.println("inputStream obtained : "+is.toString());
ByteArrayInputStream bais = null;
byte[] audioBuffer = IOUtils.toByteArray(is);
System.out.println(" is audioBuffer empty? : length = ? "+audioBuffer.length);
try {
AudioFileFormat ai = AudioSystem.getAudioFileFormat(is);
System.out.println("ai bytelength ? "+ai.getByteLength());
System.out.println("ai frame length = "+ai.getFrameLength());
Set<Map.Entry<String,Object>> audioProperties = ai.getFormat().properties().entrySet();
System.out.println("entry set is empty ? "+audioProperties.isEmpty());
for(Map.Entry me : audioProperties){
System.out.println("key = "+me.getKey());
System.out.println("value ="+me.getValue());}
bais = new ByteArrayInputStream(audioBuffer);
AudioInputStream ais = new AudioInputStream(bais, new AudioFormat(8000,8,2,true,true), 2);
AudioSystem.write(ais, AudioFileFormat.Type.WAVE,new File("testtest.wav"));
//*************************************************************************************
The audioFormat properties all turned out to be null. Are these null values giving the problem? So while creating the wave file on the server, I tried to set the properties manually once again. But even then the wav file would not play.
I have also tried quite a few approaches already mentioned on this site, but somehow they aren't working. I am sure i am missing something, but I am unable to pinpoint the exact problem.
Would be really helpful, if you guys can point out how to go about the conversion from ServletInputStream to getting a wav.
P.S (1) I know the code is shabby, because i have been under a trial and error situation for quite some time now. But I will give more details on the approaches if needed.
2) Apologise for the clumsiness, this happens to be my first post.. )

this is not how you copy a stream (from Approach 1). you have the correct code to copy a stream just above this.:
int numofbytes = inputStream.available();
byte[] buffer = new byte[numofbytes];
inputStream.read(buffer);

If all your server wants to do is get the data and write it to a file, then you do not need to use any of the audio API: simply treat the data as a stream of bytes.
So the part of approach 1 that is before any mention of AudioInputStream should be sufficient.

Although the approach chosen might not be the perfect solution, due to time constraints, I adopted a simpler approach. Using java.util.zip i simply zipped it up and sent it over to the server and then wrote a layer wherin the file gets unzipped . then i deleted the zip files. Seems like an immature solution (bcos the original challenge was to send the audio file). now i am incurring an overhead of zipping the files, but the file transfer would hapeen relatively faster. Thanks for your help guys.

Java: Reading a pdf file from URL into Byte array/ByteBuffer in an applet

I'm trying to figure out why this particular snippet of code isn't working for me. I've got an applet which is supposed to read a .pdf and display it with a pdf-renderer library, but for some reason when I read in the .pdf files which sit on my server, they end up as being corrupt. I've tested it by writing the files back out again.
I've tried viewing the applet in both IE and Firefox and the corrupt files occur. Funny thing is, when I trying viewing the applet in Safari (for Windows), the file is actually fine! I understand the JVM might be different, but I am still lost. I've compiled in Java 1.5. JVMs are 1.6. The snippet which reads the file is below.
public static ByteBuffer getAsByteArray(URL url) throws IOException {
ByteArrayOutputStream tmpOut = new ByteArrayOutputStream();
URLConnection connection = url.openConnection();
int contentLength = connection.getContentLength();
InputStream in = url.openStream();
byte[] buf = new byte[512];
int len;
while (true) {
len = in.read(buf);
if (len == -1) {
break;
}
tmpOut.write(buf, 0, len);
}
tmpOut.close();
ByteBuffer bb = ByteBuffer.wrap(tmpOut.toByteArray(), 0,
tmpOut.size());
//Lines below used to test if file is corrupt
//FileOutputStream fos = new FileOutputStream("C:\\abc.pdf");
//fos.write(tmpOut.toByteArray());
return bb;
}
I must be missing something, and I've been banging my head trying to figure it out. Any help is greatly appreciated. Thanks.
Edit:
To further clarify my situation, the difference in the file before I read then with the snippet and after, is that the ones I output after reading are significantly smaller than they originally are. When opening them, they are not recognized as .pdf files. There are no exceptions being thrown that I ignore, and I have tried flushing to no avail.
This snippet works in Safari, meaning the files are read in it's entirety, with no difference in size, and can be opened with any .pdf reader. In IE and Firefox, the files always end up being corrupted, consistently the same smaller size.
I monitored the len variable (when reading a 59kb file), hoping to see how many bytes get read in at each loop. In IE and Firefox, at 18kb, the in.read(buf) returns a -1 as if the file has ended. Safari does not do this.
I'll keep at it, and I appreciate all the suggestions so far.

Just in case these small changes make a difference, try this:
public static ByteBuffer getAsByteArray(URL url) throws IOException {
URLConnection connection = url.openConnection();
// Since you get a URLConnection, use it to get the InputStream
InputStream in = connection.getInputStream();
// Now that the InputStream is open, get the content length
int contentLength = connection.getContentLength();
// To avoid having to resize the array over and over and over as
// bytes are written to the array, provide an accurate estimate of
// the ultimate size of the byte array
ByteArrayOutputStream tmpOut;
if (contentLength != -1) {
tmpOut = new ByteArrayOutputStream(contentLength);
} else {
tmpOut = new ByteArrayOutputStream(16384); // Pick some appropriate size
}
byte[] buf = new byte[512];
while (true) {
int len = in.read(buf);
if (len == -1) {
break;
}
tmpOut.write(buf, 0, len);
}
in.close();
tmpOut.close(); // No effect, but good to do anyway to keep the metaphor alive
byte[] array = tmpOut.toByteArray();
//Lines below used to test if file is corrupt
//FileOutputStream fos = new FileOutputStream("C:\\abc.pdf");
//fos.write(array);
//fos.close();
return ByteBuffer.wrap(array);
}
You forgot to close fos which may result in that file being shorter if your application is still running or is abruptly terminated. Also, I added creating the ByteArrayOutputStream with the appropriate initial size. (Otherwise Java will have to repeatedly allocate a new array and copy, allocate a new array and copy, which is expensive.) Replace the value 16384 with a more appropriate value. 16k is probably small for a PDF, but I don't know how but the "average" size is that you expect to download.
Since you use toByteArray() twice (even though one is in diagnostic code), I assigned that to a variable. Finally, although it shouldn't make any difference, when you are wrapping the entire array in a ByteBuffer, you only need to supply the byte array itself. Supplying the offset 0 and the length is redundant.
Note that if you are downloading large PDF files this way, then ensure that your JVM is running with a large enough heap that you have enough room for several times the largest file size you expect to read. The method you're using keeps the whole file in memory, which is OK as long as you can afford that memory. :)

I thought I had the same problem as you, but it turned out my problem was that I assumed you always get the full buffer until you get nothing. But you do not assume that.
The examples on the net (e.g. java2s/tutorial) use a BufferedInputStream. But that does not make any difference for me.
You could check whether you actually get the full file in your loop. Than the problem would be in the ByteArrayOutputStream.

Have you tried a flush() before you close the tmpOut stream to ensure all bytes written out?