I want to compress byte array before saving to the file.
When I use Deflator to compress byte array, I get OutOfMemoryError:
ERROR/dalvikvm-heap(16065): Out of memory on a 921616-byte allocation.
I check the code and it is the same as android developer. But I added DeflatorOutputStream to reduce memory usage.
My code:
public static byte[] compress(byte[] data) throws IOException {
Deflater deflater = new Deflater();
deflater.setInput(data);
deflater.finish();
ByteArrayOutputStream outputStream = new ByteArrayOutputStream(data.length);
DeflaterOutputStream dos=new DeflaterOutputStream(outputStream);
byte[] buffer = new byte[1024];
while (!deflater.finished()) {
int count=deflater.deflate(buffer);
// returns the generated code... index
dos.write(buffer, 0, count);
}
deflater.end();
byte[] output = outputStream.toByteArray();
dos.finish();
dos.close();
outputStream.close();
return output;
}
I checked that error occurs in this line: int count=deflater.deflate(buffer);
I think there is a much simpler solution:
public static byte[] compress(byte[] data) throws IOException {
ByteArrayOutputStream bout = new ByteArrayOutputStream(data.length);
try (DeflaterOutputStream out = new DeflaterOutputStream(bout)) {
out.write(data);
}
return bout.toByteArray();
}
Related
In my app, I'm uploading files to the google drive using GD API. It works fine for small file sizes, but when file size is large (ex: 200MB), it throws java.lang.OutOfMemoryError: exception. I know why it crashes it loads the whole data into the memory, can anyone suggest how can I fix this problem?
This is my code:
OutputStream outputStream = result.getDriveContents().getOutputStream();
FileInputStream fis;
try {
fis = new FileInputStream(file.getPath());
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buf = new byte[8192];
int n;
while (-1 != (n = fis.read(buf)))
baos.write(buf, 0, n);
byte[] photoBytes = baos.toByteArray();
outputStream.write(photoBytes);
outputStream.close();
outputStream = null;
fis.close();
fis = null;
} catch (FileNotFoundException e) {
}
This line would allocate 200 MB of RAM and can definitely cause OutOfMemoryError exception:
byte[] photoBytes = baos.toByteArray();
Why are you not writing directly to your outputStream:
while (-1 != (n = fis.read(buf)))
outputStream.write(buf, 0, n);
Hi i am just wondering is it possible to set the deflater used in gzip output stream to use HUFFMAN_ONLY, i have it working with my own deflate method.
public static byte[] deflate(byte[] data) throws IOException {
Deflater deflater = new Deflater();
deflater.setInput(data);
ByteArrayOutputStream outputStream = new ByteArrayOutputStream(data.length);
deflater.finish();
deflater.setLevel(Deflater.BEST_COMPRESSION); //*****THESE 2 LINES HERE
deflater.setStrategy(Deflater.HUFFMAN_ONLY); // *******
byte[] buffer = new byte[1024];
while (!deflater.finished()) {
int count = deflater.deflate(buffer); // returns the generated code... index
outputStream.write(buffer, 0, count);
}
outputStream.close();
byte[] output = outputStream.toByteArray();
return output;
}
so basically i want to know how to set the deflater used in my gzip method to the same as the deflater above when i use the lines:
deflater.setLevel(Deflater.BEST_COMPRESSION);
deflater.setStrategy(Deflater.HUFFMAN_ONLY);
this is my gzip method:
//GZIP Compression method
public static byte[] compress(String data) throws IOException {
ByteArrayOutputStream bos = new ByteArrayOutputStream(data.length());
GZIPOutputStream gzip = new GZIPOutputStream(bos);
gzip.write(data.getBytes());
gzip.close();
byte[] compressed = bos.toByteArray();
bos.close();
return compressed;
}
No, but you can just use Deflater with nowrap set to true, and write your own gzip header and trailer.
I have a byte array and I want to decompress this byte array. When I run below code it gives;
java.util.zip.ZipException: Not in GZIP format
I get this byte array from a soap webservice. When I call this webservice from soap UI it returns;
<size>491520</size>
<studentData>
<dataContent>Uy0xMDAwMF90MTAwMDAtVXNlciBTZWN1cml0eSBB........</dataContent>
</studentData>
Is there a problem with data coming from web service or my decompress method?
public static byte[] decompress(final byte[] input) throws Exception{
try (ByteArrayInputStream bin = new ByteArrayInputStream(input);
GZIPInputStream gzipper = new GZIPInputStream(bin)) {
byte[] buffer = new byte[1024];
ByteArrayOutputStream out = new ByteArrayOutputStream();
int len;
while ((len = gzipper.read(buffer)) > 0) {
out.write(buffer, 0, len);
}
gzipper.close();
out.close();
return out.toByteArray();
}
}
EDIT:
I decoded base64 and write it to a file called "test.gzip". Now I can extract this file with 7zip and I can see all student files without any problem.
String encoded = Base64.getEncoder().encodeToString(studentData.getDataContent());
byte[] decoded = Base64.getDecoder().decode(encoded);
FileOutputStream fos = new FileOutputStream("test.gzip");
fos.write(decoded);
fos.close();
But when I try to decompress this decoded file it still gives same error;
decompress(decoded);
I am currently trying to read in data from a server response. I am using a Socket to connect to a server, creating a http GET request, then am using a Buffered Reader to read in data. Here is what the code looks like compacted:
Socket conn = new Socket(server, 80);
//Request made here
BufferedReader inFromServer = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String response;
while((response = inFromServer.readLine()) != null){
System.out.println(response);
}
I would like to read in the data, instead of as a String, as a byte array, and write it to a file. How is this possible? Any help is greatly appreciated, thank you.
You need to use a ByteArrayOutputStream, do something like the below code:
Socket conn = new Socket(server, 80);
//Request made here
InputStream is = conn.getInputStream();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buffer = new byte[1024];
int readBytes = -1;
while((readBytes = is.read(buffer)) > 1){
baos.write(buffer,0,readBytes);
}
byte[] responseArray = baos.toByteArray();
One way is to use Apache commons-io IOUtils
byte[] bytes = IOUtils.toByteArray(inputstream);
With plain java:
ByteArrayOutputStream output = new ByteArrayOutputStream();
try(InputStream stream = new FileInputStream("myFile")) {
byte[] buffer = new byte[2048];
int numRead;
while((numRead = stream.read(buffer)) != -1) {
output.write(buffer, 0, numRead);
}
} catch(IOException e) {
e.printStackTrace();
}
// and here your bytes
byte[] myDesiredBytes = output.toByteArray();
If you are not using Apache commons-io library in your project,I have pretty simple method to do the same without using it..
/*
* Read bytes from inputStream and writes to OutputStream,
* later converts OutputStream to byte array in Java.
*/
public static byte[] toByteArrayUsingJava(InputStream is)
throws IOException{
ByteArrayOutputStream baos = new ByteArrayOutputStream();
int reads = is.read();
while(reads != -1){
baos.write(reads);
reads = is.read();
}
return baos.toByteArray();
}
I need a code snippt for converting DataHandler to byte[].
This data handler contains Image.
It can be done by using below code without much effort using apache IO Commons.
final InputStream in = dataHandler.getInputStream();
byte[] byteArray=org.apache.commons.io.IOUtils.toByteArray(in);
You can do it like this:
public static byte[] toBytes(DataHandler handler) throws IOException {
ByteArrayOutputStream output = new ByteArrayOutputStream();
handler.writeTo(output);
return output.toByteArray();
}
private static final int INITIAL_SIZE = 1024 * 1024;
private static final int BUFFER_SIZE = 1024;
public static byte[] toBytes(DataHandler dh) throws IOException {
ByteArrayOutputStream bos = new ByteArrayOutputStream(INITIAL_SIZE);
InputStream in = dh.getInputStream();
byte[] buffer = new byte[BUFFER_SIZE];
int bytesRead;
while ( (bytesRead = in.read(buffer)) >= 0 ) {
bos.write(buffer, 0, bytesRead);
}
return bos.toByteArray();
}
Beware that ByteArrayOutputStream.toByteArray() creates a copy of the internal byte array.
I use this code:
public static byte[] getContentAsByteArray(DataHandler handler) throws IOException {
byte[] bytes = null;
ByteArrayOutputStream bos = new ByteArrayOutputStream();
handler.writeTo(bos);
bos.flush();
bos.close();
bytes = bos.toByteArray();
return bytes;
}
Is something like this what you are looking for?
public static byte[] getBytesFromDataHandler(final DataHandler data) throws IOException {
final InputStream in = data.getInputStream();
byte out[] = new byte[0];
if(in != null) {
out = new byte[in.available()];
in.read(out);
}
return out;
}
UPDATE:
Based on dkarp's comment this is incorrect. According to the docs for InputStream:
Returns the number of bytes that can be read (or skipped over) from this input stream without blocking by the next caller of a method for this input stream. The next caller might be the same thread or or another thread.
It looks like Costi has the correct answer here.