Unknown characters showing up in decompressed string via gzip - java

I have a mobile app where I am compressing a JSON string via gzip and storing the data in a database. I also have a web app that decompresses that same JSON string to display data on a web page. The problem is that the JSON string seems to have some unknown characters show up when decompressed in the web app that are not seen when decompressing in the mobile app.
The Android app is written in Java and uses the following code to compress the string:
ByteArrayOutputStream baos = new ByteArrayOutputStream();
GZIPOutputStream gzipOut = null;
try {
gzipOut = new GZIPOutputStream(baos);
ObjectOutputStream objectOut = new ObjectOutputStream(gzipOut);
objectOut.writeObject(jsonData);
objectOut.close();
// Wrapper for the byte array
ServerData nData = new ServerData();
nData.data = baos.toByteArray();
String finalData = JSONObjectStringConverter.json.toJson(nData);
return finalData;
} catch (IOException e) {
e.printStackTrace();
}
The above seems to work correctly, storing the byte array within the wrapper and then into the database.
The web app uses a nodejs backend and uses the following code to decompress the data after it is retrieved from the database and removed from the wrapper:
try{
// Convert the byte array back to JSON
const decompressedData = zlib.gunzipSync(new Uint8Array(compressedByteArray) );
jsonData = optionDataInflated.toString();
}catch(e){
console.error(e)
}
Again, this mostly seems to work but there are some unidentified characters at the beginning of the string:
"��\u0000\u0005t��{\"cloudData\": ..."
I thought it may have something to do with the header of the compressed string (i.e. the first 10 bytes in the byte array):
[31, -117, 8, 0, 0, 0, 0, 0, 0, 0,...
but I wasn't able to make much progress on that end. Does anyone have any other suggestions as to what the problem could be?

ObjectOutputStream is part of a general Java object serialization mechanism, and you're going to have a hard time making it work with nodejs.
Get rid of the object stream and write your JSON directly to the gzip stream. Assuming jsonData is a string, use:
gzipOut.write(jsonData.getBytes("UTF8"));
gzipOut.close();

Related

Is it possible to save pdf document to byte array (aspose.pdf for java)

I need to save a pdf document, generated by aspose.pdf for java library to memory (without using temporary file)
I was looking at the documentation and didn't find the save method with the appropriate signature. (I was looking for some kind of outputstream, or at least byte array).
Is it possible? If it is, how can I manage that?
Thanks
Aspose.Pdf for Java supports saving output to both file and stream. Please check following code snippet, It will help you to accomplish the task.
byte[] input = getBytesFromFile(new File("C:/data/HelloWorld.pdf"));
ByteArrayOutputStream output = new ByteArrayOutputStream();
com.aspose.pdf.Document pdfDocument = new com.aspose.pdf.Document(new ByteArrayInputStream(input));
pdfDocument.save(output);
//If you want to read the result into a Document object again, in Java you need to get the
//data bytes and wrap into an input stream.
InputStream inputStream=new ByteArrayInputStream(output.toByteArray());
I am Tilal Ahmad, developer evangelist at Aspose.
I did similar thing.
Here is method to write data to byte:
public byte[] toBytes() {
//create byte array output stream object
ByteArrayOutputStream byteOutStream = new ByteArrayOutputStream();
//create new data output stream object
DataOutputStream outStream = new DataOutputStream(byteOutStream);
try {//write data to bytes stream
if (data != null) {
outStream.write(data);//write data
}//return array of bytes
return byteOutStream.toByteArray();
}
Then you do something like
yourFileName.toBytes;

How to avoid unwanted data being read from java servlet to android

I have a Java servlet which takes some data from an android app and returns a string data back to the android app using the following code.
response.getOutputStream().write(STRING_MESSAGE.getBytes());
The value I pass here is read from the android activity as:
InputStream is = con.getInputStream();
byte[] b = new byte[1024];
while(is.read(b) != -1) {
buffer.append(new String(b));
}
The value is then converted to String using:
String result = buffer.toString();
But after doing so, the result has some added unwanted characters (they appear as a '?' inside a diamond shape) appended to the original string I have passed from the servlet. How can I avoid this?
As nafas said, the encoding is probably the error.
Try to replace the writing on your os with this :
response.getOutputStream().write(STRING_MESSAGE.getBytes(Charset.forName("UTF-8")));
And you also have to apply the mod to the InputStream :
buffer.append(new String(b, Charset.forName("UTF-8")));

How to convert byte array to buffered image

I have a server-side java code that gets a byte array from the client. In order to do some image processing, I need to convert the byte array into a BufferedImage. I have a code that's supposed to do that here:
public void processImage(byte[] data) {
ByteArrayInputStream stream = new ByteArrayInputStream(data);
BufferedImage bufferedImage;
bufferedImage = ImageIO.read(stream);
// bufferedImage is null
//...
}
But this doesn't work; bufferedImage is null. According to the ImageIO documentation:
If no registered ImageReader claims to be able to read the resulting stream, null is returned.
How do I tell the ImageReader what image type it is. For instance, if I know the image to be JPEG (which it is, in my case), what am I supposed to do?
EDIT: Thanks for the suggestion that the file is most likely not in JPEG format. This is the client-side code I have that sends the data as String over to the server:
import org.json.JSONObject;
// Client-side code that sends image to server as String
public void sendImage() {
FileInputStream inputStream = new FileInputStream(new File("myImage.jpg"));
ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
byte[] b = new byte[1024];
while ((bytesRead = inputStream.read(b)) != -1) {
byteStream.write(b,0,bytesRead);
}
byte[] byteArray = byteStream.toByteArray();
JSONObject jsonObject = new JSONObject();
jsonObject.put("data",new String(byteArray));
// ... more code here that sends jsonObject in HTTP post body
}
And this is the server-side code that calls the processImage() function:
// Server-side code that calls processImage() function
public void handleRequest(String jsonData) {
JSONObject jsonObject = new JSONObject(jsonData);
processImage(jsonObject.getString("data").getBytes());
}
The most likely explanation is that the byte array doesn't contain a JPEG image. (For instance, if you've just attempted to download it, you may have an HTML document giving an error diagnostic.) If that's the case, you'll need to find what is causing this and fix it.
However, if you "know" that the byte array contains an image with a given format, you could do something like this:
Use ImageIO.getImageReadersByFormatName or ImageIO.getImageReadersByMIMEType to get an Iterator<ImageReader>.
Pull the first ImageReader from the Iterator.
Create an MemoryCacheImageInputStream wrapping a ByteArrayInputStream for the types.
Use ImageReader.setInput to connect the reader to the ImageInputStream.
Use ImageReader.read to get the BufferedImage.

Compressing a string and storing it on a database as string for later decompression

I have a huge string that I need to cache somewhere and since I cannot write to file my only option is to store this on the data base as text, more specifically, in the clob I have I'm storing a JSON file where I'm placing the compressed string under a certain key of that JSON object.
I'm compressing the strings but somewhere across the string manipulation something happens that doesn't allow me to decompress the data, so I'm wondering if I should encode the data to base 64 but that will lose compression.
What could I do to ensure I can store the compressed string in the database so I can later fetch it?
I cannot change the database, so I'm stuck with that CLOB field
These are my compression functions:
public static String compress(String text) {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
try {
OutputStream out = new DeflaterOutputStream(baos);
out.write(text.getBytes("UTF-8"));
out.close();
} catch (IOException e) {
//ooops
}
return baos.toString();
}
public static String decompress(String bytes) {
InputStream in = new InflaterInputStream(new ByteArrayInputStream(bytes.getBytes()));
ByteArrayOutputStream baos = new ByteArrayOutputStream();
try {
byte[] buffer = new byte[8192];
int len;
while ((len = in.read(buffer)) > 0)
baos.write(buffer, 0, len);
return new String(baos.toByteArray(), "UTF-8");
} catch (IOException e) {
//ooops
}
}
As you found out, you can't store binary data in a CLOB without some corruption, so encoding to text will be required.
Base 64 will, on average add 33% to the size of your binary data. So you will lose some compression, but if your compression ratio is greater than 25% (this is often easy with particular types of text strings), then compression followed by base 64 encoding may provide you with a net storage gain. Lots of CPU use though.....
You can't convert arbitrary binary data to a String without breaking it. As you've already stated, if you want to store the data in a clob, you need to base64 encode the data (or use some other valid binary to text encoding).
Have you thought of other solutions, such as using memcached or other caching system? Or do you really want to mess around with compression?

Java - Image encoding in XML

I thought I would find a solution to this problem relatively easily, but here I am calling upon the help from ye gods to pull me out of this conundrum.
So, I've got an image and I want to store it in an XML document using Java. I have previously achieved this in VisualBasic by saving the image to a stream, converting the stream to an array, and then VB's xml class was able to encode the array as a base64 string. But, after a couple of hours of scouring the net for an equivalent solution in Java, I've come back empty handed. The only success I have had has been by:
import it.sauronsoftware.base64.*;
import java.awt.image.BufferedImage;
import org.w3c.dom.*;
...
BufferedImage img;
Element node;
...
java.io.ByteArrayOutputStream os = new java.io.ByteArrayOutputStream();
ImageIO.write(img, "png", os);
byte[] array = Base64.encode(os.toByteArray());
String ss = arrayToString(array, ",");
node.setTextContent(ss);
...
private static String arrayToString(byte[] a, String separator) {
StringBuffer result = new StringBuffer();
if (a.length > 0) {
result.append(a[0]);
for (int i=1; i<a.length; i++) {
result.append(separator);
result.append(a[i]);
}
}
return result.toString();
}
Which is okay I guess, but reversing the process to get it back to an image when I load the XML file has proved impossible. If anyone has a better way to encode/decode an image in an XML file, please step forward, even if it's just a link to another thread that would be fine.
Cheers in advance,
Hoopla.
I've done something similar (encoding and decoding in Base64) and it worked like a charm. Here's what I think you should do, using the class Base64 from the Apache Commons project:
// ENCODING
BufferedImage img = ImageIO.read(new File("image.png"));
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ImageIO.write(img, "png", baos);
baos.flush();
String encodedImage = Base64.encodeToString(baos.toByteArray());
baos.close(); // should be inside a finally block
node.setTextContent(encodedImage); // store it inside node
// DECODING
String encodedImage = node.getTextContent();
byte[] bytes = Base64.decode(encodedImage);
BufferedImage image = ImageIO.read(new ByteArrayInputStream(bytes));
Hope it helps.
Apache Commons has a Base64 class that should be helpful to you:
From there, you can just write out the bytes (they are already in a readable format)
After you get your byte array
byte[] array = Base64.encode(os.toByteArray());
use an encoded String :
String encodedImg = new String( array, "utf-8");
Then you can do fun things in your xml like
<binImg string-encoding="utf-8" bin-encoding="base64" img-type="png"><![CDATA[ encodedIImg here ]]></binImg>
With Java 6, you can use DatatypeConverter to convert a byte array to a Base64 string:
byte[] imageData = ...
String base64String = DatatypeConverter.printBase64Binary(imageData);
And to convert it back:
String base64String = ...
byte[] imageData = DatatypeConverter.parseBase64Binary(base64String);
Your arrayToString() method is rather bizarre (what's the point of that separator?). Why not simply say
String s = new String(array, "US-ASCII");
The reverse operation is
byte[] array = s.getBytes("US-ASCII");
Use the ASCII encoding, which should be sufficient when dealing with Base64 encoded data. Also, I'd prefer a Base64 encoder from a reputable source like Apache Commons.
You don't need to invent your own XML data type for this. XML schema defines standard binary data types, such as base64Binary, which is exactly what you are trying to do.
Once you use the standard types, it can be converted into binary automatically by some parsers (like XMLBeans). If your parser doesn't handle it, you can find classes for base64Binary in many places since the datatype is widely used in SOAP, XMLSec etc.
most easy implementation I was able to made is as below, And this is from Server to Server XML transfer containing binary data Base64 is from the Apache Codec library:
- Reading binary data from DB and create XML
Blob blobData = oRs.getBlob("ClassByteCode");
byte[] bData = blobData.getBytes(1, (int)blobData.length());
bData = Base64.encodeBase64(bData);
String strClassByteCode = new String(bData,"US-ASCII");
on requesting server read the tag and save it in DB
byte[] bData = strClassByteCode.getBytes("US-ASCII");
bData = Base64.decodeBase64(bData);
oPrStmt.setBytes( ++nParam, bData );
easy as it can be..
I'm still working on implementing the streaming of the XML as it is generated from the first server where the XML is created and stream it to the response object, this is to take care when the XML with binary data is too large.
Vishesh Sahu
The basic problem is that you cannot have an arbitrary bytestream in an XML document, so you need to encode it somehow. A frequent encoding scheme is BASE64, but any will do as long as the recipient knows about it.
I know that the question was aking how to encode an image via XML, but it is also possible to just stream the bytes via an HTTP GET request instead of using XML and encoding an image. Note that input is a FileInputStream.
Server Code:
File f = new File(uri_string);
FileInputStream input = new FileInputStream(f);
OutputStream output = exchange.getResponseBody();
int c = 0;
while ((c = input.read()) != -1) {
output.write(c); //writes each byte to the exchange.getResponseBody();
}
result = new DownloadFileResult(int_list);
if (input != null) {input.close();}
if (output != null){ output.close();}
Client Code:
InputStream input = connection.getInputStream();
List<Integer> l = new ArrayList<>();
int b = 0;
while((b = input.read()) != -1){
l.add(b);//you can do what you wish with this list of ints ie- write them to a file. see code below.
}
Here is how you would write the Integer list to a file:
FileOutputStream out = new FileOutputStream("path/to/file.png");
for(int i : result_bytes_list){
out.write(i);
}
out.close();
node.setTextContent( base64.encodeAsString( fileBytes ) )
using org.apache.commons.codec.binary.Base64

Categories