We Keep Coding

Unknown characters showing up in decompressed string via gzip

I have a mobile app where I am compressing a JSON string via gzip and storing the data in a database. I also have a web app that decompresses that same JSON string to display data on a web page. The problem is that the JSON string seems to have some unknown characters show up when decompressed in the web app that are not seen when decompressing in the mobile app.
The Android app is written in Java and uses the following code to compress the string:
ByteArrayOutputStream baos = new ByteArrayOutputStream();
GZIPOutputStream gzipOut = null;
try {
gzipOut = new GZIPOutputStream(baos);
ObjectOutputStream objectOut = new ObjectOutputStream(gzipOut);
objectOut.writeObject(jsonData);
objectOut.close();
// Wrapper for the byte array
ServerData nData = new ServerData();
nData.data = baos.toByteArray();
String finalData = JSONObjectStringConverter.json.toJson(nData);
return finalData;
} catch (IOException e) {
e.printStackTrace();
}
The above seems to work correctly, storing the byte array within the wrapper and then into the database.
The web app uses a nodejs backend and uses the following code to decompress the data after it is retrieved from the database and removed from the wrapper:
try{
// Convert the byte array back to JSON
const decompressedData = zlib.gunzipSync(new Uint8Array(compressedByteArray) );
jsonData = optionDataInflated.toString();
}catch(e){
console.error(e)
}
Again, this mostly seems to work but there are some unidentified characters at the beginning of the string:
"��\u0000\u0005t��{\"cloudData\": ..."
I thought it may have something to do with the header of the compressed string (i.e. the first 10 bytes in the byte array):
[31, -117, 8, 0, 0, 0, 0, 0, 0, 0,...
but I wasn't able to make much progress on that end. Does anyone have any other suggestions as to what the problem could be?

ObjectOutputStream is part of a general Java object serialization mechanism, and you're going to have a hard time making it work with nodejs.
Get rid of the object stream and write your JSON directly to the gzip stream. Assuming jsonData is a string, use:
gzipOut.write(jsonData.getBytes("UTF8"));
gzipOut.close();

Apache Beam/Dataflow: KVCoder corrupting Inputstream for decode

I have custom objects: CustomKey, CustomValue which I provided coder via Avro: CustomKeyCoder, CustomValueCoder.
Since I need to group by KV[CustomKey, CustomValue], I registered KVCoder.of(new CustomKeyCoder, new CustomValueCoder). Custom coders wraps in/out stream to data in/out stream and uses Avro Datum Writer/Reader.
Issue I am having is in the decode of the KVCoder, when we attempt to decode value part of KV I get Forbidden IOException when reading from InputStream. As noted, key part of decoding works properly, error is thrown when input stream is passed into decoding value. KVCoder reuses same input stream for both key and value I am guessing key decoding reads entire stream. Why would this be happening? Is usage of Avro a problem?
Here is some code to showcase above:
//Coder
override def decode(inputStream: InputStream): CustomValue = {
val dataInputStream = new DataInputStream(inputStream)
val id = dataInputStream.readShort
underlying.decode(dataInputStream)
}
//Underlying
override def decode(inputStream: InputStream): CustomValue = {
val decoder = DecoderFactory.get().binaryDecoder(inputStream, null)
val record = datumReader.read(null, decoder)
CustomValue.decode(record)
}

Convert byteArray to XSSFWorkbook using Apache POI

I am using Apache POI and I am trying to send a xlsx file as HTTP request and get it back as response. I am using jayway restassured for making HTTP requests.
Here is the part of the code where I send the request
File file = new File("path");
String response = given().multipart(file).when().post("URL").getBody().asString();
byte[] bytes = response.getBytes("ISO-8859-1");
InputStream stream = new ByteArrayOutputStream(bytes);
try
{
XSSFWorkbook workbook = new XSSFWorkbook(stream);
}
catch(Exception e){
e.printStackTrace();
}
Here is the code where the response is generated for the request
XSSFWorkbook workBook; //this workBook has the workbook sent as HTTP request
//code to make changes in workBook
ByteArrayOutputStream outStream = new ByteArrayOutputStream();
workBook.write(outStream);
byte[] byteArray = outStream.toByteArray();
String responseBody = new String(byteArray, "ISO-8859-1");
context.response().putHeader("Content-Type", "multipart/form-data").setStatusCode(200).end(responseBody);
So, what I am trying to do is send a xlsx file as request make some changes and get it back as a string response, convert it back to xssfworkbook. When converting back I get error in the following line-
XSSFWorkbook workbook = new XSSFWorkbook(stream);
The error I get is
java.util.zip.ZipException: invalid code lengths set

You cannot simply send the byte-array as ISO-8859-1 encoded text the way you attempt.
There will be special characters that might get replaced/truncated/modified.
Currently you mix binary data and a text-only channel (HTTP). You will need to do it differently, either use a binary data transfer and not convert it to String or use some binary-to-text representation, see e.g. https://en.wikipedia.org/wiki/Binary-to-text_encoding, the most common one being Base64

Use
InputStream stream = new ByteArrayInputStream(bytes);

How can I See the content of a MultipartForm request?

I am using Apache HTTPClient 4. I am doing very normal multipart stuff like this:
val entity = new MultipartEntity(HttpMultipartMode.BROWSER_COMPATIBLE);
entity.addPart("filename", new FileBody(new File(fileName), "application/zip").asInstanceOf[ContentBody])
entity.addPart("shared", new StringBody(sharedValue, "text/plain", Charset.forName("UTF-8")));
val post = new HttpPost(uploadUrl);
post.setEntity(entity);
I want to see the contents of the entity (or post, whatever) before I send it. However, that specific method is not implemented:
entity.getContent() // not defined for MultipartEntity
How can I see what I am posting?

Use the org.apache.http.entity.mime.MultipartEntity writeTo(java.io.OutputStream) method to write the content to an java.io.OutputStream, and then convert that stream to a String or byte[]:
// import java.io.ByteArrayOutputStream;
// import org.apache.http.entity.mime.MultipartEntity;
// ...
// MultipartEntity entity = ...;
// ...
ByteArrayOutputStream out = new ByteArrayOutputStream(entity.getContentLength());
// write content to stream
entity.writeTo(out);
// either convert stream to string
String string = out.toString();
// or convert stream to bytes
byte[] bytes = out.toByteArray();
Note: this only works for multipart entities small enough to be read into memory, and smaller than 2Gb which is the maximum size of a byte array in Java.

Following would help for sure:
ByteArrayOutputStream content = new ByteArrayOutputStream();
httpEntity.writeTo(content);
logger.info("Calling "+url+" with data: "+content.toString());
The above has a fix in comparison to the first answer, there is no need to pass any parameter to ByteArrayOutputStream constructor.

Do you not know the content? Although, you are building the StringBody by supplying sharedValue. So, how could it be different than sharedValue.

I have printed the Multipart request by following code, You can try like
ByteArrayOutputStream bytes = new ByteArrayOutputStream();
entity.writeTo(bytes);
String content = bytes.toString();
Log.e("MultiPartEntityRequest:",content);

Java - Image encoding in XML

I thought I would find a solution to this problem relatively easily, but here I am calling upon the help from ye gods to pull me out of this conundrum.
So, I've got an image and I want to store it in an XML document using Java. I have previously achieved this in VisualBasic by saving the image to a stream, converting the stream to an array, and then VB's xml class was able to encode the array as a base64 string. But, after a couple of hours of scouring the net for an equivalent solution in Java, I've come back empty handed. The only success I have had has been by:
import it.sauronsoftware.base64.*;
import java.awt.image.BufferedImage;
import org.w3c.dom.*;
...
BufferedImage img;
Element node;
...
java.io.ByteArrayOutputStream os = new java.io.ByteArrayOutputStream();
ImageIO.write(img, "png", os);
byte[] array = Base64.encode(os.toByteArray());
String ss = arrayToString(array, ",");
node.setTextContent(ss);
...
private static String arrayToString(byte[] a, String separator) {
StringBuffer result = new StringBuffer();
if (a.length > 0) {
result.append(a[0]);
for (int i=1; i<a.length; i++) {
result.append(separator);
result.append(a[i]);
}
}
return result.toString();
}
Which is okay I guess, but reversing the process to get it back to an image when I load the XML file has proved impossible. If anyone has a better way to encode/decode an image in an XML file, please step forward, even if it's just a link to another thread that would be fine.
Cheers in advance,
Hoopla.

I've done something similar (encoding and decoding in Base64) and it worked like a charm. Here's what I think you should do, using the class Base64 from the Apache Commons project:
// ENCODING
BufferedImage img = ImageIO.read(new File("image.png"));
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ImageIO.write(img, "png", baos);
baos.flush();
String encodedImage = Base64.encodeToString(baos.toByteArray());
baos.close(); // should be inside a finally block
node.setTextContent(encodedImage); // store it inside node
// DECODING
String encodedImage = node.getTextContent();
byte[] bytes = Base64.decode(encodedImage);
BufferedImage image = ImageIO.read(new ByteArrayInputStream(bytes));
Hope it helps.

Apache Commons has a Base64 class that should be helpful to you:
From there, you can just write out the bytes (they are already in a readable format)

After you get your byte array
byte[] array = Base64.encode(os.toByteArray());
use an encoded String :
String encodedImg = new String( array, "utf-8");
Then you can do fun things in your xml like
<binImg string-encoding="utf-8" bin-encoding="base64" img-type="png"><![CDATA[ encodedIImg here ]]></binImg>

With Java 6, you can use DatatypeConverter to convert a byte array to a Base64 string:
byte[] imageData = ...
String base64String = DatatypeConverter.printBase64Binary(imageData);
And to convert it back:
String base64String = ...
byte[] imageData = DatatypeConverter.parseBase64Binary(base64String);

Your arrayToString() method is rather bizarre (what's the point of that separator?). Why not simply say
String s = new String(array, "US-ASCII");
The reverse operation is
byte[] array = s.getBytes("US-ASCII");
Use the ASCII encoding, which should be sufficient when dealing with Base64 encoded data. Also, I'd prefer a Base64 encoder from a reputable source like Apache Commons.

You don't need to invent your own XML data type for this. XML schema defines standard binary data types, such as base64Binary, which is exactly what you are trying to do.
Once you use the standard types, it can be converted into binary automatically by some parsers (like XMLBeans). If your parser doesn't handle it, you can find classes for base64Binary in many places since the datatype is widely used in SOAP, XMLSec etc.

most easy implementation I was able to made is as below, And this is from Server to Server XML transfer containing binary data Base64 is from the Apache Codec library:
- Reading binary data from DB and create XML
Blob blobData = oRs.getBlob("ClassByteCode");
byte[] bData = blobData.getBytes(1, (int)blobData.length());
bData = Base64.encodeBase64(bData);
String strClassByteCode = new String(bData,"US-ASCII");
on requesting server read the tag and save it in DB
byte[] bData = strClassByteCode.getBytes("US-ASCII");
bData = Base64.decodeBase64(bData);
oPrStmt.setBytes( ++nParam, bData );
easy as it can be..
I'm still working on implementing the streaming of the XML as it is generated from the first server where the XML is created and stream it to the response object, this is to take care when the XML with binary data is too large.
Vishesh Sahu

The basic problem is that you cannot have an arbitrary bytestream in an XML document, so you need to encode it somehow. A frequent encoding scheme is BASE64, but any will do as long as the recipient knows about it.

I know that the question was aking how to encode an image via XML, but it is also possible to just stream the bytes via an HTTP GET request instead of using XML and encoding an image. Note that input is a FileInputStream.
Server Code:
File f = new File(uri_string);
FileInputStream input = new FileInputStream(f);
OutputStream output = exchange.getResponseBody();
int c = 0;
while ((c = input.read()) != -1) {
output.write(c); //writes each byte to the exchange.getResponseBody();
}
result = new DownloadFileResult(int_list);
if (input != null) {input.close();}
if (output != null){ output.close();}
Client Code:
InputStream input = connection.getInputStream();
List<Integer> l = new ArrayList<>();
int b = 0;
while((b = input.read()) != -1){
l.add(b);//you can do what you wish with this list of ints ie- write them to a file. see code below.
}
Here is how you would write the Integer list to a file:
FileOutputStream out = new FileOutputStream("path/to/file.png");
for(int i : result_bytes_list){
out.write(i);
}
out.close();

node.setTextContent( base64.encodeAsString( fileBytes ) )
using org.apache.commons.codec.binary.Base64