We Keep Coding

How to deserialize avro files

I would like to read a hdfs folder containing avro files with spark . Then I would like to deserialize the avro events contained in these files. I would like to do it without the com.databrics library (or any other that allow to do it easely).
The problem is that I have difficulties with the deserialization.
I assume that my avro file is compressed with snappy because at the begining of the file (just after the schema), I have
avro.codecsnappy
written. Then it's followed by readable or unreadable charaters.
My first attempt to deserialize the avro event is the following :
public static String deserialize(String message) throws IOException {
Schema.Parser schemaParser = new Schema.Parser();
Schema avroSchema = schemaParser.parse(defaultFlumeAvroSchema);
DatumReader<GenericRecord> specificDatumReader = new SpecificDatumReader<GenericRecord>(avroSchema);
byte[] messageBytes = message.getBytes();
Decoder decoder = DecoderFactory.get().binaryDecoder(messageBytes, null);
GenericRecord genericRecord = specificDatumReader.read(null, decoder);
return genericRecord.toString();
}
This function works when I want to deserialise an avro file that doesn't have the avro.codecsbappy in it. When it's the case I have the error :
Malformed data : length is negative : -50
So I tried another way of doing it which is :
private static void deserialize2(String path) throws IOException {
DatumReader<GenericRecord> reader = new GenericDatumReader<>();
DataFileReader<GenericRecord> fileReader =
new DataFileReader<>(new File(path), reader);
System.out.println(fileReader.getSchema().toString());
GenericRecord record = new GenericData.Record(fileReader.getSchema());
int numEvents = 0;
while (fileReader.hasNext()) {
fileReader.next(record);
ByteBuffer body = (ByteBuffer) record.get("body");
CharsetDecoder decoder = Charsets.UTF_8.newDecoder();
System.out.println("Positon of the index " + body.position());
System.out.println("Size of the array : " + body.array().length);
String bodyStr = decoder.decode(body).toString();
System.out.println("THE BODY STRING ---> " bodyStr);
numEvents++;
}
fileReader.close();
}
and it returns the follwing output :
Positon of the index 0
Size of the array : 127482
THE BODY STRING --->
I can see that the array isn't empty but it just return an empty string.
How can I proceed ?

Use this when converting to string:
String bodyStr = new String(body.array());
System.out.println("THE BODY STRING ---> " + bodyStr);
Source: https://www.mkyong.com/java/how-do-convert-byte-array-to-string-in-java/

Well, it seems that you are on a good way. However, your ByteBuffer might not have a proper byte[] array to decode, so let's try the following instead:
byte[] bytes = new byte[body.remaining()];
buffer.get(bytes);
String result = new String(bytes, "UTF-8"); // Maybe you need to change charset
This should work, you have shown in your question that ByteBuffer contains actual data, as given in the code example you might have to change the charset.
List of charsets: https://docs.oracle.com/javase/7/docs/api/java/nio/charset/Charset.html
Also usful: https://docs.oracle.com/javase/7/docs/api/java/nio/ByteBuffer.html

Add byte array inside a json object using java

I have below situation. It needs to be implemented in Java.
Take input from a text file, convert the content into a byte array.
Use the above byte array as a part of a JSON object , create a .json file
For point1, i have done something like this.
InputStream is = new ClassPathresource("file.txt").getInputStream();
byte[] ip = IOUtils.toByteArray(is);
For point2, my Json file (containing json object), should look like below.
{
"name": "xyz",
"address: "address here",
"ipdata": ""
}
The ipdata should contain the byte array created in step 1.
How can i create a json object with the byte array created in step 1 as a part of it ? And then write the entire content to a separate .json file ?
Also is the byte array conversion done in step1 an optimum way, or do we need to use any other API(may be to take care of encoding)?Please suggest.
Any help is appreciated. Thanks in advance.

You can simply convert the byte array ip using ip.toString()
Or if you know the encoding you can use ipString = new String(ip, "UTF8")
And then take that string to add to your json object.

Since you are reading a JSON string from file and want to write it back to a new json file you dont need the JSON Object conversion in-between. Just convert the byte[] to String as
String ips = new String(ip);
Now create a JSON Object with the data you want to write to the new file. And then you can write the data to file using FileWriter. PFB the code-
JSONObject obj = new JSONObject();
obj.put("name", "xyz");
obj.put("address", "address here");
obj.put("ipdata", ips);
try(FileWriter fileWriter =
new FileWriter("newFileName.json") ){
fileWriter.write(obj.toString());
}

how to covert json string to byte array in java

String json="{"FROM_JID":"6bc24cac4eaf304ce1731bd5aebe9b0419052701","TO_JID":"dfc8d53f402373a1d3622dde50e180b388b36bc1","TYPE_ID":"1","PLATFORM":"IOS","CONTENT":"{\"FROM_JID\":\"6bc24cac4eaf304ce1731bd5aebe9b0419052701\",\"FROM_HOST\":\"ssdevim.mtouche-mobile.com\",\"FROM_JNAME\":\"test1\",\"TO_JID\":\"dfc8d53f402373a1d3622dde50e180b388b36bc1\",\"TO_HOST\":\"ssdevim.mtouche-mobile.com\",\"MESSAGE_ID\":\"LiYaU-39\",\"MESSAGE_TYPE\":\"enc\",\"MESSAGE\":\"test1 has sent you an encrypted message.\",\"STAMP\":\"2015-11-12 12:04:54.252241\",\"BADGE\":3,\"CONTENT-AVAILABLE\":1,\"SOUND\":\"dafault\"}","DEVICE_ID":"AC53D4F0-DAAA-475E-9668-5E9E7485797C","PUSH_ID":"c9544c8db2117f02f3edc8af9058b3d54c15500302bf6f47c487193876f6dc23","CREATE_DATE":"2015-11-12","CREATE_TIME":"04:04:54"}";
JSONParser parser = new JSONParser();
Object obj = parser.parse(json);
but it showing error

First, this won't compile:
String json="{"FROM_JID":"6bc24cac4eaf304ce1731bd5aebe9b0419052701","TO_JID":"dfc8d53f402373a1d3622dde50e180b388b36bc1","TYPE_ID":"1","PLATFORM":"IOS","CONTENT":"{\"FROM_JID\":\"6bc24cac4eaf304ce1731bd5aebe9b0419052701\",\"FROM_HOST\":\"ssdevim.mtouche-mobile.com\",\"FROM_JNAME\":\"test1\",\"TO_JID\":\"dfc8d53f402373a1d3622dde50e180b388b36bc1\",\"TO_HOST\":\"ssdevim.mtouche-mobile.com\",\"MESSAGE_ID\":\"LiYaU-39\",\"MESSAGE_TYPE\":\"enc\",\"MESSAGE\":\"test1 has sent you an encrypted message.\",\"STAMP\":\"2015-11-12 12:04:54.252241\",\"BADGE\":3,\"CONTENT-AVAILABLE\":1,\"SOUND\":\"dafault\"}","DEVICE_ID":"AC53D4F0-DAAA-475E-9668-5E9E7485797C","PUSH_ID":"c9544c8db2117f02f3edc8af9058b3d54c15500302bf6f47c487193876f6dc23","CREATE_DATE":"2015-11-12","CREATE_TIME":"04:04:54"}";
You even can notice that its syntax is not highlighted properly.
You need to escape your quotes in order to make Java recognize it as a part of a string, but not your code:
String json="{\"FROM_JID\":\"6bc24cac4eaf304ce1731bd5aebe9b0419052701\",\"TO_JID\":\"dfc8d53f402373a1d3622dde50e180b388b36bc1\",\"TYPE_ID\":\"1\",\"PLATFORM\":\"IOS\",\"CONTENT\":\"{\\\"FROM_JID\\\":\\\"6bc24cac4eaf304ce1731bd5aebe9b0419052701\\\",\\\"FROM_HOST\\\":\\\"ssdevim.mtouche-mobile.com\\\",\\\"FROM_JNAME\\\":\\\"test1\\\",\\\"TO_JID\\\":\\\"dfc8d53f402373a1d3622dde50e180b388b36bc1\\\",\\\"TO_HOST\\\":\\\"ssdevim.mtouche-mobile.com\\\",\\\"MESSAGE_ID\\\":\\\"LiYaU-39\\\",\\\"MESSAGE_TYPE\\\":\\\"enc\\\",\\\"MESSAGE\\\":\\\"test1 has sent you an encrypted message.\\\",\\\"STAMP\\\":\\\"2015-11-12 12:04:54.252241\\\",\\\"BADGE\\\":3,\\\"CONTENT-AVAILABLE\\\":1,\\\"SOUND\\\":\\\"dafault\\\"}\",\"DEVICE_ID\":\"AC53D4F0-DAAA-475E-9668-5E9E7485797C\",\"PUSH_ID\":\"c9544c8db2117f02f3edc8af9058b3d54c15500302bf6f47c487193876f6dc23\",\"CREATE_DATE\":\"2015-11-12\",\"CREATE_TIME\":\"04:04:54\"}";
Second, if you already have a String and you want to convert it to byte[], why do you deserialize it? Just convert it to byte array:
byte[] bytes = json.getBytes();

Convert String representation of bytes to byte[] in java

My application get the String representation of bytes. I need to convert it byte[] array. I am using below code but it is not working.
byte[] bytesArray = myString.getBytes();
Can anyone help what is the correct way to convert it to byte[].
EDIT:
hi all, My code is here http://pastebin.com/87jGprtD/. I have one base64 code. This base64 has content for text and imagedata both. I want to download/create an image from this code. When I decode I get the byte[] for both text and imagedata. I convert it string because I have to differentiate the each part. I used spilt with some delimiter now i have an array of string. This string contains the imagedata. I have to convert it back to bytes to create an image. please check code for the same. please
Here is the relevant code:
byte[] imageByteArray = Base64.decodeBase64(imageDataString);
System.out.println(new String(imageByteArray));
String[] contentArray = new String(imageByteArray).split("--1_520B30B0_E358708");
for (int i = 0; i < contentArray.length; i++) {
if (i == 2) {
String[] parts = contentArray[i].split("binary");
InputStream is = new ByteArrayInputStream((parts[1].trim()).getBytes());
ImageInputStream iis = ImageIO.createImageInputStream(is);
System.out.println(iis);
image = ImageIO.read(iis);
ImageIO.write(image, "JPG", new File("E:/test1.JPG"));
}
}

You are decoding Base64 data into byte[], then converting that to String. You can't do that -- "binary" data cannot be converted to String and back to "binary" without data loss.

Java - Image encoding in XML

I thought I would find a solution to this problem relatively easily, but here I am calling upon the help from ye gods to pull me out of this conundrum.
So, I've got an image and I want to store it in an XML document using Java. I have previously achieved this in VisualBasic by saving the image to a stream, converting the stream to an array, and then VB's xml class was able to encode the array as a base64 string. But, after a couple of hours of scouring the net for an equivalent solution in Java, I've come back empty handed. The only success I have had has been by:
import it.sauronsoftware.base64.*;
import java.awt.image.BufferedImage;
import org.w3c.dom.*;
...
BufferedImage img;
Element node;
...
java.io.ByteArrayOutputStream os = new java.io.ByteArrayOutputStream();
ImageIO.write(img, "png", os);
byte[] array = Base64.encode(os.toByteArray());
String ss = arrayToString(array, ",");
node.setTextContent(ss);
...
private static String arrayToString(byte[] a, String separator) {
StringBuffer result = new StringBuffer();
if (a.length > 0) {
result.append(a[0]);
for (int i=1; i<a.length; i++) {
result.append(separator);
result.append(a[i]);
}
}
return result.toString();
}
Which is okay I guess, but reversing the process to get it back to an image when I load the XML file has proved impossible. If anyone has a better way to encode/decode an image in an XML file, please step forward, even if it's just a link to another thread that would be fine.
Cheers in advance,
Hoopla.

I've done something similar (encoding and decoding in Base64) and it worked like a charm. Here's what I think you should do, using the class Base64 from the Apache Commons project:
// ENCODING
BufferedImage img = ImageIO.read(new File("image.png"));
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ImageIO.write(img, "png", baos);
baos.flush();
String encodedImage = Base64.encodeToString(baos.toByteArray());
baos.close(); // should be inside a finally block
node.setTextContent(encodedImage); // store it inside node
// DECODING
String encodedImage = node.getTextContent();
byte[] bytes = Base64.decode(encodedImage);
BufferedImage image = ImageIO.read(new ByteArrayInputStream(bytes));
Hope it helps.

Apache Commons has a Base64 class that should be helpful to you:
From there, you can just write out the bytes (they are already in a readable format)

After you get your byte array
byte[] array = Base64.encode(os.toByteArray());
use an encoded String :
String encodedImg = new String( array, "utf-8");
Then you can do fun things in your xml like
<binImg string-encoding="utf-8" bin-encoding="base64" img-type="png"><![CDATA[ encodedIImg here ]]></binImg>

With Java 6, you can use DatatypeConverter to convert a byte array to a Base64 string:
byte[] imageData = ...
String base64String = DatatypeConverter.printBase64Binary(imageData);
And to convert it back:
String base64String = ...
byte[] imageData = DatatypeConverter.parseBase64Binary(base64String);

Your arrayToString() method is rather bizarre (what's the point of that separator?). Why not simply say
String s = new String(array, "US-ASCII");
The reverse operation is
byte[] array = s.getBytes("US-ASCII");
Use the ASCII encoding, which should be sufficient when dealing with Base64 encoded data. Also, I'd prefer a Base64 encoder from a reputable source like Apache Commons.

You don't need to invent your own XML data type for this. XML schema defines standard binary data types, such as base64Binary, which is exactly what you are trying to do.
Once you use the standard types, it can be converted into binary automatically by some parsers (like XMLBeans). If your parser doesn't handle it, you can find classes for base64Binary in many places since the datatype is widely used in SOAP, XMLSec etc.

most easy implementation I was able to made is as below, And this is from Server to Server XML transfer containing binary data Base64 is from the Apache Codec library:
- Reading binary data from DB and create XML
Blob blobData = oRs.getBlob("ClassByteCode");
byte[] bData = blobData.getBytes(1, (int)blobData.length());
bData = Base64.encodeBase64(bData);
String strClassByteCode = new String(bData,"US-ASCII");
on requesting server read the tag and save it in DB
byte[] bData = strClassByteCode.getBytes("US-ASCII");
bData = Base64.decodeBase64(bData);
oPrStmt.setBytes( ++nParam, bData );
easy as it can be..
I'm still working on implementing the streaming of the XML as it is generated from the first server where the XML is created and stream it to the response object, this is to take care when the XML with binary data is too large.
Vishesh Sahu

The basic problem is that you cannot have an arbitrary bytestream in an XML document, so you need to encode it somehow. A frequent encoding scheme is BASE64, but any will do as long as the recipient knows about it.

I know that the question was aking how to encode an image via XML, but it is also possible to just stream the bytes via an HTTP GET request instead of using XML and encoding an image. Note that input is a FileInputStream.
Server Code:
File f = new File(uri_string);
FileInputStream input = new FileInputStream(f);
OutputStream output = exchange.getResponseBody();
int c = 0;
while ((c = input.read()) != -1) {
output.write(c); //writes each byte to the exchange.getResponseBody();
}
result = new DownloadFileResult(int_list);
if (input != null) {input.close();}
if (output != null){ output.close();}
Client Code:
InputStream input = connection.getInputStream();
List<Integer> l = new ArrayList<>();
int b = 0;
while((b = input.read()) != -1){
l.add(b);//you can do what you wish with this list of ints ie- write them to a file. see code below.
}
Here is how you would write the Integer list to a file:
FileOutputStream out = new FileOutputStream("path/to/file.png");
for(int i : result_bytes_list){
out.write(i);
}
out.close();

node.setTextContent( base64.encodeAsString( fileBytes ) )
using org.apache.commons.codec.binary.Base64