I'm looping through the Java POJO (fetching data from the database) and then using protostuff library to covert it into byte[] and again encoding it into ProtoBuff classes so that I can send it as a content-type →application/x-protobuf
My biggest concern is in the folloiwing line, where I need to encode for each row:
for (InstrumentHistory instrumentHistory : instrumentHistoryRepository.getAllInstrumentHistorys()) {
Schema<InstrumentHistory> schema = RuntimeSchema.getSchema(InstrumentHistory.class);
LinkedBuffer buffer = LinkedBuffer.allocate(LinkedBuffer.DEFAULT_BUFFER_SIZE);
final byte[] protostuff;
try {
protostuff = ProtostuffIOUtil.toByteArray(instrumentHistory, schema, buffer);
instrumentHistoryProtos.add(InstrumentHistoryProto.InstrumentHistory.parseFrom(protostuff));
} finally {
buffer.clear();
}
}
return InstrumentHistoryProto.InstrumentHistorys.newBuilder().addAllInstrumentHistory(instrumentHistoryProtos).build();
Is there any workaround or simple way to do this?
Related
I'm trying to write some unit tests for Kafka Streams and have a number of quite complex schemas that I need to incorporate into my tests.
Instead of just creating objects from scratch each time, I would ideally like to instantiate using some real data and perform tests on that. We use Confluent with records in Avro format, and can extract both schema and a text JSON-like representation from the Control Center application. The JSON is valid JSON, but it's not really in the form that you'd write it in if you were just writing JSON representations of the data, so I assume it's some representation of the underlying AVRO in text form.
I've already used the schema to create a Java SpecificRecord class (price_assessment) and would like to use the JSON string copied from the Control Center message to populate a new instance of that class to feed into to my unit test InputTopic.
The code I've tried so far is
var testAvroString = "{JSON copied from Control Center topic}";
Schema schema = price_assessment.getClassSchema();
DecoderFactory decoderFactory = new DecoderFactory();
Decoder decoder = null;
try {
DatumReader<price_assessment> reader = new SpecificDatumReader<price_assessment>();
decoder = decoderFactory.get().jsonDecoder(schema, testAvroString);
return reader.read(null, decoder);
} catch (Exception e)
{
return null;
}
which is adapted from another SO answer that was using GenericRecords. When I try running this though I get the exception Cannot invoke "org.apache.avro.Schema.equals(Object)" because "writer" is null on the reader.read(...) step.
I'm not massively familiar with streams testing or Java and I'm not sure what exactly I've done wrong. Written in Java 17, streams 3.1.0, though flexible with version
The solution that I've managed to come up with is the following, which seems to work:
private static <T> T avroStringToInstance(Schema classSchema, String testAvroString) {
DecoderFactory decoderFactory = new DecoderFactory();
GenericRecord genericRecord = null;
try {
Decoder decoder = decoderFactory.jsonDecoder(classSchema, testAvroString);
DatumReader<GenericData.Record> reader =
new GenericDatumReader<>(classSchema);
genericRecord = reader.read(null, decoder);
} catch (Exception e)
{
return null;
}
var specific = (T) SpecificData.get().deepCopy(genericRecord.getSchema(), genericRecord);
return specific;
}
I used the following function to create a base64 encoded string of my Gravatar image (https://www.gravatar.com/avatar/cd5415f97afbe0177ba35ae31fbfd0db):
final BASE64Encoder encoder = new BASE64Encoder();
String encoded = encoder.encode(inputStreamToByteArray(is));
encoded = encoded.replaceAll("\r?\n", "");
return encoded;
I ran the method a couple of days ago and got the following base64 encoded string:
/9j/4AAQSkZJRgABAQAAAQABAAD//gA7Q1JFQVRPUjogZ2QtanBlZyB2MS4wICh1c2luZyBJSkcgSlBFRyB2ODApLCBxdWFsaXR5ID0gOTAK/9sAQwADAgIDAgIDAwMDBAMDBAUIBQUEBAUKBwcGCAwKDAwLCgsLDQ4SEA0OEQ4LCxAWEBETFBUVFQwPFxgWFBgSFBUU/9sAQwEDBAQFBAUJBQUJFA0LDRQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQU/8AAEQgAUABQAwEiAAIRAQMRAf/EAB8AAAEFAQEBAQEBAAAAAAAAAAABAgMEBQYHCAkKC//EALUQAAIBAwMCBAMFBQQEAAABfQECAwAEEQUSITFBBhNRYQcicRQygZGhCCNCscEVUtHwJDNicoIJChYXGBkaJSYnKCkqNDU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6g4SFhoeIiYqSk5SVlpeYmZqio6Slpqeoqaqys7S1tre4ubrCw8TFxsfIycrS09TV1tfY2drh4uPk5ebn6Onq8fLz9PX29/j5+v/EAB8BAAMBAQEBAQEBAQEAAAAAAAABAgMEBQYHCAkKC//EALURAAIBAgQEAwQHBQQEAAECdwABAgMRBAUhMQYSQVEHYXETIjKBCBRCkaGxwQkjM1LwFWJy0QoWJDThJfEXGBkaJicoKSo1Njc4OTpDREVGR0hJSlNUVVZXWFlaY2RlZmdoaWpzdHV2d3h5eoKDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uLj5OXm5+jp6vLz9PX29/j5+v/aAAwDAQACEQMRAD8A7j4ZeIl1PRljvrcCMgorw3Qdk78qBnb6Zzjiusv4o7nS45w3lzwuY2heQSMV/hbOBXjNtfXPhLxe088l3dWwlVpYIJdxEWCR8gGQAfavdPCGseH/AB3oyjTLiKSaVNyzs3mOrDu2R6jGCO1fk2Hr1cqzOFap8F+WXktv6+89NxVSnZHj/wATvjP4V+EotR4hvGhmulZooYkLsVHUn0H168+lebw/tleC7yVRDFcmJiB5uDgc8549xXnn7Xutaz4d8Z6PofiAt420OGRb37PqreXLDlirxo0OwKrBey8e9fcf7OP7J37Ofifwjp/irw/4dj1y0v40lMGo3klyttJgFo2jLlAyng5B6V+3xqqK+FP+vU8SSbejseZeD/iBonju3MulXQkYDcYmGGA9cfjXRstdZ+1b4K+GPwWs/C2paBo1hoHiTUdVgsoYdOIhSaE/LKWiX5doDD5gAc7cnHB5VwRwRg1UnF2cNLmkHJq0uhXkWoWqw/Sq7nFZvY0K83ArNmOWNX5zyazp25NYy2NojPFZudI1nTtTsYRLdSMLfPlq2ATyDuxx616Tph0REsbhNIsP7WuEjZ57aArsY/eX5slV3cZzxmuT8RaY2qaNNDG8cUvVZJeFU+ua6n4ezWHw88HiyjbyU3pcRXZAjWUsvzcnhmcqfpkYHGa/NuIaVHDYr2s/tdF1umn0en+Z0YeTnT5ex86/Ff8AZ+8Y/tRfG3Wz4Z0+OHStIEWmT6pNMEtklSPc4LYyzb2YfKCenarul/s/eOf2VLK7udJ+LI0+5lw76dplq8y3EgB2g5HTJxkrjkntX1D8OfEx8O/CHS/LjFtNqN1d3l0I+SJpbiVn3H+8Cdp9NuKxfitb2vivwBLcwzeXdWrl4Z9wDK3v1yp9K9PDZnX9tTwVLSMUle127JdX6HJUpRUXUe58t2XgnU/iJ8QLLxR8Q/GT+IruCRJI4F34HQqnzAbVz1Cjt719CTne5cYIbn5eleD2vjxb5Uju/LM6nCSqQenrj6GvT/BfjK31m3NnMPLmAGwlcHJGfyznn39q9NYnEUsYp15Xi1ba1tf8zGDjy2RvyniqshxU0zEMQRgjjBqpK1fTN3LIJm4NZ0p5q5O3HvVKY1lM2idrGoKkMoZT1B6GvKPEnjrUvBl241jVQumvGFgjlgKQRSbyCS6nOQpyByTgdB19ZQ5HFeL/ABS1CHRNemuNb0WbVbCWSK3s7p5P3NkSC7MFKsDkryuBnOCSK8jM8FDFypTlG/K/6/Ezo1HC9j1HwH4p+1+Af7Pub5LzUVZtQDecZGkjldm3EnkHfvBB5BA9RXN+NPGl1L4I1/w9HNJZy36D7NeQgFoJBgHIPVSOo46cGsPSPjv4Q1HSYbmfV2g1ESO0apYFFuFJIkjdB91mO3BXIBReBk5f4rt0nQPGjrvAfZIpV146EHkGvi8cq2WY32kPglqnvZ9V/X6HTHlqx13PAvCV7o+m+IZdO8WXOoWjRDIGmQq0kpONpXewAU/j074zX0B4b8efDee0msdB0DXJNTGVS61K8jDRKQBvVVBztPOORyc+leVeJvCula9EttqcMgmXJiuoW2yRH1Uj+VcVLpWp+EtlxPcST28bDy7+3AEkR7GRVOQP9pfXkCvrsPmNHGwUau/4HBOi6b90+trPVrm+maC9WM3UaDEsSlfNUBfmZcna3I6Eg8EEg1LKpFcH8JviQvikLpV/NFb64YTDZXwbEF4pO4RMR6kBgCOD8y9WDegK63EKyLnawyMgg/kele3SleNuxUXdGfPmqUvNak0QOaozRYzVSTNU7HXxvjg9KzPE/jvSdN8QaNpl5pch02ZD9q86JJY5FG4ZDblbcdwyMcADBrQArkPivp2pX/hZf7Nd0khmEkuwDJi2sGGeoHIzjrjuK58WpujJ03ZoxpuzsePeL9Js/EscCpp7WFg1286G+XZEoyCoJU5RmA6bjz25OPUJvDaeJPAFpL4h1KWxurNT5etRP5UjRq3yPICSDlcAg5z14JrwTx18XNai1ia9i12dbVFMZsZ5i0ZAJyhThcDkfKB2r1H4NeP7Pxo9kPEWItXMatptqUC2oUA5MYAC+ZweTz2B7V4+BVGo5q10901pf8f8/M2lzIl8LeCNY8QTTfb8nS1YLDfXFv5Es692EW4kf8Cxnr7VvXvwYtTG3karI67SvkzJwR6d69GbNQSZxXRHJcJCXPBNPybt92qIdVtWZ8pa18IPF3gG8lv9FRNQst5d9PcEDbnOVI6EcnH4jnp714D8ZyeLtAgmvIZbbUUQCSKcgsw9cjhvTI9u+a6t2IrktSspvD9019p6O+nyuHvLKLqP+mkYHf1UD1wDyretQoqm/el8/wDP/Mwcn0N+SQiqNxLwabFfxXao0UqTRSr5kMyEFZU9R7jIyO2R1BBMc4JNdMouO41K+p3axDFO8oEEEZHoasLEKf5dZXLsfKPxZ+GVlfeMrmxFjiBvmVjIAHZlJznPB5xgjGBWD4d+Gt5oeq2l5oms2cJjuFnitLpyQcbWYg9uADgAn5RwK+ivi9o8j6db6jbxgyxN5Urs+1VQ8hm4PAOfT72OQa86svDGm3FpHcXEz3+pO8bxf6RujR1fIaRVQuUGOQOoI46Gvgsfia2Aqypxdo9Pn/W/kdtOKnG57mgE0Ecisrh1DBk6HPce1RPHioPCt9HqOmsFuorp43OfLcNhTyue447MAeOlackXtX2uFrrE0IVVpdf8P+JxTg4yaMiWPFVJIyDWxLD7VTlhxXXcyOB8ReHrm0El3pTMil/Ne2QZ2SdfNjHqedydHBP8RO6Tw34ii8R2sn3Uu4CFmiU5APZl9VPb8R1BrsJIPSudj8H29p4kk1i3/cyTxGOeID5XOc7vY+v+JJovpYLH/9k=
Just today I executed the method again, using the same image. But instead of retrieving the same encoded string, I received:
/9j/4AAQSkZJRgABAQAAAQABAAD//gA7Q1JFQVRPUjogZ2QtanBlZyB2MS4wICh1c2luZyBJSkcgSlBFRyB2NjIpLCBxdWFsaXR5ID0gOTAK/9sAQwADAgIDAgIDAwMDBAMDBAUIBQUEBAUKBwcGCAwKDAwLCgsLDQ4SEA0OEQ4LCxAWEBETFBUVFQwPFxgWFBgSFBUU/9sAQwEDBAQFBAUJBQUJFA0LDRQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQU/8AAEQgAUABQAwEiAAIRAQMRAf/EAB8AAAEFAQEBAQEBAAAAAAAAAAABAgMEBQYHCAkKC//EALUQAAIBAwMCBAMFBQQEAAABfQECAwAEEQUSITFBBhNRYQcicRQygZGhCCNCscEVUtHwJDNicoIJChYXGBkaJSYnKCkqNDU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6g4SFhoeIiYqSk5SVlpeYmZqio6Slpqeoqaqys7S1tre4ubrCw8TFxsfIycrS09TV1tfY2drh4uPk5ebn6Onq8fLz9PX29/j5+v/EAB8BAAMBAQEBAQEBAQEAAAAAAAABAgMEBQYHCAkKC//EALURAAIBAgQEAwQHBQQEAAECdwABAgMRBAUhMQYSQVEHYXETIjKBCBRCkaGxwQkjM1LwFWJy0QoWJDThJfEXGBkaJicoKSo1Njc4OTpDREVGR0hJSlNUVVZXWFlaY2RlZmdoaWpzdHV2d3h5eoKDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uLj5OXm5+jp6vLz9PX29/j5+v/aAAwDAQACEQMRAD8A7j4ZeIl1PRljvrcCMgorw3Qdk78qBnb6Zzjiusv4o7nS45w3lzwuY2heQOxX+Fs4FeM219c+EvF7TzyXd1bCVWlggl3ERYJHyAZAB9q908Iax4f8d6Mo0y4ikmlTcs7N5jqw6Fsj1GMEdq/JsPXq5XmcK1T4L8r8uh6biqlOyPH/AInfGjwr8JRajxDeNDNdKzRQxRl2KjqTjoPr159K83h/bK8F3koEMNyYmIHm4OBzznj3Feefte61rPh3xno+h+IC3jbQ4ZFvfs+qt5csOWKvGjQ7AqsF7Lx719x/s4/snfs5+J/COn+KvD/h2PW7S/jSU2+o3klyttJgFo2jLlAyng5B6V+3xqqK+FP+vU8SSbejseZeD/iBonju3MulXQkYDJiYYYD1x+NdGy11n7Vvgr4Y/Baz8LaloGjWGgeI9R1WCyhh04iFJoT8spaJflwAw+YAHO3JxweVcEcEYNVJxdnHqaQcmrSK8i1C1WH6VXc4rN7GhXm4FZsxyxq/OeTWdO3WsJbG0Rnis3Okazp2p2MIlupGFvny1bAJ5B3Y49a9J0w6IiWNwmkWH9rXCRs81tAV2MfvL82Sq54znjNcn4i0xtU0aaGN44peqyS8Kp9c11Pw9msPh54OFlG3kpvSeK7IEayll+bk8MzFT9MjA4zX5vxDSo4bE+0n9rout9H8v8zfDyc6fL2PnX4r/s/eMf2ovjbrZ8M6fHDpWkCLTJ9UmmCWySpHucFsZZt7MPlBPTtV3S/2fvHP7Klld3Ok/Fkafcy4d9O0y1eZbiQA7Qcjpk4yVxyT2r6g+HPiY+HfhDpflxi2m1G6u7y68vkiaW4lZ9x/vAnafTbisb4rW9r4q8AS3MM3l3Vq5eGfcAyt79cqfSvTw2Z1/bU8FS0jFJXtduy7vuctSlFRdR7ny3ZeCdT+InxAsvFHxD8Zv4iu4JEkjgXfgdCqfMF2rnqFHb3r6EnIdy4wQ3Py9K8HtfHi3ypHd+WZ1OElUg9PXH0Nen+C/GVvrNubOYeXMANhK4OSM/lnPPv7V6axNeljFKvK8WrbWsYwceWyN+U8VVkOKmmYhiCMEcYNVJWr6Zu5ZBM3BrOlPNXJ2496pTGspm0TtY1BUhlDKeoPQ15R4k8dal4Mu3GsaqF014wsEcsBSCKTeQSXU5yFOQOScDoOvrKHI4rxf4pahDomvT3Gt6LNqthLJFb2d08n7myJBdmClWByV5XAznBJFeRmeDhi5U5yjflf9fkZ0ZuF7HqPgPxSbvwD/Z9zfJeairNqAbzzI0kcrs24k8g794IPIwPUVzfjTxpdS+CNf8PRzPZy36A215CAWgkGAcg9VI6jjpwaw9I+O/hDUdJhuZtXaDURI7RqlgUW4UkiSN0H3WY7SCuQCi8DJy/xXbpOgeNHXeA+yRSrrx0IPINfF45VssxntI/BLVPez6o6Y8tWOu54F4SvdH03xDLp3iy51C0aIZA0yFWklJxtK72ACn8enfGa+gPDfjz4bz2k1joOg65LqYysd1qV5GGiUgDeqqDnaeccjk59K8q8TeFdK16JbbU4ZBMuTFdQsVkiPqpH8q4qXStT8JbLie4knt42Hl39uAJIj2Miqcgf7S+vIFfXYbMaWNharucE6Lpv3T62s9Wub6ZoL1YzcxoMSxKV81QF+ZlydrcjoSDwQSDUsqkVwfwm+JC+KQulX80VvrhhMNlfBsQXik7hExHqQGAI4PzL1YN6ArrcQrIudrDIyCD+R6V7dKV427FRdzPnzVKXmtSaIHNUZosZqpI1TsdfG+OD0rM8T+O9J03xBo2mXmlyHTZkP2rzokljkUbhkNuVtx3DIxwAMGtACuQ+K+n6lf8AhZf7Md0khmEkuwDJi2sGGeoHIzjrjuK58Wpui3TdmjGm7M8e8X6TZ+JY4FTT2sLBrt50N8uyJRkFQSpyjMB03HntyceoTeG08SeALSTxDqUtjdWany9aifypGjVvkeQEkHK4BBznrwTXgnjr4ua1FrE17Frs6WsamM2M8xaMgE5QpwuByPlA7V6j8GvH9n40eyHiPEWr+WrabalAtqFAOTGAAvmfKeTz2B7V4+CVKo5q10901pf8TaV0SeFvBGseIJpvt+TpasFhvri38iWde7CLcSP+BYz19q3734MWpjbyNVkddpXyZk4I9O9ejNmoJM4reOTYSEueCafk3+WxDqtqzPlLWvhB4u8A3kt/oqJqFlvLvp7ggbQc5UjoRycfiOenvXgPxnJ4u0CCa8hlttRVAJIpyCzD1yOG9Mj275rq3YiuS1Kym8P3TX2no76fK4e8souo/wCmkYHf1UD1wDyrevQoqm/el8/8/wDMwcmtjflkIqjcS8GmxX8V2qNFKk0Uq+ZDMhBWVPUe4yMjtkdQQTHOCTXTKLjuNSO7WIYpxiBBBGR6GrCxCn+XWVy7Hyj8WfhlZX3jK4sRY4gb5lYyAB2ZSc5zwecYIxgVg+HfhreaHqtpeaJrNnAY7hZ4rS6ckHG1mIPbgA4AJO0cCvor4vaPI+m2+o28YMsTeVK7PtVUPIZuDwDnnj72OQa86svDGm3FpHcXEz3+pO8bxf6RujR1fIaRVQuUGOQOoI46Gvgsfia2Aqypxfu9PmdtOKmrnuaATQRyKyuHUMGToc9x7VE8eKg8K30eo6awW6iunjc58tw2FPK57j6MAeOlackXtX2uFrrE0Y1V1RxTg4yaMiWPFVJIyK2JYfaqcsOK67mRwXiLw9c2gku9KZkUv5r2yDOyTr5sY9TzuTo4J7k7n+G/EUXiO1k+6l3AQs0SnIB7Mvqp7fiOoNdhJB6VzsfhC3tPEkmsW/7mSeIxzxAfK5znd7HPX/Ek0X0sFj//2Q==
As you may notice, the first couple of characters are the same, but if you look at the end of the strings, you notice directly that they are completely different. Why is that? The image is still my Gravatar image (e.g., using http://codebeautify.org/base64-to-image-converter shows the same image for both strings).
Thanks for any hint, explanation! Is there anything wrong with my implementation? Is some of the implementation time or location dependent? How can I get the same base64 string for the same image?
PS: inputStreamToByteArray is implemented as follows:
public static byte[] inputStreamToByteArray(final InputStream is) {
final ByteArrayOutputStream buffer = new ByteArrayOutputStream();
final byte[] data = new byte[16384];
try {
int nRead;
while ((nRead = is.read(data, 0, data.length)) != -1) {
buffer.write(data, 0, nRead);
}
buffer.flush();
} catch (final IOException e) {
return null;
} finally {
try {
buffer.close();
} catch (final IOException e) {
// ignore
}
}
return buffer.toByteArray();
}
Converting both images back to .jpg, and using http://regex.info/exif.cgi, the following header comments appear:
One:
CREATOR: gd-jpeg v1.0 (using IJG JPEG v80), quality = 90
Other:
CREATOR: gd-jpeg v1.0 (using IJG JPEG v62), quality = 90
(can't remember which order I converted the images in, but there is a version change either way)
As you can see, they have upgraded the JPEG compressor (or spec?) to a newer version, resulting in a different JPEG encoding, or at least difference in the comments.
So to answer your question, there does not seem to be an issue with your base64 converter, but just that the two images actually are different
I have a huge string that I need to cache somewhere and since I cannot write to file my only option is to store this on the data base as text, more specifically, in the clob I have I'm storing a JSON file where I'm placing the compressed string under a certain key of that JSON object.
I'm compressing the strings but somewhere across the string manipulation something happens that doesn't allow me to decompress the data, so I'm wondering if I should encode the data to base 64 but that will lose compression.
What could I do to ensure I can store the compressed string in the database so I can later fetch it?
I cannot change the database, so I'm stuck with that CLOB field
These are my compression functions:
public static String compress(String text) {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
try {
OutputStream out = new DeflaterOutputStream(baos);
out.write(text.getBytes("UTF-8"));
out.close();
} catch (IOException e) {
//ooops
}
return baos.toString();
}
public static String decompress(String bytes) {
InputStream in = new InflaterInputStream(new ByteArrayInputStream(bytes.getBytes()));
ByteArrayOutputStream baos = new ByteArrayOutputStream();
try {
byte[] buffer = new byte[8192];
int len;
while ((len = in.read(buffer)) > 0)
baos.write(buffer, 0, len);
return new String(baos.toByteArray(), "UTF-8");
} catch (IOException e) {
//ooops
}
}
As you found out, you can't store binary data in a CLOB without some corruption, so encoding to text will be required.
Base 64 will, on average add 33% to the size of your binary data. So you will lose some compression, but if your compression ratio is greater than 25% (this is often easy with particular types of text strings), then compression followed by base 64 encoding may provide you with a net storage gain. Lots of CPU use though.....
You can't convert arbitrary binary data to a String without breaking it. As you've already stated, if you want to store the data in a clob, you need to base64 encode the data (or use some other valid binary to text encoding).
Have you thought of other solutions, such as using memcached or other caching system? Or do you really want to mess around with compression?
This is an example XML file that would come from the Android Client.
<test>
<to>Mee</to>
<from>Youuu</from>
<img src="http://www.domain.com/path/to/my/image.jpg" />
</test>
I have written a XML parser about this. My problem is while passing it to the Android Client, I need to have the image binary data instead of the image path. How can I accomplish this and how can I update the above said XML with the binary data.
You could use Base64 to encode your image binary data (represented by a byte[]) and include it in the xml as CDATA.
Then on the Android machine, you just decode it to a byte array, and render the image.
You can use Apache Commons to encode/decode.
Edit:
You need to get a byte representation of the image data in order to convert it. See my example. This is using sun.misc.BASE64Decoder and sun.misc.BASE64Encoder, you may need to adapt depending on what you have at your disposal on Android (see Apache Commons).
public class SO11096275 {
public static byte[] readImage(URL url) throws IOException {
final ByteArrayOutputStream bais = new ByteArrayOutputStream();
final InputStream is = url.openStream();
try {
int n;
byte[] b = new byte[4096];
while ((n = is.read(b)) > 0) {
bais.write(b, 0, n);
}
return bais.toByteArray();
} finally {
if (is != null) {
is.close();
}
}
}
public static void main(String[] args) throws Exception {
URL url = new URL("http://upload.wikimedia.org/wikipedia/en/b/bc/Wiki.png");
byte[] imgData = readImage(url);
String imgBase64 = new BASE64Encoder().encode(imgData);
System.out.println(imgBase64);
byte[] decodedData = new BASE64Decoder().decodeBuffer(imgBase64);
FileUtils.writeByteArrayToFile(new File("/path/to/wikipedia-logo.png"), decodedData); // apache commons
}
}
Then you have your image data as a string in imgBase64, you just have to append a node to your xml using the DOM implementation you want, for example dom4j. There are methods to add CDATA to the XML. Finally, on your Android, you just need to retrieve the node content and you're good to decode it like above and do what you want with the image.
XML-alternatives like JSON, Protocol Buffer could help you.
I thought I would find a solution to this problem relatively easily, but here I am calling upon the help from ye gods to pull me out of this conundrum.
So, I've got an image and I want to store it in an XML document using Java. I have previously achieved this in VisualBasic by saving the image to a stream, converting the stream to an array, and then VB's xml class was able to encode the array as a base64 string. But, after a couple of hours of scouring the net for an equivalent solution in Java, I've come back empty handed. The only success I have had has been by:
import it.sauronsoftware.base64.*;
import java.awt.image.BufferedImage;
import org.w3c.dom.*;
...
BufferedImage img;
Element node;
...
java.io.ByteArrayOutputStream os = new java.io.ByteArrayOutputStream();
ImageIO.write(img, "png", os);
byte[] array = Base64.encode(os.toByteArray());
String ss = arrayToString(array, ",");
node.setTextContent(ss);
...
private static String arrayToString(byte[] a, String separator) {
StringBuffer result = new StringBuffer();
if (a.length > 0) {
result.append(a[0]);
for (int i=1; i<a.length; i++) {
result.append(separator);
result.append(a[i]);
}
}
return result.toString();
}
Which is okay I guess, but reversing the process to get it back to an image when I load the XML file has proved impossible. If anyone has a better way to encode/decode an image in an XML file, please step forward, even if it's just a link to another thread that would be fine.
Cheers in advance,
Hoopla.
I've done something similar (encoding and decoding in Base64) and it worked like a charm. Here's what I think you should do, using the class Base64 from the Apache Commons project:
// ENCODING
BufferedImage img = ImageIO.read(new File("image.png"));
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ImageIO.write(img, "png", baos);
baos.flush();
String encodedImage = Base64.encodeToString(baos.toByteArray());
baos.close(); // should be inside a finally block
node.setTextContent(encodedImage); // store it inside node
// DECODING
String encodedImage = node.getTextContent();
byte[] bytes = Base64.decode(encodedImage);
BufferedImage image = ImageIO.read(new ByteArrayInputStream(bytes));
Hope it helps.
Apache Commons has a Base64 class that should be helpful to you:
From there, you can just write out the bytes (they are already in a readable format)
After you get your byte array
byte[] array = Base64.encode(os.toByteArray());
use an encoded String :
String encodedImg = new String( array, "utf-8");
Then you can do fun things in your xml like
<binImg string-encoding="utf-8" bin-encoding="base64" img-type="png"><![CDATA[ encodedIImg here ]]></binImg>
With Java 6, you can use DatatypeConverter to convert a byte array to a Base64 string:
byte[] imageData = ...
String base64String = DatatypeConverter.printBase64Binary(imageData);
And to convert it back:
String base64String = ...
byte[] imageData = DatatypeConverter.parseBase64Binary(base64String);
Your arrayToString() method is rather bizarre (what's the point of that separator?). Why not simply say
String s = new String(array, "US-ASCII");
The reverse operation is
byte[] array = s.getBytes("US-ASCII");
Use the ASCII encoding, which should be sufficient when dealing with Base64 encoded data. Also, I'd prefer a Base64 encoder from a reputable source like Apache Commons.
You don't need to invent your own XML data type for this. XML schema defines standard binary data types, such as base64Binary, which is exactly what you are trying to do.
Once you use the standard types, it can be converted into binary automatically by some parsers (like XMLBeans). If your parser doesn't handle it, you can find classes for base64Binary in many places since the datatype is widely used in SOAP, XMLSec etc.
most easy implementation I was able to made is as below, And this is from Server to Server XML transfer containing binary data Base64 is from the Apache Codec library:
- Reading binary data from DB and create XML
Blob blobData = oRs.getBlob("ClassByteCode");
byte[] bData = blobData.getBytes(1, (int)blobData.length());
bData = Base64.encodeBase64(bData);
String strClassByteCode = new String(bData,"US-ASCII");
on requesting server read the tag and save it in DB
byte[] bData = strClassByteCode.getBytes("US-ASCII");
bData = Base64.decodeBase64(bData);
oPrStmt.setBytes( ++nParam, bData );
easy as it can be..
I'm still working on implementing the streaming of the XML as it is generated from the first server where the XML is created and stream it to the response object, this is to take care when the XML with binary data is too large.
Vishesh Sahu
The basic problem is that you cannot have an arbitrary bytestream in an XML document, so you need to encode it somehow. A frequent encoding scheme is BASE64, but any will do as long as the recipient knows about it.
I know that the question was aking how to encode an image via XML, but it is also possible to just stream the bytes via an HTTP GET request instead of using XML and encoding an image. Note that input is a FileInputStream.
Server Code:
File f = new File(uri_string);
FileInputStream input = new FileInputStream(f);
OutputStream output = exchange.getResponseBody();
int c = 0;
while ((c = input.read()) != -1) {
output.write(c); //writes each byte to the exchange.getResponseBody();
}
result = new DownloadFileResult(int_list);
if (input != null) {input.close();}
if (output != null){ output.close();}
Client Code:
InputStream input = connection.getInputStream();
List<Integer> l = new ArrayList<>();
int b = 0;
while((b = input.read()) != -1){
l.add(b);//you can do what you wish with this list of ints ie- write them to a file. see code below.
}
Here is how you would write the Integer list to a file:
FileOutputStream out = new FileOutputStream("path/to/file.png");
for(int i : result_bytes_list){
out.write(i);
}
out.close();
node.setTextContent( base64.encodeAsString( fileBytes ) )
using org.apache.commons.codec.binary.Base64