I have a system designed around a broker such that my producer is in Java and consumer in Go.
I am using apache-pulsar as my broker
Java - Producer
MessageJava class is converted to byte array before sending it to pulsar: An object of MessageJava class calls getBytes() method defined in same class to convert it to byte[] and then this array is sent to apache-pulsar
class MessageJava {
String id;
int entityId;
Date timestamp;
public byte[] getBytes() throws Exception{
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(bos);
oos.writeObject(this);
oos.flush();
return bos.toByteArray();
}
}
My consumer is written in Go.
Go - Consumer
byte array is read from pulsar and converted to MessageGo struct using ConvertAfterReceiving method [defined below], I am using gob for decoding:
type MessageGo struct {
Id string
EntityId int
Timestamp time.Time
}
func ConvertAfterReceiving(msg pulsar.Message) *MessageGo {
payload := msg.Payload()
messageBytes := bytes.NewBuffer(payload)
dec := gob.NewDecoder(messageBytes)
var message MessageGo
err := dec.Decode(&message)
if err != nil {
logging.Log.Error("error occurred in consumer while decoding message:", err)
}
return &message
}
The issue is I am not able to decode byte[] and convert it to MessageGo struct. It shows error encoded unsigned integer out of range
I have tried changing MessageJava.entityId to short/long and MessageGo.EntityId to int8/int16/int32/int64/uint8/uint16/uint32/uint64 [all permutations], but all in vain.
A Java ObjectOutputStream and a Go Decoder do not speak the same language, even if at the base they're both made up of bytes; the same way that "these words" and "эти слова" are made up of lines yet knowing one doesn't let you know the other.
An ObjectOutputStream transforms objects into a form meant to be read by a Java ObjectInputStream, while a Go Decoder expects data in a format created by a Go Encoder.
What's needed is a language that they both speak like JSON, which both Java and Go know how to work with. Then instead of serializing the object straight into bytes, you transform it into a string representation, send the bytes of that string and convert that string in Go into the desired struct.
I have custom objects: CustomKey, CustomValue which I provided coder via Avro: CustomKeyCoder, CustomValueCoder.
Since I need to group by KV[CustomKey, CustomValue], I registered KVCoder.of(new CustomKeyCoder, new CustomValueCoder). Custom coders wraps in/out stream to data in/out stream and uses Avro Datum Writer/Reader.
Issue I am having is in the decode of the KVCoder, when we attempt to decode value part of KV I get Forbidden IOException when reading from InputStream. As noted, key part of decoding works properly, error is thrown when input stream is passed into decoding value. KVCoder reuses same input stream for both key and value I am guessing key decoding reads entire stream. Why would this be happening? Is usage of Avro a problem?
Here is some code to showcase above:
//Coder
override def decode(inputStream: InputStream): CustomValue = {
val dataInputStream = new DataInputStream(inputStream)
val id = dataInputStream.readShort
underlying.decode(dataInputStream)
}
//Underlying
override def decode(inputStream: InputStream): CustomValue = {
val decoder = DecoderFactory.get().binaryDecoder(inputStream, null)
val record = datumReader.read(null, decoder)
CustomValue.decode(record)
}
I have a binary file, in my jar, and I want to slurp its contents in binary mode, not into a string of characters. Following this example
private byte[] readBinaryFile(String fileName) throws IOException {
InputStream input = getClass().getResourceAsStream(fileName);
ByteArrayOutputStream output = new ByteArrayOutputStream();
for (int read = input.read(); read >= 0; read = input.read())
output.write(read);
byte[] buffer = output.toByteArray();
input.close ();
output.close();
return buffer;
}
It's pretty trivial, but the calling context is expecting and Object. How do I pass this binary contents back to the caller, but not as a primitive array? I am trying to deliver this binary data as a response to a web service using jaxrs.
As #Jon notes, the caller should be just fine:
byte[] b = new byte[10];
Object o = b;
That works because as he points out a byte[] is an instance of Object.
Don't confuse bytes themselves, which are indeed primitives, with the array. All arrays are objects no matter what they contain.
So the caller should receive his Object and then send it back to his caller as application/octet-stream.
I'm developing an application running on Android and iOS devices. For this app I need to get an XML stream from a URL. This XML is not really safe, because some lines, for example :
Révélation
Will become :
R�v�lation
Of course I know the best thing to do is to fix the XML generator script. But I'm only working as a developer for a firm and don't have access to it, so for the moment I'm trying to do what I can with what I have.
Now here is the reason of this topic. When I put this data in an Objective-C's NSData object :
NSData *data = [[NSData alloc] initWithContentsOfURL:[NSURL URLWithString:url]];
And then try to read every byte :
NSUInteger len = [data length];
Byte *byteData = (Byte*)malloc(len);
memcpy(byteData, [data bytes], len);
for(int i = 0 ; i < len ; i++)
{
NSLog(#"%d",byteData[i]);
}
It correctly displays the int value of the char, special character or not. Then I just have to handle (unichar)byteData[i] to solve it.
No with Java and Android, I'm trying to do a basic BufferedReader operation.
URL myURL = new URL(url);
BufferedReader in = new BufferedReader(new InputStreamReader(myURL.openStream()));
Then print every char's int one by one :
int i;
while((i = in.read()) != -1) System.out.print(i);
But with Java, by doing this I immediately get the replacement char's id (65533) instead of the good one, and can't manage to replace it.
Any idea? Thanks for reading me.
BufferedReader in = new BufferedReader(
new InputStreamReader(myURL.openStream(), "UTF-8"));
InputStreams are for bytes, binary data.
Readers are for characters, String, text.
The InputStreamReader bridges this conceptual difference, saying which encoding the binary data is in, and has an optional parameter for the encoding. The default encoding is that of the current platform - so not very portable.
I thought I would find a solution to this problem relatively easily, but here I am calling upon the help from ye gods to pull me out of this conundrum.
So, I've got an image and I want to store it in an XML document using Java. I have previously achieved this in VisualBasic by saving the image to a stream, converting the stream to an array, and then VB's xml class was able to encode the array as a base64 string. But, after a couple of hours of scouring the net for an equivalent solution in Java, I've come back empty handed. The only success I have had has been by:
import it.sauronsoftware.base64.*;
import java.awt.image.BufferedImage;
import org.w3c.dom.*;
...
BufferedImage img;
Element node;
...
java.io.ByteArrayOutputStream os = new java.io.ByteArrayOutputStream();
ImageIO.write(img, "png", os);
byte[] array = Base64.encode(os.toByteArray());
String ss = arrayToString(array, ",");
node.setTextContent(ss);
...
private static String arrayToString(byte[] a, String separator) {
StringBuffer result = new StringBuffer();
if (a.length > 0) {
result.append(a[0]);
for (int i=1; i<a.length; i++) {
result.append(separator);
result.append(a[i]);
}
}
return result.toString();
}
Which is okay I guess, but reversing the process to get it back to an image when I load the XML file has proved impossible. If anyone has a better way to encode/decode an image in an XML file, please step forward, even if it's just a link to another thread that would be fine.
Cheers in advance,
Hoopla.
I've done something similar (encoding and decoding in Base64) and it worked like a charm. Here's what I think you should do, using the class Base64 from the Apache Commons project:
// ENCODING
BufferedImage img = ImageIO.read(new File("image.png"));
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ImageIO.write(img, "png", baos);
baos.flush();
String encodedImage = Base64.encodeToString(baos.toByteArray());
baos.close(); // should be inside a finally block
node.setTextContent(encodedImage); // store it inside node
// DECODING
String encodedImage = node.getTextContent();
byte[] bytes = Base64.decode(encodedImage);
BufferedImage image = ImageIO.read(new ByteArrayInputStream(bytes));
Hope it helps.
Apache Commons has a Base64 class that should be helpful to you:
From there, you can just write out the bytes (they are already in a readable format)
After you get your byte array
byte[] array = Base64.encode(os.toByteArray());
use an encoded String :
String encodedImg = new String( array, "utf-8");
Then you can do fun things in your xml like
<binImg string-encoding="utf-8" bin-encoding="base64" img-type="png"><![CDATA[ encodedIImg here ]]></binImg>
With Java 6, you can use DatatypeConverter to convert a byte array to a Base64 string:
byte[] imageData = ...
String base64String = DatatypeConverter.printBase64Binary(imageData);
And to convert it back:
String base64String = ...
byte[] imageData = DatatypeConverter.parseBase64Binary(base64String);
Your arrayToString() method is rather bizarre (what's the point of that separator?). Why not simply say
String s = new String(array, "US-ASCII");
The reverse operation is
byte[] array = s.getBytes("US-ASCII");
Use the ASCII encoding, which should be sufficient when dealing with Base64 encoded data. Also, I'd prefer a Base64 encoder from a reputable source like Apache Commons.
You don't need to invent your own XML data type for this. XML schema defines standard binary data types, such as base64Binary, which is exactly what you are trying to do.
Once you use the standard types, it can be converted into binary automatically by some parsers (like XMLBeans). If your parser doesn't handle it, you can find classes for base64Binary in many places since the datatype is widely used in SOAP, XMLSec etc.
most easy implementation I was able to made is as below, And this is from Server to Server XML transfer containing binary data Base64 is from the Apache Codec library:
- Reading binary data from DB and create XML
Blob blobData = oRs.getBlob("ClassByteCode");
byte[] bData = blobData.getBytes(1, (int)blobData.length());
bData = Base64.encodeBase64(bData);
String strClassByteCode = new String(bData,"US-ASCII");
on requesting server read the tag and save it in DB
byte[] bData = strClassByteCode.getBytes("US-ASCII");
bData = Base64.decodeBase64(bData);
oPrStmt.setBytes( ++nParam, bData );
easy as it can be..
I'm still working on implementing the streaming of the XML as it is generated from the first server where the XML is created and stream it to the response object, this is to take care when the XML with binary data is too large.
Vishesh Sahu
The basic problem is that you cannot have an arbitrary bytestream in an XML document, so you need to encode it somehow. A frequent encoding scheme is BASE64, but any will do as long as the recipient knows about it.
I know that the question was aking how to encode an image via XML, but it is also possible to just stream the bytes via an HTTP GET request instead of using XML and encoding an image. Note that input is a FileInputStream.
Server Code:
File f = new File(uri_string);
FileInputStream input = new FileInputStream(f);
OutputStream output = exchange.getResponseBody();
int c = 0;
while ((c = input.read()) != -1) {
output.write(c); //writes each byte to the exchange.getResponseBody();
}
result = new DownloadFileResult(int_list);
if (input != null) {input.close();}
if (output != null){ output.close();}
Client Code:
InputStream input = connection.getInputStream();
List<Integer> l = new ArrayList<>();
int b = 0;
while((b = input.read()) != -1){
l.add(b);//you can do what you wish with this list of ints ie- write them to a file. see code below.
}
Here is how you would write the Integer list to a file:
FileOutputStream out = new FileOutputStream("path/to/file.png");
for(int i : result_bytes_list){
out.write(i);
}
out.close();
node.setTextContent( base64.encodeAsString( fileBytes ) )
using org.apache.commons.codec.binary.Base64