I have a system designed around a broker such that my producer is in Java and consumer in Go.
I am using apache-pulsar as my broker
Java - Producer
MessageJava class is converted to byte array before sending it to pulsar: An object of MessageJava class calls getBytes() method defined in same class to convert it to byte[] and then this array is sent to apache-pulsar
class MessageJava {
String id;
int entityId;
Date timestamp;
public byte[] getBytes() throws Exception{
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(bos);
oos.writeObject(this);
oos.flush();
return bos.toByteArray();
}
}
My consumer is written in Go.
Go - Consumer
byte array is read from pulsar and converted to MessageGo struct using ConvertAfterReceiving method [defined below], I am using gob for decoding:
type MessageGo struct {
Id string
EntityId int
Timestamp time.Time
}
func ConvertAfterReceiving(msg pulsar.Message) *MessageGo {
payload := msg.Payload()
messageBytes := bytes.NewBuffer(payload)
dec := gob.NewDecoder(messageBytes)
var message MessageGo
err := dec.Decode(&message)
if err != nil {
logging.Log.Error("error occurred in consumer while decoding message:", err)
}
return &message
}
The issue is I am not able to decode byte[] and convert it to MessageGo struct. It shows error encoded unsigned integer out of range
I have tried changing MessageJava.entityId to short/long and MessageGo.EntityId to int8/int16/int32/int64/uint8/uint16/uint32/uint64 [all permutations], but all in vain.
A Java ObjectOutputStream and a Go Decoder do not speak the same language, even if at the base they're both made up of bytes; the same way that "these words" and "эти слова" are made up of lines yet knowing one doesn't let you know the other.
An ObjectOutputStream transforms objects into a form meant to be read by a Java ObjectInputStream, while a Go Decoder expects data in a format created by a Go Encoder.
What's needed is a language that they both speak like JSON, which both Java and Go know how to work with. Then instead of serializing the object straight into bytes, you transform it into a string representation, send the bytes of that string and convert that string in Go into the desired struct.
Related
I am trying to de-serialize bytes into an object in Go, which was serialized into bytes in Java, in the following way:
//myMap is an instance of Java TreeMap<String, Object>
ByteArrayOutputStream a = new ByteArrayOutputStream();
GZIPOutputStream b = new GZIPOutputStream(a);
ObjectOutputStream c = new ObjectOutputStream(b);
c.writeObject(myMap);
c.close();
byte[] bytes = a.toByteArray()
Below are the attempts I made
step1 - uncompressed the bytes (in the variable result) using
//att is the byte array received
buf := bytes.NewBuffer(att)
reader, _ := gzip.NewReader(buf)
defer reader.Close()
result , _ := ioutil.ReadAll(reader)
step2 - read object out of uncompressed bytes - but failed
var decodedMap map[string]interface{}
d := gob.NewDecoder(bytes.NewBuffer(*result*))
err = d.Decode(&decodedMap)
if err != nil {
panic(err)
}
error = gob: encoded unsigned integer out of range
But when I convert the (byte array) result to string in Go, I see the encoded treemap details and the contents
map: �� sr java.util.TreeMap��>-%j� Lt NAMEt JOHNt AGEt 32t LOCODEsr java.lang.Long;���̏#� J valuexr java.lang.Number������ xp y
Can someone help me out here?
You can't (easily) deserialize those maps in Go, because the serialized data contains Java-specific data, data required to instantiate and initialize the original Java class (java.util.TreeMap in this case), which is obviously unknown to a Go app. Java object serialization and the encoding implemented by encoding/gob have nothing to do with each other; the former is specific to Java and the latter is specific to Go.
Instead try to serialize the Java object in a language-neutral way, e.g. to JSON, which you can decode in Go (or in any other language).
I'm building a custom file creater app with Android. I'm attempting to write the contents of a Bytebuffer, which are String members from a custom class I created, to a file in byte type. However, whenever I do so I get the contents of the file in String format. I've tried several alternatives such as using get method, BufferedOutputStream class, ByteArrayOutputStream class, DataOutputStream, Filechannel class, etc. Here is my code:
ByteBuffer byteBuffer = ByteBuffer.allocate(totalSize);
byteBuffer.put(hm.getDocID().getBytes());
byteBuffer.put(hm.getFextension().getBytes());
byteBuffer.put(hm.getMagic().getBytes());
byteBuffer.put(hm.getFversion().getBytes());
byteBuffer.put(hm.getFsize().getBytes());
byteBuffer.flip();
byte[] bablock = new byte[byteBuffer.remaining()];
BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(idHeader));
bos.write(bablock);
bos.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
Keep getting the following contents on my idHeader file:
12345678-1234-1234-1234-123456789abcjpgIDF1.00000166178
Which is all of my Strings concatenated. What I would like to do is write the same contents as bytes to the file, not as a human-readable string. What I am missing here? Any help is appreciated.
So instead of building Strings and getting byte arrays from them and putting those to the ByteBuffer and writing that to the file, you need to write the data directly. I would get rid of the ByteBuffer and use the various APIs of DataOutputStream to write the individual bytes or ints or whatever.
It being entirely unclear whether you want to write 12345678 as 8 bytes or as a 4-byte integer, for example, it is impossible to help you further without clarification.
In BluetoothChat source code i cannot understand some part of the code-
private void sendMessage(String message) {
// Check that we're actually connected before trying anything
if (mChatService.getState() != BluetoothChatService.STATE_CONNECTED) {
Toast.makeText(this, R.string.not_connected, Toast.LENGTH_SHORT).show();
return;
}
// Check that there's actually something to send
if (message.length() > 0) {
// Get the message bytes and tell the BluetoothChatService to write
byte[] send = message.getBytes();
mChatService.write(send);
// Reset out string buffer to zero and clear the edit text field
mOutStringBuffer.setLength(0);
mOutEditText.setText(mOutStringBuffer);
}
}
Here, I undersatnd that byte[] send is a array but cannot understand why i am intializing this array = message.getBytes();
May be its a very silly question but as i am a beginner so i think i should clear this part.
Java experts need your suggestion.
The 'send' has to be a byte array as the mChatService.write() method accepts byte array.
you may read a bit more on the following question:
Java Byte Array to String to Byte Array
The chat service sends binary data, the bytes.
In java text (String, char, Reader/Writer) is a black box of Unicode text, so one may combine all kinds of scripts and languages.
To get the bytes for a specific encoding one does:
String s = "...";
byte[] b = s.getBytes(s, encoding);
Those bytes are in that given encoding.
and reversed:
s = new String(b, encoding);
The version of String.getBytes() without encoding can cause an error: it uses the default, platform encoding, which differs per computer.
Best would have been to return bytes in a Unicode format, like UTF-8.
byte[] b = s.getBytes(StandardCharsets.UTF_8);
I'm wondering what the objections are to using what I'll call the 'String constructor method' to convert an InputStream into a String.
Edit: added emphasis. In particular, I'm wondering why we have to mess with Streams and Buffers and Scanners and whatnot when this method seems to work fine.
private String readStream(InputStream in) {
byte[] buffer = new byte[1024];
try {
return new String(buffer, 0, in.read(buffer));
} catch (IOException e) {
Log.d(DEBUG_TAG, "Error reading input stream!");
return "";
}
}
I've seen this other helpful post and tried the methods I could:
Method 1, Apache commons, is a no-go, since I can't use and don't want libraries right now.
Method 2, The Scanner one, looks promising, but then you'd have to be able to set delimiters in the stream, which isn't always possible, right? E.g. right now I'm using an InputStream from a web API.
Method 3, the InputStreamReader in the slurp function, didn't work either - it gives me a bunch of numbers, where I'm sending a string with all types of characters, so I may be messing something up in my encoding.
But after many Google searches, I finally found the String constructor method, which is the only one that works for me.
From comments on the thread I linked, I know there are issues with encoding in the method I'm using. I've been coding for a while now and know what encodings are and why they're around. But I still lack any knowledge about what kinds of encodings are used where, and how to detect and handle them. Any resources/help on that topic would also be very appreciated!
Here is one method using only standard libraries:
use a ByteArrayOutputStream and copy all the bytes you receive in it;
wrap this ByteArrayOutputStream's bytes into a ByteBuffer;
use a CharsetDecoder to decode the ByteBuffer into a CharBuffer;
.toString() the CharBuffer after rewinding it.
Code (note: doesn't handle closing the input):
// Step 1: read all the bytes
final ByteArrayOutputStream out = new ByteArrayOutputStream();
final byte[] buffer = new byte[8196];
int count;
while ((count = in.read(buffer)) != -1)
out.write(buf, 0, count);
// Step 2: wrap the array
final ByteBuffer byteBuffer = ByteBuffer.wrap(out.toByteArray());
// Step 3: decode
final CharsetDecoder decoder = StandardCharsets.UTF_8.newDecoder()
.onUnmappableCharacter(CodingErrorAction.REPORT)
.onMalformedInput(CodingErrorAction.REPORT);
final CharBuffer charBuffer = decoder.decode(byteBuffer);
charBuffer.flip();
return charBuffer.toString();
I have a binary file, in my jar, and I want to slurp its contents in binary mode, not into a string of characters. Following this example
private byte[] readBinaryFile(String fileName) throws IOException {
InputStream input = getClass().getResourceAsStream(fileName);
ByteArrayOutputStream output = new ByteArrayOutputStream();
for (int read = input.read(); read >= 0; read = input.read())
output.write(read);
byte[] buffer = output.toByteArray();
input.close ();
output.close();
return buffer;
}
It's pretty trivial, but the calling context is expecting and Object. How do I pass this binary contents back to the caller, but not as a primitive array? I am trying to deliver this binary data as a response to a web service using jaxrs.
As #Jon notes, the caller should be just fine:
byte[] b = new byte[10];
Object o = b;
That works because as he points out a byte[] is an instance of Object.
Don't confuse bytes themselves, which are indeed primitives, with the array. All arrays are objects no matter what they contain.
So the caller should receive his Object and then send it back to his caller as application/octet-stream.