Sorry, but I think I am going mad. I have this in C++:
std::stringstream message;
protoMsg.SerializeToOstream(&message);
boost::system::error_code ignored_error;
std::cout << "Writing response to socket. Byte size " << protoMsg.ByteSize() << " Buffer " << message.str().length() << std::endl;
for(int i(0);i<(int)message.str().length();i++)
std::cout << (int)message.str().at(i);
std::cout << std::endl;
boost::asio::write(socket, boost::asio::buffer(message.str()), ignored_error);
std::cout << "Done writing." << std::endl;
Which yields this output
Writing response to socket. Byte size 88 Buffer 88
1886811891161111001113278971091012500000000320400480560640730000000081000000008859621061211611110011132115121109789710910111416691201161011141109710832114101113117101115116
Done writing.
And this in Java:
try {
System.out.println("Parsing");
int lenbytes = is.read(buffer);
System.out.println("Read bytes " + lenbytes);
for(int i=0;i<lenbytes;i++)
System.out.print(Integer.toString((int)buffer[i]));
System.out.println("");
EnvInfoProto envInfoProto = EnvInfoProto.parseFrom(buffer);
System.out.println("Done");
return envInfoProto;
} catch(IOException ignore) { System.out.println("Ex: " + ignore.toString(); }
Which yields
Parsing
Read bytes 88
1886811891161111001113278971091012500000000320400480560640730000000081000000008859621061211611110011132115121109789710910111416691201161011141109710832114101113117101115116
Ex: com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
The binary data is the same. I checked that I am using the right version of the proto files. I am a bit at a loss tbh. Any help appreciated.
You're asking the message to be parsed from the whole of buffer - and my guess is that buffer is more than 88 bytes long.
I can't remember offhand whether parseFrom allows you to specify the maximum amount of data to read, but an alternative would be:
ByteArrayIntputStream stream = new ByteArrayInputStream(buffer, 0, lenbytes);
EnvInfoProto envInfoProto = EnvInfoProto.parseFrom(stream);
Note that this still has the problem that you're assuming you can read all the data from the stream in a single call to read, which is never a good idea - but that's a very separate issue. If you're going to close the socket after writing, you could just parse stream from the socket's InputStream in the Java code, of course. If you're not going to close the stream, I would suggest writing the message length to the socket first (as a 32-bit integer) so you can then read that in Java, and read exactly the right amount of data, knowing when you're finished.
Related
My app sends small lines of text (commands to the server) or 32 bytes chunks of voice data. Right now I'm just using the socket's OutputStream's write. However, the problem is that Android Java seems to like to send the first byte by itself. Example:
Send: "Call Iron Man"
Received: "C", "all Iron Man"
To work around this splitting I prefix each line with a "throw away" character #. So the previous example is sent as:
Send: "#Call Iron Man"
Received: "#", "Call Iron Man" --> "Call Iron Man" will be used and "#" is ignored.
The problem becomes when I want to send 32 bytes of voice, it is sent as one packet of 1 byte, and then one packet of 31 byte. These 1 byte packets waste a lot of data because of the TCP/IP overhead. According to Sizing Source that means I will use (64+1) + (64+31) = 160 bytes for my 32 byte voice chunk when I could be using 64+32 = 96. That means I will be using 1.67X more LTE data than I should which (in Canada) will cost me a very pretty penny.
Is there a way to force all 32 bytes to be sent as one packet?
Here is the Android code for sending:
int totalRead = 0, dataRead;
while (totalRead < WAVBUFFERSIZE)
{//although unlikely to be necessary, buffer the mic input
dataRead = wavRecorder.read(wavbuffer, totalRead, WAVBUFFERSIZE - totalRead);
totalRead = totalRead + dataRead;
}
int encodeLength = AmrEncoder.encode(AmrEncoder.Mode.MR122.ordinal(), wavbuffer, amrbuffer);
try
{
Vars.mediaSocket.getOutputStream().write(amrbuffer, 0, encodeLength);
}
catch (Exception e)
{
Utils.logcat(Const.LOGE, encTag, "Cannot send amr out the media socket");
}
Here is the C/C++ code for receiving:
mediaRead = 0; //to know how much media is ACTUALLY received. don't always assume MAXMEDIA amount was received
bzero(bufferMedia, MAXMEDIA+1);
alarm(ALARMTIMEOUT);
do
{//wait for the media chunk to come in first before doing something
returnValue = SSL_read(sdssl, bufferMedia, MAXMEDIA-mediaRead);
if(returnValue > 0)
{
mediaRead = mediaRead + returnValue;
}
int sslerr = SSL_get_error(sdssl, returnValue);
switch (sslerr)
{
case SSL_ERROR_NONE:
waiting = false;
eventful = true; //an ssl operation completed this round, something did happen
break;
//other cases when necessary. right now only no error signals a successful read
}
} while(waiting && SSL_pending(sdssl));
alarm(0);
if(alarmKilled)
{
alarmKilled = false;
cout << "Alarm killed SSL read of media socket\n";
}
where MAXMEDIA = 1024 so there is definetly enough place for the 32 bytes of voice data
I am trying to create a simple messaging application, using a C++ client and a Java server.
It seems I can't get around because the C++ client fails to get information from the Java server, yet I can't track down the problem.
I have tried connecting with a Java client to the Java server and it works well.
I have tried connecting with the C++ Client to a simple C++ echo server I made for this purpose, and everything is going well (it reads the information).
I have tried connecting with the java client to the c++ echo server and it works well.
Keep in mind that the Java server get all the information from the client, and is responding (eg: when I try to Log in, the server gets it, logs me in, and "sends" a http response setting the cookie and displaying a welcome message, but the client never gets it).
Here is the java code which send the reply:
while ((msg = tokenizer.nextMessage()) != null)
{
System.out.println("Received \"" + msg + "\" from client");
String response = (String)protocol.processMessage(msg);
System.out.println(response); // used for testing
if (response != null)
{
clientSocket.getOutputStream().write(response.getBytes("UTF-8"));
//the Out below this line is being initialized on connection, just put it here for you to read
//Also the out.println(response) doesn't work as well, those are 2 attempts i have made
//out = new PrintWriter(new OutputStreamWriter(clientSocket.getOutputStream(), "UTF-8"), true);
//out.println(response);
}
if (protocol.isEnd(msg))
{
break;
}
}
Here is the client side code (C++):
//while (_socket.available() == 0){
// boost::this_thread::sleep(boost::posix_time::milliseconds(10));
// std::cout << "test";
//}
char reply[max_length];
size_t reply_length = boost::asio::read(_socket,boost::asio::buffer(reply,10));
//size_t reply_length = boost::asio::read(_socket, boost::asio::buffer(reply, _socket.available()));
std::cout << "Reply is:\n";
std::cout.write(reply, reply_length);
std::cout << "\n";
Note that the while above in the start of the code is used in order to wait for the response after each sent message, I have tried replacing it with a longer sleep time so I wont have to check the size of the incoming buffer, as you can see just after it I am trying to read a buffer the size of 10, just from testing, and i put the "real" read line in a comment just after it.
Any help would be appreciated.
Thank you.
EDIT - Forgot to mention that if I close the socket after sending the information passes, but doing so fails the purpose, as I am trying to keep the socket open until the client performs a log out.
EDIT #2- I have tried a diffrent method of reading, by using a delimiter char and reaing the buffer 1 char at the time, it just get stuck blocking with the empty buffer.
Here is the code for the second type of reading which i have tried:
std::string respone = "";
char ch='0';
boost::system::error_code error;
try {
while(!error && ch!='$'){
size_t tmp=0;
try {
std::cout << "going to read:";
tmp = _socket.read_some(boost::asio::buffer(&ch + tmp, 1 - tmp), error);
std::cout << "finish reading 1 char";
if (error)
throw boost::system::system_error(error);
}
catch (std::exception& e) {
std::cerr << "recv failed (Error: " << e.what() << ')' << std::endl;
}
respone.append(1, ch);
}
}
A bit silly, but apparntly my anti virus was blocking all the packets from the java server (and not the c++ for some reason), turning it off solved every thing.
I have an android java app sending bytes over a socket which is connected to a host machine running a server in Python. I need to receive these bytes as they were sent from the python socket. I see that in Python 'socket.recv' only returns a string. When I send an ASCII string from the java app, I am able to receive the data correctly in the python server, but when I send binary data using java byte, I see the data received is not same. I need to receive raw bytes in Python for my protocol to work correctly. Please point me in right direction.
Code snippet for Sending data on socket:
private void sendFrameMessage(byte[] data) {
byte[] lengthInfo = new byte[4];
Log.v(TAG, "sendFrameMessage");
for(int i=0; i<data.length; i++) {
Log.v(TAG, String.format("data[%d] = %d", i, data[i]));
}
try {
lengthInfo[0] = (byte) data.length;
lengthInfo[1] = (byte) (data.length >> 8);
lengthInfo[2] = (byte) (data.length >> 16);
lengthInfo[3] = (byte) (data.length >> 24);
DataOutputStream dos;
dos = new DataOutputStream(mSocket.getOutputStream());
dos.write(lengthInfo, 0, 4);
dos.write(data, 0, data.length);
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
Python Code on receiver side
def recvFrameMessage(self, s):
recv_count = 4;
data = s.recv(recv_count)
if data == 0:
return None
total_rx = len(data)
lenInfo = data
while total_rx < recv_count:
data = s.recv(recv_count - total_rx)
if data == 0:
return None
total_rx += len(data)
lenInfo = lenInfo + data
recv_count = self.decodeFrameLen(lenInfo)
logger.info("length = %d" % recv_count)
data = s.recv(recv_count)
total_rx = len(data)
msg = data
while total_rx < recv_count:
data = s.recv(recv_count - total_rx)
if data == 0:
return None
total_rx += len(data)
msg = msg + data
logger.info("msg = " + msg)
for i in range(0, len(msg)-1):
logger.info("msg[%d] = %s" % (i, msg[i]))
return msg
#SteveP makes good points for binary data "with some structure", but if this is a plain stream of bytes, in Python 2 simply apply the ord() function to each "character" you get from the socket. For example, if the Java end sends a NUL byte, that will show up on the Python end as the character "\x00", and then:
>>> ord("\x00")
0
To convert a whole string s,
map(ord, s)
returns a list of the corresponding 8-bit unsigned integers.
I'm assuming Python 2 here.
Reading binary data is perfectly doable, but what if the binary representation from your android app is different than the byte representation on the Python server? From the Python documentation:
It is perfectly possible to send binary data over a socket. The major
problem is that not all machines use the same formats for binary data.
For example, a Motorola chip will represent a 16 bit integer with the
value 1 as the two hex bytes 00 01. Intel and DEC, however, are
byte-reversed - that same 1 is 01 00. Socket libraries have calls for
converting 16 and 32 bit integers - ntohl, htonl, ntohs, htons where
“n” means network and “h” means host, “s” means short and “l” means
long. Where network order is host order, these do nothing, but where
the machine is byte-reversed, these swap the bytes around
appropriately.
Without code and example input/output, this question is going to be really difficult to answer. I assume the issue is that the representation is different. The most likely issue is that Java uses big endian, whereas Python adheres to whatever machine you are running it off of. If your server uses little endian, then you need to account for that. See here for a more thorough explanation on endianness.
I created a SocketChannel to a remote server to send and receive messages on Tomcat. To receive messages from a remote computer, I used a thread dedicated to task (only this thread will read from the socket, nothing else).
When some bytes are received at the SocketChannel (I keep polling the SocketChannel on non-blocking mode for new data), I first read 4 bytes to get the length of the next message, then allocate and read x bytes from the SocketChannel, which is then decoded and reconstructed into a message.
Below is my code for the receiving thread:
#Override
public void run() {
while (true) { //Don't exit thread
//Attempt to read the size of the incoming message
ByteBuffer buf = ByteBuffer.allocate(4);
int bytesread = 0;
try {
while (buf.remaining() > 0) {
bytesread = schannel.read(buf);
if (bytesread == -1) { //Socket was terminated
}
if (quitthread) break;
}
} catch (IOException ex) {
}
if (buf.remaining() == 0) {
//Read the header
byte[] header = buf.array();
int msgsize = (0xFF & (int)header[0]) + ((0xFF & (int)header[1]) << 8)
+ ((0xFF & (int)header[2]) << 16) + ((0xFF & (int)header[3]) << 24);
//Read the message coming from the pipeline
buf = ByteBuffer.allocate(msgsize);
try {
while (buf.remaining() > 0) {
bytesread = schannel.read(buf);
if (bytesread == -1) { //Socket was terminated
}
if (quitthread) break;
}
} catch (IOException ex) {
}
parent.recvMessage(buf.array());
}
if (quitthread) {
break;
}
}
}
The first bytes I received from the SocketChannel is fine, and I successfully decoded the message. However, the next time I read from the SocketChannel, the socket skipped ahead about 100 bytes, which caused the wrong bytes to be read and interpreted as length, causing everything to become corrupted.
What is wrong with the code? No other thread is reading from the SocketChannel.
Your parenthesis are off, the code is:
(0xFF & ((int)header[1] << 8))
which is always 0 (same with << 16 and << 24), my guess is you meant:
((0xFF & ((int)header[1])) << 8)
This would lead to reading not enough message bytes, also leading to a mismatch in synchronisation (as opposed to reading too many.)
Edit: now you fixed the above, I cannot see anything wrong. Could you tell us the relation between the length of the first message and the exact number of bytes that are eaten?
Based on the code shown, my only guess is that you edited some of the behaviour out of the sample shown which might influence the schannel, is the schannel referenced elsewhere?
If the line:
ByteBuffer buf = ByteBuffer.allocate(4);
would be outside of the while that would result in behaviour you describe, but in your sample code it isn't.
I presume when you say you're polling the socket in non-blocking mode you mean you're using a the "standard" Selector.select() approach?
When select returns and indicates that there's data available for reading from the socket you should only read the bytes that are available before re-entering the call to select(). If read() returns -1 it indicates that no more bytes are available for immediate reading in the buffer - It does not mean that the socket has been closed. Hence I suspect your of attempting to completely fill the buffer before returning is incorrect. Even if it does work your I/O thread will be constantly spinning whilst data arrives. In particular, it looks like you're simply ignoring a return value of -1.
Consider re-architecting your code to use a finite state machine approach. For example, I've implemented this in the past using a 3-state model: IDLE, READ_MESSAGE_LENGTH and READ_MESSAGE.
What is the best possible way to send an int through a socket in Java? Right now I'm looking at
sockout.write((byte)( length >> 24 ));
sockout.write((byte)( (length << 8) >> 24 ));
sockout.write((byte)( (length << 16) >> 24 ));
sockout.write((byte)( (length << 24) >> 24 ));
and then trying to rebuild the int from bytes on the other side, but it doesn't seem to work. Any ideas?
Thanks.
Wrap your OutputStream with a DataOutputStream and then just use the writeInt() method.
Something else which may be useful is that on the other end you can wrap our InputStream in a DataInputStream and use readInt() to read an int back out.
Both classes also contain a number of other useful methods for reading and writing other raw types.
There are other type of streams you can use, which can directly send integers. You can use DataOutputStream. Observe,
DataOutputStream out;
try {
//create write stream to send information
out=new DataOutputStream(sock.getOutputStream());
} catch (IOException e) {
//Bail out
}
out.writeInt(5);
If you are sending small amounts of data, then encoding as character data (e.g. Integer.toString(length)) and decoding at the other end is not unreasonable.