I am reading TCP packets that I use to display an image. Each packet is supposed to have a 1025 length. And each time, I get 128 lines, I draw an image.
SO I begin by initializing
s = new Socket(ip, port);
stream = s.getInputStream();
byte[] tmpBuffer = new byte[1025];
byte[] finalBuffer = new byte[128 * 1025];
int count_lines =0
and then I read stream by stream of length 1025,
while((read = stream.read(tmpBuffer, 0, tmpBuffer.length)) != -1){
System.arraycopy(tmpBuffer, 1, finalBuffer, count_lines * 1025, 1025);
count_lines++;
if(count_lines == 128) break;
}
The problem is when I log the read integer, I get a bunch of 1025 but sometimes (apparently randomly) 423 or 602 (noticing that 423+602=1025)
Am I going wrong with the TCP reading or is there a problem on the server side ?
Am I going wrong with the TCP reading or is there a problem on the server side ?
In TCP you have only a stream of bytes. You have messages nor control over packets which are typically no more than 1532 bytes long.
You have to have your own protocol to handle sending of messages.
inputStream.read(buffer) will read between 1 byte and buffer.length and you can't ignore the actual length read as you are doing.
If you want to read into a buffer of a known length, you can use
DataInputStream dis = new DataInputStream(socket.getInputStream());
byte[] finalBuffer = new byte[128 * 1025];
dis.readFully(finalBuffer);
The readFully reads as much as it can at a time until the byte[] has been fully read.
A more flexible protocol is to send the length first.
public static void write(DataOutputStream out, byte[] bytes) {
out.writeInt(bytes.length());
out.write(bytes);
}
public static byte[] read(DataInputStream in) {
int length = in.readInt();
byte[] bytes = new byte[length];
in.readFully(bytes);
return bytes;
}
This way, whatever length of data you send, you will recieve in the same manner.
Related
Trying to send a byte[] through TCP Socket in Java, it sends a specific array as two separate messages, while other arrays are sending as one message.
In more details, I convert a hex String to a byte[] using the following function:
public static byte[] hexStringToByteArray(String s) {
byte[] b = new byte[s.length() / 2];
for (int i = 0; i < b.length; i++) {
int index = i * 2;
int v = Integer.parseInt(s.substring(index, index + 2), 16);
b[i] = (byte) v;
}
return b;
}
And then I send it through Java TCP Socket:
ServerSocket server = new ServerSocket(3030);
Socket socket = server.accept();
OutputStream out = socket.getOutputStream();
// sample hex string
String msg = "F0700F8000F42400001544952414E00000000000000000000000662000E00060000";
out.write(HexUtiles.hexStringToByteArray(msg));
In debugging the code I found out it separate the array in byte number 1024. Also, increasing and decreasing the socket buffer size made no differences.
In addition, there is no 0A (Hex String of \n) in the message! I guess there is some strange behavior in the write method that sends a byte[] array as two messages! How can I send the byte[] as only one message?
There are no "messages" in TCP. Everything is sent as a stream and it's undefined how it will be chunked on the other end. Chunks might be split or joined together.
Send the byte size before each array using DataOutputStream. Read the size on the other end using DataInputStream and then that many bytes. This way you will be able to recreate your arrays at the other end.
Another option is to use ObjectOutputStream and ObjectInputStream and send the arrays as objects.
I have a Java application that is reading data from a TCP socket that is receiving XML of varying size. The first 5 bytes of a given packet are supposed to indicate the size of the remaining message. I can read the message and xml successfully if I manually create a large byte[] and read the data.
Here are the instructions from the manual for the application that is generating the data:
Each message is preceded by the message size indicator which is a
32-bit unsinged integer using the network bytes order method. For
example: \x05\x00\x00\x00\x30\x31\x30\x32\x00 indicates the message
size of an ack which is 5 bytes included the fifth message byte '\0'. The
size indicator specifies everything following the size indicator
itself.
However I can't figure out how to decode the first 5 bytes to an integer that I can use to correctly size a byte[] for reading the rest of the message. I get random results:
Here is the code I'm using to parse the message:
DataOutputStream out = new DataOutputStream(clientSocket.getOutputStream());
BufferedInputStream inFromServer = new BufferedInputStream(clientSocket.getInputStream());
byte[] data = new byte[10];
inFromServer.read(data);
String result = new String(data, "ISO-8859-1");
Logger.info(data+"");
//PROBLEM AREA: Tried reading different byte lengths but no joy
//This should be a number but it never is. Often strange symbols
byte[] numeric = Arrays.copyOfRange(data,1,5);
String numericString = new String(numeric, "ISO-8859-1");
//Create a huge array to make sure everything gets captured.
//Want to use the parsed value from the start here
byte[] message = new byte[1000000];
inFromServer.read(message);
//This works as expected and returns correctly formatted XML
String fullMessage = new String(message, "ISO-8859-1");
Logger.info("Result "+result+ " Full message "+fullMessage);
The length looks like it's little endian. You can still use DataInputStream but you have to swap the bytes. If you used NIO's SocketChannel and a ByteBuffer you could set the byte order, but this is likely to be harder to use.
// only do this once per socket.
DataInputStream in = new DataInputStream(
new BufferedInputStream(clientSocket.getInputStream()));
// for each message.
int len0 = in.readInt();
int len = Integer.reverseBytes(len0);
assert len < 1 << 24;
byte[] bytes = new byte[len];
in.readFully(bytes);
String text = new String(bytes, "ISO-8859-1").trim();
int number = Integer.parseInt(text);
Network byte order is aka big-endian. But seeing your data it seems, that actually little-endian is used. At least 5 will look like those first 4 bytes in little-endian, but not in big-endian. So you need to read those bytes, consider little-endian and convert to long to consider "unsigned-ness".
public static void main(String[] args) throws IOException {
DataInputStream inFromServer = new DataInputStream(new BufferedInputStream(null));
int iSize = inFromServer.readInt();
iSize = Integer.reverseBytes(iSize); //read as little-endian
long count = Integer.toUnsignedLong(iSize); //unsigned int
}
Hi I have created a server socket for reading byte array from socket using getInputStream, But getInputStream.read is not exiting after endof data reaches. Below is my code.
class imageReciver extends Thread {
private ServerSocket serverSocket;
InputStream in;
public imageReciver(int port) throws IOException
{
serverSocket = new ServerSocket(port);
}
public void run()
{
Socket server = null;
server = serverSocket.accept();
in = server.getInputStream();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte buffer[] = new byte[1024];
while(true){
int s = 0;
s = in.read(buffer); //Not exiting from here
if(s<0) break;
baos.write(buffer, 0, s);
}
server.close();
return;
}
}
From the client if I sent 2048 bytes, the line in.read(buffer) should return -1 after reading two times, but it waiting there to read for the third time. How can I solve this ?
Thanks in advance....
Your server will need to close the connection, basically. If you're trying to send multiple "messages" over the same connection, you'll need some way to indicate the size/end of a message - e.g. length-prefixing or using a message delimiter. Remember that you're using a stream protocol - the abstraction is just that this is a stream of data; it's up to you to break it up as you see fit.
See the "network packets" in Marc Gravell's IO blog post for more information.
EDIT: Now that we know that you have an expected length, you probably want something like this:
int remainingBytes = expectedBytes;
while (remainingBytes > 0) {
int bytesRead = in.read(buffer, 0, Math.min(buffer.length, remainingBytes));
if (bytesRead < 0) {
throw new IOException("Unexpected end of data");
}
baos.write(buffer, 0, bytesRead);
remainingBytes -= bytesRead;
}
Note that this will also avoid overreading, i.e. if the server starts sending the next bit of data, we won't read into that.
If I send 2048 bytes, the line 'in.read(buffer)' should return -1 after reading two times.
You are mistaken on at least two counts here. If you send 2048 bytes, the line 'in.read(buffer)' should execute an indeterminate number of times, to read a total of 2048 bytes, and then block. It should only return -1 when the peer has closed the connection.
I have this TCP server running Because UDP server was able to accept large packets but ISP blocks my UDP packets even i have Public IP and Static services. But now decided to change it to TCP but i have a large stack now to replace with UDP to TCP.
Here is the server which is running but it does not at once receive a large packet, how can i increase it to unlimited or maximum size or etc?
1) Is there any way with this?
public void run()
{
while(true)
{
byte[] buf=new byte[5024]; <<< not helping!!
int bytes_read = 0;
try {
bytes_read = sockInput.read(buf, 0, buf.length);
String data = null;
data = new String(buf, 0, bytes_read);
System.out.println("[TCP]: incomeing data: " + bytes_read + " bytes, data=" +data);
}
}
2) Is there any way with this?
public TCPHandler(Socket sock) throws IOException
{
sock.setReceiveBufferSize(10*1024 +1024); //<<<<< not helping !!!
sock.setSendBufferSize(10*1024+1024);
this.sock = sock;
sockInput = sock.getInputStream();
sockOutput = sock.getOutputStream();
this.myThread = new Thread(this);
}
None is allowing me to handle large so that i can switch from UDP to TCP. Any ideas!!
TCP and UDP are different. TCP is not a packet-based protocol, it is a stream based protocol. Your data will arrive in order, and un-corrupted, but it will not necessarily all arrive at once.
What you should do is prefix each set 'packet' with a length. When you receive data, first read the length value, and then continue reading chunks from the socket until you have the amount of data required.
You shouldn't care how TCP splits your messages into packets and how many times you must read to get a full message. Just read until the end of the message has been reached, and then return all the bytes of the message.
Knowing when the message has been read completely depends on your protocol. You could choose to send only one message per connection, or you could send the length of the message before the message, or use some marker byte.
Note that you shouldn't use new String(byte[]) without specifying an encoding (and use the same when calling String.getBytes()), especially if you transfer from one machine to another, since both machines could have a different default encoding.
The trouble you're having is how InputStreams work. When you call read(buf,offset,length) it doesn't always read length bytes everytime. It simply reads what is available() on the InputStream, and returns the number of bytes it read. So you need to continue to call read until you've actually read length bytes.
int len = 0;
int expectedLength = sockInput.readInt();
byte[] buf = new byte[ expectedLength ];
while( len != buf.length ) {
len += sockInput.read( buf, len, buf.length );
}
// now you have a buffer with expectedLength data in it.
You can also look at using DataInputStream.readFully() by wrapping your
DataInputStream ds = new DataInputStream( sock.getInputStream() );
ds.readFully( buf, 0, buf.length );
http://docs.oracle.com/javase/6/docs/api/java/io/DataInputStream.html
But if you're really just after a String you should do this:
Reader reader = new InputStreamReader( sockInput.getInputStream(), "UTF-8" );
Same rules for read( char[], offset, length ) as read( byte[], offset, length )
A TCP is a stream of bytes you can get a portion of, one or more messages is a single read.
With UDP packets of more than 532 bytes, the packet can be broken up, so you have the same potential problem.
A simple way to encode message is to send the length first.
final DataOutputStream out = new DataOutputStream(new BufferedOutputStream(socket.getOutputStream()));
final DataInputStream in = new DataInputStream(new BufferedInputStream(socket.getInputStream()));
public void writeMessage(String text) {
out.writeUTF(text);
}
public String readMessage() {
return in.readUTF();
}
read/write UTF sends the length first and followed by the content
I am writing a java TCP client that talks to a C server.
I have to alternate sends and receives between the two.
Here is my code.
The server sends the length of the binary msg(len) to client(java)
Client sends an "ok" string
Server sends the binary and client allocates a byte array of 'len' bytes to recieve it.
It again sends back an "ok".
step 1. works. I get "len" value. However the Client gets "send blocked" and the server waits to receive data.
Can anybody take a look.
In the try block I have defined:
Socket echoSocket = new Socket("192.168.178.20",2400);
OutputStream os = echoSocket.getOutputStream();
InputStream ins = echoSocket.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(ins));
String fromPU = null;
if( (fromPU = br.readLine()) != null){
System.out.println("Pu returns as="+fromPU);
len = Integer.parseInt(fromPU.trim());
System.out.println("value of len from PU="+len);
byte[] str = "Ok\n".getBytes();
os.write(str, 0, str.length);
os.flush();
byte[] buffer = new byte[len];
int bytes;
StringBuilder curMsg = new StringBuilder();
bytes =ins.read(buffer);
System.out.println("bytes="+bytes);
curMsg.append(new String(buffer, 0, bytes));
System.out.println("ciphertext="+curMsg);
os.write(str, 0, str.length);
os.flush();
}
UPDATED:
Here is my code. At the moment, there is no recv or send blocking on either sides. However, both with Buffered Reader and DataInput Stream reader, I am unable to send the ok msg. At the server end, I get a large number of bytes instead of the 2 bytes for ok.
Socket echoSocket = new Socket("192.168.178.20",2400);
OutputStream os = echoSocket.getOutputStream();
InputStream ins = echoSocket.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(ins));
DataInputStream dis = new DataInputStream(ins);
DataOutputStream dos = new DataOutputStream(os);
if( (fromPU = dis.readLine()) != null){
//if( (fromPU = br.readLine()) != null){
System.out.println("PU Server returns length as="+fromPU);
len = Integer.parseInt(fromPU.trim());
byte[] str = "Ok".getBytes();
System.out.println("str.length="+str.length);
dos.writeInt(str.length);
if (str.length > 0) {
dos.write(str, 0, str.length);
System.out.println("sent ok");
}
byte[] buffer = new byte[len];
int bytes;
StringBuilder curMsg = new StringBuilder();
bytes =ins.read(buffer);
System.out.println("bytes="+bytes);
curMsg.append(new String(buffer, 0, bytes));
System.out.println("binarytext="+curMsg);
dos.writeInt(str.length);
if (str.length > 0) {
dos.write(str, 0, str.length);
System.out.println("sent ok");
}
Using a BufferedReader around a stream and then trying to read binary data from the stream is a bad idea. I wouldn't be surprised if the server has actually sent all the data in one go, and the BufferedReader has read the binary data as well as the line that it's returned.
Are you in control of the protocol? If so, I suggest you change it to send the length of data as binary (e.g. a fixed 4 bytes) so that you don't need to work out how to switch between text and binary (which is basically a pain).
If you can't do that, you'll probably need to just read a byte at a time to start with until you see the byte representing \n, then convert what you've read into text, parse it, and then read the rest as a chunk. That's slightly inefficient (reading a byte at a time instead of reading a buffer at a time) but I'd imagine the amount of data being read at that point is pretty small.
Several thoughts:
len = Integer.parseInt(fromPU.trim());
You should check the given size against a maximum that makes some sense. Your server is unlikely to send a two gigabyte message to the client. (Maybe it will, but there might be a better design. :) You don't typically want to allocate however much memory a remote client asks you to allocate. That's a recipe for easy remote denial of service attacks.
BufferedReader br = new BufferedReader(new InputStreamReader(ins));
/* ... */
bytes =ins.read(buffer);
Maybe your BufferedReader has sucked in too much data? (Does the server wait for the Ok before continuing?) Are you sure that you're allowed to read from the underlying InputStreamReader object after attaching a BufferedReader object?
Note that TCP is free to deliver your data in ten byte chunks over the next two weeks :) -- because encapsulation, differing hardware, and so forth makes it very difficult to tell the size of packets that will eventually be used between two peers, most applications that are looking for a specific amount of data will instead populate their buffers using code somewhat like this (stolen from Advanced Programming in the Unix Environment, an excellent book; pity the code is in C and your code is in Java, but the principle is the same):
ssize_t /* Read "n" bytes from a descriptor */
readn(int fd, void *ptr, size_t n)
{
size_t nleft;
ssize_t nread;
nleft = n;
while (nleft > 0) {
if ((nread = read(fd, ptr, nleft)) < 0) {
if (nleft == n)
return(-1); /* error, return -1 */
else
break; /* error, return amount read so far */
} else if (nread == 0) {
break; /* EOF */
}
nleft -= nread;
ptr += nread;
}
return(n - nleft); /* return >= 0 */
}
The point to take away is that filling your buffer might take one, ten, or one hundred calls to read(), and your code must be resilient against slight changes in network capabilities.