We Keep Coding

When to register OP_WRITE

I use NIO with reactor pattern to connect a server to a client. My codes are as follows:
Server side codes, in the block of if(selectionKey.isWritable){} :
public void isWritable(SelectionKey selectionKey) throws Exception {
SocketChannel socketChannel =
(SocketChannel) selectionKey.channel();
Integer myInteger = (Integer) selectionKey.attachment();
if (myInteger == null){
int myJob = jobFacade.isAnyJob(socketChannel, 100 /*deadline*/);
if (myJob > 0){
ByteBuffer inputBuffer = ByteBuffer.wrap("available\n".getBytes("UTF-8"));
socketChannel.write(inputBuffer);
myInteger = myJob;
socketChannel.register(
selector, SelectionKey.OP_WRITE, myInteger);
}else if (myJob == -1){
ByteBuffer inputBuffer = ByteBuffer.wrap("unavailable\n".getBytes("UTF-8"));
socketChannel.write(inputBuffer);
socketChannel.close();
UnsupportedOperationException un = new UnsupportedOperationException();
throw un;
}else if (myJob == -2){
ByteBuffer inputBuffer = ByteBuffer.wrap("pending\n".getBytes("UTF-8"));
inputBuffer.flip();
socketChannel.write(inputBuffer);
myInteger = null;
socketChannel.register(
selector, SelectionKey.OP_WRITE, myInteger);
}
// is there any new job to do?
}else{
int myInt = myInteger.intValue();
if ( myInt > 0 ){
long startRange = jobFacade.findByID(myInt);
sendTextFile(startRange, Integer.parseInt(properties.getProperty("workUnit")),
properties.getProperty("textPath"), socketChannel);
myInteger = -3;
socketChannel.register(
selector, SelectionKey.OP_WRITE, myInteger);
}else if (myInt == -3){
sendAlgorithmFile(socketChannel, properties.getProperty("algorithmPath"));
myInteger = -4;
socketChannel.register(
selector, SelectionKey.OP_WRITE, myInteger);
// send algorithm file
}else if (myInt == -4){
int isOK = jobFacade.isAccepted(socketChannel.socket().getInetAddress().toString(),
Long.parseLong(properties.getProperty("deadline")));
if(isOK == -1){
ByteBuffer inputBuffer = ByteBuffer.wrap("notaccepted\n".getBytes("UTF-8"));
socketChannel.write(inputBuffer);
myInteger = null;
socketChannel.register(
selector, SelectionKey.OP_WRITE, myInteger);
}else {
ByteBuffer inputBuffer = ByteBuffer.wrap("accepted\n".getBytes("UTF-8"));
socketChannel.write(inputBuffer);
myInteger = isOK;
socketChannel.register(
selector, SelectionKey.OP_READ, myInteger);
}
// send "accepted" or "not accepted"
}
}
}
It is no need to know what my methods in each block do except that these methods generate a number with this order at first. 1)myInteger=null, 2) myInteger > 0, 3) myInteger = -3, 4) myInteger = -4
In this order, OP-WRITE will register consecutively for four times. And this part is so important. So lets see my Client side code and then I will tell you my problem:
BufferedReader inFromServer = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
sentence = inFromServer.readLine();
System.out.println("Response from Server : " + sentence);
if (sentence.equals("available")){
BufferedReader inFromServer1 = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
while ((sentence = inFromServer1.readLine()) != null) {
myJob = myJob + sentence ;
}
inFromServer = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
String acception = inFromServer.readLine();
if (acception.equals("accepted")){
File file = new File("account.json");
byte[] bytes = new byte[2048];
InputStream inputStream = new FileInputStream(file);
OutputStream outputStream = clientSocket.getOutputStream();
int count;
try {
while ((count = inputStream.read(bytes)) > 0){
outputStream.write(bytes, 0, count);
}
outputStream.close();
inputStream.close();
}catch (IOException io){}
continue;
}else if (acception.equals("notaccepted")){
continue;
}
Now, my problem is that when I run my server and then my client, my server will run without waiting for my client to get input stream. First, the client get "available" but when the second getInputStream will be reached in client, the server paced all the phase of OP-WRITE registering and wait for client to get streams of data (As I defined in my code).
Actually, my server do its job well. It will pass all the stages in required order. But the problem is that sending and receiving data is not synchronous.
I do not know what my problem is. But I guess when I register OP-WRITE consecutively, it means that my server did not send all bytes of data, so just the first getInputStream will get the data.
On the other hand, I need this order to run my program. So, Is there any Idea?

I find out my problem. There is no problem with my code. OP_WRITE can be registered any time with any order. The most important thing is to write to buffer and read from socket correctly.
Actually, when I send something for the second time to my client, I did not clear the buffer. In this case I found it, and correct it.
But when I send some characters to my client and then want to send a file, because in my client side I have a loop to get all characters, the content if the file is gotten by the same loop.
The question here is that how I can make them separate?

I will help you clarify the problem before thinking about patterns:
You have one thread/process that passes a message asking another thread/ process to act upon the message.
The receiver needs to read the message and maybe start some child threads of its own to perform that work because it can receive other requests.
It would be nice to tell the sender that an acknowledgment that the request was received.
It seems necessary that the message passing is protected. Because if another requests comes in while you are reading you could end up processing garbage.
You can configure nio to have several readers and just one writer, just read one portion of a buffer, etc. Check the how-tos, api docs. It is plenty powerful

exactly after sending a message
There is no such thing as a message in TCP. It is a byte stream. Two writes at the sender are very likely to by read by one read at the receiver. If you want messages you have to implement them yourself, with count words, terminators, STX/ETX, XML, etc.

Java server can't send bytes below 0 to PHP website

I'm running a Java server that uses the Twitter API and collects search results about any given keyword. My goal is to send the results to my website in PHP. However some Tweets have text with bytes less than 0, these appear to be unicode characters or similar. I've had to replace all of those characters with a space for the packet to be sent at all. If a byte less than 0 is sent the PHP script just reads "null". I need to be able to send bytes of any value, even if they're below 0.
Java: Replace bytes below value 0 with a space
// Get the Tweet text
String text = content.getData(2);
// Get the bytes
byte [] bytes = text.getBytes();
// Replace any byte below 0 with a space
for(int a = 0; a < bytes.length; ++a) {
if(bytes[a] < 0) {
bytes[a] = " ".getBytes()[0];
}
}
// Put the bytes back into a String
text = new String(bytes);
Java: Server that listens to commands and replies with output
ServerSocket socket = null;
InputStreamReader inputStream = null;
BufferedReader input = null;
try {
socket = new ServerSocket(port);
Logger.log("Server running on port " + port);
while(running) {
connection = socket.accept();
inputStream = new InputStreamReader(connection.getInputStream(), StandardCharsets.UTF_8);
input = new BufferedReader(inputStream);
// Run the command we're given, in this case the command will request Twitter search results
String reply = runCommand(input.readLine());
// Reply(String) will reply with the results
reply(reply);
}
} catch(Exception e) {
e.printStackTrace();
} finally {
try {
connection.close();
response.close();
if(inputStream != null) {
inputStream.close();
}
if(input != null) {
input.close();
}
if(socket != null) {
socket.close();
}
} catch(IOException e) {
e.printStackTrace();
}
}
Java: Reply method to send the results of the command execution (in this case it'll send the Twitter search results)
private void reply(String reply) {
try {
response = new DataOutputStream(connection.getOutputStream());
response.writeUTF(reply);
response.flush();
} catch(IOException e) {
e.printStackTrace();
}
}
PHP: Send commands (Twitter search query) via sockets and get the reply (Tweet data)
$socket = socket_create(AF_INET, SOCK_STREAM, getprotobyname('tcp'));
try {
socket_connect($socket, $address, $port);
// Encode the message in UTF-8, is this correct do to?
$message = utf8_encode($message);
// Get the result of sending this message to my Java server
$status = socket_sendto($socket, $message, strlen($message), MSG_EOF, $address, $port);
// Decode and return the results, is this the correct way to do this?
if($status != false) {
if($next = utf8_decode(socket_read($socket, $port))) {
return substr($next, 2);
}
}
} catch(Exception $e) {
}
// Even when I console.log the return of this method and it comes out "null" it is never this. I've tried changing this to return "-1" and other values and it still always returned "null" as if it was returning the string "null" in the above if statements.
return null;
I believe it may be important to note that bottom comment in PHP. I've tried looking over Google for a while now about this and I'm not sure if I'm doing things wrong or if I'm searching for the wrong thing.
How would I send bytes that are less than 0 through this system?

Why doesnt java tcp socket send all data when called repeatetly in a while-loop?

I am sending data over a socket but the java socket seems to change ordering and loose data and I can't fix it.
Here is my java code:
Socket socket;
...
while(isSending){
try {
DataOutputStream out = new DataOutputStream(socket.getOutputStream());
String data = getMyData();
out.writeBytes(data);//data is a csv string parsed on server-side
out.flush();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Server.cpp:
while(1){
char recv_buffer[4096];
memset(recv_buffer,0,4096);
//receive data from socket
int ret = recv(socket , recv_buffer , 4095 , 0);
if (ret == 0){
error_print("Socket not connected");
ret = 0;
} else if (ret < 0) {
error_print("Error reading from socket!");
ret = 0;
}
if(ret<=0) break;
recv_buffer[ret]='\0';
//parse recv_buffer
}
If I put a Thread.sleep(2000) in the java while-loop, the values are received correctly. What could be the reason for this behavior and how can I fix it?

Just as I suspected. You are completely ignoring the value returned by the recv() function. It can be -1 indicating an error, or zero indicating end of stream, or a positive integer indicating the length received. Instead you are assuming not only that the read aucceeded but also that it returns a null-terminated string.

Is there a difference in Java's writeInt when executed on Windows vs an Intel based Mac

I currently writing a Java TCP server to handle the communication with a client (which I didn't write). When the server, hosted on windows, responds to the client with the number of records received the client doesn't read the integer correctly, and instead reads it as an empty packet. When the same server code, hosted on my Mac, responds to the client with the number of records received the client reads the packet and responds correctly. Through my research I haven't found an explanation that seems to solve the issue. I have tried reversing the bytes (Integer.reverseBytes) before calling the writeInt method and that didn't seem to resolve the issue. Any ideas are appreciated.
Brian
After comparing the pcap files there are no obvious differences in how they are sent. The first byte is sent followed by the last 3. Both systems send the correct number of records.
Yes I'm referring to the DataOutputStream.writeInt() method. //Code added
public void run() {
try {
InputStream in = socket.getInputStream();
DataOutputStream datOut = new DataOutputStream(socket.getOutputStream());
datOut.writeByte(1); //sends correctly and read correctly by client
datOut.flush();
//below is used to read bytes to determine length of message
int bytesRead=0;
int bytesToRead=25;
byte[] input = new byte[bytesToRead];
while (bytesRead < bytesToRead) {
int result = in.read(input, bytesRead, bytesToRead - bytesRead);
if (result == -1) break;
bytesRead += result;
}
try {
inputLine = getHexString(input);
String hexLength = inputLine.substring(46, 50);
System.out.println("hexLength: " + hexLength);
System.out.println(inputLine);
//used to read entire sent message
bytesRead = 0;
bytesToRead = Integer.parseInt(hexLength, 16);
System.out.println("bytes to read " + bytesToRead);
byte[] dataInput = new byte[bytesToRead];
while (bytesRead < bytesToRead) {
int result = in.read(dataInput, bytesRead, bytesToRead - bytesRead);
if (result == -1) break;
bytesRead += result;
}
String data = getHexString(dataInput);
System.out.println(data);
//Sends received data to class to process
ProcessTel dataValues= new ProcessTel(data);
String[] dataArray = new String[10];
dataArray = dataValues.dataArray();
//assigns returned number of records to be written to client
int towrite = Integer.parseInt(dataArray[0].trim());
//Same write method on Windows & Mac...works on Mac but not Windows
datOut.writeInt(towrite);
System.out.println("Returned number of records: " + Integer.parseInt(dataArray[0].trim()) );
datOut.flush();
} catch (Exception ex) {
Logger.getLogger(ServerThread.class.getName()).log(Level.SEVERE, null, ex);
}
datOut.close();
in.close();
socket.close();
} catch (IOException e) {
e.printStackTrace();
}
}

As described in its Javadoc, DataOutputStream.writeInt() uses network byte order as per the TCP/IP RFCs. Is that the method you are referring to?

No, x86 processors only support little-endian byte order, it doesn't vary with the OS. Something else is wrong.
I suggest using wireshark to capture the stream from a working Mac server and a non-working Windows server and compare.

Some general comments on your code:
int bytesRead=0;
int bytesToRead=25;
byte[] input = new byte[bytesToRead];
while (bytesRead < bytesToRead) {
int result = in.read(input, bytesRead, bytesToRead - bytesRead);
if (result == -1) break;
bytesRead += result;
}
This EOF handling is hokey. It means that you don't know whether or not you've actually read the full 25 bytes. And if you don't, you'll assume that the bytes-to-send is 0.
Worse, you copy-and-paste this code lower down, relying on proper initialization of the same variables. If there's a typo, you'll never know it. You could refactor it into its own method (with tests), or you could call DataInputStream.readFully().
inputLine = getHexString(input);
String hexLength = inputLine.substring(46, 50);
You're converting to hex in order to extract an integer? Why? And more important, if you have any endianness issues this is probably the reason
I was originally going to recommend using a ByteBuffer to extract values, but on a second look I think you should wrap your input stream with a DataInputStream. That would allow you to read complete byte[] buffers without the need for a loop, and it would let you get rid of the byte-to-hex-to-integer conversions: you'd simply call readInt().
But, continuing on:
String[] dataArray = new String[10];
dataArray = dataValues.dataArray();
Do you realize that the new String[10] is being thrown away by the very next line? Is that what you want?
int towrite = Integer.parseInt(dataArray[0].trim());
datOut.writeInt(towrite);
System.out.println("Returned number of records: " + Integer.parseInt(dataArray[0].trim()) );
If you're using logging statements, print what you're actually using (towrite). Don't recalculate it. There's too much of a chance to make a mistake.
} catch (Exception ex) {
Logger.getLogger(ServerThread.class.getName()).log(Level.SEVERE, null, ex);
}
// ...
} catch (IOException e) {
e.printStackTrace();
}
Do either or both of these catch blocks get invoked? And why do they send their output to different places? For that matter, if you have a logger, why are you inserting System.out.println() statements?

File transfer from C++ client to Java server

I have a c++ client which needs to send a file to a c++ server. I'm splitting the file to chunks of PACKET_SIZE (=1024) bytes and send them over a TCP socket. At the server side I read at most PACKET_SIZE bytes to a buffer. When the client sends files which are less than PACKET_SIZE, the server receives more bytes than sent. Even when I limit the number of bytes to be exactly the size of the file, the files differ. I know the problem does not have to do with the client because I've tested it with a c++ server and it works flawlessly.
Thanks.
Server:
public void run() {
DataInputStream input = null;
PrintWriter output = null;
try {
input = new DataInputStream (_client.getInputStream());
}
catch (Exception e) {/* Error handling code */}
FileHeader fh = recvHeader(input);
size = fh._size;
filename = fh._name;
try {
output = new PrintWriter(_client.getOutputStream(), true);
}
catch (Exception e) {/* Error handling code */}
output.write(HEADER_ACK);
output.flush();
FileOutputStream file = null;
try {
file = new FileOutputStream(filename);
}
catch (FileNotFoundException fnfe) {/* Error handling code */}
int total_bytes_rcvd = 0, bytes_rcvd = 0, packets_rcvd = 0;
byte [] buf = new byte [PACKET_DATA_SIZE];
try {
int max = (size > PACKET_DATA_SIZE)? PACKET_DATA_SIZE: size;
bytes_rcvd = input.read(buf,0, max);
while (total_bytes_rcvd < size) {
if (-1 == bytes_rcvd) {...}
++packets_rcvd;
total_bytes_rcvd += bytes_rcvd;
file.write (buf,0, bytes_rcvd);
if (total_bytes_rcvd < size)
bytes_rcvd = input.read(buf);
}
file.close();
}
catch (Exception e) {/* Error handling code */}
}
Client:
char packet [PACKET_SIZE] ;
file.open (filename, ios::in | ios::binary);//fopen (file_path , "rb");
int max = 0;
if (file.is_open()) {
if (size > PACKET_SIZE)
max = PACKET_SIZE;
else
max = size;
file.read (packet , max);
}
else {...}
int sent_packets = 0;
while (sent_packets < (int) ceil (((float)size)/PACKET_SIZE) ) {
_write=send(_sd , packet, max,0);
if (_write <0) {...}
else {
++sent_packets;
if (size > PACKET_SIZE* sent_packets) {
if (size - PACKET_SIZE* sent_packets >= PACKET_SIZE)
max = PACKET_SIZE;
else
max = size - PACKET_SIZE* sent_packets;
file.read (packet , max);
}
}
}

Is the sending socket closed at the end of the file, or is the next file streamed over the same socket? If more than one file is streamed, you could pick up data from the next file if you have the endedness wrong for the file size in recvHeader(), i.e. you send a file of length 0x0102 and try to read one of length 0x0201.
Other question, why do you provide a max for the first read, but not for the following reads on the same file?

One issue I see is that it appears that you assume that if the send returns a non-error, that it sent the entire chunk you requested it to send. This is not necessarily true, especially with stream sockets. How large are the packets you are sending, and how many? The most likely reason this could occur would be if the sndbuf for the socket filled, and your socket _sd is set to non-blocking. I'm not positive (depends on stack implementation), but I believe it could also likely occur if the TCP transmit window was full for your connection, and tcp couldn't enqueue your entire packet.
You should probably loop on the send until max is sent.
Thusly:
int send_ct=0;
while( (_write = send(_sd, packet + send_ct, max-send_ct, 0)) > 0) {
send_ct += _write;
if(send_ct >= max) {
break;
} else {
// Had to do another send
}
}

the code is not complete. E.g. you have omitted the sending of the filename and the filesize, as well as the parsing of those values. Are those values correct? If not first ensure that these values are the right ones before investigating further.