We Keep Coding

Java pattern for subsequent Input & OutputStream

Hello stack overflow world, I've been struggling with the most straight forward and common problem within Java IO, for some time, and now need your help to tackle it.
Check out this piece of code I have in a try block, within a thread.run():
// connect to client socket, and setup own server socket
clientSocket = new Socket(serverHostname, CLIENT_PORT);
//send a test command to download a file
String downloadFileName = "sample.txt";
DataOutputStream dataOutputStream = new DataOutputStream(clientSocket.getOutputStream());
System.out.println("Sending a request to download file : " + downloadFileName + " from user: Arsa node"); //todo: replace with node user later
dataOutputStream.writeUTF("D/sample.txt");
//close socket if host isn't detected anymore, and if socket doesn't become null suddenly
dataOutputStream.flush();
dataOutputStream.close();
System.out.println("****File has been sent****");
in = new DataInputStream(clientSocket.getInputStream());
byte[] retrievedFileData = new byte[8036];
if (in.readInt() > 0) {
System.out.println("Starting file download!");
in.read(retrievedFileData);
System.out.println("File data has been read, converting to file now");
//closing input stream will close socket also
in.close();
}
clientSocket.close();
2 Main questions that have been confusing me to death:
Why does dataOutputStream.close() need to be run for writeUTF to actually send my string to the server socket, I find that when I don't have dos.close(), data isn't retrieved on the other side, further because I close it, I no longer can read from the socket - as it seems the socket connection becomes closed when the Output Stream is previously closed...
What's a better way, following some sort of pattern to do this? For context, all I'm trying to do is write the filename I'm looking to download to my client, then read the response right away, which I expect to be bytes with the file, any error handling I will consider as a part of my development.
Overall, it shouldn't be complicated to write something to a socket, then read and ingest it's response...which doesn't seem to be the case here,
any help would be greatly appreciated! If the ServerSocket code snippet is needed I'm happy to share.

The observed behavior is just a side-effect of close(), as it calls flush() before closing to make sure any buffered data is sent. To solve your problem, you need to call the flush() method instead of closing.
This behavior is not unique to DataOutputStream: a lot of other OutputStream (or Writer) implementations apply buffering, and you will need to flush when you want to ensure the data is sent to the client, written to disk or otherwise processed.
BTW: The DataOutputStream and DataInputStream is for a very specific type of data serialization protocol that is particular to Java. You may want to consider carefully if this is the right protocol to use.

Socket blocked in transmitting json object from Java to Python

I have a problem with a Java-Python Socket. My objective is to send a Json object from java application to python script via socket tcp and receive a response but the socket is blocked after Json sending. In the following there is my code:
try {
Socket socket = new Socket(dstAddress, dstPort);
is = new DataInputStream(socket.getInputStream());
os = new DataOutputStream(socket.getOutputStream());
PrintWriter pw = new PrintWriter(os, true);
pw.println(jsonObject.toString());
System.out.println("Send to the socket jsonObject.");
BufferedReader in = new BufferedReader(new InputStreamReader(is));
String response = in.readLine();
System.out.println("Response: " + response);
is.close();
os.close();
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
In the following lines the python code:
HOST = "192.168.1.101" #localhost
PORT = 7011
s = socket(AF_INET, SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
while (1):
print("\n\nAttending for client.....\n\n")
conn, addr = s.accept()
print("Connected by: " , addr)
data = ""
while 1:
temp = conn.recv(1024).decode()
if not temp:
break
data = data + temp
print("JSON Received!!!!!")
imageJson = {}
imageJson = json.loads(data)
# responding to the client
response = DbImages[elem[0]]
resp = "Prova"
conn.send(resp.encode())
If I terminate the java code (ctrl+C) the socket exit from block and json arrive to python. What is the problem? The problem seems to be in.readLine (). If I delete that statement then the socket works without blocks.

Your Python code is waiting for the Java side to finish and send EOF before responding (that’s what it means to recv until you get an empty bytes).
Your Java code is waiting for the Python side to respond before closing the socket.
So, they’re both waiting for each other.
Removing the readLine means the Java code is no longer waiting for anything, so it just hangs up on the Python code as soon as it’s done sending, which does make the problem go away—but it isn’t much of a solution if you actually needed a response.
So, what should they be doing? Well, there are a few different options.
Use a framed protocol, where the Java side either sends a “message-done” delimiter after each message or sends a header (with, e.g., the byte length of the message) before each one. So, the Python code can read until it has a complete message, instead of until EOF.
If you encode your JSON in compact format with everything but printable ASCII escaped, then the delimiter can just be a newline (at which point you’re using JSONlines as your protocol), and the Python code can use makefile on the socket and call readline instead of looping over recv.
Cheat and use JSON as if it were a framed protocol. It isn’t, but as long as the only top-level values you ever send are objects and arrays, it works. Then the Python code can use raw_decode (see the json module docs) after each receive until it succeeds.
If you’re only ever going to send a single message, you can just half-shutdown the socket (close the write end) from Java, and then Python will get its EOF and respond on the still-open other side of the socket. (This may sound hacky, but it’s actually perfectly common—it’s how web browsers traditionally work, although HTTP 1.1 made things a bit more complicated.)

Your response is not a line, as it doesn't seem to contain a line ending.
That means it readLine will read forever.
Try adding a newline to your response to make readLine happy:
resp = "Prova\n"

java.net.Socket > outputStream > BufferedOutputStream flush() confirmation

is there a way of knowing when or whether the flush() method of a BufferedOutputStream thread has finished successfully? In my case I'm using it for sending a simple string through a java.net.Socket. In the following code, the flush() method is run in parallel with the BufferedReader.read() method and the socket output is immediately blocked by the input read resulting in something that resembles a deadlock. What I would like to do is wait for the output to end, and then start reading the input.
Socket sk = new Socket("192.168.0.112", 3000);
BufferedOutputStream bo = new BufferedOutputStream(sk.getOutputStream());
bo.write(message.getBytes());
bo.flush();
BufferedReader br = new BufferedReader(new InputStreamReader(sk.getInputStream()));
String line = br.readLine();
if (line.equals("ack")) {
System.out.println("ack");
}
sk.close();
Update
ServerSocket:
ServerSocket ss = new ServerSocket(3000);
System.out.println("server socket open");
while (true) {
Socket sk = ss.accept();
System.out.println("new connection");
BufferedReader br = new BufferedReader(new InputStreamReader(sk.getInputStream()));
String line = br.readLine();
System.out.println("received line: " + line);
BufferedOutputStream bo = new BufferedOutputStream(sk.getOutputStream());
bo.write("ack".getBytes()); bo.flush();
sk.close();
}
Update:
#Global Variable - the reason that read was blocking the socket is that it was waiting for the \n, indeed. Using
bo.write("ack\n".getBytes());
instead of
bo.write("ack".getBytes());
made it work.
Regarding the initial question, is there a way of knowing if flush() method has finished successfully, #Stephen C provided the answer:
there is no way to know that based on the Socket or OutputStream APIs.
The normal way to get that sort of assurance is to have the remote
application send an "reply" in response, and read it in the local
side.
This "reply" is implemented in the code sample and it works.

Is there a way of knowing when or whether the flush() method of a BufferedOutputStream thread has finished successfully?
It depends on what you mean by "finished successfully".
The flush() method ensures that all unsent data in the pipeline has been pushed as far as the operating system network stack. When that is done, then you could say that flush() has finished successfully. The way that you know that that has happened is that the flush() call returns.
On the other hand, if you want some assurance that the data has (all) been delivered to the remote machine, or that the remote application has read it (all) ... there is no way to know that based on the Socket or OutputStream APIs. The normal way to get that sort of assurance is to have the remote application send an "reply" in response, and read it in the local side.
In the following code, the flush() method is run in parallel with the BufferedReader.read() method and the socket output is immediately blocked by the input read resulting in something that resembles a deadlock.
The code that you are talking about is basically the correct approach. The way to wait for the response is to read it like that.
If it is not working, then you need to compare what the client and server side are doing:
Is the server waiting for the client to send something more? Maybe an end of line sequence?
Did the server sends the response?
Did it flush() the response?
A mismatch between what the client and server are doing can lead to a form or deadlock, but the solution is to fix the mismatch. Waiting for some kind of hypothetical confirmation of the flush() is not the answer.
UPDATE
The problem is indeed a mismatch. For example, the server writes "ack" but the client expects "ack\n". The same happens in the client -> server case ... unless message always ends with a newline.

Your code is reading reader.readLine() . Are your writing \n when writing? You may want to append \n to the string your are writing.

I tried to reproduce your problem. First, I ran in to some kind of blocking state too, until I realized, I was using readLine at Server-side, too. But the message I was sending did not have a concluding \n. Therefore, the serversocket was still waiting at its InputStream without sending the client the ACK through its OutputStream. I think, #Global Variable is right.

purpose of socket.shutdownOutput()

I am using the below code to send data to a tcp server. I am assuming that I need to use socket.shutdownOutput() to properly indicate that the client is done sending the request. Is my assumption correct? If not please let me know the purpose of shutdownOutput(). Also appreciate any further optimizations I can make.
Client
def address = new InetSocketAddress(tcpIpAddress, tcpPort as Integer)
clientSocket = new Socket()
clientSocket.connect(address, FIVE_SECONDS)
clientSocket.setSoTimeout(FIVE_SECONDS)
// default to 4K when writing to the server
BufferedOutputStream outputStream = new BufferedOutputStream(clientSocket.getOutputStream(), 4096)
//encode the data
final byte[] bytes = reqFFF.getBytes("8859_1")
outputStream.write(bytes,0,bytes.length)
outputStream.flush()
clientSocket.shutdownOutput()
Server
ServerSocket welcomeSocket = new ServerSocket(6789)
while(true)
{
println "ready to accept connections"
Socket connectionSocket = welcomeSocket.accept()
println "accepted client req"
BufferedInputStream inFromClient = new BufferedInputStream(connectionSocket.getInputStream())
BufferedOutputStream outToClient = new BufferedOutputStream(connectionSocket.getOutputStream())
ByteArrayOutputStream bos=new ByteArrayOutputStream()
println "reading data byte by byte"
byte b=inFromClient.read()
while(b!=-1)
{
bos.write(b)
b=inFromClient.read()
}
String s=bos.toString()
println("Received request: [" + s +"]")
def resp = "InvalidInput"
if(s=="hit") { resp = "some data" }
println "Sending resp: ["+resp+"]"
outToClient.write(resp.getBytes());
outToClient.flush()
}

I am using the below code to send data to a tcp server. I am assuming
that I need to use socket.shutdownOutput() to properly indicate that
the client is done sending the request. Is my assumption correct?
YES Your assumption is correct. And this output ShutDown is known as half close . Using half close the TCP provides the ability for one end of the connection to terminate its output, while still receiving data from the other end. Let me walk you through the effects of socket.shutdownOutput() method :
Locally, the local socket and its input stream behave normally for reading
purposes, but for writing purposes the socket and its output stream behave
as though the socket had been closed by this end: subsequent writes to the
socket will throw an IOException
TCP’s normal connection-termination sequence (a - FIN acknowledged by
an ACK) is queued to be sent after any pending data has been sent and acknowledged.
Remotely, the remote socket behaves normally for writing purposes, but for
reading purposes the socket behaves as though it had been closed by this
end: further reads from the socket return an EOF condition, i.e. a read count
of -1 or an EOFException , depending on the method being called.
When the local socket is finally closed, the connection-termination sequence
has already been sent, and is not repeated; if the other end has already
done a half-close as well, all protocol exchanges on the socket are now
complete.
Hence we see that When the EOF is received, that end is assured that the other end has done the output shutdown. And this scenario is perfectly achieved by socket.shutDownOutput() on the other side.
Source: Fundamental Networking in Java, Esmond Pitt

Socket.shutdownOutput() means that the client is finished sending any data through the TCP connection. It will send the remaining data followed by a termination sequence which will completely close its OUTGOING connection. It is not possible to send any further data, which will also indicate to your program that the request is completely finished. So its recommended if you are sure you don't have to send any more data.
But it's not needed to indicate that the request is finished (you don't have to open/close the output all the time if you have multiple requests), there are other ways.

Forwarding incoming TCP "Commands" to a function?

In Java, how would you set up a socket listener that listened to a socket for a series of bytes that represented a command and on recieving called a method which parsed the incoming data and invoked the appropriate command?
Clarification:
My issue is not with handling the commands (Which might also be error codes or responses to commands from the server) but with creating the socket and listening to it.
More Clarification:
What I want to do is mimic the following line of .Net (C#) code:
_stream.BeginRead(_data,0, _data.Length, new
AsyncCallback(this.StreamEventHandler), _stream);
Where:
_stream is a network stream created from a socket
_data is an array of Byte of length 9
this.StreamHandler is a delegate (function pointer) which get executed when data is read.
I am rewriting a library from C# into Java and the component I am currently writing passes commands to a server over TCPIP but also has to be able to bubble up events/responses to the layer above it.
In C# this seems to be trivial and it's looking less and less so in Java.

Starting from my other answer: The specific part you request is the one that goes into the section: "Magic goes here". It can be done in ohh so many ways, but one is:
final InputStream in = socket.getInputStream();
// This creates a new thread to service the request.
new Thread(new Runnable(){
public void run(){
byte[] retrievedData= new byte[ITEM_LENGTH];
in.read(retrievedData, 0, ITEM_LENGTH);
in.close();
// Here call your delegate or something to process the data
callSomethingWithTheData(retrievedData);
}
}).start();

Have a small main method which sets up the socket and listens for incoming connections. Pass each connection to a worker object (possibly in its own thread).
The worker object should have two APIs: The server and the client. The client API gets a connection and reads data from it, the server API takes a connection and writes data to it.
I like to keep these two in a single class because that makes it much more simple to keep the two in sync. Use a helper class to encode/decode the data for transmission, so you have single point to decide how to transmit integers, commands, options, etc.
If you want to go further, define a command class and write code to serialize that to a socket connection and read it from it. This way, you worker objects just need to declare which command class they handle and the server/client API gets even more simple (at the expense of the command class).

I would
put each command into a class of its own, where each class implements a specific interface (e.g. Command)
create a Map<String,Command> which contains a lookup table from each command string to an instance of the class that implements that command

This should help.
Lesson 1: Socket Communications

The TCP connection provides you with one InputStream and one OutputStream. You could just poll the InputStream continuously for the next command (and its inputs) on a dedicated thread. ByteBuffer.wrap(byte[] array) may be useful in interpreting the bytes as chars, ints, longs, etc. You could also pass objects around using serialization.

Any naive approach most likely will not scale well.
Consider using a REST-approach with a suitable small web-server. Jetty is usually a good choice.

To create an listen to a socket, in a very naive way:
mServerSocket = new ServerSocket(port);
listening = true;
while (listening) {
// This call blocks until a connection is made
Socket socket = serverSocket.accept();
OutputStream out = socket.getOutputStream();
InputStream in = socket.getInputStream();
// Here you do your magic, reading and writing what you need from the streams
// You would set listening to true if you have some command to close the server
// remotely
out.close();
in.close();
socket.close();
}
Normally it is a good idea to delegate the processing of the input stream to some other thread, so you can answer the next request. Otherwise, you will answer all requests serially.
You also need to define some kind of protocol of what bytes you expect on the input and output streams, but from your question it looks like you already have one.

You could create an enum with one member per command
interface Comamnd {
// whatever you expect all command to know to perform their function
void perform(Context context);
}
enum Commands implements Command{
ACTIONONE() {
void perform(Context context) {
System.out.println("Action One");
}
},
ACTIONTWO() {
void perform(Context context) {
System.out.println("Action Two");
}
}
}
// initialise
DataInputStream in = new DataInputStream(socket.getInputStream());
// in a loop
byte[] retrievedData= new byte[ITEM_LENGTH];
in.readFully(retrievedData);
String command = new String(retrievedData, 0);
Commands.valueOf(command).perform(context);