We Keep Coding

Unsuccessful in trying to reuse Java client socket

I have a software driver which communicates with a third-party controller; I have an API for using the latter but no visibility of its source code, and the supplier is not co-operative in trying to improve things!
The situation is as follows.
To send a request to the controller, I send an XML packet as the content of an HTTP POST to a servlet, which then sends me the response. The original code, implemented by a previous developer, works stably using java.net.Socket. However, our driver is implemented such that a new socket is created for EVERY request sent and, if the driver gets busy, the third-party controller struggles to keep up in terms of socket handling. In fact, their support guy said to me: "You really need to leave 5 seconds between each request...". This simply isn't commercially acceptable.
To improve performance, I wanted to try leaving our end of the socket open and reusing the socket pretty much indefinitely (given that connections can drop unexpectedly of course, but that's the least of my concerns and is manageable). However, whatever I seem to do, the effect is that if I use Comms.getSocket(false), a new socket is created for each request and everything works OK but bottlenecks when busy. If I use Comms.getSocket(true), the following happens:
Controller is sent first request
Controller responds to first request
Controller is sent second request (maybe 5 seconds later)
Controller never responds to second request or anything after it
postRequest() keeps getting called: for the first 12 seconds, the console outputs "Input shut down ? false" but, after that, the code no longer reaches there and doesn't get past the bw.write() and bw.flush() calls.
The controller allows both HTTP 1.0 and 1.1 but their docs say zilch about keep-alive. I've tried both and the code below shows that I've added Keep-Alive headers as well but the controller, as server, I'm guessing is ignoring them -- I don't think I have any way of knowing, do I ? When in HTTP 1.0 mode, the controller certainly returns a "Connection: close" but doesn't do that in HTTP 1.1 mode.
The likelihood is then that the server side is insisting on a "one socket per request" approach.
However, I wondered if I might be doing anything wrong (or missing something) in the following code to achieve what I want:
private String postRequest() throws IOException {
String resp = null;
String logMsg;
StringBuilder sb = new StringBuilder();
StringBuilder sbWrite = new StringBuilder();
Comms comms = getComms();
Socket socket = comms.getSocket(true);
BufferedReader br = comms.getReader();
BufferedWriter bw = comms.getWriter();
if (null != socket) {
System.out.println("Socket closed ? " + socket.isClosed());
System.out.println("Socket bound ? " + socket.isBound());
System.out.println("Socket connected ? " + socket.isConnected());
// Write the request
sbWrite
.append("POST /servlet/receiverServlet HTTP/1.1\r\n")
.append("Host: 192.168.200.100\r\n")
.append("Connection: Keep-Alive\r\n")
.append("Keep-Alive: timeout=10\r\n")
.append("Content-Type: text/xml\r\n")
.append("Content-Length: " + requestString.length() + "\r\n\r\n")
.append(requestString);
System.out.println("Writing:\n" + sbWrite.toString());
bw.write(sbWrite.toString());
bw.flush();
// Read the response
System.out.println("Input shut down ? " + socket.isInputShutdown());
String line;
boolean flag = false;
while ((line = br.readLine()) != null) {
System.out.println("Line: <" + line + ">");
if (flag) sb.append(line);
if (line.isEmpty()) flag = true;
}
resp = sb.toString();
}
else {
System.out.println("Socket not available");
}
return resp; // Another method will parse the response
}
To ease testing, I provide the socket using an extra Comms helper class and a method called getSocket(boolean reuse) where I can choose to always create a new socket or reuse the one that Comms creates for me, as follows:
public Comms(String ip, int port) {
this.ip = ip;
this.port = port;
initSocket();
}
private void initSocket() {
try {
socket = new Socket(ip, port);
socket.setKeepAlive(true);
socket.setPerformancePreferences(1, 0, 0);
socket.setReuseAddress(true);
bw = new BufferedWriter(new OutputStreamWriter(socket.getOutputStream(), StandardCharsets.UTF_8));
br = new BufferedReader(new InputStreamReader(socket.getInputStream(), StandardCharsets.UTF_8));
System.out.println("### CREATED NEW SOCKET");
}
catch (UnknownHostException uhe) {
System.out.println("### UNKNOWN HOST FOR SOCKET");
}
catch (IOException ioe) {
System.out.println("### SOCKET I/O EXCEPTION");
}
}
public BufferedReader getReader() { return br; }
public BufferedWriter getWriter() { return bw; }
public Socket getSocket(boolean reuse) {
if (! reuse) initSocket();
return socket;
}
Can anyone help ?

If we assume that keep-alive thing is working as expected, I think the line while ((line = br.readLine()) != null) is a faulty one, as this is kind of infinity loop.
readline() returns null when there is no more data to read, e.g. a EOF, or when server/client closes the connection, that will break-down your reusing socket solution, since an open stream will never cause a null to a readLine() call, but blocking.
You need to fix the alg about reading a response (why not using implemented http client?), checking content-length, and when read the amount of required data from body, go for next loop by keeping the socket alive.
After that setting flag to true, you have to know what kind of data should be read(considering mime/content-type), besides that, the length of data, so reading data using readLine() may not be a good practice here.
Also make sure server allow for persistence connection, by checking if it respects it by responsing the same connection:keep-alive header.

Skype Bot webhook - receive nothing

I am developing a bot for Skype in Java. Right now, I only want the bot to authenticate (get bearer token) and to echo received messages.
I did last test on 31st July 2017 and it worked fine. But, since August, the strangest thing is happening:
The bot can authenticate correctly, and I got the bearen token and its timeout. After that, the SslSocket starts listening. When I chat the bot, it gets a connection (initiates handshake and all that) but receives nothing. No headers, no payload, nothing.
I tried reading documentation and see if the latest update to Microsoft Bot Framework had anything to do with it, but I couldn't find anything.
Anyone can help me about what's going on?
InputStream inputStream = sslSocket.getInputStream();
OutputStream outputStream = sslSocket.getOutputStream();
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
PrintWriter printWriter = new PrintWriter(new OutputStreamWriter(outputStream));
String line = null;
String jsonLine = null;
String challenge = null;
while(((line = bufferedReader.readLine()) != null)){
System.out.println("iline = "+line);
if (!line.isEmpty()){
if(line.substring(0,1).equals("{")){
jsonLine = line;
}
}
}
This is the part of the code which reads the incoming message. JsonLine and line are always null.

Certificate was expired. Once renewed, everything started working again.

readLine() never stops reading

inFromClientR.readLine() never stops. any ideas? Am I forgetting something?
Server:
/*{ some code:
send a file with a dataoutputstream to client using a new port(4000) and when transfer is done i want a responce message (e.g. OK) send back to server in the old port(6000)
}*/
ServerSocket listenTransferSocket = new ServerSocket(6000);
Socket connectionTransferSocket = listenTransferSocket.accept();
BufferedReader inFromClientR =
new BufferedReader(new InputStreamReader(connectionTransferSocket.getInputStream()));
System.out.println("Client's response to Transfer: " +inFromClientR.readLine());
Client:
/*{ some code:
receive the file on port (4000) and then the responce is sent to server using the following commands
}*/
Socket fileTransferSocket = new Socket("localhost", 6000);
DataOutputStream outToServerR =
new DataOutputStream(fileTransferSocket.getOutputStream());
outToServerR.writeBytes("Transfer completed " +'\n');

BufferedReader#readLine() tries to fill its buffer with 8192 bytes, regradless of any linefeeds it find meanwhile. Since you have the connection open, the receiving side will wait until 1) you have sent 8192 bytes, or 2) closes the connection.
You would be better off using some other framing mechanism, maybe an ObjectOutputStream/ObjectInputStream.

String line = null;
while ((line = inFromClientR.readLine()) != null) {
// do sth
}

BufferedReader.readLine() method Blocking alternative

Is there any way to make the method readline() (from bufferedReader) block until it has in fact something to read??
I'm creating an client/server application socketBased and I have this
The Server reads and then writes. The Client writes and then reads.. this is the communication between client/server (based on sockets)
The only problem is that on the server I'm reading with bufferedReader.readLine() which isn't a blocking method. I have already tried replacing the bufferedReader.readLine() by dataInputstream().read() (handle all the byte stuffs and getting out when -1 is received) and it isn't working either.
The problem in all this is that the server is expecting to read but do not blocks. the client is only going to write to the server (through a socket) when the user press a button (so the server must be waiting for reading and it is not)
On the server side i have this:
BufferedReader READ = new BufferedReader(new InputStreamReader(skt.getInputStream()));
BufferedWriter WRITE = new BufferedWriter(new OutputStreamWriter(skt.getOutputStream()));
FileInputStream fstream = new FileInputStream(file);
DataInputStream in = new DataInputStream(fstream);
boolean FLAG = true;
String s = new String("#$");
while (FLAG) {
if (skt == null)
FLAG = false;
// READING FROM THE CLIENT
while (!READ.ready()){System.out.println("Blocking")};
while ((s = READ.readLine()) != null) {
System.out.println("RECEIVING: " + s);
System.out.println("$$$");
}
System.out.print("hi");
// READING FROM THE FILE AND SENDING TO THE CLIENT
//WRITTING TO THE CLIENT
while ((strLine = in.readLine()) != null) {
System.out.println("SENDING: " + strLine);
strLine += "\n";
WRITE.write(strLine);
WRITE.flush();
strLine = new String();
}
}
On the Client side i have this:
//READING FROM THE FILE AND SENDING TO THE SERVER
while((strLine = in.readLine()) != null)
{
strLine += "\n";
soma+=strLine.length();
WRITE.write(strLine);
WRITE.flush();
TextArea1.append(strLine);
ProgressBar.setValue(soma);
ProgressBar.repaint();
};
in.close();
ProgressBar.setValue(0);
soma=0;
//READING FROM THE SERVER
while((strLine=READ.readLine()) != null)
{
strLine+="\n";
soma+=strLine.length();
TextArea2.append(strLine);
strLine = new String();
ProgressBar.setValue(soma);
ProgressBar.repaint();
}
in.close();
fstream.close();
WRITE.close();
READ.close();
Skt.close();
NOTE: Skt is a socket connecting client to server.
the problem here is that. When the client connects to the server, the server expects to read from the socket right after the connection is established.. however the code on the client (writing to the server) is only expected to run when the user presses a button.. So, the server must be blocked until the client in fact writes something to the socket which is not (the server is not blocking). :-/ hope to make my self more clear this time.. Sorry

BufferedReader.readLine() always blocks until it finds '\n'.
If you can give examples of data you're sending around, I might help you further.

Try to use file lock approach like:
file = new RandomAccessFile("file.txt", "rw");
FileChannel channel = file.getChannel();
fl = channel.tryLock();
// file now is locked
fl.release();

Incrementally handling twitter's streaming api using apache httpclient?

I am using Apache HTTPClient 4 to connect to twitter's streaming api with default level access. It works perfectly well in the beginning but after a few minutes of retrieving data it bails out with this error:
2012-03-28 16:17:00,040 DEBUG org.apache.http.impl.conn.SingleClientConnManager: Get connection for route HttpRoute[{tls}->http://myproxy:80->https://stream.twitter.com:443]
2012-03-28 16:17:00,040 WARN com.cloudera.flume.core.connector.DirectDriver: Exception in source: TestTwitterSource
java.lang.IllegalStateException: Invalid use of SingleClientConnManager: connection still allocated.
at org.apache.http.impl.conn.SingleClientConnManager.getConnection(SingleClientConnManager.java:216)
Make sure to release the connection before allocating another one.
at org.apache.http.impl.conn.SingleClientConnManager$1.getConnection(SingleClientConnManager.java:190)
I understand why I am facing this issue. I am trying to use this HttpClient in a flume cluster as a flume source. The code looks like this:
public Event next() throws IOException, InterruptedException {
try {
HttpHost target = new HttpHost("stream.twitter.com", 443, "https");
new BasicHttpContext();
HttpPost httpPost = new HttpPost("/1/statuses/filter.json");
StringEntity postEntity = new StringEntity("track=birthday",
"UTF-8");
postEntity.setContentType("application/x-www-form-urlencoded");
httpPost.setEntity(postEntity);
HttpResponse response = httpClient.execute(target, httpPost,
new BasicHttpContext());
BufferedReader reader = new BufferedReader(new InputStreamReader(
response.getEntity().getContent()));
String line = null;
StringBuffer buffer = new StringBuffer();
while ((line = reader.readLine()) != null) {
buffer.append(line);
if(buffer.length()>30000) break;
}
return new EventImpl(buffer.toString().getBytes());
} catch (IOException ie) {
throw ie;
}
}
I am trying to buffer 30,000 characters in the response stream to a StringBuffer and then return this as the data received. I am obviously not closing the connection - but I do not want to close it just yet I guess. Twitter's dev guide talks about this here It reads:
Some HTTP client libraries only return the response body after the
connection has been closed by the server. These clients will not work
for accessing the Streaming API. You must use an HTTP client that will
return response data incrementally. Most robust HTTP client libraries
will provide this functionality. The Apache HttpClient will handle
this use case, for example.
It clearly tells you that HttpClient will return response data incrementally. I've gone through the examples and tutorials, but I haven't found anything that comes close to doing this. If you guys have used a httpclient (if not apache) and read the streaming api of twitter incrementally, please let me know how you achieved this feat. Those who haven't, please feel free to contribute to answers. TIA.
UPDATE
I tried doing this: 1) I moved obtaining stream handle to the open method of the flume source. 2) Using a simple inpustream and reading data into a bytebuffer. So here is what the method body looks like now:
byte[] buffer = new byte[30000];
while (true) {
int count = instream.read(buffer);
if (count == -1)
continue;
else
break;
}
return new EventImpl(buffer);
This works to an extent - I get tweets, they are nicely being written to a destination. The problem is with the instream.read(buffer) return value. Even when there is no data on the stream, and the buffer has default \u0000 bytes and 30,000 of them, so this value is getting written to the destination. So the destination file looks like this.. " tweets..tweets..tweeets.. \u0000\u0000\u0000\u0000\u0000\u0000\u0000...tweets..tweets... ". I understand the count won't return a -1 coz this is a never ending stream, so how do I figure out if the buffer has new content from the read command?

The problem is that your code is leaking connections. Please make sure that no matter what you either close the content stream or abort the request.
InputStream instream = response.getEntity().getContent();
try {
BufferedReader reader = new BufferedReader(
new InputStreamReader(instream));
String line = null;
StringBuffer buffer = new StringBuffer();
while ((line = reader.readLine()) != null) {
buffer.append(line);
if (buffer.length()>30000) {
httpPost.abort();
// connection will not be re-used
break;
}
}
return new EventImpl(buffer.toString().getBytes());
} finally {
// if request is not aborted the connection can be re-used
try {
instream.close();
} catch (IOException ex) {
// log or ignore
}
}

It turns out that it was a flume issue. Flume is optimized to transfer events of size 32kb. Anything beyond 32kb, Flume bails out. (The workaround is to tune event size to be greater than 32KB). So, I've changed my code to buffer 20,000 characters at least. It kind of works, but it is not fool proof. This can still fail if the buffer length exceeds 32kb, however, it hasn't failed so far in an hour of testing - I believe it has to do with the fact that Twitter doesn't send a lot of data on its public stream.
while ((line = reader.readLine()) != null) {
buffer.append(line);
if(buffer.length()>20000) break;
}