We Keep Coding

Vertx NetServer control read flow

I am trying to mimic a TCP server for tests with Vertx based on existing infrastructure that I have to work with.
The server I am mimicking works completely async and knows the length of the incoming buffer based on a pre-header in the buffer that indicates the length of the request.
I need to read the first 6 characters of the incoming request on each client socket that connect to my mock TCP server. from this pre-header I read the actual length of the request (e.g. for xx3018, i know the full length of the request is 3018).
Then I need to read the rest of the buffer according to the length, match it to a map of responses and return the right response for the request.
Example for a working mock server with plain java (fast implementation so other development won't be blocked :) )
public void run(String... args) throws Exception {
log.info("Starting TCP Server");
ServerSocket serverSocket = new ServerSocket(1750);
while (true) {
try {
Socket socket = serverSocket.accept();
CompletableFuture.runAsync(() -> {
Exception e = null;
while (e == null) {
try {
InputStream inputStream = socket.getInputStream();
OutputStream outputStream = socket.getOutputStream();
byte[] preHeader = new byte[6];
inputStream.read(preHeader);
String preHeaderValue = new String(preHeader);
log.info("Pre header: {}", preHeaderValue);
int length = Integer.valueOf(preHeaderValue.substring(2));
log.info("Request full length: {}", length);
byte[] request = new byte[length - 6];
inputStream.read(request);
String requestValue = new String(request);
log.info("Request: {}", requestValue);
String response = this.requestResponseProvider.getResponse(preHeaderValue + requestValue);
log.info("Response: {}", response);
outputStream.write(response.getBytes());
} catch (Exception ex) {
log.error("Encountered a problem: {}", e.getMessage());
e = ex;
}
}
});
} catch (Exception e) {
log.error("Encountered a problem: {}", e.getMessage());
}
}
}
I can't seem to find a way to control the input stream the same way I control it with plain java.

After a very long time of leaving this issue aside, I decided to play with it a bit.
I remembered using the following module for a different project: https://github.com/vert-x3/vertx-tcp-eventbus-bridge
I also remembered that in the tcp bridge's internal protocol, it appends the length of the payload to the buffer that is being sent via the tcp bridge, I looked into the source code to find out how it handles chunks (aka frames)
I found the following: https://github.com/vert-x3/vertx-tcp-eventbus-bridge/blob/master/src/main/java/io/vertx/ext/eventbus/bridge/tcp/impl/protocol/FrameParser.java which does exactly what I wanted to achieve :)
I modified it a bit, converted to Kotlin, and made it so I can control the header size and the way it extracts the payload length.
The following is a rough quick and dirty example of controlling the read flow with Vert.x NetServer:
suspend fun main() {
val vertx = Vertx.vertx()
initServer(vertx)
initClient(vertx)
}
suspend fun initServer(vertx: Vertx) {
val server = vertx.createNetServer(netServerOptionsOf(port = 8888, host = "localhost"))
server
.connectHandler { socket ->
val parser = FrameParser(
headerSize = 4,
headerHandler = {
it.getInt(0)
},
handler = {
println(it.toString())
println("---")
}
)
socket.handler(parser)
socket.exceptionHandler {
it.printStackTrace()
socket.close()
}
}
.listenAwait()
}
suspend fun initClient(vertx: Vertx) {
val client = vertx.createNetClient()
val socket = client.connectAwait(port = 8888, host = "localhost")
val message = "START|${"foobarfoobar".repeat(100)}|END"
val length = message.length
repeat(5) {
repeat(100) {
vertx.setPeriodic(10) {
socket.write(
Buffer.buffer()
.appendInt(length)
.appendString(message)
)
}
}
delay(1000)
}
}
/**
* Based on: https://github.com/vert-x3/vertx-tcp-eventbus-bridge/blob/master/src/main/java/io/vertx/ext/eventbus/bridge/tcp/impl/protocol/FrameParser.java
*/
class FrameParser(
private val headerSize: Int,
private val headerHandler: (Buffer) -> Int,
private val handler: (Buffer) -> Unit
) : Handler<Buffer?> {
private var _buffer: Buffer? = null
private var _offset = 0
override fun handle(buffer: Buffer?) {
append(buffer)
var offset: Int
while (true) {
// set a rewind point. if a failure occurs,
// wait for the next handle()/append() and try again
offset = _offset
// how many bytes are in the buffer
val remainingBytes = bytesRemaining()
// at least expected header size
if (remainingBytes < headerSize) {
break
}
// what is the length of the message
val length: Int = headerHandler(_buffer!!.getBuffer(_offset, _offset + headerSize))
_offset += headerSize
if (remainingBytes - headerSize >= length) {
// we have a complete message
handler(_buffer!!.getBuffer(_offset, _offset + length))
_offset += length
} else {
// not enough data: rewind, and wait
// for the next packet to appear
_offset = offset
break
}
}
}
private fun append(newBuffer: Buffer?) {
if (newBuffer == null) {
return
}
// first run
if (_buffer == null) {
_buffer = newBuffer
return
}
// out of data
if (_offset >= _buffer!!.length()) {
_buffer = newBuffer
_offset = 0
return
}
// very large packet
if (_offset > 0) {
_buffer = _buffer!!.getBuffer(_offset, _buffer!!.length())
}
_buffer!!.appendBuffer(newBuffer)
_offset = 0
}
private fun bytesRemaining(): Int {
return if (_buffer!!.length() - _offset < 0) {
0
} else {
_buffer!!.length() - _offset
}
}
}

Java, Memcached - how to pass commands to server?

So ... I have a bit of software that's supposed to communicate with a memcached server (using no external libraries).
For testing purposes, let's settle on a simple get hello\r\n command.
I start memcached with the -vv option, this is what the command produces via telnet:
<15 new auto-negotiating client connection
15: Client using the ascii protocol
<15 get hello
>15 END
Now here is what the same command issued from my software produces:
<15 new auto-negotiating client connection
I'm connecting as following:
private void reconnect(){
InetSocketAddress remote;
int nofServers = m.servers.size();
for(int i = 0; i < R; ++i){
boolean success = false;
while(!success) {
try {
SocketChannel oldConnection = connections.get(i);
if (oldConnection != null) oldConnection.close();
remote = m.servers.get((myServerIndex + i) % nofServers).address();
SocketChannel chan = SocketChannel.open(remote);
chan.configureBlocking(false);
chan.register(this.selector, SelectionKey.OP_READ);
connections.set(i, chan);
success = true;
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
}
After that, the software falls into simple enough a NIO loop:
#Override
public void run() {
MyRequest curr = null;
this.canHandleNewRequest = true;
while (true) {
if (canHandleNewRequest) {
curr = myQueue.poll();
}
if (canHandleNewRequest && curr != null) {
canHandleNewRequest = false;
for (int i = 0; i < R; ++i) {
connections.get(i).keyFor(this.selector).interestOps(SelectionKey.OP_WRITE);
}
}
try {
selector.select();
Iterator<SelectionKey> it = selector.selectedKeys().iterator();
while (it.hasNext()) {
SelectionKey k = it.next();
it.remove();
if (!k.isValid()) continue;
if (k.isConnectable()) finishConnection(k);
else if (k.isReadable()) this.read(k, curr);
else if (k.isWritable()) this.write(k, curr);
}
} catch (IOException ex) {
ex.printStackTrace();
reconnect();
}
if(curr != null && /*have sent to all servers I need to*/){
curr = null;
this.canHandleNewRequest = true;
}
}
}
where
private void write(SelectionKey k, MyRequest currentRequest){
try {
SocketChannel chan = (SocketChannel) k.channel();
ByteBuffer out = currentRequest.getSendBuffer(); //DO NOT FLIP (to resume sending from last sent position)
assert(chan != null);
assert(out != null);
//System.out.println(new String(out.array()));
chan.write(out); //TODO: make this work!
if(!out.hasRemaining()) { //do not enable read until whole command has been sent
currentRequest.partiallySent();
k.interestOps(SelectionKey.OP_READ);
}
}catch(IOException ex){
ex.printStackTrace();
}
//TODO: create response structure
}
I even tried to substitute the write method for a dummy command provider:
else if(k.isWritable()){
SocketChannel chan = (SocketChannel)k.channel();
ByteBuffer msg = ByteBuffer.wrap("get hello\r\n".getBytes());
msg.flip();
while(msg.hasRemaining()) {
System.out.println("writing ...");
chan.write(msg);
}
k.interestOps(SelectionKey.OP_READ);
}
but this only gets stuck in the "writing" loop (never terminates).
You should think that at least the server should react to that command but it doesn't.
So ... how do I get this working?
The second line from the log providing the command via telnet produces,
15: Client using the ascii protocol
makes me think there might be something I need to send to the server prior to engaging in actual memcached commands... except I seem to miss it in the protocol.
Help would be appreciated.
EDIT
This seems to be the issue: flipping a buffer in the getSendBuffer method and then returning it is not the same as returning it unflipped and then flipping it in the write method.
I find this rather strange. Can this be or is this merely masking a different error?

With NIO you should always check whether all of the buffer has been written, which is not being done doing in the first write block. Having said that, unless there's a lot of data being written, the whole buffer is usually written in a single call to write. So, it's unlikely to be the root problem here.
In the alternative writing code block the hasRemaining condition is negated, it should be:
while(msg.hasRemaining()) {
System.out.println("writing ...");
chan.write(msg);
}
Could you include what's being sent first? Is the first command terminated with \r\n?

Is it possible to use SSLEngine with *Blocking* I/O

I am trying to integrate some two code bases. One code base uses blocking I/O. The other code base uses non-blocking I/O.
The hook where I can integrate the two of these is a plain old-fashioned acceptor thread.
This acceptor thread reads the sub-protocol information from the socket and then based on the sub-protocol name forwards to the corresponding handler.
The other side of the code has it's own selector thread but only exposes a higher level set of constructs.
So basically I need to - in the acceptor thread's spawned worker thread - fire up the SSLEngine validate some sub-protocol information and then hand the whole thing off to the other code base's selector thread.
To make things more complicated, there is a fall-back path on the second code base whereby if it gets a Socket that was not opened with a SocketChannel it will drop down to blocking mode... and this is the bit that is causing me issues...
Namely it is not safe for me to assume that Socket.getChannel()!=null
So my SSLEngine code needs to take account of that possibility and set up the SSLEngine without using the non-blocking I/O APIs...
So far, I keep hitting blocking read calls that stall the engine...
Question does anyone know of any examples where SSLEngine has been used with a traditional InputStream/OutputStream rather than with a SocketChannel

Yes for handshake.
It is sad that we can't use SSLSocket.getSession for SSLEngine.
But we cat use SSLEngine in blocking mode.
See jdk1.8.0_112/sample/nio/server/ChannelIOSecure.java
Blocking:
method1(){
....
socketChannel.configureBlocking(true);
SSLEngine engine = sslContext.createSSLEngine();
engine.setUseClientMode(false);//server
engine.setNeedClientAuth(true);
this.outNetBB = ByteBuffer.allocate(engine.getSession().getPacketBufferSize());
outNetBB.position(0);
outNetBB.limit(0);
this.inNetBB = ByteBuffer.allocate(engine.getSession().getPacketBufferSize());
this.requestBB = ByteBuffer.allocate(engine.getSession().getApplicationBufferSize());
this.hsBB = ByteBuffer.allocate(engine.getSession().getApplicationBufferSize());
initialHSComplete = false;
while(initialHSComplete != true)
doHandshake(socketChannel, engine, null);
}
private boolean tryFlush(ByteBuffer bb, SocketChannel socketChannel) throws IOException {
socketChannel.write(bb);
return !bb.hasRemaining();
}
private SSLEngineResult.HandshakeStatus doTasks(SSLEngine sslEngine) {
Runnable runnable;
/*
* We could run this in a separate thread, but
* do in the current for now.
*/
while ((runnable = sslEngine.getDelegatedTask()) != null) {
runnable.run();
}
return sslEngine.getHandshakeStatus();
}
private ByteBuffer outNetBB;
int netBBSize;
private ByteBuffer inNetBB;
int appBBSize;
private ByteBuffer requestBB;
private ByteBuffer hsBB;
private boolean initialHSComplete; // Handshake complete status
HandshakeStatus initialHSStatus = HandshakeStatus.NEED_UNWRAP; //server
private boolean doHandshake(SocketChannel sc, SSLEngine sslEngine, SelectionKey sk) throws IOException {
SSLEngineResult result;
if (initialHSComplete) {
return initialHSComplete;
}
/*
* Flush out the outgoing buffer, if there's anything left in
* it.
*/
if (outNetBB.hasRemaining()) {
System.out.println("doha wtf");
if (!tryFlush(outNetBB, sc)) {
return false;
}
// See if we need to switch from write to read mode.
switch (initialHSStatus) {
// Is this the last buffer?
case FINISHED:
initialHSComplete = true;
// Fall-through to reregister need for a Read.
case NEED_UNWRAP:
if (sk != null) {
sk.interestOps(SelectionKey.OP_READ);
}
break;
}
return initialHSComplete;
}
switch (initialHSStatus) {
case NEED_UNWRAP:
System.out.println("before read");
if (sc.read(inNetBB) == -1) {
sslEngine.closeInbound();
return initialHSComplete;
}
System.out.println("after read");
needIO:
while (initialHSStatus == HandshakeStatus.NEED_UNWRAP) {
System.out.println("initialHSStatus"+initialHSStatus);
resizeRequestBB(); // expected room for unwrap
inNetBB.flip();
result = sslEngine.unwrap(inNetBB, requestBB);
inNetBB.compact();
System.out.println("result"+result);
initialHSStatus = result.getHandshakeStatus();
switch (result.getStatus()) {
case OK:
switch (initialHSStatus) {
case NOT_HANDSHAKING:
throw new IOException(
"Not handshaking during initial handshake");
case NEED_TASK:
initialHSStatus = doTasks(sslEngine);
break;
case FINISHED:
initialHSComplete = true;
break needIO;
}
break;
case BUFFER_UNDERFLOW:
// Resize buffer if needed.
netBBSize = sslEngine.getSession().getPacketBufferSize();
if (netBBSize > inNetBB.capacity()) {
resizeResponseBB();
}
/*
* Need to go reread the Channel for more data.
*/
if (sk != null) {
sk.interestOps(SelectionKey.OP_READ);
}
break needIO;
case BUFFER_OVERFLOW:
// Reset the application buffer size.
appBBSize =
sslEngine.getSession().getApplicationBufferSize();
break;
default: //CLOSED:
throw new IOException("Received" + result.getStatus() +
"during initial handshaking");
}
System.out.println("bottom of needIO");
} // "needIO" block.
System.out.println("after needIO "+initialHSStatus);
/*
* Just transitioned from read to write.
*/
if (initialHSStatus != HandshakeStatus.NEED_WRAP) {
break;
}
// Fall through and fill the write buffers.
case NEED_WRAP:
/*
* The flush above guarantees the out buffer to be empty
*/
outNetBB.clear();
result = sslEngine.wrap(hsBB, outNetBB);
outNetBB.flip();
initialHSStatus = result.getHandshakeStatus();
System.out.println("result wrap="+result);
switch (result.getStatus()) {
case OK:
if (initialHSStatus == HandshakeStatus.NEED_TASK) {
initialHSStatus = doTasks(sslEngine);
}
System.out.println("here");
if (sk != null) {
sk.interestOps(SelectionKey.OP_WRITE);
}
System.out.println("here2");
break;
default: // BUFFER_OVERFLOW/BUFFER_UNDERFLOW/CLOSED:
throw new IOException("Received" + result.getStatus() +
"during initial handshaking");
}
break;
default: // NOT_HANDSHAKING/NEED_TASK/FINISHED
throw new RuntimeException("Invalid Handshaking State" +
initialHSStatus);
} // switch
return initialHSComplete;
}
/*private void tryFlush(SocketChannel sc) throws IOException {
System.out.println("flush"+outNetBB);
sc.write(outNetBB);
if (!outNetBB.hasRemaining())
outNetBB.clear();
}*/
//}
//}
private void resizeResponseBB() {
ByteBuffer bb = ByteBuffer.allocate(netBBSize);
inNetBB.flip();
bb.put(inNetBB);
inNetBB = bb;
}
protected void resizeRequestBB() {
int remaining = appBBSize;
if (requestBB.remaining() < remaining) {
// Expand buffer for large request
ByteBuffer bb = ByteBuffer.allocate(requestBB.capacity() * 2);
requestBB.flip();
bb.put(requestBB);
requestBB = bb;
}
}
To change to non-blocking add:
socketChannel.configureBlocking(true);
Selector selector = Selector.open();
SelectionKey sk = socketChannel.register(selector, SelectionKey.OP_READ);
while(initialHSComplete != true){
selector.select();
doHandshake(socketChannel, engine, sk);

Broken pipe error communicating with Java server & C# client in multi thread env

I hope to find any help on my old annoying problem.
I have a TCP sever program with java and client program with c#
packet protocol between those two is simply consist of 4byte length & body ASCII data.
The Problem is that C# client faces FormatException which is from parsing fail on length byte. If I look into an error from client side, then client is trying to parse somewhere in the body which is not length header.
But apparently, Server does not send broken packet.
meanwhile, at the server, I could find an Broken pipe error whenever this kind of problem happens.
Unfortunately this error does not always happen and was not able to recreate the problem situation. it makes me difficult to find exact cause of this problem
Please see below codes for server side
public class SimplifiedServer {
private Map<InetAddress, DataOutputStream> outMap;
private Map<InetAddress,DataInputStream> inMap;
protected void onAcceptNewClient(Socket client) {
DataOutputStream out = null;
DataInputStream in = null;
try {
out = new DataOutputStream(client.getOutputStream());
in = new DataInputStream(client.getInputStream());
} catch (IOException e) {
e.printStackTrace();
}
outMap.put(client.getInetAddress(), out);
inMap.put(client.getInetAddress(), in);
}
public void writeToAll(String packet) {
outMap.forEach((key, out) -> {
try {
byte[] body = packet.getBytes("UTF-8");
int len = body.length;
if (len > 9999) {
throw new IllegalArgumentException("packet length is longer than 10000, this try will be neglected");
}
String lenStr = String.format("%04d%s", len, packet);
byte[] obuf = lenStr.getBytes();
synchronized (out) {
out.write(obuf);
out.flush();
}
} catch (IOException e) {
e.printStackTrace();
}
});
}
public void listenClient(Socket client) {
try {
DataOutputStream out = outMap.get(client.getInetAddress());
DataInputStream in = inMap.get(client.getInetAddress());
while (true) {
byte[] received = SimplePacketHandler.receiveLpControlerData(in);
byte[] lenBytes = new byte[4];
for( int i = 0 ; i < 4 ; i ++){
lenBytes[i] = in.readByte();
}
String lenString = new String(lenBytes);
int length = Integer.parseInt(lenString);
byte[] data = new byte[length];
for ( int i = 0 ; i < length ; i ++){
data[i] = in.readByte();
}
if ( data == null ){
System.out.println("NetWork error, closing socket :" + client.getInetAddress());
in.close();
out.close();
outMap.remove(client.getInetAddress());
inMap.remove(client.getInetAddress());
return;
}
doSomethingWithData(out, data);
}
} catch (NumberFormatException e) {
e.printStackTrace();
} catch ( Exception e ) {
e.printStackTrace();
} finally {
try {
System.out.println(client.getRemoteSocketAddress().toString() + " closing !!! ");
// remove stream handler from map
outMap.remove(client.getInetAddress());
inMap.remove(client.getInetAddress());
//close socket.
client.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
And here is client side code
public class ClientSide
{
public TcpClient client;
public String ip;
public int port;
public NetworkStream ns;
public BinaryWriter writer;
public BinaryReader reader;
public Boolean isConnected = false;
public System.Timers.Timer t;
public String lastPacketSucceeded = String.Empty;
public ClientSide(String ip, int port)
{
this.ip = ip;
this.port = port;
client = new TcpClient();
}
public bool connect()
{
try
{
client.Connect(ip, port);
}
catch (SocketException e)
{
Console.WriteLine(e.ToString());
return false;
}
Console.WriteLine("Connection Established");
reader = new BinaryReader(client.GetStream());
writer = new BinaryWriter(client.GetStream());
isConnected = true;
return true;
}
public void startListen()
{
Thread t = new Thread(new ThreadStart(listen));
t.Start();
}
public void listen()
{
byte[] buffer = new byte[4];
while (true)
{
try
{
reader.Read(buffer, 0, 4);
String len = Encoding.UTF8.GetString(buffer);
int length = Int32.Parse(len);
byte[] bodyBuf = new byte[length];
reader.Read(bodyBuf, 0, length);
String body = Encoding.UTF8.GetString(bodyBuf);
doSomethingWithBody(body);
}
catch (FormatException e)
{
Console.WriteLine(e.Message);
}
}
}
public void writeToServer(String bodyStr)
{
byte[] body = Encoding.UTF8.GetBytes(bodyStr);
int len = body.Length;
if (len > 10000)
{
Console.WriteLine("Send Abort:" + bodyStr);
}
len = len + 10000;
String lenStr = Convert.ToString(len);
lenStr = lenStr.Substring(1);
byte[] lengthHeader = Encoding.UTF8.GetBytes(lenStr);
String fullPacket = lenStr + bodyStr;
byte[] full = Encoding.UTF8.GetBytes(fullPacket);
try
{
writer.Write(full);
}
catch (Exception)
{
reader.Close();
writer.Close();
client.Close();
reader = null;
writer = null;
client = null;
Console.WriteLine("Send Fail" + fullPacket);
}
Console.WriteLine("Send complete " + fullPacket);
}
}
Considering it is impossible to recreate problem, I would guess this problem is from multithread issue. but I could not find any further clue to fix this problem.
Please let me know if you guys need any more information to solve this out.
Any help will be great appreciated, thanks in advance.

A broken pipe exception is caused by closing the connection on the other side. Most likely the C# client has a bug, causing the format exception which causes it to close the connection and therefore the broken pipe on the server side. See what is the meaning of Broken pipe Exception?.
Check the return value of this read:
byte[] bodyBuf = new byte[length];
reader.Read(bodyBuf, 0, length);
According to Microsoft documentation for BinaryReader.Read https://msdn.microsoft.com/en-us/library/ms143295%28v=vs.110%29.aspx
[The return value is ] The number of bytes read into buffer. This might be less than the number of bytes requested if that many bytes are not available, or it might be zero if the end of the stream is reached.
If it reads less than the length bytes then next time it will be parsing the length using data somewhere in the middle of the last message.

These broke pipe exceptions happen when the client (browser) has closed the connection, but the server (your tag) continues to try to write to the stream.
This usually happens when someone clicks Back, Stop, etc. in the browser and it disconnects from the server before the request is finished. Sometimes, it can happen because, for example, the Content-Length header is incorrect (and the browser takes its value as true).
Usually, this is a non-event, and nothing to worry about. But if you are seeing them in your dev environment when you know you have not interrupted your browser, you might dig a bit more to find out why.
WLS server will try to filter these exceptions from the web container out of the log, since it is due to client (browser) action and we can't do anything about it. But the server doesn't catch all of them.
refer from :: https://community.oracle.com/thread/806884

Client to close the connection

I got a socket listener which keep listening for data. The problem now is that the client which send data will finally close the connection by itself. Based on my codes below I am wondering do I still need to perform this part of the codes where it does writeBuffer.close();?
Should I remove the final part and just put the socket closing the catch?
public void run()
{
BufferedWriter writeBuffer = null;
BufferedReader readBuffer = null;
String message="";
try {
writeBuffer = new BufferedWriter(new OutputStreamWriter(receivedSocketConn1.getOutputStream()));
readBuffer = new BufferedReader(new InputStreamReader(receivedSocketConn1.getInputStream()));
int m = 0, count=0;
int nextChar=0;
while ((nextChar=readBuffer.read()) != -1)
{
message += (char) nextChar;
if (nextChar == '#')
{
System.out.println("\n\nSending PA : "+message);
writeBuffer.write("$PA\r\n");
writeBuffer.flush();
message="";
}
}
}
catch (Exception ex)
{
System.out.println("MyError:Exception has been caught in in the main first try");
ex.printStackTrace(System.out);
}
/*finally
{
try
{
if ( writeBuffer != null )
{
writeBuffer.close();
}
else
{
System.out.println("MyError:writeBuffer is null in finally close");
}
}
catch(IOException ex)
{
ex.printStackTrace(System.out);
}
}*/
}

It's always a good idea to explicitly close the connections you're using. Think about it, it might be possible that the client never closes the connection (of course, then you'd have to implement some kind of timeout mechanism that closes the connection on the server side after a certain amount of time, but that's a different matter).
My point is - it never hurts to be careful, and manage your resources in a conservative fashion.