We Keep Coding

A function cannot be accessed when the program is running

I am working on homework about thread synchronization. The method reduceLoad() cannot be accessed when I call this method. All print function inside the reduceLoad() method are never called. If I can't run this method, the key will never be released.
It might be caused by Semaphore.accquire() and Semaphore.realse(). So I tried to delete all Semaphore methods that may cause the problem.
Protion of the program result below:
As you can see the program is held after all threads print "disembarks from ferry at port "
..... omit some result here....
Arrive at port 1 with a load of 5 vehicles
Auto 4 arrives at port 0
Ambulance 1 arrives at port 1
Ambulance 1 boards the ferry at port 1
Auto 6 boards on the ferry at port 1
Auto 8 boards on the ferry at port 1
Auto 0 disembarks from ferry at port 1
Auto 2 disembarks from ferry at port 1
Auto 1 disembarks from ferry at port 1
Ambulance 0 disembarks the ferry at port 1
Auto 7 boards on the ferry at port 1
Auto 3 disembarks from ferry at port 1
This is where the function is called
// Arrive at the next port
port = 1 - port;
// wait for ferry arrives
while (fry.getPort() != port) {
try {
sleep(1000);
} catch (Exception e) {
}
}
// disembarking
System.out.println("Auto " + id_auto + " disembarks from ferry at port " + port);
logger.check(fry.getPort() == port, "error unloading at wrong port");
fry.reduceLoad(); // Reduce load
System.out.println("Auto " + id_auto + " successfully disembarks from ferry at port " + port);
This method reduceLoad() should release a key when the ferry is empty, this key is used to notify other threads to continue adding more cars into the ferry
public synchronized void reduceLoad() {
logger.check(load > 0, "error unloading an empty Ferry!");
load = load - 1;
System.out.println("removed load, now " + load);
if (load == 0) {
unloadingDone.release();
}
}

socket.io java client - does not connect when used threaded

I'm using socket.io-client java library version 1.0.0 (included it via pom.xml)
I want to do a stress/performance-test to a socket.io based server in nodejs (the server seems to work well, not of interest here).
I'm experiencing a very strange behaviour when I establish my connections doing
a Thread.sleep between each new connection.
If establish the connection all at once, there is no problem. The problems occur when I wait (Thead.sleep()) some milliseconds.
Always after the 5th established connection, the connections need unusual long until they are established (connected callback received). The next connection seems to wait until the first one disconnects, but thereafter several connections are established. And a lot of connections aren't established at all, no matter how long I wait. This is really strange. Why does this happen?
It does not matter, if I establish the connections asynychronously (doing each connection in a separate thread starting all threads immediately) or synchronously one after another - it works fine if I don't wait between each new socket/thread creation.
This is the complete code:
import io.socket.client.IO;
import io.socket.client.Socket;
import io.socket.emitter.Emitter;
import java.net.URISyntaxException;
public class SocketStressTest
{
private static final String SOCKET_URI = "http://test.mydomain.com/";
public static int AMOUNT_OF_SOCKETS = 100;
IO.Options opts;
private int connectionCounter = 0;
private int disConnectionCounter = 0;
public static void main(String[] args)
{
new SocketStressTest().start();
}
private void start()
{
opts = new IO.Options();
opts.forceNew = true;
opts.reconnection = false;
Thread thread = new Thread(new Runnable()
{
public void run()
{
initSockets();
}
});
thread.start();
System.out.println("Generated all sockets");
}
private void initSockets()
{
for (int i = 0; i < AMOUNT_OF_SOCKETS; i++)
{
//when I comment out sleeping and try-catch below, everything works fine
try
{
Thread.sleep(400);
}
catch (InterruptedException e)
{
e.printStackTrace();
}
createSocket(i);
}
}
private void createSocket(final int nr)
{
//does not matter if I do this threaded or not:
Thread thread = new Thread(new Runnable()
{
public void run()
{
try
{
System.out.println("nr " + nr + " creating socket");
final Socket socket = IO.socket(SOCKET_URI, opts);
socket
.on(Socket.EVENT_CONNECT, new Emitter.Listener()
{
public void call(Object... args)
{
connectionCounter++;
System.out.println("nr " + nr + " connected id:" + socket.id() + " concounter:" + connectionCounter);
if (connectionCounter == AMOUNT_OF_SOCKETS)
System.out.println("===> ALL connected!");
}
})
.on(Socket.EVENT_DISCONNECT, new Emitter.Listener()
{
public void call(Object... args)
{
disConnectionCounter++;
System.out.println("nr " + nr + " disconnected. discounter=" + disConnectionCounter);
if (disConnectionCounter == AMOUNT_OF_SOCKETS)
System.out.println("<=== ALL DISCONNECTED!");
}
});
socket.connect();
System.out.println("nr " + nr + " connect called");
}
catch (URISyntaxException e)
{
e.printStackTrace();
}
}
});
thread.start();
}
}
This is example output(shortened), waiting 400ms inside the loop:
Generated all sockets
nr 0 creating socket
nr 0 connect called
nr 1 creating socket
nr 1 connect called
nr 0 connected id:7JLvH0hHNF0pg36mAAW3 concounter:1
nr 1 connected id:5fj3I_bFIa1JeUlXAAW4 concounter:2
nr 2 creating socket
nr 2 connect called
nr 2 connected id:RQTLEjftWna2JPuFAAW5 concounter:3
nr 3 creating socket
nr 3 connect called
nr 3 connected id:dg1xL9ddnLqwAlDsAAW6 concounter:4
nr 4 creating socket
nr 4 connect called
nr 4 connected id:y_zIvI4BXdhmEiuwAAW7 concounter:5
nr 5 creating socket
nr 5 connect called
nr 6 creating socket
nr 6 connect called
...
nr 25 creating socket
nr 25 connect called
nr 26 creating socket
nr 26 connect called
nr 0 disconnected. discounter=1
nr 1 disconnected. discounter=2
nr 5 connected id:zCoCg1qG1vJA7pezAAW8 concounter:6
nr 6 connected id:QZJA3yhcXzpRzCwgAAW9 concounter:7
nr 7 connected id:aNZMGdiY8bTeylz3AAW- concounter:8
nr 8 connected id:vitG7xSlEXO5AhnoAAW_ concounter:9
nr 9 connected id:kWirqWwxE5V4ITRiAAXA concounter:10
nr 10 connected id:gCbDdV62pzPRq71qAAXB concounter:11
nr 11 connected id:4ERh1JvC654ky96AAAXC concounter:12
nr 12 connected id:4QMQni7Ohjk0IO7XAAXD concounter:13
nr 27 creating socket
nr 27 connect called
nr 13 connected id:Xb8i-VeDaE_G9N2PAAXE concounter:14
nr 14 connected id:AccfpvKWkWoGY7TEAAXF concounter:15
example output without waiting (commented out the sleep(400) (shortened):
Generated all sockets
nr 0 creating socket
nr 1 creating socket
...
nr 97 creating socket
nr 99 creating socket
nr 24 connect called
nr 21 connect called
nr 52 connect called
...
nr 78 connect called
nr 18 connect called
nr 24 connected id:N4MyGCp4IIWwMtJhAAXV concounter:1
nr 85 connected id:0ahc2QOlGpzPqUwjAAXW concounter:2
nr 68 connected id:W280V6PpH-gUxogOAAXX concounter:3
...
nr 38 connected id:_4c8Ll0cCM_1oQYzAAY2 concounter:98
nr 11 connected id:lIyRiuxK8pmb9voAAAY3 concounter:99
nr 12 connected id:pZkW0Y5DxXgw-Sy6AAY4 concounter:100
===> ALL connected!
I tried it on 2 different machines. Same behaviour.
My goal was to adjust my stresstests with some waiting, but with this behaviour no testing is possible. Anyone any ideas? Am I doing something significantly wrong?
I'm doing the tests on windows 10.
I also created a jar from it and tested it on a debian system. Same behaviour.
I also created a similar javascript test for doing simultanously over 500 socket-connections - works like a charm.
So is there something strange coded in this socket.io java client library that disallows more than 5 connections simultanously?

SSH Server Identification never received - Handshake Deadlock [SSHJ]

We're having some trouble trying to implement a Pool of SftpConnections for our application.
We're currently using SSHJ (Schmizz) as the transport library, and facing an issue we simply cannot simulate in our development environment (but the error keeps showing randomly in production, sometimes after three days, sometimes after just 10 minutes).
The problem is, when trying to send a file via SFTP, the thread gets locked in the init method from schmizz' TransportImpl class:
#Override
public void init(String remoteHost, int remotePort, InputStream in, OutputStream out)
throws TransportException {
connInfo = new ConnInfo(remoteHost, remotePort, in, out);
try {
if (config.isWaitForServerIdentBeforeSendingClientIdent()) {
receiveServerIdent();
sendClientIdent();
} else {
sendClientIdent();
receiveServerIdent();
}
log.info("Server identity string: {}", serverID);
} catch (IOException e) {
throw new TransportException(e);
}
reader.start();
}
isWaitForServerIdentBeforeSendingClientIdent is FALSE for us, so first of all the client (we) send our identification, as appears in logs:
"Client identity String: blabla"
Then it's turn for the receiveServerIdent:
private void receiveServerIdent() throws IOException
{
final Buffer.PlainBuffer buf = new Buffer.PlainBuffer();
while ((serverID = readIdentification(buf)).isEmpty()) {
int b = connInfo.in.read();
if (b == -1)
throw new TransportException("Server closed connection during identification exchange");
buf.putByte((byte) b);
}
}
The thread never gets the control back, as the server never replies with its identity. Seems like the code is stuck in this While loop. No timeouts, or SSH exceptions are thrown, my client just keeps waiting forever, and the thread gets deadlocked.
This is the readIdentification method's impl:
private String readIdentification(Buffer.PlainBuffer buffer)
throws IOException {
String ident = new IdentificationStringParser(buffer, loggerFactory).parseIdentificationString();
if (ident.isEmpty()) {
return ident;
}
if (!ident.startsWith("SSH-2.0-") && !ident.startsWith("SSH-1.99-"))
throw new TransportException(DisconnectReason.PROTOCOL_VERSION_NOT_SUPPORTED,
"Server does not support SSHv2, identified as: " + ident);
return ident;
}
Seems like ConnectionInfo's inputstream never gets data to read, as if the server closed the connection (even if, as said earlier, no exception is thrown).
I've tried to simulate this error by saturating the negotiation, closing sockets while connecting, using conntrack to kill established connections while the handshake is being made, but with no luck at all, so any help would be HIGHLY appreciated.
: )

I bet following code creates a problem:
String ident = new IdentificationStringParser(buffer, loggerFactory).parseIdentificationString();
if (ident.isEmpty()) {
return ident;
}
If the IdentificationStringParser.parseIdentificationString() returns empty string, it will be returned to the caller method. The caller method will keep calling the while ((serverID = readIdentification(buf)).isEmpty()) since the string is always empty. The only way to break the loop would be if call to int b = connInfo.in.read(); returns -1... but if server keeps sending the data (or resending the data) this condition is never met.
If this is the case I would add some kind of artificial way to detect this like:
private String readIdentification(Buffer.PlainBuffer buffer, AtomicInteger numberOfAttempts)
throws IOException {
String ident = new IdentificationStringParser(buffer, loggerFactory).parseIdentificationString();
numberOfAttempts.incrementAndGet();
if (ident.isEmpty() && numberOfAttempts.intValue() < 1000) { // 1000
return ident;
} else if (numberOfAttempts.intValue() >= 1000) {
throw new TransportException("To many attempts to read the server ident").
}
if (!ident.startsWith("SSH-2.0-") && !ident.startsWith("SSH-1.99-"))
throw new TransportException(DisconnectReason.PROTOCOL_VERSION_NOT_SUPPORTED,
"Server does not support SSHv2, identified as: " + ident);
return ident;
}
This way you would at least confirm that this is the case and can dig further why .parseIdentificationString() returns empty string.

Faced a similar issue where we would see:
INFO [net.schmizz.sshj.transport.TransportImpl : pool-6-thread-2] - Client identity string: blablabla
INFO [net.schmizz.sshj.transport.TransportImpl : pool-6-thread-2] - Server identity string: blablabla
But on some occasions, there were no server response.
Our service would typically wake up and transfer several files simultaneously, one file per connection / thread.
The issue was in the sshd server config, we increased maxStartups from default value 10
(we noticed the problems started shortly after batch sizes increased to above 10)
Default in /etc/ssh/sshd_config:
MaxStartups 10:30:100
Changed to:
MaxStartups 30:30:100
MaxStartups
Specifies the maximum number of concurrent unauthenticated connections to the SSH daemon. Additional connections will be dropped until authentication succeeds or the LoginGraceTime expires for a connection. The default is 10:30:100. Alternatively, random early drop can be enabled by specifying the three colon separated values start:rate:full (e.g. "10:30:60"). sshd will refuse connection attempts with a probability of rate/100 (30%) if there are currently start (10) unauthenticated connections. The probability increases linearly and all connection attempts are refused if the number of unauthenticated connections reaches full (60).
If you cannot control the server, you might have to find a way to limit your concurrent connection attempts in your client code instead.

How to write multithreaded algorithm to discovery available socket

For now I have sequential single threaded algorithm like this:
boolean conn = false;
for (InetSocketAddress addr : addrs) {
while (!conn) {
SocketChannel ch = SocketChannel.open();
ch.configureBlocking(true);
ch.socket().setTcpNoDelay(tcpNoDelay);
ch.socket().setKeepAlive(true);
try {
ch.socket().connect(addr, connTimeout);
conn = true;
....
}
catch (...) {
log("Not available: " + addr);
}
if (conn)
break;
}
}
But in some situations I could have quite a large list of addresses and available address could be in the end of this list - this will result into slow discovery of available address because I need to check all addresses one by one.
So my question - how I could implement algorithm that will try to connect to several addresses from different threads and will exit once first available address is found?
Also this algorithm should exit if no address available.
Any ideas how to code this?

Do all the connects in non-blocking mode, and then use a Selector on OP_CONNECT to find out when and if the connections succeed.

Using JZMQ with EPGM Transport Is Not Sending or Receiving Data

I'm experimenting with java flavored zmq to test the benefits of using PGM over TCP in my project. So I changed the weather example, from the zmq guide, to use the epgm transport.
Everything compiles and runs, but nothing is being sent or received. If I change the transport back to TCP, the server receives the messages sent from the client and I get the console output I'm expecting.
So, what are the requirements for using PGM? I changed the string, that I'm passing to the bind and connect methods, to follow the zmq api for zmq_pgm: "transport://interface;multicast address:port". That didn't work. I get and invalid argument error whenever I attempt to use this format. So, I simplified it by dropping the interface and semicolon which "works", but I'm not getting any results.
I haven't been able to find a jzmq example that uses pgm/epgm and the api documentation for the java binding does not define the appropriate string format for an endpoint passed to bind or connect. So what am I missing here? Do I have to use different hosts for the client and the server?
One thing of note is that I'm running my code on a VirtualBox VM (Ubuntu 14.04/OSX Mavericks host). I'm not sure if that has anything to do with the issue I'm currently facing.
Server:
public class wuserver {
public static void main (String[] args) throws Exception {
// Prepare our context and publisher
ZMQ.Context context = ZMQ.context(1);
ZMQ.Socket publisher = context.socket(ZMQ.PUB);
publisher.bind("epgm://xx.x.x.xx:5556");
publisher.bind("ipc://weather");
// Initialize random number generator
Random srandom = new Random(System.currentTimeMillis());
while (!Thread.currentThread ().isInterrupted ()) {
// Get values that will fool the boss
int zipcode, temperature, relhumidity;
zipcode = 10000 + srandom.nextInt(10000) ;
temperature = srandom.nextInt(215) - 80 + 1;
relhumidity = srandom.nextInt(50) + 10 + 1;
// Send message to all subscribers
String update = String.format("%05d %d %d", zipcode, temperature, relhumidity);
publisher.send(update, 0);
}
publisher.close ();
context.term ();
}
}
Client:
public class wuclient {
public static void main (String[] args) {
ZMQ.Context context = ZMQ.context(1);
// Socket to talk to server
System.out.println("Collecting updates from weather server");
ZMQ.Socket subscriber = context.socket(ZMQ.SUB);
//subscriber.connect("tcp://localhost:5556");
subscriber.connect("epgm://xx.x.x.xx:5556");
// Subscribe to zipcode, default is NYC, 10001
String filter = (args.length > 0) ? args[0] : "10001 ";
subscriber.subscribe(filter.getBytes());
// Process 100 updates
int update_nbr;
long total_temp = 0;
for (update_nbr = 0; update_nbr < 100; update_nbr++) {
// Use trim to remove the tailing '0' character
String string = subscriber.recvStr(0).trim();
StringTokenizer sscanf = new StringTokenizer(string, " ");
int zipcode = Integer.valueOf(sscanf.nextToken());
int temperature = Integer.valueOf(sscanf.nextToken());
int relhumidity = Integer.valueOf(sscanf.nextToken());
total_temp += temperature;
}
System.out.println("Average temperature for zipcode '"
+ filter + "' was " + (int) (total_temp / update_nbr));
subscriber.close();
context.term();
}
}

There are a couple possibilities:
You need to make sure ZMQ is compiled with the --with-pgm option: see here - but this doesn't appear to be your issue if you're not seeing "protocol not supported"
Using raw pgm requires root privileges because it requires the ability to create raw sockets... but epgm doesn't require that, so it shouldn't be your issue either (I only bring it up because you use the term "pgm/epgm", and you should be aware that they are not equally available in all situations)
What actually appears to be the problem in your case is that pgm/epgm requires support along the network path. In theory, it requires support out to your router, so your application can send a single message and have your router send out multiple messages to each client, but if your server is aware enough, it can probably send out multiple messages immediately and bypass this router support. The problem is, as you correctly guessed, trying to do this all on one host is not supported.
So, you need different hosts for client and server.

Another bit to be aware of is that some virtualization environments--RHEV/Ovirt and libvirt/KVM with the mac_filter option enabled come to mind-- that, by default, neuter one's abilities via (eb|ip)tables to utilize mcast between guests. With libvirt, of course, the solution is to simply set the option to '0' and restart libvirtd. RHEV/Ovirt require a custom plugin.
At any rate, I would suggest putting a sniffer on the network devices on each system you are using and watching to be sure traffic that is exiting the one host is actually visible on the other.