We Keep Coding

SSH Server Identification never received - Handshake Deadlock [SSHJ]

We're having some trouble trying to implement a Pool of SftpConnections for our application.
We're currently using SSHJ (Schmizz) as the transport library, and facing an issue we simply cannot simulate in our development environment (but the error keeps showing randomly in production, sometimes after three days, sometimes after just 10 minutes).
The problem is, when trying to send a file via SFTP, the thread gets locked in the init method from schmizz' TransportImpl class:
#Override
public void init(String remoteHost, int remotePort, InputStream in, OutputStream out)
throws TransportException {
connInfo = new ConnInfo(remoteHost, remotePort, in, out);
try {
if (config.isWaitForServerIdentBeforeSendingClientIdent()) {
receiveServerIdent();
sendClientIdent();
} else {
sendClientIdent();
receiveServerIdent();
}
log.info("Server identity string: {}", serverID);
} catch (IOException e) {
throw new TransportException(e);
}
reader.start();
}
isWaitForServerIdentBeforeSendingClientIdent is FALSE for us, so first of all the client (we) send our identification, as appears in logs:
"Client identity String: blabla"
Then it's turn for the receiveServerIdent:
private void receiveServerIdent() throws IOException
{
final Buffer.PlainBuffer buf = new Buffer.PlainBuffer();
while ((serverID = readIdentification(buf)).isEmpty()) {
int b = connInfo.in.read();
if (b == -1)
throw new TransportException("Server closed connection during identification exchange");
buf.putByte((byte) b);
}
}
The thread never gets the control back, as the server never replies with its identity. Seems like the code is stuck in this While loop. No timeouts, or SSH exceptions are thrown, my client just keeps waiting forever, and the thread gets deadlocked.
This is the readIdentification method's impl:
private String readIdentification(Buffer.PlainBuffer buffer)
throws IOException {
String ident = new IdentificationStringParser(buffer, loggerFactory).parseIdentificationString();
if (ident.isEmpty()) {
return ident;
}
if (!ident.startsWith("SSH-2.0-") && !ident.startsWith("SSH-1.99-"))
throw new TransportException(DisconnectReason.PROTOCOL_VERSION_NOT_SUPPORTED,
"Server does not support SSHv2, identified as: " + ident);
return ident;
}
Seems like ConnectionInfo's inputstream never gets data to read, as if the server closed the connection (even if, as said earlier, no exception is thrown).
I've tried to simulate this error by saturating the negotiation, closing sockets while connecting, using conntrack to kill established connections while the handshake is being made, but with no luck at all, so any help would be HIGHLY appreciated.
: )

I bet following code creates a problem:
String ident = new IdentificationStringParser(buffer, loggerFactory).parseIdentificationString();
if (ident.isEmpty()) {
return ident;
}
If the IdentificationStringParser.parseIdentificationString() returns empty string, it will be returned to the caller method. The caller method will keep calling the while ((serverID = readIdentification(buf)).isEmpty()) since the string is always empty. The only way to break the loop would be if call to int b = connInfo.in.read(); returns -1... but if server keeps sending the data (or resending the data) this condition is never met.
If this is the case I would add some kind of artificial way to detect this like:
private String readIdentification(Buffer.PlainBuffer buffer, AtomicInteger numberOfAttempts)
throws IOException {
String ident = new IdentificationStringParser(buffer, loggerFactory).parseIdentificationString();
numberOfAttempts.incrementAndGet();
if (ident.isEmpty() && numberOfAttempts.intValue() < 1000) { // 1000
return ident;
} else if (numberOfAttempts.intValue() >= 1000) {
throw new TransportException("To many attempts to read the server ident").
}
if (!ident.startsWith("SSH-2.0-") && !ident.startsWith("SSH-1.99-"))
throw new TransportException(DisconnectReason.PROTOCOL_VERSION_NOT_SUPPORTED,
"Server does not support SSHv2, identified as: " + ident);
return ident;
}
This way you would at least confirm that this is the case and can dig further why .parseIdentificationString() returns empty string.

Faced a similar issue where we would see:
INFO [net.schmizz.sshj.transport.TransportImpl : pool-6-thread-2] - Client identity string: blablabla
INFO [net.schmizz.sshj.transport.TransportImpl : pool-6-thread-2] - Server identity string: blablabla
But on some occasions, there were no server response.
Our service would typically wake up and transfer several files simultaneously, one file per connection / thread.
The issue was in the sshd server config, we increased maxStartups from default value 10
(we noticed the problems started shortly after batch sizes increased to above 10)
Default in /etc/ssh/sshd_config:
MaxStartups 10:30:100
Changed to:
MaxStartups 30:30:100
MaxStartups
Specifies the maximum number of concurrent unauthenticated connections to the SSH daemon. Additional connections will be dropped until authentication succeeds or the LoginGraceTime expires for a connection. The default is 10:30:100. Alternatively, random early drop can be enabled by specifying the three colon separated values start:rate:full (e.g. "10:30:60"). sshd will refuse connection attempts with a probability of rate/100 (30%) if there are currently start (10) unauthenticated connections. The probability increases linearly and all connection attempts are refused if the number of unauthenticated connections reaches full (60).
If you cannot control the server, you might have to find a way to limit your concurrent connection attempts in your client code instead.

Scala - Cassandra: cluster read fails with error "Can't use this Cluster instance because it was previously closed"

I'm getting this error when reading from a table in a 5 node cluster using datastax drivers.
2015-02-19 03:24:09,908 ERROR [akka.actor.default-dispatcher-9] OneForOneStrategy akka://user/HealthServiceChecker-49e686b9-e189-48e3-9aeb-a574c875a8ab Can't use this Cluster instance because it was previously closed
java.lang.IllegalStateException: Can't use this Cluster instance because it was previously closed
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1128) ~[cassandra-driver-core-2.0.4.jar:na]
at com.datastax.driver.core.Cluster.init(Cluster.java:149) ~[cassandra-driver-core-2.0.4.jar:na]
at com.datastax.driver.core.Cluster.connect(Cluster.java:225) ~[cassandra-driver-core-2.0.4.jar:na]
at com.datastax.driver.core.Cluster.connect(Cluster.java:258) ~[cassandra-driver-core-2.0.4.jar:na]
I am able to connect using cqlsh and perform read operations.
Any clue what could be the problem here?
settings:
Consistency Level: ONE
keyspace replication strategy:
'class': 'NetworkTopologyStrategy',
'DC2': '1',
'DC1': '1'
cassandra version: 2.0.6
The code managing cassandra sessions is central and it is;
trait ConfigCassandraCluster
extends CassandraCluster
{
def cassandraConf: CassandraConfig
lazy val port = cassandraConf.port
lazy val host = cassandraConf.host
lazy val cluster: Cluster =
Cluster.builder()
.addContactPoints(host)
.withReconnectionPolicy(new ExponentialReconnectionPolicy(100, 30000))
.withPort(port)
.withSocketOptions(new SocketOptions().setKeepAlive(true))
.build()
lazy val keyspace = cassandraConf.keyspace
private lazy val casSession = cluster.connect(keyspace)
val session = new SessionProvider(casSession)
}
class SessionProvider(casSession: => Session) extends Logging {
var lastSuccessful: Long = 0
var firstSuccessful: Long = -1
def apply[T](fn: Session => T): T = {
val result = retry(fn, 15)
if(firstSuccessful < 0)
firstSuccessful = System.currentTimeMillis()
lastSuccessful = System.currentTimeMillis()
result
}
private def retry[T](fn: Session => T, remainingAttempts: Int): T = {
//retry logic
}

The problem is, cluster.connect(keyspace) will close the cluster itself if it experiences NoHostAvailableException. Due to that during retry logic, you are experiencing IllegalStateException.
Have a look at Cluster init() method and you will understand more.
The solution for your problem would be, in the retry logic, do Cluster.builder.addContactPoint(node).build.connect(keyspace). This will enable to have a new cluster object while you retry.

Search your code for session.close().
You are closing your connection somewhere as stated in the comments. Once a session is closed, it can't be used again. Instead of closing connections, pool them to allow for re-use.

Asterisk AMI originate call

I have configured the analog local phone with cisco adapter, so I can make any outbound call from SIP phone. But I can't achieve this by AMI which calls to outbound channel through trunk then plays prompt.
manager.conf:
[asteriskjava]
secret = asteriskjava
deny = 0.0.0.0/0.0.0.0
permit = 127.0.0.1/255.255.255.0
read = all
write = all
extensions.conf:
[bulk]
exten => 8,1,Playback(thank-you-cooperation)
exten => h,1,Hangup
source code:
public class HelloManager
{
private ManagerConnection managerConnection;
public HelloManager() throws IOException
{
ManagerConnectionFactory factory = new ManagerConnectionFactory(
"localhost", "asteriskjava", "asteriskjava");
this.managerConnection = factory.createManagerConnection();
}
public void run() throws IOException, AuthenticationFailedException,
TimeoutException
{
OriginateAction originateAction;
ManagerResponse originateResponse;
originateAction = new OriginateAction();
originateAction.setChannel("SIP/405/7000000");
originateAction.setContext("bulk");
originateAction.setExten("8");
originateAction.setPriority(new Integer(1));
originateAction.setAsync(true);
// connect to Asterisk and log in
managerConnection.login();
// send the originate action and wait for a maximum of 30 seconds for Asterisk
// to send a reply
originateResponse = managerConnection.sendAction(originateAction, 30000);
// print out whether the originate succeeded or not
System.out.println("---" + originateResponse.getResponse());
// and finally log off and disconnect
managerConnection.logoff();
}
}
Where 405 is the UserID of CISCO adapter for outgoing calls, 7000000 is a sample cell phone number.
Here is the logs:
== Manager 'asteriskjava' logged on from 127.0.0.1
== Manager 'asteriskjava' logged off from 127.0.0.1
== Using SIP RTP CoS mark 5
> Channel SIP/405-0000000c was answered.
-- Executing [8#bulk:1] Playback("SIP/405-0000000c", "thank-you-cooperation") in new stack
-- <SIP/405-0000000c> Playing 'thank-you-cooperation.gsm' (language 'en')
-- Auto fallthrough, channel 'SIP/405-0000000c' status is 'UNKNOWN'
-- Executing [h#bulk:1] Hangup("SIP/405-0000000c", "") in new stack
== Spawn extension (bulk, h, 1) exited non-zero on 'SIP/405-0000000c'
I think SIP/405 is answering, executing Playback then hangs up, not redirecting to sample number.
Any suggestions?
EDIT: How can I configure my cisco adapter in order to redirect outgoing calls, not to answer and make the bridge?

You have configure ring, answer and busy recognition on your ATA.
Asterisk work as you requested as far as i can see from your trace.
If adapter not calling, you have check with your adapater settings. For example it can be calling in tone, why you line expect it is pulse.
Also can be incorrect adapter type for your task. For calling out via PSTN line you need FXO adapter,not FXS.

Get the number of open connections in mongoDB using java

My program requires a large number of connections to be open (Mongo). I get the error :
Too many connections open, can't open anymore
after 819 connections. I already know we can increase this limit. But that's not what I have in mind. I'm thinking of closing the MongoClient object, and then creating a new one again after 800 connections.
My thinking is that with a new mongoClient object all the connections will be closed and when I start/create it again, the connections will be opened again until 800.
Thus not giving the error. (Let me know if this approach is totally wrong/ won't give the required results.)
For this I need to know the number of connections opened ATM. Is there any way to get this information using java?

You can get connection information by using the db.serverStatus() command. It has a connections subdocument which contains the total/available connections information.
For more information :
Documentation of server status
Details of connections block

Check the number of MongoDB connections using MongoDB Scala driver:
Create a MongoDB client:
import org.mongodb.scala._
import scala.collection.JavaConverters._
import scala.concurrent.Await
import scala.concurrent.duration._
import scala.util.{Failure, Success, Try}
// To directly connect to the default server localhost on port 27017
val mongodbClient: MongoClient = MongoClient()
// Use a Connection String
val mongodbClient: MongoClient = MongoClient("mongodb://localhost")
// or provide custom MongoClientSettings
val settings: MongoClientSettings = MongoClientSettings.builder()
.applyToClusterSettings(b => b.hosts(List(new ServerAddress("localhost")).asJava).
.build()
val mongodbClient: MongoClient = MongoClient(settings)
Call getNoOfMongodbConnection by passing mongodbClient:
val result = getNoOfMongodbConnection(mongodbClient)
Method to get the number of connections(current, available and total)
def getNoOfMongodbConnection(mongodbClient: MongoClient) = {
val adminDatabase = mongodbClient.getDatabase("admin")
val serverStatus = adminDatabase.runCommand(Document("serverStatus" -> 1)).toFuture()
Try {
Await.result(serverStatus, 10 seconds)
} match {
case Success(x) => {
val connection = x.get("connections")
logger.info("Number of mongodb connection:--> " + connection)
connection
}
case Failure(ex) => {
logger.error("Got error while getting the number of Mongodb connection:---> " + ex.printStackTrace())
None
}
}
}

Write a String value (10 chars) with the Modbus protocol to tag 400001

I have the following case: I need to send information to the tags 400001 through 400006 that will displayed on a PTV screen. I do not have much experience using modbus, but from what I understand is that I should use Function code 16 to set the tag value.
I currently have the following code and that seems to work in a simulator:
// Open the connection
TCPMasterConnection connection = new TCPMasterConnection(address);
connection.setPort(Modbus.DEFAULT_PORT);
connection.connect();
ModbusTCPTransaction trans = new ModbusTCPTransaction(connection); //the transaction
// Prepare the request
SimpleRegister[] registers = new SimpleRegister[1];
registers[0] = new SimpleRegister(15);
WriteMultipleRegistersRequest req = new WriteMultipleRegistersRequest(400001, registers);
req.setUnitID(1);
trans.setRequest(req);
//No we are ready for action. The last part is executing the prepared transaction the given (repeat) number of times and then for cleanup, close the connection:
log.debug("ModbusSlave: FC" + req.getFunctionCode() + " ref=" + req.getReference() + " value=" + registers[0].getValue());
trans.execute();
// Close the connection
connection.close();
But how to set a string like "0010323"?