We Keep Coding

HDFS: namenode.FSEditLog (JournalSet.java:mapJournalsAndReportErrors(390)) - Error: flush failed for required journal

On one of our platforms, HDFS namenode is shutting down with following error message every 1 or 3 days
FATAL namenode.FSEditLog (JournalSet.java:mapJournalsAndReportErrors(390)) - Error: flush failed for required journal (JournalAndStream(mgr=QJM to [<ip1>:<port>,<ip2>:<port>, etc], stream=QuorumOutputStream starting at txid 29873171))
java.io.IOException: Timed out waiting 20000ms for a quorum of nodes to respond.
at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:137)
at org.apache.hadoop.hdfs.qjournal.client.QuorumOutputStream.flushAndSync(QuorumOutputStream.java:109)
at org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:113)
at org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:107)
at org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream$8.apply(JournalSet.java:525)
at org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:385)
at org.apache.hadoop.hdfs.server.namenode.JournalSet.access$100(JournalSet.java:55)
at org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream.flush(JournalSet.java:521)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:710)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.run(FSEditLogAsync.java:188)
at java.lang.Thread.run(Thread.java:748)
Before this FATAL log we can see following kind of logs, on which we can detect a degradation of response time
WARN client.QuorumJournalManager (QuorumCall.java:waitFor(185)) - Waited 18014 ms (timeout=20000 ms) for a response for sendEdits. Succeeded so far: [<ip1>:<port>,<ip2>:<port>]
Have you already encountered this problem, and do you have any advices to fix it ?
We have already:
checked that our VMs are time synchronized
detected that when the problem occurs a burst of data on the network is in progress, without detecting the root cause yet
checked our network devices. Except a problem on a port which goes from UP state to DOWN state quickly that we are going to fix, the network seems correct
Thanks in advance

java.net.BindException: Address already in use: connect (I did read the others before posting)

I know there are a ton of these posts, but this is a little different. We are using vended code for part of our data processing system, and part of the system sends emails to clients if certain events take place on data insertion or deletion. Recently we have started getting address already in use exceptions. We checked the repository history, and nothing has changed in our code in the last 6 months for this system. We have already tried the typical solutions for this issue including increasing the number of connections allowed to the port with little success. We had a meeting with the vendor, and I asked if anything had changed in their code, and if they would assure that all connections in their code are explicitly closed. They indicated that they are explicitly closing all sockets. However they didn't show us the code so there is no way for us to know if this is true other than taking their word for it. So, the only thing I can think of to do is continue to increase the number of connections to the port until we stop getting bind exceptions. So, what is the industry standard for max number of connections to port 25; is there one? Also if anyone has any other suggestions I would greatly appreciate it? Thanks so much in advance, Robert
20210505112127.716 ERROR m.fiserv.ppx.business.notification.EmailNotifier : MessagingException from notify
javax.mail.MessagingException: Could not connect to SMTP host: SERVER.URL.COM, port: 25;
nested exception is:
java.net.BindException: Address already in use: connect
at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:1545)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:453)
Caused by:
java.net.BindException: Address already in use: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:90)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:380)
20210505131529.950 ERROR erv.ppx.web.controller.AuditReportViewController : Error while generating HTML
net.sf.jasperreports.engine.JRException: Error writing to OutputStream writer : CorpAdminAuditReport
at net.sf.jasperreports.engine.export.JRHtmlExporter.exportReport(JRHtmlExporter.java:496)
at com.fiserv.ppx.web.controller.AuditReportViewController.generateReport(AuditReportViewController.java:184)
Caused by:
com.ibm.wsspi.webcontainer.ClosedConnectionException: OutputStream encountered error during write
at com.ibm.ws.webcontainer.channel.WCCByteBufferOutputStream.write(WCCByteBufferOutputStream.java:188)
at com.ibm.ws.webcontainer.srt.SRTOutputStream.write(SRTOutputStream.java:97)
20210505140706.240 ERROR com.fiserv.ppx.business.db.DBConnectionUtil : Exception in getting for AppServer connection from DataSource.
com.ibm.websphere.ce.cm.ConnectionWaitTimeoutException: J2CA1010E: Connection not available; timed out waiting for 180,005 seconds.
at com.ibm.ws.rsadapter.AdapterUtil.toSQLException(AdapterUtil.java:1680)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:661)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:611)
Caused by:
com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException: J2CA1010E: Connection not available; timed out waiting for 180,005 seconds.
at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1781)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3834)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3082)
20210505140731.341 ERROR com.fiserv.ppx.business.db.DBConnectionUtil : Exception in getting for AppServer connection from DataSource.
com.ibm.websphere.ce.cm.ConnectionWaitTimeoutException: J2CA1010E: Connection not available; timed out waiting for 180,010 seconds.
at com.ibm.ws.rsadapter.AdapterUtil.toSQLException(AdapterUtil.java:1680)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:661)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:611)
Caused by:
com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException: J2CA1010E: Connection not available; timed out waiting for 180,010 seconds.
at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1781)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3904)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3082)
20210505140731.341 ERROR com.fiserv.ppx.sso.controller.SSOController : SSO Configuration error
java.lang.NullPointerException
at com.fiserv.ppx.business.db.PPXDbTransactionManager.<init>(PPXDbTransactionManager.java:60)
at com.fiserv.ppx.sso.impl.SSOLoginAuthenticator.authenticateSSOUser(SSOLoginAuthenticator.java:157)

Websocket blocks without timing out on client plug pull/power loss

I have an odd issue with my Tomcat + Spring websocket application. When a user disconnects without sending a "closing" signal, due to power loss or a plug pull, the thread will block about 10 seconds later.
The thread blocks on this function :
org.springframework.web.socket.WebSocketSession.sendMessage(WebSocketMessage<?> wsm) throws IOException;
I have tried putting a line in my AppConfig to try and set a timeout of 3 seconds but it does not seem to work properly as the block seems to go on for upwards of 15 minutes before throwing an exception.
#Bean(name="servletServerContainerFactoryBean")
public int maxSessionIdleTimeout() {
return 3000;
}
Here is the stack trace after an eventual SocketTimeoutException
Step: 2304
SendB -> test isOpen -> sendMes -> Done -> Finished Send.
SendB -> test2 isOpen -> sendMes -> User closed connection during packet sending: s01
Propogating exception up!
java.io.IOException: java.net.SocketTimeoutException
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlock(WsRemoteEndpointImplBase.java:315)
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlock(WsRemoteEndpointImplBase.java:250)
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendPartialString(WsRemoteEndpointImplBase.java:223)
at org.apache.tomcat.websocket.WsRemoteEndpointBasic.sendText(WsRemoteEndpointBasic.java:49)
at org.springframework.web.socket.adapter.standard.StandardWebSocketSession.sendTextMessage(StandardWebSocketSession.java:197)
at org.springframework.web.socket.adapter.AbstractWebSocketSession.sendMessage(AbstractWebSocketSession.java:102)
at org.infpls.noxio.game.module.game.session.NoxioSession.sendPacket(NoxioSession.java:40)
at org.infpls.noxio.game.module.game.dao.lobby.GameLobby.step(GameLobby.java:117)
at org.infpls.noxio.game.module.game.dao.lobby.GameLobby$GameLoop.run(GameLobby.java:274)
Caused by: java.net.SocketTimeoutException
at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.doWrite(WsRemoteEndpointImplServer.java:81)
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.writeMessagePart(WsRemoteEndpointImplBase.java:494)
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlock(WsRemoteEndpointImplBase.java:309)
... 8 more
## CRITICAL ## Ejecting player: test2 :: Exception thrown on packet send...
java.io.IOException: java.net.SocketTimeoutException
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlock(WsRemoteEndpointImplBase.java:315)
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlock(WsRemoteEndpointImplBase.java:250)
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendPartialString(WsRemoteEndpointImplBase.java:223)
at org.apache.tomcat.websocket.WsRemoteEndpointBasic.sendText(WsRemoteEndpointBasic.java:49)
at org.springframework.web.socket.adapter.standard.StandardWebSocketSession.sendTextMessage(StandardWebSocketSession.java:197)
at org.springframework.web.socket.adapter.AbstractWebSocketSession.sendMessage(AbstractWebSocketSession.java:102)
at org.infpls.noxio.game.module.game.session.NoxioSession.sendPacket(NoxioSession.java:40)
at org.infpls.noxio.game.module.game.dao.lobby.GameLobby.step(GameLobby.java:117)
at org.infpls.noxio.game.module.game.dao.lobby.GameLobby$GameLoop.run(GameLobby.java:274)
Caused by: java.net.SocketTimeoutException
at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.doWrite(WsRemoteEndpointImplServer.java:81)
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.writeMessagePart(WsRemoteEndpointImplBase.java:494)
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlock(WsRemoteEndpointImplBase.java:309)
... 8 more
Finished Send. Step Finished.
Having threads be blocked for 15 minutes at a time is a major problem. Any info on why this happens and how to fix it would be greatly appreciated. Thank you!

Found an answer finally. It's actually a system setting.
Source
How many times to retry before deciding that something is wrong and
it is necessary to report this suspicion to network layer. Minimal RFC
value is 3, it is default, which corresponds to 3sec-8min depending on
RTO.
/proc/sys/net/ipv4/tcp_retries2

Signature length not correct: got 127 but was expecting 128

I am facing a strange problem after java 1.8 upgrade. I am using jsch-0.1.54.jar in one of our utility programs to download files from various places. This particular utility was being used for almost 4-5 years without any problem(back then it jsch-0.1.48). At that time environment was java 1.6. Recently we upgraded to java 1.8 and as a result we upgraded this particular utility. Now we are encountering a strange problem and it occurs occasionally, and most of the time the download of files are prefect.
Error log
INFO: SSH_MSG_KEXDH_INIT sent
INFO: expecting SSH_MSG_KEXDH_REPLY
INFO: Disconnecting from SRV2000 port 22
2016-10-28 08:42:18:0576 ERROR [main] net.AerisAbstractMethod - Failed to open connection
com.jcraft.jsch.JSchException: Session.connect: java.security.SignatureException: Signature length not correct: got 127 but was expecting 128
at com.jcraft.jsch.Session.connect(Session.java:565)
at com.jcraft.jsch.Session.connect(Session.java:183)
at com.aeris.net.AerisSFTPMethod.connectToServer(AerisSFTPMethod.java:65)
at com.aeris.net.AerisAbstractMethod.getListOfFiles(AerisAbstractMethod.java:143)
at com.aeris.worker.AerisUploaderDownloader.performUploadDownloadListing(AerisUploaderDownloader.java:112)
at com.aeris.main.AerisCommonSftpUtility.main(AerisCommonSftpUtility.java:102)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.simontuffs.onejar.Boot.run(Boot.java:340)
at com.simontuffs.onejar.Boot.main(Boot.java:166)
Success Log: (in most of the cases it is success)
INFO: SSH_MSG_KEXDH_INIT sent
INFO: expecting SSH_MSG_KEXDH_REPLY
INFO: ssh_rsa_verify: signature true
WARN: Permanently added 'SRV2000' (RSA) to the list of known hosts.
INFO: SSH_MSG_NEWKEYS sent
INFO: SSH_MSG_NEWKEYS received
INFO: SSH_MSG_SERVICE_REQUEST sent
INFO: SSH_MSG_SERVICE_ACCEPT received
INFO: Authentications that can continue: publickey,password,keyboard-interactive
INFO: Next authentication method: publickey
INFO: Authentication succeeded (publickey).
2016-10-28 08:36:48:0794 INFO [main] net.AerisAbstractMethod - Session connected to server
2016-10-28 08:36:48:0794 INFO [main] net.AerisAbstractMethod - Opening SFTP channel..
2016-10-28 08:36:48:0810 INFO [main] net.AerisAbstractMethod - Connecting to server through channel.
2016-10-28 08:36:48:0857 INFO [main] net.AerisAbstractMethod - Connection successful.
2016-10-28 08:36:48:0857 INFO [main] net.AerisAbstractMethod - Changing to directory:C:/interfaces/ib/wf/work
2016-10-28 08:36:48:0888 INFO [main] net.AerisAbstractMethod - Start file Listing of the remote directory:C:/interfaces/ib/wf/work
0 Oct 28, 2016 04:15 ./
0 Oct 28, 2016 04:15 ../
I did a complete analysis with Vandyke (sftp software provider) but did not find any error at that end. I also tried to sftp using different tools but I am not getting any error. Following is the code snippet to connect of SFTP server. Can any one help in this matter?
protected void connectToServer() throws AerisConnectionException {
JSch jSch =(JSch)this.client;
try {
session = jSch.getSession(config.getUsername(), config.getRemoteserver(), config.getPort());
LOGGER.info("Creating SSH Session using Username:"+config.getUsername()+ " Server :" +config.getRemoteserver()+ " at PORT:"+config.getPort());
if(config.getAuth().equalsIgnoreCase("PASSWD")||config.getAuth().equalsIgnoreCase("KEYPASS")){
LOGGER.info("Setting password ...");
session.setPassword(config.getPassword());
}
Properties jShconfig = new Properties();
jShconfig.put("StrictHostKeyChecking", "no");
jShconfig.put("PreferredAuthentications",
"publickey,password,keyboard-interactive");
jShconfig.put("LogLevel", "VERBOSE");
LOGGER.info("Setting timeout to "+config.getTimeOut());
session.setTimeout(config.getTimeOut()*1000);
session.setConfig(jShconfig);
session.connect();
LOGGER.info("Session connected to server");
this.connected=true;
} catch (JSchException e) {
LOGGER.error("Failed to open connection ",e);
throw new AerisConnectionException("Failed to open connection.");
}
}

Although it would be nice to have the stacktrace to confirm, I'll bet the server is using an RSA 'host' key to authenticate and is wrongly 'trimming' leading zero in rare cases.
RSA signature values (and encrypted values also) defined by PKCS#1, which SSH uses (as do many other things including SSL), are required to be encoded as an octet string of fixed length 'k' equal to the length required to encode the modulus, or informally 'the same size as the modulus'. However, since the underlying mathematical value is a large nonnegative (aka unsigned) integer, specifically modexp(s,d,n), historically some implementations have omitted leading zero octets -- an omission which is valid when treating the value as an integer -- resulting in an encoded value that is sometimes shorter than it should be.
The RSA signature (or encrypted) value is effectively a uniform random number in (1,n). Thus when the RSA key used by the server has a 'round binary' size like 1024 as here, this trimming will happen approximately 1 in 200 times at random, or 400 if trimmed as a signed number.
I did not know, but on testing I confirm that (Oracle) Java 6 does accept such a 'short' value for Signature type RSA or as actually used here SHA1withRSA, both of which imply the PKCS1-v1_5 scheme, but Java 7 and 8 throw the exception you saw.
OTOH both OpenSSH and PuTTY (also used by WinSCP and FileZilla) do accept 'short' values, while always sending correct length values; this Postelian behavior may make it more difficult to detect when a peer is misbehaving this way. (Note: I checked OpenSSH 5.5 and 7.3, the oldest and newest I have conveniently at hand, but only the current PuTTY 0.67 because that's all I keep online.)
You can try pointing the server software implementor to the published standards, but it may not do any good. You could ask jcraft to special-case this; they already have logic in the DSA and ECDSA cases for mpint/ASN.1 that I might argue is equally ugly.
Or if the server has another (usable) key, request that by configuring "server_host_key" to NOT include ssh-rsa -- easiest just get the existing/default list, split, remove "ssh-rsa" (and check not empty) and rejoin, instead of possibly confusing your users and/or (co)maintainers by listing today's specific algorithms.

java.net.SocketException: Broken pipe with SMTP

I got strange behavior when my application tries to send an email.
20:59:08,926 ERROR [release.com.mycompany.mail.GenericMail] (EJB default - 5) [MY_EJB INBOUND] Sending message failed!: javax.mail.MessagingException: Can't send command to SMTP host;
nested exception is:
java.net.SocketException: Broken pipe
at com.sun.mail.smtp.SMTPTransport.sendCommand(SMTPTransport.java:2106) [mail-1.4.4-redhat-2.jar:1.4.4-redhat-2]
at com.sun.mail.smtp.SMTPTransport.sendCommand(SMTPTransport.java:2093) [mail-1.4.4-redhat-2.jar:1.4.4-redhat-2]
at com.sun.mail.smtp.SMTPTransport.close(SMTPTransport.java:1184) [mail-1.4.4-redhat-2.jar:1.4.4-redhat-2]
at javax.mail.Transport.send0(Transport.java:197) [mail-1.4.4-redhat-2.jar:1.4.4-redhat-2]
at javax.mail.Transport.send(Transport.java:124) [mail-1.4.4-redhat-2.jar:1.4.4-redhat-2]
at com.mycompany.MailUtils.sendMail(MailUtils.java:258) [classes:]
Before this exception some timeout exception has been thrown:
20:57:50,291 ERROR [org.jboss.as.ejb3] (EJB default - 9) [ ] JBAS014122: Error during
retrying timeout for timer: [id=6be904b5-c1ef-4f0e-a277-d4c9f93e21b3 timedObjectId=SOME_EJB
auto-timer?:false persistent?:false timerService=org.jboss.as.ejb3.timerservice.TimerServiceImpl#99fdab1 initialExpiration=/* date */ 00:00:00 UTC 2015 intervalDuration(in milli sec)=0 nextExpiration=/* other date */ 21:00:00 UTC 2015
timerState=RETRY_TIMEOUT: javax.ejb.EJBTransactionRolledbackException: JBAS014373:
EJB 3.1 PFD2 4.8.5.5.1 concurrent access timeout on org.jboss.invocation.InterceptorContext$Invocation#66668127 - could not obtain lock within 5000MILLISECONDS
To be honest i don't know whats happened. I got these kind of exception for 5 hours.
I want to know whats happened and be able to avoid exceptions in future.
UPDATE 1
SOME_EJB is an ejb with works with timerService. Runs every 3 minutes, when the conditions are met sends an email.
My only idea is that there was some network/database issue and it caused that single execution of task took more than 3 minutes.
MailUtils is a #Stateless ejb

Is the mail sent or not?
It looks like JavaMail is closing the connection to the mail server when the exception occurs. If a previous error caused the server to drop the connection immediately, JavaMail may be getting this exception while trying to send the SMTP BYE command before closing the connection.
Turn on JavaMail Session debugging to see what error the server might have reported before this exception occurs.