Jenkins headless linux slave connection error - java

I get java.net.SocketException: Connection reset when trying to connect a headless debian slave to my main Jenkins master. It's been running good for a couple of weeks without error and suddenly I can't get it to connect.
I can telnet into the master on port 6256.
Slave output:
#> java -jar ~/slave.jar -jnlpUrl https://test.tv/jenkins/computer/debian_slave_node_1/slave-agent.jnlp
Aug 12, 2013 10:10:46 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Hudson agent is running in headless mode.
Aug 12, 2013 10:10:46 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [https://test.tv/jenkins/, http://test.tv/jenkins/]
Aug 12, 2013 10:10:46 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to test.tv:6256
Aug 12, 2013 10:10:46 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Aug 12, 2013 10:10:46 AM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: Connection reset
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:189)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at hudson.remoting.ClassicCommandTransport.create(ClassicCommandTransport.java:98)
at hudson.remoting.Channel.<init>(Channel.java:391)
at hudson.remoting.Channel.<init>(Channel.java:387)
at hudson.remoting.Channel.<init>(Channel.java:348)
at hudson.remoting.Channel.<init>(Channel.java:344)
at hudson.remoting.Channel.<init>(Channel.java:332)
at hudson.remoting.Engine.run(Engine.java:238)
Jenkins Master output:
java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2596)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1316)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at hudson.remoting.Command.readFrom(Command.java:92)
at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:72)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

Hard to say what happens before knowing more. In particular:
does it fail at every connection request ? If so please check your auth.log on your server as it seems the failure happens after "status INFO: Handshaking" and before "Connected"
are your clock properly synchronized ?
what OSes are you using ? Which java version on your slaves ? check your PATH and JAVA_HOME
have you updated jenkins lately ? If so, have you updated the slave(s) ? Also have you tried reverting the update ? Which version are you using ? 1.520 took out support for JDK 1.5.
how do you run your java command ? From a terminal in an SSH session ? Is this scripted from the master ?
See also Jenkins slave jobs failing on "Unexpected termination of channel"

I've run into the same issue, which seemed to be worked-around by running jenkins-slave as root. Now, the thing periodically disconnects with that very same error, which I'm wondering is due to a LB keepalive. I'll update this answer as soon as I get more info.

In my case everything was working under root account. But under user account, node client failed to connect to server with message "failed Unexpected termination of the channel".
The problem was with access-or-write to working folder. As soon as I've reassigned owner throug "chown -R", node starts successfully.

Related

Jenkins failed to obtain

I am new to Jenkins and was trying to figure out Nodes. After I made a new node and run from agent command line I got this. Could it be related to this supposed issue I got in the node status "Java Web Start is not available for the JVM version running Jenkins"?
INFO: Using C:\Data\Slave\remoting as a remoting work directory
эю с. 14, 2021 11:06:48 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to C:\Data\Slave\remoting
Failed to obtain http://8080/computer/DESKTOP/jenkins-agent.jnlp?encrypt=true
java.net.SocketException: Network is unreachable: connect
at java.base/java.net.PlainSocketImpl.connect0(Native Method)
at java.base/java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:101)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)
at java.base/java.net.Socket.connect(Socket.java:608)
at java.base/java.net.Socket.connect(Socket.java:557)
at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:182)
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
at java.base/sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:341)
at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:362)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1253)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)
at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1015)
at hudson.remoting.Launcher.parseJnlpArguments(Launcher.java:509)
at hudson.remoting.Launcher.run(Launcher.java:347)
at hudson.remoting.Launcher.main(Launcher.java:298)
Waiting 10 seconds before retry'

Sesame 2.7.11 Change Server message

After I added more RAM to my machine, I restarted it. I discovered that Sesame is not working anymore and I keep getting the Change Server message.
I use Sesame 2.7.11, with Apache Tomcat 7-0.53 and Java 1.7.0_51_b13 on Ubuntu 13.10.
Here is the Tomcat error log:
May 20, 2014 6:29:55 PM org.openrdf.workbench.proxy.ServerValidator checkServerPrefixes
WARNING: server URL null does not have a prefix file: http: https:
May 20, 2014 6:29:55 PM org.openrdf.workbench.proxy.ServerValidator canConnect
WARNING: java.net.UnknownHostException: localhost
java.net.UnknownHostException: localhost
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)
...
at org.openrdf.workbench.proxy.ServerValidator.canConnect(ServerValidator.java:121)
at org.openrdf.workbench.proxy.ServerValidator.isValidServer(ServerValidator.java:76)
at org.openrdf.workbench.proxy.WorkbenchGateway.findWorkbenchServlet(WorkbenchGateway.java:251)
at org.openrdf.workbench.proxy.WorkbenchGateway.service(WorkbenchGateway.java:121)
at org.openrdf.workbench.base.BaseServlet.service(BaseServlet.java:136)
...
I am aware that there is a similar question (sesame 2.6.9 with tomcat 7 set up won't work), but the Tomcat error in my case seems to be different. Instead of localhost in my actual log you can find the address of my web server (so let's say data.something.com).
The answer can be found here: http://openrdf.callimachus.net/sesame/2.7/docs/users.docbook?view#Setting_the_Server__Repository_and_User_Credentials
Essentially if the workbench can't find the server and keeps loading the Change Server page, it's ok to add the IP in the Change Server page, and it should work.

Tomcat is not shutting down

I am unable to shutdown tomcat. When I do ./shutdown.sh, I don't see any error, but the process still seems to be running. When I kill the process (kill -9), the process gets killed. After this, if I do a startup, and shutdown again, I see the same problem. How do I gracefully shut it down? Why is this happening in the first place? Please help me.
devServer:bin dev$ ps -ef|grep tomcat
501 34933 1 0 11:24PM test 0:02.02 /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -Djava.util.logging.config.file=/Users/dev/admin/runtime/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/Users/dev/admin/runtime/endorsed -classpath /Users/dev/admin/runtime/bin/bootstrap.jar:/Users/dev/admin/runtime/bin/tomcat-juli.jar -Dcatalina.base=/Users/dev/admin/runtime -Dcatalina.home=/Users/dev/admin/runtime -Djava.io.tmpdir=/Users/dev/admin/runtime/temp org.apache.catalina.startup.Bootstrap start
501 34982 33786 0 11:45PM test 0:00.00 grep tomcat
devServer:bin dev$ ./shutdown.sh
Using CATALINA_BASE: /Users/dev/admin/runtime
Using CATALINA_HOME: /Users/dev/admin/runtime
Using CATALINA_TMPDIR: /Users/dev/admin/runtime/temp
Using JRE_HOME: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
Using CLASSPATH: /Users/dev/admin/runtime/bin/bootstrap.jar:/Users/dev/admin/runtime/bin/tomcat-juli.jar
devServer:bin dev$ ps -ef|grep tomcat
501 34933 1 0 11:24PM test 0:02.03 /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -Djava.util.logging.config.file=/Users/dev/admin/runtime/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/Users/dev/admin/runtime/endorsed -classpath /Users/dev/admin/runtime/bin/bootstrap.jar:/Users/dev/admin/runtime/bin/tomcat-juli.jar -Dcatalina.base=/Users/dev/admin/runtime -Dcatalina.home=/Users/dev/admin/runtime -Djava.io.tmpdir=/Users/dev/admin/runtime/temp org.apache.catalina.startup.Bootstrap start
501 34992 33786 0 11:45PM test 0:00.00 grep tomcat
devServer:bin dev$
Also, I see my catalina.out is rather too short. There is no port number info getting written to it at all. This is all I see everytime I startup.
Feb 15, 2014 11:24:23 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
Here is my log file -
devServer:logs dev$ cat catalina.2014-02-15.log
Feb 15, 2014 11:24:23 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
Something seems to be terribly messed up. I can't understand what. Please advise.
I am running tomcat 7.0.50 on Mac OS.
Judging by the logs, it is evident that tomcat is not starting up at all as it should be. I can advise a couple of other tests as well like to
Verify whether or not the ports are getting bound by the tomcat service or not. You can do this by issueing the command netstat -tuplen on the terminal and check to see if port 8080 is getting bound by the tomcat instance or not.
Check to see if you can ping the instance. Try to ping the tomcat instance from a local machine. Then try to ping the public ip of the server to verify whether that server is responding correctly or not
Verify that you have, infact, the complete tomcat template as is used by your other servers. Verify the tomcat installtion folder and compare it with the other tomcat templates as used in rest of your servers. Make sure all folders are there, all configuration (default tomcat config) files are present.
(Just a hunch) Verify read/write/execute privileges as well
Remove your application, shutdown tomcat, empty the logs, and then restart tomcat and re-deploy your application and then view the logs to see if you get some error
Point 3 seems more likely that you have an incomplete tomcat instance, so I advise you to make a copy of the tomcat instance used in any other server and use it in this server to make sure that the server environment is infact how it should be

google app engine service unavailable

I'm was following http://googcloudlabs.appspot.com/ tutorial to create new Google App Engine project. It throws me an error as per below when I try to run the from my local host, but when I deployed it working fine. (http://mynewcloudcom.appspot.com/). Please help.
HTTP ERROR: 503
Problem accessing /. Reason:
SERVICE_UNAVAILABLE
Powered by Jetty://
Eclipse Console
Mar 05, 2012 10:42:46 AM com.google.apphosting.utils.jetty.JettyLogger info
INFO: Logging to JettyLogger(null) via com.google.apphosting.utils.jetty.JettyLogger
Mar 05, 2012 10:42:46 AM com.google.apphosting.utils.config.AppEngineWebXmlReader readAppEngineWebXml
INFO: Successfully processed C:\Smartag_Eclipse_Project\DontEditProject\war\WEB-INF/appengine-web.xml
Mar 05, 2012 10:42:46 AM com.google.apphosting.utils.config.AbstractConfigXmlReader readConfigXml
INFO: Successfully processed C:\Smartag_Eclipse_Project\DontEditProject\war\WEB-INF/web.xml
Mar 05, 2012 6:42:48 PM com.google.appengine.tools.development.DevAppServerImpl start
INFO: The server is running at http://localhost:8888/
Mar 05, 2012 6:42:48 PM com.google.appengine.tools.development.DevAppServerImpl start
INFO: The admin console is running at http://localhost:8888/_ah/admin
Found the answer. I need to run in JDK1.6. Once i changed the Java Compiler(right click project -> properties -> Java Compiler -> (Check) Enable project specific settings -> change the Compiler compliance level), its works fine now. Thanks.
I had the same problem. I justed changed the port from 8888 to 80 in the Debug Configuration and it worked for me.
I had the same issue.
for me the problem was one of my filters had an exception. specifically objectify - it was not loaded properly. you can delete the filter and see if that's it
When using Java 1.7, also check for the Execution Environment and set it to JavaSE-1.7.
You can look into the Execution Environment in:
Right Click -> Properties -> Java Compiler
And you have the option above Restore Defaults and Apply.
I faced the same issue and resolved it by selecting "Automatically select an unused port" in run configuration.
Sometimes it happens there is a "hanging" process that blocks the port. Therefore changing port in Run configuration helps, as well as checking "Automatically select an unused port" in run configuration.
It happened with me too. In my case, I had Run the project and due to some error in my code, the Project ran with errors. So, I debugged it and Run it again but without cancelling the earlier one in the console and it gave me that error. After cancelling or stopping the Project, it ran fine.
i had the same issue. Seems that Plugin is stocked on creating and opening the socket for the server. i don't really know why happens, I've resolved in this way.
Open Run Configurations.
Open Server tab. Change the Embedded server Port.
Open Arguments tab.
3.1 - VM arguments box. i've deleted all that stuff there and just let this
line and save changes and run it.
-Xmx512m -javaagent:/Applications/eclipse/plugins/com.google.appengine.eclipse.sdkbundle_1.7.5/appengine-java-sdk-1.7.5/lib/agent/appengine-agent.jar
It has worked pretty good 4me.
If someone know which events or configuration change are the reason for . please let us know it
NOTE: "I'm not responsible for unexpected behave. I'm don't know if there is side effects ahead"
The solution working for me is to run appEngineServer from command line using Gradle, as comments in generated build.xml for endpoints module proposes. And gradle config includes option for Java 1.7 compatibility, so it's no reason to use Java 1.6 if You want 1.7:
gradlew modulename:appengineRun
Currently, the appengine gradle plugin's appengine devappserver
launch doesn't interact well with Intellij/AndroidStudio's
Gradle integration. As a temporary solution, please launch from the command
line. ./gradlew modulename:appengineRun
I had a condition where a servlet defined in the web.xml with non existent class name. I specified proper class name, worked!

Launching Jade In Ubuntu10.04 Linux

When I run the below command in terminal of ubuntu linux and also I set the ClassPath properly but I wasn't successful.
java jade.Boot -gui
I got following errors in terminal window:
15 Jun, 2011 6:33:10 PM jade.core.Runtime beginContainer
INFO: ----------------------------------
This is JADE snapshot - revision 6357 of 2010/07/06 16:27:34
downloaded in Open Source, under LGPL restrictions,
at http://jade.tilab.com/
----------------------------------------
Retrieving CommandDispatcher for platform null
15 Jun, 2011 6:33:11 PM jade.imtp.leap.CommandDispatcher addICP
WARNING: Error adding ICP jade.imtp.leap.JICP.JICPPeer#ae506e[Error: Not possible to launch JADE on a remote host (127.0.1.1). Check the -host and -local-host options.].
15 Jun, 2011 6:33:11 PM jade.core.AgentContainerImpl joinPlatform
SEVERE: Communication failure while joining agent platform: No ICP active
jade.core.IMTPException: No ICP active
at jade.imtp.leap.LEAPIMTPManager.initialize(LEAPIMTPManager.java:138)
at jade.core.AgentContainerImpl.init(AgentContainerImpl.java:316)
at jade.core.AgentContainerImpl.joinPlatform(AgentContainerImpl.java:482)
at jade.core.Runtime.createMainContainer(Runtime.java:165)
at jade.Boot.main(Boot.java:89)
15 Jun, 2011 6:33:11 PM jade.core.Runtime$1 run
INFO: JADE is closing down now.
help me to recover from this error.
I am so tired of this problem. It is one of many quirks and problems with JADE.
The problem you are facing is that you need to supply correct host information in the command line. Example
java jade.Boot -gui -local-host
This is all described here: http://jade.tilab.com/doc/tutorials/JADEAdmin/startJade.html
In many cases you don't have this problem and then it is sufficient to use the -gui option only. However I have discovered that Jade does not work very well when your resolves to 127.0.1.1. You can fix this by setting your IP address in or if you are on Linux edit your /etc/hosts file for a more permanent solution.
just type in JADE path..
java -cp lib/jade.jar jade.Boot -gui -local-host 127.0.0.1
(JADE has some problems to set local host adress)
The problem might be is previous java process might be still running on the local port , Make sure to check if port is in use or not if in use kill the java process before you rerun.
It's simple.
Modify /etc/hosts
Modify the line for the address 127.0.0.1, as follows:
127.0.0.1 localhost.localdomain localhost <machine_name>
Remove or comment the line with 127.0.1.1:
# 127.0.1.1 <machine_name>
and you're done.
The error isn't in the jade. It's a bug used by jade to start the default parameters with a bugged java method that should be fixed.
When jade runs the java method, it will retrieve 127.0.1.1 instead the loop-back 127.0.0.1.
I'm saying it because I tested. I read the JADE src that do this and I executed the java method that I didn't remember now.
Unfortunately it's happening in a few linux machines. In windows I didn't see it happens.
What I did to solve this problem was to edit the /etc/hosts file, like this:
Original file:
127.0.0.1 localhost
127.0.1.1 machine-name
Modified file:
127.0.0.1 localhost machine-name
127.0.1.1 machine-name
What I did was to put an alias of the machine-name on the loop-back ip address. This will bypass the bug.

Categories