I have an application running on Apache Tomcat, I am getting a connection timed out error when I try to telnet to that port. Here is the server.xml config:
<Service name="Catalina">
<!-- Define a non-SSL Coyote HTTP/1.1 Connector on port 8080 -->
<Connector port="8868" maxThreads="200" minSpareThreads="25" maxSpareThreads="75" enableLookups="true" redirectPort="8804" acceptCount="100" debug="0" connectionTimeout="60000" disableUploadTimeout="true"/>
When I do netstat -tupln, it shows that the port is listening:
netstat -tulpn | grep :8868
tcp 0 0 0.0.0.0:8868 0.0.0.0:* LISTEN 15629/java
I am able to ping the IP address, but when I do a telnet to the port I get a connection timed out. I don't think it is a firewall issue because I ran service iptables status and got
iptables: Firewall is not running.
same thing with service ip6tables status
ip6tables: Firewall is not running.
Related
I have tomcat 10.0.18 on macOS (via homebrew)
In my server.xml I have
<Connector port="8443" protocol="org.apache.coyote.http11.Http11Nio2Protocol"
maxThreads="20" SSLEnabled="true" scheme="https" secure="true" clientAuth="true" defaultSSLHostConfigName="test">
<UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
<SSLHostConfig>
<hostName>test</hostName>
<protocols>TLSv1.2</protocols>
<certificateVerification>required</certificateVerification>
<Certificate>
<caCertificateFile>/Users/jonesn/src/egeria/master/open-metadata-resources/open-metadata-deployment/certificates/EgeriaRootCA.p12</caCertificateFile>
<certificateFile>/Users/jonesn/src/egeria/master/open-metadata-resources/open-metadata-deployment/certificates/EgeriaServerChassis.p12</certificateFile>
<certificateKeyPassword>egeria</certificateKeyPassword>
</Certificate>
</SSLHostConfig>
</Connector>
However when I start & connect I see an error:
30-Mar-2022 09:51:08.547 INFO [main] org.apache.coyote.http11.AbstractHttp11Protocol.configureUpgradeProtocol The ["https-jsse-nio2-8443"] connector has been configured to
support negotiation to [h2] via ALPN
30-Mar-2022 09:51:08.547 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-jsse-nio2-8443"]
30-Mar-2022 09:51:21.162 WARNING [main] org.apache.tomcat.util.net.Nio2Endpoint.shutdownExecutor The executor associated with thread pool [https-jsse-nio2-8443] has not ful
ly shutdown. Some application threads may still be running.
30-Mar-2022 09:51:21.163 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector[org.apache.coyote.http11.Htt
p11Nio2Protocol-8443]]
org.apache.catalina.LifecycleException: Protocol handler initialization failed
at org.apache.catalina.connector.Connector.initInternal(Connector.java:1055)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:556)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1042)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.startup.Catalina.load(Catalina.java:747)
at org.apache.catalina.startup.Catalina.load(Catalina.java:769)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at java.base/java.lang.reflect.Method.invoke(Method.java:577)
at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:305)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:475)
Caused by: java.lang.IllegalArgumentException: SSLHostConfig attribute certificateFile must be defined when using an SSL connector
at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:99)
at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:71)
at org.apache.tomcat.util.net.Nio2Endpoint.bind(Nio2Endpoint.java:132)
at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1192)
at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1205)
at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:580)
at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:82)
at org.apache.catalina.connector.Connector.initInternal(Connector.java:1052)
... 11 more
Caused by: java.io.IOException: SSLHostConfig attribute certificateFile must be defined when using an SSL connector
at org.apache.tomcat.util.net.SSLUtilBase.getKeyManagers(SSLUtilBase.java:310)
at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:245)
at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:97)
... 18 more
30-Mar-2022 09:51:21.164 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [13255] milliseconds
Yes as best I can see, I do have the value assigned.
Background -- just testing some mutual SSL config for our open-source project with some self-signed certs I generated. We use springboot server-side, a variety of java, python, go, node etc client-side -- but for debugging/understanding I'm just trying base tomcat, with curl/httpie clients
The .p12 files above should contain the right combination of certs/keys, but this is the area I'm exploring. I can adapt, use individual PEMs, try a chain via a directory.. but this first issue seems more about the basics of defining a config and is what I'm trying to understand - being fairly new to tomcat.
I figured it out - so will answer my own question for the benefit of future readers.
The parsing issue was down to format. I'm so used to editing other XML files such as maven, where the above format I tried is typical. tomcat was new to me, so I realised it needs to be:
<Connector port="8443" protocol="org.apache.coyote.http11.Http11Nio2Protocol"
maxThreads="20" SSLEnabled="true" scheme="https" secure="true" clientAuth="true" defaultSSLHostConfigName="test">
<UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
<SSLHostConfig hostName="test" protocols="TLSv1.2" certificateVerification="required">
<Certificate
caCertificateFile="/Users/jonesn/src/egeria/master/open-metadata-resources/open-metadata-deployment/certificates/EgeriaRootCA.p12"
certificateFile="/Users/jonesn/src/egeria/master/open-metadata-resources/open-metadata-deployment/certificates/EgeriaServerChassis.p12"
certificateKeyPassword="egeria"
/>
</SSLHostConfig>
</Connector>
I now get other exceptions - but that is specific to the certs, something I'm debugging. I share this to answer the question specifically on why the properties could not be found.
1. What I want to do
I am currently creating a dynamic web application (JSP/Servlet), and try to setup SSL certificate in Tomcat (v8.5) to access from "https://[hostname]:8443/[AppName]".
2. Current Problem
I cannot reach the page "https://[hostname]:8443/[AppName]" and the following message is showing.
The connection timeout.
the response from the localhost server was not received at a certain time.
Cannot connect to my app using https
But somehow when I try to access the default Tomcat page, I can reach it successfully by "https://[hostname]:8443/"
Can connect to Tomcat default using https
And of course I could connect to my app using "http", like "http://[hostname]:8080/TennisDatabase"
Can connect using http
3. What I did
(1)Generated certificate.
keytool -genkey -alias tomcat -keyalg RSA -keypass [password] -keystore mycertificate.cer -storepass [password]
(2)Modify server.xml
<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
scheme="https"
secure="true"
clientAuth="false"
sslProtocol="TLS"
sslEnabledProtocols="TLSv1.2,TLSv1.1,TLSv1"
keystoreFile="conf/mycertificate.cer" keystorePass="[password]" />
(3)Allow port 8443 from iptables.
-A INPUT -m state --state NEW -m tcp -p tcp --dport 8443 -j ACCEPT
(4)Modify server.xml
Also changed permission of file to 755.
<security-constraint>
<web-resource-collection>
<web-resource-name>HTTPSOnly</web-resource-name>
<url-pattern>/*</url-pattern>
</web-resource-collection>
<user-data-constraint>
<transport-guarantee>CONFIDENTIAL</transport-guarantee>
</user-data-constraint>
(5)Restart Tomcat
4. Additional Info
When I try crul, I faced the following error.
$ curl -v --tlsv1.1 https://[hostname]:8443/[appname]
* About to connect() to [hostname] port 8443 (#0)
* Trying [IP address]... connected
* Connected to [hostname] ([IP address]) port 8443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* NSS error -5990
* Closing connection #0
* SSL connect error
curl: (35) SSL connect error
CentOS : 6.10
Tomcat : 8.5
Make sure your server.xml file has the right permission... and the main thing is you mention that you are using JSP/servlet so you have web.xml
add these lines in web.xml
<security-constraint>
<web-resource-collection>
<web-resource-name>HTTPSOnly</web-resource-name>
<url-pattern>/*</url-pattern>
</web-resource-collection>
<user-data-constraint>
<transport-guarantee>CONFIDENTIAL</transport-guarantee>
</user-data-constraint>
I have a tomcat service configured to run on port 8380. This site was running fine until recently. I started to see this in the mod_jk.log on apache:
[3409:140197396281344] [info] jk_open_socket::jk_connect.c (627): connect to 127.0.0.1:8309 failed (errno=111)
[3409:140197396281344] [info] ajp_connect_to_endpoint::jk_ajp_common.c (995): Failed opening socket to (127.0.0.1:8309) (errno=111)
[3409:140197396281344] [error] ajp_send_request::jk_ajp_common.c (1630): (worker1) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111)
[3409:140197396281344] [info] ajp_service::jk_ajp_common.c (2623): (worker1) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1)
[3409:140197396281344] [info] jk_open_socket::jk_connect.c (627): connect to 127.0.0.1:8309 failed (errno=111)
[3409:140197396281344] [info] ajp_connect_to_endpoint::jk_ajp_common.c (995): Failed opening socket to (127.0.0.1:8309) (errno=111)
[3409:140197396281344] [error] ajp_send_request::jk_ajp_common.c (1630): (worker1) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111)
[3409:140197396281344] [info] ajp_service::jk_ajp_common.c (2623): (worker1) sending request to tomcat failed (recoverable), because of error during request sending (attempt=2)
[3409:140197396281344] [error] ajp_service::jk_ajp_common.c (2643): (worker1) connecting to tomcat failed.
[3409:140197396281344] [info] jk_handler::mod_jk.c (2788): Service error=-3 for worker=worker1
I'm able to start up the tomcat service but the port it's supposed to be listening on is not accepting connections. nmap and netstat don't show the ports as available. I've also tried telnetting to to the port just in case but as i expected there's no response. I've stopped the firewall in case it's a iptables rule issue but that too had no effect. I've googled similar issues resolved by what i tried above but none of them apply to my issues. I'm not sure what else to try at this point and so I'm turning to you all!
For this particular host the CATALINA_BASE is from a mounted nfs export. I have 2 tomcat sites hosted on this server. For the sake of testing I've disabled one from starting up since they both have same issue. i figured if i can get 1 running the other will be work as well.
server.xml:
<Server port="8105" shutdown="SHUTDOWN">
<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
<Listener className="org.apache.catalina.core.JasperListener" />
<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
<Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
<GlobalNamingResources>
<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml" />
</GlobalNamingResources>
<Service name="Catalina">
<Connector port="8380" protocol="HTTP/1.1"
connectionTimeout="20000"
URIEncoding="UTF-8"
redirectPort="8443" />
<Connector port="8309" protocol="AJP/1.3" redirectPort="8443" />
<Engine name="Catalina" defaultHost="localhost">
<Realm className="org.apache.catalina.realm.LockOutRealm">
<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
resourceName="UserDatabase"/>
</Realm>
<Host name="localhost" appBase="webapps"
unpackWARs="true" autoDeploy="true">
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="localhost_access_log." suffix=".txt"
pattern="%h %l %u %t "%r" %s %b" />
</Host>
<Host name="mytomcat.site.org" appBase="webapps"
unpackWARs="true" autoDeploy="true">
</Host>
</Engine>
</Service>
</Server>
My worker.properties in apache. I have 2 workers 1 for each tomcat site.
workers.tomcat_home=/opt/tomcat
workers.java_home=/usr/local/jdk/jre
ps=/
# Define workers
worker.list=worker1
worker.list=worker2
# Set properties for worker1 - localhost
worker.worker1.type=ajp13
worker.worker1.host=127.0.0.1
worker.worker1.port=8309
worker.worker1.socket_keepalive=True
worker.worker1.socket_timeout=300
# Set properties for worker2 - localhost
worker.worker2.type=ajp13
worker.worker2.host=127.0.0.1
worker.worker2.port=8409
worker.worker2.socket_keepalive=True
worker.worker2.socket_timeout=300
Apache starts up ok and is serving static pages ok. tomcat starts ok but like i said it's not listening on any port.
Any help would be appreciated. TIA!
MORE INFO:
I ran the init script with -debug in the JAVA_OPTS var. and I see the following towards the bottom of the output that scrolls out to stdout:
...
user changed to 'daemon'
User 'daemon' validated
Attempting to locate Java Home in /usr/local/jdk
Attempting to locate VM configuration file /usr/local/jdk/jre/lib/jvm.cfg
Attempting to locate VM configuration file /usr/local/jdk/lib/jvm.cfg
Attempting to locate VM configuration file /usr/local/jdk/jre/lib/amd64/jvm.cfg
Found VM configuration file at /usr/local/jdk/jre/lib/amd64/jvm.cfg
Found VM server definition in configuration
Checking library /usr/local/jdk/jre/lib/amd64/server/libjvm.so
Found VM client definition in configuration
Checking library /usr/local/jdk/jre/lib/amd64/client/libjvm.so
Checking library /usr/local/jdk/lib/amd64/client/libjvm.so
Cannot locate library for VM client (skipping)
Found VM hotspot definition in configuration
Checking library /usr/local/jdk/jre/lib/amd64/hotspot/libjvm.so
Checking library /usr/local/jdk/lib/amd64/hotspot/libjvm.so
Cannot locate library for VM hotspot (skipping)
Found VM classic definition in configuration
Checking library /usr/local/jdk/jre/lib/amd64/classic/libjvm.so
Checking library /usr/local/jdk/lib/amd64/classic/libjvm.so
Cannot locate library for VM classic (skipping)
Found VM native definition in configuration
Checking library /usr/local/jdk/jre/lib/amd64/native/libjvm.so
Checking library /usr/localjdk/lib/amd64/native/libjvm.so
Cannot locate library for VM native (skipping)
Found VM green definition in configuration
Checking library /usr/local/jdk/jre/lib/amd64/green/libjvm.so
Checking library /usr/local/jdk/lib/amd64/green/libjvm.so
Cannot locate library for VM green (skipping)
Java Home located in /usr/local/jdk
+-- DUMPING JAVA HOME STRUCTURE ------------------------
| Java Home: "/usr/local/jdk"
| Java VM Config.: "/usr/local/jdk/jre/lib/amd64/jvm.cfg"
| Found JVMs: 1
| JVM Name: "server"
| "/usr/local/jdk/jre/lib/amd64/server/libjvm.so"
+-------------------------------------------------------
Running w/ LD_LIBRARY_PATH=/usr/local/jdk/jre/lib/amd64/server:/usr/local/jdk/jre/lib/amd64
wait_child 12035
redirecting stdout to /opt/www/mytomcat/logs/catalina-daemon.out and stderr to &1
Then it goes back to the shell prompt. If it completed successfully it would have additional lines scrolling out:
get_pidf: 5 in /opt/www/mytomcat/logs/catalina-daemon.pid
get_pidf: pid 97253
check_tmp_file: /tmp/97253.jsvc_up
get_pidf: 5 in /opt/www/mytomcat/logs/catalina-daemon.pid
get_pidf: pid 97253
check_tmp_file: /tmp/97253.jsvc_up
get_pidf: 5 in /opt/www/mytomcat/logs/catalina-daemon.pid
get_pidf: pid 97253
check_tmp_file: /tmp/97253.jsvc_up
get_pidf: 5 in /opt/www/mytomcat/logs/catalina-daemon.pid
get_pidf: pid 97253
check_tmp_file: /tmp/97253.jsvc_up
get_pidf: 5 in /opt/www/mytomcat/logs/catalina-daemon.pid
get_pidf: pid 97253
check_tmp_file: /tmp/97253.jsvc_up
get_pidf: 5 in /opt/www/mytomcat/logs/catalina-daemon.pid
get_pidf: pid 97253
check_tmp_file: /tmp/97253.jsvc_up
I also noticed both tomcat process are running as root when i expect the child process to be running as daemon. Not sure why this is happening but it may be related to the issue.
Based on the debug output I started to try a bunch of different things specifically around catalina-daemon.out and catalina-daemon.pid files. As it turns out because these files are on the nfs location it causes the odd issue of the service starting but not listening. Once I moved these to the local filesystem the service started up AND it was listening on the correct port again. I also noticed the child tomcat process switched to the unprivileged user daemon. The page is reachable again.
Not entirely sure why it was working before but all of a sudden its not when i made no changes to the server. Something must have changed in the nfs export from the zfs machine that was serving up the CATALINA_BASE. Somehow this may have exposed a bug that causes this issue on the tomcat server.
I get the following error when I try to initiate Tomcat connector on port 443 :
16-Jul-2018 09:30:08.394 SEVERE [main] org.apache.catalina.core.StandardService.initInternal Failed to initialize connector [Connector[HTTP/1.1-443]]
org.apache.catalina.LifecycleException: Failed to initialize component [Connector[HTTP/1.1-443]]
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:112)
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:549)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:875)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
at org.apache.catalina.startup.Catalina.load(Catalina.java:632)
at org.apache.catalina.startup.Catalina.load(Catalina.java:655)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:309)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:492)
I expect this error and knows why it occurred.
The issue is that although the connector has failed to start, it seems that Tomcat is still listing on port 443 and every message sent to this port hangs until I get read timeout.
How can i configure Tomcat not to listen when connector failed to start ?
I can't comment... but still, the configuration related to the ports ins't in the server.xml file?
Something like this?
<!-- Define a non-SSL HTTP/1.1 Connector on port 8180 -->
<Connector port="8180" maxHttpHeaderSize="8192"
maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
enableLookups="false" redirectPort="8443" acceptCount="100"
connectionTimeout="20000" disableUploadTimeout="true" />
Look up for the 443 port and modify it?
I know the similar questions are already asked, but unfortunately I can't find the good answer to find the solution for this problem. When I try to upload any file (XML/CSV formats accepted only), it works in local, but on the server it doesn’t work.
I don't have any timeout or read timeout error. Here you can find more details.
IMPORTANT : I have this error after correction of Hibernate Session's error, in fact I had always Session Close error in log file so I changed my code and now the Sessions will be closed after finishing the task, but I have this new error only in Server and not in LOCAL !!
Server log :
ERROR [org.apache.struts.upload.CommonsMultipartRequestHandler] (ajp-127.0.0.1-8009-6) Failed to parse multipart request
org.apache.commons.fileupload.FileUploadException: Processing of multipart/form-data request failed. null
at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:384)
at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:268)
at org.apache.struts.upload.CommonsMultipartRequestHandler.handleRequest(CommonsMultipartRequestHandler.java:182)
at org.apache.struts.util.RequestUtils.populate(RequestUtils.java:389)
at org.apache.struts.chain.commands.servlet.PopulateActionForm.populate(PopulateActionForm.java:45)
at org.apache.struts.chain.commands.AbstractPopulateActionForm.execute(AbstractPopulateActionForm.java:57)
at org.apache.struts.chain.commands.ActionCommandBase.execute(ActionCommandBase.java:48)
at org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:190)
at org.apache.commons.chain.generic.LookupCommand.execute(LookupCommand.java:304)
at org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:190)
at org.apache.struts.chain.ComposableRequestProcessor.process(ComposableRequestProcessor.java:280)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1858)
Server.xml file :
<Service name="jboss.web">
<!-- A HTTP/1.1 Connector on port 8080 -->
<Connector protocol="HTTP/1.1" port="8080" address="${jboss.bind.address}"
connectionTimeout="20000" redirectPort="8443" />
I'm assuming that you are uploading a big file and it's taking time to upload.
I would suggest to add the following to Connector with the following:
<!-- A HTTP/1.1 Connector on port 8080 -->
<Connector protocol="HTTP/1.1" port="8080" address="${jboss.bind.address}"
connectionTimeout="60000" redirectPort="8443"
connectionUploadTimeout="36000000" disableUploadTimeout="false" />
(See this related SO Question).
I hope this helps.