Tomcat 8 speed gets degraded over time - java

I am using Tomcat 8.5 to host a WAR which is used for java REST services.
In my rest service, I create a connection and take a multi-part form data file from user, scan it using a scan engine and return the result. At the start, tomcat is running fine and giving a speed of almost 57-58 Mbps but degrades over time (degrades to nearly half in 5-8 min)
My setenv.bat file looks like this.
"set "JAVA_OPTS=%JAVA_OPTS% -Xms1024m -Xmx5120m -XX:MaxMetaspaceSize=512m -Xincgc -server""
JVM is using ParNewGC for garbage collection.
my server.xml file looks like this
<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
maxThreads="100" minSpareThreads="8" maxSpareThreads="10" acceptorThreadCount="16" acceptCount="500"/>
<!--acceptCount :The maximum queue length for incoming connection requests when all possible request processing threads are in use. Any requests received when the queue is full will be refused. The default value is 100.
A "Connector" represents an endpoint by which requests are received
and responses are returned. Documentation at :
Java HTTP Connector: /docs/config/http.html
Java AJP Connector: /docs/config/ajp.html
APR (HTTP/AJP) Connector: /docs/apr.html
Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
-->
<Connector executor="tomcatThreadPool" port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" socket.rxBufSize="10000000" socket.txBufSize="3000000" socket.directBuffer="true" />
<!-- A "Connector" using the shared thread pool-->
As my response is completely dynamic. I am not using any type of caching. please help me with this issue.

It may be a error due to a large number of open tcp/ip connections .Try connecting with server for once and send data check for sockets when you see a performance degradation.
In windows, you can use netstat-an to check the open sockets.

Related

Tomcat 9 slow response

I have a web application(jsp) which was running fine on Tomcat 8.0.46 for more than a year. Few weeks back we upgraded to Tomcat 9.0.10, after couple of days of upgrade tomcat is responding with a delay of 8-16 seconds for some of the request.
I saw more than 800 request/sec in localhostaccess log, so I increased maxThreads to 512 as below and max heap memory to 4096MB.
<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
maxThreads="512" minSpareThreads="4"/>
<Connector executor="tomcatThreadPool" port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" />
But the issue was not resolved, so I compared all the configuration with old Tomcat and found that tomcat9 is using tomcat executor where old was not using it. will executor impact request handling time?
Old tomcat configuration
<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" />
FYI the webapp consist of only jsps and few of them used to interact with DB using DBCP and gives XML response.
I am not suspecting DB connection pool because it was already used and no
change.
CPU : Xenon
RAM: 8GB
OS : Window 2012 server
JDK : jdk1.8.0_144
Added response time duration to localhost access log, can see the delay in some request, but the request prior and after are having quick response withing 15 milliseconds(bold).
10.50.29.27 - - [17/Dec/2018:09:27:23 -0500] "GET /App1/sendevent.jsp?TNAME=Transfer1 HTTP/1.1" 200 90 270BA450469B7AA71D22252711CA288A **0.015** http-nio-8080-exec-3
10.50.29.26 - - [17/Dec/2018:09:27:23 -0500] "GET /App1/Start.jsp?ACTION=START&ID=3154583920&SID=$num$&SESSIONID=63AA673E-B6EF-447E-AAB9-3B5B7260EB03&ScriptID=$sid$&ScriptData=$scriptdata$ HTTP/1.1" 200 2948 D97741884AD1005359430A3307D5D44E **6.031** http-nio-8080-exec-5
10.50.29.27 - - [17/Dec/2018:09:27:23 -0500] "GET /App1/sendevent.jsp?TNAME=Transfer1&TRANSFER_RESULT=S&LAST_ACTION=1&TRANSFER_REASON=connection.disconnect.transfer&TRANSFER_NOTE=undefined HTTP/1.1" 200 90 270BA450469B7AA71D22252711CA288A **0.000** http-nio-8080-exec-9
acceptorThreadCount=2 solved productivity problems for me in two cases:
Tomcat 8 under Debian on a small virtual machine (application is XWiki)
Tomcat 9 under Centos on a big production server (application is DSpace:jspui/xmlui/oai/solr).
The third case when I saw the sufficiently better productivity is:
Tomcat 8 under Windows Server 2008 Standard SP2 on a very small old Dell server (application is DSpace:jspui/xmlui/oai/solr). It is the previous instance of Case 2 kept until end of transition.

Tomcat 6 and TLSv1.2 In Java

I have a Java app deployed in tomcat 6. The app sends messages to another service via socket and it needs to use ONLY TLSv1.2 protocol.
In my tomcat6.conf file I put this configuration:
JAVA_HOME=/usr/lib/jvm/jre1.7.0_75
JAVA_OPTS="${JAVA_OPTS} -Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory -Dhttps.protocols=TLSv1.2"
But stll use the older tls version.
It there any configuration to apply in java or tomcat to force use TLSv1.2?
Edit 1:
The answer provided by #Peter Walser is good and could work. The problem is I can't modify the code because is a jar provided by third party, and I can only configure the enviroment, not the code.
The https.protocols system property is only considered for HttpsURLConnection and URL.openStream(), as stated in Diagnosing TLS, SSL, and HTTPS
Controls the protocol version used by Java clients which obtain https connections through use of the HttpsURLConnection class or via URL.openStream() operations. ...
For non-HTTP protocols, this can be controlled through the SocketFactory's SSLContext.
You can configure the SSLSocket as follows:
SSLSocketFactory factory = (SSLSocketFactory) SSLSocketFactory.getDefault();
SSLSocket socket = (SSLSocket) factory.createSocket(host, port);
socket.setEnabledProtocols(new String[] {"TLSv1.2"});
When working with REST-clients, most of them support configuring the protocols over the SSLContext. Example (JAX-RS client):
Client client = ClientBuilder.newBuilder()
.sslContext(SSLContext.getInstance("TLSv1.2"))
// more settings, such as key/truststore, timeouts, logging
.build();
If you are trying to force the server to use TLSv1.2 the following link may provide what you need.
The Apache Tomcat 5.5 Servlet/JSP Container - SSL Configuration HOW-TO
As the doc specifies edit the Tomcat Configuration File as below,
The implementation of SSL used by Tomcat is chosen automatically unless it is overridden as described below. If the installation uses APR - i.e. you have installed the Tomcat native library - then it will use the APR SSL implementation, otherwise it will use the Java JSSE implementation.
To avoid auto configuration you can define which implementation to use by specifying a classname in the protocol attribute of the Connector.
To define a Java (JSSE) connector, regardless of whether the APR library is loaded or not do:
<Connector protocol="org.apache.coyote.http11.Http11AprProtocol" port="8443" .../>
Configure the Connector in the $CATALINA_BASE/conf/server.xml file, where $CATALINA_BASE represents the base directory for the Tomcat 6 instance. An example <Connector> element for an SSL connector is included in the default server.xml file installed with Tomcat. For JSSE, it should look something like this:
<!--
<Connector
port="8443" maxThreads="200"
scheme="https" secure="true" SSLEnabled="true"
SSLCertificateFile="/usr/local/ssl/server.crt"
SSLCertificateKeyFile="/usr/local/ssl/server.pem"
clientAuth="optional" SSLProtocol="TLSv1"/>
-->
You will note that the example SSL connector elements are commented out by default. You can either remove the comment tags from around the the example SSL connector you wish to use or add a new Connector element of your own. In either case, you will need to configure the SSL Connector for your requirements and environment.
The port attribute (default value is 8443) is the TCP/IP port number on which Tomcat will listen for secure connections. You can change this to any port number you wish (such as to the default port for https communications, which is 443). However, special setup (outside the scope of this document) is necessary to run Tomcat on port numbers lower than 1024 on many operating systems.
After completing these configuration changes, you must restart Tomcat as you normally do, and you should be in business. You should be able to access any web application supported by Tomcat via SSL.
Try changing the SSLProtocol attribute in <Connector> element to SSLProtocol="TLSv1.2".
<Connector
port="8443" maxThreads="200"
scheme="https" secure="true" SSLEnabled="true"
SSLCertificateFile="/usr/local/ssl/server.crt"
SSLCertificateKeyFile="/usr/local/ssl/server.pem"
clientAuth="optional" SSLProtocol="TLSv1.2"/>

Mitigating Slow HTTP Post Vulnerability on Tomcat 8

The third party tool we used for security test is giving Slow HTTP POST Vulnerability on Tomcat 8. We have a simple Spring Controller and JSP in the application.
Existing Tomcat connector config is below:
<Connector port="8643" protocol="HTTP/1.1" SSLEnabled="true"
maxThreads="150" scheme="https" secure="true" compression="on"
clientAuth="false" sslProtocol="TLS" maxPostSize="20480"
maxSwallowSize="20480" maxHeaderCount="25" maxParameterCount="100"/>
Note that we don't have Apache or Nginx in front of tomcat. Please suggest the configs that we can use directly on Tomcat.
An example of Slow HTTP Attack is SLOWLORIS
To mitigate it with Tomcat, the solution is to use the NIO Connector, as explained in this tutorial.
What is unclear with your problem, is that Tomcat already uses the NIO connector by default on Tomcat 8, which is your configuration :
The default value is HTTP/1.1 which uses an auto-switching mechanism
to select either a non blocking Java NIO based connector or an
APR/native based connector.
Maybe should you set some other Connector parameters to specifically limit POST abuse, I suggest :
maxPostSize="1048576" (1 MByte)
connectionTimeout="10000" (10 seconds between the connection and the URI request)
disableUploadTimeout="false" (activate the POST maximum time allowed)
connectionUploadTimeout="20000" (maximum POST of 20 seconds)
An option is also to limit the headers number (default being 100), but this can have side effects with people using smartphones (which are known to send many headers) :
maxHeaderCount="25"
But it depends if your traffic is coming from Internet, or if it is a pro intranet with known users. In this latter case you could adjust the settings to be more permissive.
Edit 1: hardening with MultipartConfig
As stated on some other posts, maxPostSize might not work for limitting uploads. When using Java 7 built-in uploads, it is possible to configure limits by an annotation to the Servlet, or by configuration. It's not a pure Tomcat configuration as you asked, but it is necessary to know about it and talk with the DEV team as security must be taken in account since the early stages of development.
Edit 2: disabling chunked Transfer-Encoding
Some Slow HTTP POST attacks are based on requests sent with a Transfer-Encoding : chunked header, and then send many or an infinite number of chunks. To counter this attack, I suggest configuring a Rewrite Valve.
To achieve this, add the valve in your Host definition in server.xml :
<Valve className="org.apache.catalina.valves.rewrite.RewriteValve" />
Supposing your host name is the default one (localhost), you need to create $CATALINA_BASE/conf/Catalina/localhost/rewrite.config file with this content :
RewriteCond %{HTTP:Transfer-Encoding} chunked
RewriteRule ^(.*)$ / [F]
If necessary, you can adapt the RewriteRule to reply with something else than a 403 Forbidden which is due to the F flag. This is pure Tomcat config and flexible.

Connections stuck in CLOSE_WAIT status

I have a Java application running in JBOSS EAP 6.4.5 on Linux.
Over a period of time as multiple users logs in application then it become inaccessible( connection failed error ) with the warning message on.
JBWEB003008: Maximum number of threads (1024) created for connector with **address * and port *.******
We have noticed is that most of the connections as in CLOSE_WAIT state.
Server restart helps to resolve this issue temporary.
Not sure what's causing this.
You need to increase the maximum number of threads created for a connector in EAP6. For non-native APR connectors, the maximum number of threads which a connector can handle can be defined by adding the attribute max-connections in the subsytem as:
<subsystem xmlns="urn:jboss:domain:web:1.5" default-virtual-server="default-host" native="false">
<connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http" max-connections="2048"/>
...
</subsystem>
For APR connectors (native="true"), the max thread pool size is sized through the org.apache.tomcat.util.net.MAX_THREADS system property instead

Request entity too large error by Tomcat

Request entity too large error, The requested resource does not allow request Data with GET(OR POST) Requests, or the amount of the data provided in the request exceeds the capacity limit.
I use tomcat 7 and apache in my project. I made changes in tomcat's server.xml as
<Connector port="8109" protocol="AJP/1.3" redirectPort="8443" maxPostSize="-1" packetSize="65536"/>
still getting the same error.

Categories