We're running a web app on Tomcat 6 and Apache mod_proxy 2.2.3. Seeing a lot of 502 errors like this:
Bad Gateway!
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /the/page.do.
Reason: Error reading from remote server
If you think this is a server error, please contact the webmaster.
Error 502
Tomcat has plenty of threads, so it's not thread-constrained. We're pushing 2400 users via JMeter against the app. All the boxes are sitting inside our firewall on a fast unloaded network, so there shouldn't be any network problems.
Anyone have any suggestions for things to look at or try? We're heading to tcpdump next.
UPDATE 10/21/08: Still haven't figured this out. Seeing only a very small number of these under load. The answers below haven't provided any magical answers...yet. :)
Just to add some specific settings, I had a similar setup (with Apache 2.0.63 reverse proxying onto Tomcat 5.0.27).
For certain URLs the Tomcat server could take perhaps 20 minutes to return a page.
I ended up modifying the following settings in the Apache configuration file to prevent it from timing out with its proxy operation (with a large over-spill factor in case Tomcat took longer to return a page):
Timeout 5400
ProxyTimeout 5400
Some backgound
ProxyTimeout alone wasn't enough. Looking at the documentation for Timeout I'm guessing (I'm not sure) that this is because while Apache is waiting for a response from Tomcat, there is no traffic flowing between Apache and the Browser (or whatever http client) - and so Apache closes down the connection to the browser.
I found that if I left the Timeout setting at its default (300 seconds), then if the proxied request to Tomcat took longer than 300 seconds to get a response the browser would display a "502 Proxy Error" page. I believe this message is generated by Apache, in the knowledge that it's acting as a reverse proxy, before it closes down the connection to the browser (this is my current understanding - it may be flawed).
The proxy error page says:
Proxy Error
The proxy server received an invalid
response from an upstream server. The
proxy server could not handle the
request GET.
Reason: Error reading from remote server
...which suggests that it's the ProxyTimeout setting that's too short, while investigation shows that Apache's Timeout setting (timeout between Apache and the client) that also influences this.
So, answering my own question here. We ultimately determined that we were seeing 502 and 503 errors in the load balancer due to Tomcat threads timing out. In the short term we increased the timeout. In the longer term, we fixed the app problems that were causing the timeouts in the first place. Why Tomcat timeouts were being perceived as 502 and 503 errors at the load balancer is still a bit of a mystery.
You can use
proxy-initial-not-pooled
See http://httpd.apache.org/docs/2.2/mod/mod_proxy_http.html :
If this variable is set no pooled connection will be reused if the client connection is an initial connection. This avoids the "proxy: error reading status line from remote server" error message caused by the race condition that the backend server closed the pooled connection after the connection check by the proxy and before data sent by the proxy reached the backend. It has to be kept in mind that setting this variable downgrades performance, especially with HTTP/1.0 clients.
We had this problem, too. We fixed it by adding
SetEnv proxy-nokeepalive 1
SetEnv proxy-initial-not-pooled 1
and turning keepAlive on all servers off.
mod_proxy_http is fine in most scenarios but we are running it with heavy load and we still got some timeout problems we do not understand.
But see if the above directive fits your needs.
I know this does not answer this question, but I came here because I had the same error with nodeJS server. I am stuck a long time until I found the solution. My solution just adds slash or /in end of proxyreserve apache.
my old code is:
ProxyPass / http://192.168.1.1:3001
ProxyPassReverse / http://192.168.1.1:3001
the correct code is:
ProxyPass / http://192.168.1.1:3001/
ProxyPassReverse / http://192.168.1.1:3001/
Sample from apache conf:
#Default value is 2 minutes
**Timeout 600**
ProxyRequests off
ProxyPass /app balancer://MyApp stickysession=JSESSIONID lbmethod=bytraffic nofailover=On
ProxyPassReverse /app balancer://MyApp
ProxyTimeout 600
<Proxy balancer://MyApp>
BalancerMember http://node1:8080/ route=node1 retry=1 max=25 timeout=600
.........
</Proxy>
I'm guessing your using mod_proxy_http (or proxy balancer).
Look in your tomcat logs (localhost.log, or catalina.log) I suspect your seeing an exception in your web stack bubbling up and closing the socket that the tomcat worker is connected to.
You can avoid global timeouts or having to virtual hosts by specifying the proxy timeouts in the ProxyPass directive as follows:
ProxyPass /svc http://example.com/svc timeout=600
ProxyPassReverse /svc http://example.com/svc timeout=600
Notice timeout=600 seconds.
However this does not always work when you have load balancer. In that case you must add the timeouts in both the places (tested in Apache 2.2.31)
Load Balancer example:
<Proxy "balancer://mycluster">
BalancerMember "http://member1:8080/svc" timeout=600
BalancerMember "http://member2:8080/svc" timeout=600
</Proxy>
ProxyPass /svc "balancer://mycluster" timeout=600
ProxyPassReverse /svc "balancer://mycluster" timeout=600
A side note: the timeout=600 on ProxyPass was not required when Chrome was the client (I don;t know why) but without this timeout on ProxyPass Internet Explorer (11) aborts saying connection reset by server.
My theory is that the :
ProxyPass timeout is used between the client(browser) and the Apache.
BalancerMember timeout is used between the Apache and the backend.
To those who use Tomcat or other backed you may also want to pay attention to the HTTP Connector timeouts.
you should be able to get this problem resolved through a timeout and proxyTimeout parameter set to 600 seconds. It worked for me after battling for a while.
Most likely you should increase Timeout parameter in apache conf (default value 120 sec)
If you want to handle your webapp's timeout with an apache load balancer, you first have to understand the different meaning of timeout.
I try to condense the discussion I found here: http://apache-http-server.18135.x6.nabble.com/mod-proxy-When-does-a-backend-be-considered-as-failed-td5031316.html :
It appears that mod_proxy considers a backend as failed only when the
transport layer connection to that backend fails. Unless failonstatus/failontimeout is used. ...
So, setting failontimeout is necessary for apache to consider a timeout of the webapp (e.g. served by tomcat) as a fail (and consecutively switch to the hot spare server). For the proper configuration, note the following misconfiguration:
ProxyPass / balancer://localbalance/ failontimeout=on timeout=10 failonstatus=50
This is a misconfiguration because:
You are defining a balancer here, so the timeout parameter relates to
the balancer (like the two others).
However for a balancer, the timeout parameter is not a connection
timeout (like the one used with BalancerMember), but the maximum time
to wait for a free worker/member (e.g. when all the workers are busy
or in error state, the default being to not wait).
So, a proper configuration is done like this
set timeout at the BalanceMember level:
<Proxy balancer://mycluster>
BalancerMember http://member1:8080/svc timeout=6
... more BalanceMembers here
</Proxy>
set the failontimeout on the balancer
ProxyPass /svc balancer://mycluster failontimeout=on
Restart apache.
Related
I have Microservices implemented using Spring Boot hosted in VM. I have Apache HTTP Server which act as load balancer and have configured reverse proxy to redirect the requests to appropriate service.
Below is the entry from mod_prox.conf
<proxy:balancer://app-user>
BalancerMember https://host-name1:port loadfactor=1 route=node1 keepAlive=On smax=1 connectiontimeout=10 retry=600 timeout=600 ttl=900
BalancerMember https://host-name2:port loadfactor=1 route=node2 keepAlive=On smax=1 connectiontimeout=10 retry=600 timeout=600 ttl=900
</proxy>
ProxyPass /service/user balancer://app-user
ProxyPassReverse /service/user balancer://app-user
When performing load test I am facing below errors, request is not reaching the service, when I looked into apache error logs,
client AH00898: Error reading from remote server returned by /service/user/All referer https://my-app.com/user
AH01102: error reading status line from remote server referer https://my-app.com/user
After analysing I see it could be because of my service server got overloaded and due it which Apache server throws error ? To resolve this I need to increase extra VM or run another instance with different port ?
I am setting up my tomcat 8 server to use a SSL connection and the application is working fine - the redirect from HTTP to HTTPS is good, but I need to find a way to allow HTTP for some pages (API calls).
Why do I need that? because those API calls are trying to upload/download something to/from the server and because the connection is secure, those files are firstly - encrypted, secondly - decrypted and finally - used. And because the CPU has low performance, the upload/download speed is very poor.
I've tried to change configuration from conf/web.xml file, with no success.
If I change the parameter from CONFIDENTIAL to NONE, but both connection types (HTTP /HTTPS) will be enabled - and this is not what I need.
Any help in this direction is appreciated.
Thank you,
If you are using the Linux system like Ubuntu, then, instead of setting up an SSL connection setup in Tomcat, you will use the Nginx server. Use the following link for installation. In the Nginx server configuration file, nginx.conf, you can define a location inside the server name setting and then you can filter the APIs that you do not want to enable as HTTPS.
For setting SSL in Nginx, you have to use the ssl_certificate & ssl_certificate_key setting.
Environment:
Apache2
nanohttpd (web server (java))
The site itself is and has been working fine, I have added alot of entries to a database which the nanohttpd web server returns on some endpoints
I have figured out that is has todo with the length of the site being returned since when I move entries from one status to another, the one with alot of entries will always give me this error.
I already looked around a bit and think it may be a timeout issue but I have added timeouts to my virtual host and this problem still persists
Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /foo
Reason: Error reading from remote server
Apache/2.4.10 (Debian) Server at foo.foo.foo Port 443
My current VirtualHost file for this:
<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerAdmin webmaster#localhost
DocumentRoot /var/www/directory
ServerName foo.foo.foo
ServerAlias foo.foo.foo
ProxyPass "/" "http://foo.com:8080/"
ProxyPassReverse "/" "http://foo.com:8080/"
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
Include /etc/path/options-ssl-apache.conf
SSLCertificateFile /etc/path/foo.com-0001/fullchain.pem
SSLCertificateKeyFile /etc/path/foo.com-0001/privkey.pem
</VirtualHost>
</IfModule>
What I had tried earlier was adding timeout=x and connectiontimeout=x behind ProxyPass but this didnt seem to help
Edit:
Since there is still no answer here is some more info, I will add a bounty once eligible.
As you can see in my VirtualHost file I am using Apache Reverse Proxy to enable SSL for my application
So my little nanohttpd server basically returns a very simplistic html page with a table and a row for each database entry. If the specific category has more than around 100 entries I will get the Proxy Error message.
At this point I am very certain that it has todo with something timeing out but I have went through so many posts and tried to add configs to mine and (yes I restarted Apache2 after every change) nothing worked so far.
Could it be something on my nanohttpd server that I am missing?
A simple Header or Cookie, I am sending none of those as of now.
If any required information is missing please let me know.
Some of your input may have been malformed.
Well funnily enough I figured it out and I feel horrible now..
The table data I was inserting included a dollar sign and apparently I did not think of escaping the replacement String
I was looking over this guide to setup tomcat + apache with SSL: http://www.mulesoft.com/tcat/tomcat-ssl
Under section, "When To Use SSL With Tomcat" it says:
"...In other words, if you're fronting Tomcat with a web server and using it only as
an application server or Tomcat servlet container, in most cases you should let the web server function as a proxy for all SSL requests"
Since I already have a webserver set up with SSL, I decided to be lazy. I installed tomcat with default settings, and started it up. In my httpd.conf, I redirected all 80 traffic to 443, and then proxypass and proxypassreverse to ajp://hostname.com:8009. I restarted httpd and it "appears" to redirect to tomcat server over ssl. Is this completely broken or did I actually manage to do what I intended on first go? Any test suggestions are much appreciated.
<VirtualHost *:80>
ServerName hostname_DNS_alias.com
Redirect / https://hostname_DNS_alias.com
</VirtualHost>
<VirtualHost *:443>
SSLEngine On
SSLCertificateFile /etc/pki/tls/certs/thecrt.crt
SSLCertificateKeyFile /etc/pki/tls/private/thekey.key
SSLCertificateChainFile /etc/pki/tls/certs/CA.crt
ServerName hostname_DNS_alias.com
DocumentRoot /var/www/html
<Proxy *>
AddDefaultCharset off
Order deny,allow
Allow from all
</Proxy>
ProxyPass / ajp://hostname.com:8009/
ProxyPassReverse / ajp://hostname.com:8009/
</VirtualHost>
I think you've got it, but you can look at the access logs on HTTPD & Tomcat to confirm the request is being proxied. You should see an access log entry on both systems.
A couple quick notes...
As mentioned in the comment, you can remove the HTTP connector from Tomcat. It's not a must though. Sometimes it nice to keep open for testing purposes (i.e. you can hit the server directly) or if you want to run the Manager app on it. If you do keep it around, especially if you use it to run the Manager app, you should probably restrict access to it. Two easy ways to do that are by setting the address attribute on the HTTP connector to localhost or by configuring a RemoteAddressFilter.
Keep in mind that the AJP connection from your HTTPD server to Tomcat is not encrypted (SSL is terminated at HTTPD), so you want to make sure that traffic never goes over an insecure network (like the Internet).
Since you already have HTTPD in the mix, you can also use it to serve up your static files. If you deploy them to your document root, you can then add a "ProxyPass !" directive to exclude that path from being proxied to Tomcat. This will offer slightly less latency on the request as HTTPD does need to get the static file from Tomcat.
I have this error in my apache acces log.
I work with apache 2.2 and tomcat 7. In the access file i see that apache send request to tomcat but tomcat can't treat them all. And when the error occures, i must restart tomcat to make my application work.
I have 2 j2ee applications application on tomcat. That application use connexion to posgres database via jdbc connexion pool. I see in my application log their is at least 4 free connexions. So i don't think there is trouble with database.
Does someone know what could cause this error ? Thank you in advence !
Try adding following apache parameters inside the virtual host section which should be in httpd.conf file or any other place per your server configuration
Timeout 2400
ProxyTimeout 2400
ProxyBadHeader Ignore
This resolved problem in my case. I am using Apache 2.4 and tomcat ajp/http configurations.
It seems that your application take more time to respond that apache ProxyTimeout
ProxyTimeout
try to increase proxytimeout in you vhost/httpd configuration and try again
The root cause is slowness in you application. you need to find the root cause of slowness
The settings for the packet data transfer needs to be increased which is now low and the ab tool is not able to serve any data after the completion of the mentioned packets.
You can change the configuration as below and then can enjoy running ab tool for your applications.
On Ubuntu Server you can find them in /etc/sysctl.conf
net.ipv4.netfilter.ip_conntrack_max = 32768
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_orphan_retries = 1
net.ipv4.tcp_fin_timeout = 25
net.ipv4.tcp_max_orphans = 8192
net.ipv4.ip_local_port_range = 32768 61000