I have Microservices implemented using Spring Boot hosted in VM. I have Apache HTTP Server which act as load balancer and have configured reverse proxy to redirect the requests to appropriate service.
Below is the entry from mod_prox.conf
<proxy:balancer://app-user>
BalancerMember https://host-name1:port loadfactor=1 route=node1 keepAlive=On smax=1 connectiontimeout=10 retry=600 timeout=600 ttl=900
BalancerMember https://host-name2:port loadfactor=1 route=node2 keepAlive=On smax=1 connectiontimeout=10 retry=600 timeout=600 ttl=900
</proxy>
ProxyPass /service/user balancer://app-user
ProxyPassReverse /service/user balancer://app-user
When performing load test I am facing below errors, request is not reaching the service, when I looked into apache error logs,
client AH00898: Error reading from remote server returned by /service/user/All referer https://my-app.com/user
AH01102: error reading status line from remote server referer https://my-app.com/user
After analysing I see it could be because of my service server got overloaded and due it which Apache server throws error ? To resolve this I need to increase extra VM or run another instance with different port ?
I'm trying to deploy a solution using Open Trip Planner, and everything is OK if I use HTTP, but apparently the HTTPS connection doesn't work.
I've followed the official docs but with no success, apparently the internal server is running, it logs that the expected HTTPS port is listening and the port is actually shown as listening by the OS (Windows 10 Pro) but no secure connection can be established (I tried the "curl" and "open-ssl" tests in the page but both failed)
This is the document I refer to:
http://docs.opentripplanner.org/en/latest/Security/#security
Please any help is appreciated, thanks in advance
Is using a reverse proxy like nginx an option for you? That way nginx can handle the HTTPS requests, and then pass them onto opentripplanner.
Here's an example nginx configuration:
server {
listen 443;
ssl on;
ssl_certificate /etc/ssl/cacert.pem;
ssl_certificate_key /etc/ssl/privkey.pem;
server_name opentripplanner.example.com;
proxy_pass http://127.0.0.1:8000;
}
References:
https://manual.seafile.com/deploy/https_with_nginx.html
https://nginx.org/en/docs/beginners_guide.html
I have serverA with haproxy and configuration:
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 4096
daemon
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
maxconn 2000
contimeout 5000
clitimeout 50000
srvtimeout 50000
frontend http-in
bind *:80
default_backend servers
backend servers
option httpchk OPTIONS /
option forwardfor
stats enable
stats refresh 10s
stats hide-version
stats scope .
stats uri /admin?stats
stats realm Haproxy\ Statistics
stats auth admin:pass
cookie JSESSIONID prefix
server tomcat1 serverB:10039 cookie JSESSIONID_SERVER_1
server tomcat2 serverC:10039 cookie JSESSIONID_SERVER_2
Now, i goes to http://serverA/admin?stats and got statistic. On servers serverB and serverC installed WebSphere Application Server and WebSphere Portal Server (WAS it is like Tomcat and WPS it is like any application deployed to Tomcat). It hosts on port 100039. Now i goes to http://serverA/wps/portal and got my portal, but when i click on any link to any page, i got redirect to http://serverA:100039/wps/portal/bla/bla, this happens because WPS response with its port - 100039, but my haproxy configuration listen only 80 port. What i've tried:
option forwardfor
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
option httpchk HEAD / HTTP/1.1\r\nHost:localhost
For an example, in nginx i got like this:
My application hosts on 3000 port and usefull part of my nginx configuration looks like this:
location #ruby {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_read_timeout 300;
proxy_pass http://app; #upstream app
}
How i can do this in HAProxy?
This question is similar to WebSphere Portal behind reverse proxy and getServerPort()
I think the issue is that WebSphere Application Server (traditional) doesn't honor host headers properly, which can impact getting reverse proxies to work.
Try the settings recommended in that other answer (adjust the apache configuration setting for haproxy), and all should be well.
In your backend section, use "http-request set-header" to set $WSSP and $WSSN to your client-visible hostname and port. They will then be used for self-referential redirects.
Or, set the websphere custom properties trusthostheaderport and com.ibm.ws.webcontainer.extractHostHeaderPort (http://www-01.ibm.com/support/docview.wss?uid=swg21569667) to respect the port in the Host: header.
With this option you may need to ask HAProxy to set the host header to the clients view with "http-request set-header Host" also in the backend section. I'm not sure what the default is.
I've been running a site for the past 3 years with out any issues. The current SSL certificate was applied the past December. It is running on the FreeBSD 10 operating system. Apache is configured as a reverse proxy passing to Tomcat 6.0. As of this week the initial loading of pages has become extremely slow and does not leave any messages for the user. Nothing seems to stand out in my logs either. I will try to post the configuration below in hopes someone with a fresh set of eyes sees something I'm overlooking. There have been no modifications to the configuration since the site was stood up. If you need any additional info I'll be happy to provide.
Thanks
<VirtualHost *:443>
# General setup for the virtual host
DocumentRoot "/usr/local/apache-tomcat-6.0/webapps"
ServerName hsc.myissinc.com
ServerAdmin support#myissinc.com
ServerSignature Off
Timeout 60
#KeepAlive On
ErrorLog /var/log/apache2/ajp.error.log
LogLevel warn
CustomLog /var/log/apache2/ajp.log combined
RewriteEngine On
#RewriteOptions inherit
ProxyVia On
ProxyRequests Off
ProxyPreserveHost Off
#ProxyPassReverseCookieDomain localhost hsc.myissinc.com
ProxyPass /easehsc/PublicTempStorage/ http://localhost:8080/easehsc/PublicTempStorage/ retry=10 acquire=3000 connectiontimeout=300 timeout=300 Keepalive=on
ProxyPassReverse /easehsc/PublicTempStorage/ http://localhost:8080/easehsc/PublicTempStorage/
ProxyPass /easehsc/ http://localhost:8080/easehsc/ retry=10 acquire=3000 connectiontimeout=300 timeout=300 Keepalive=on
ProxyPassReverse /easehsc/ http://localhost:8080/easehsc/
ProxyPass / http://localhost:8080/ retry=10 acquire=3000 connectiontimeout=300 timeout=300 Keepalive=on
ProxyPassReverse / https://localhost:8080/
# SSL Engine Switch:
# Enable/Disable SSL for this virtual host.
SSLEngine on
SSLProxyEngine on
# Server Certificate:
SSLCertificateFile "/root/sslCA/myissinc.cer"
# Server Private Key:
SSLCertificateKeyFile "/root/sslCA/myissinc.key"
# Server Chain File (Intermediate Cert)
SSLCertificateChainFile "/root/sslCA/myissincint.cer"
</VirtualHost>
Turns out the issue was not with the Apache configuration nor tomcat at all. The web app was calling a REST API that was suddenly taken offline due to getting DDoS attack. We commented this call out and all works great. Thanks for those who looked at this and made suggestions.
We're running a web app on Tomcat 6 and Apache mod_proxy 2.2.3. Seeing a lot of 502 errors like this:
Bad Gateway!
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /the/page.do.
Reason: Error reading from remote server
If you think this is a server error, please contact the webmaster.
Error 502
Tomcat has plenty of threads, so it's not thread-constrained. We're pushing 2400 users via JMeter against the app. All the boxes are sitting inside our firewall on a fast unloaded network, so there shouldn't be any network problems.
Anyone have any suggestions for things to look at or try? We're heading to tcpdump next.
UPDATE 10/21/08: Still haven't figured this out. Seeing only a very small number of these under load. The answers below haven't provided any magical answers...yet. :)
Just to add some specific settings, I had a similar setup (with Apache 2.0.63 reverse proxying onto Tomcat 5.0.27).
For certain URLs the Tomcat server could take perhaps 20 minutes to return a page.
I ended up modifying the following settings in the Apache configuration file to prevent it from timing out with its proxy operation (with a large over-spill factor in case Tomcat took longer to return a page):
Timeout 5400
ProxyTimeout 5400
Some backgound
ProxyTimeout alone wasn't enough. Looking at the documentation for Timeout I'm guessing (I'm not sure) that this is because while Apache is waiting for a response from Tomcat, there is no traffic flowing between Apache and the Browser (or whatever http client) - and so Apache closes down the connection to the browser.
I found that if I left the Timeout setting at its default (300 seconds), then if the proxied request to Tomcat took longer than 300 seconds to get a response the browser would display a "502 Proxy Error" page. I believe this message is generated by Apache, in the knowledge that it's acting as a reverse proxy, before it closes down the connection to the browser (this is my current understanding - it may be flawed).
The proxy error page says:
Proxy Error
The proxy server received an invalid
response from an upstream server. The
proxy server could not handle the
request GET.
Reason: Error reading from remote server
...which suggests that it's the ProxyTimeout setting that's too short, while investigation shows that Apache's Timeout setting (timeout between Apache and the client) that also influences this.
So, answering my own question here. We ultimately determined that we were seeing 502 and 503 errors in the load balancer due to Tomcat threads timing out. In the short term we increased the timeout. In the longer term, we fixed the app problems that were causing the timeouts in the first place. Why Tomcat timeouts were being perceived as 502 and 503 errors at the load balancer is still a bit of a mystery.
You can use
proxy-initial-not-pooled
See http://httpd.apache.org/docs/2.2/mod/mod_proxy_http.html :
If this variable is set no pooled connection will be reused if the client connection is an initial connection. This avoids the "proxy: error reading status line from remote server" error message caused by the race condition that the backend server closed the pooled connection after the connection check by the proxy and before data sent by the proxy reached the backend. It has to be kept in mind that setting this variable downgrades performance, especially with HTTP/1.0 clients.
We had this problem, too. We fixed it by adding
SetEnv proxy-nokeepalive 1
SetEnv proxy-initial-not-pooled 1
and turning keepAlive on all servers off.
mod_proxy_http is fine in most scenarios but we are running it with heavy load and we still got some timeout problems we do not understand.
But see if the above directive fits your needs.
I know this does not answer this question, but I came here because I had the same error with nodeJS server. I am stuck a long time until I found the solution. My solution just adds slash or /in end of proxyreserve apache.
my old code is:
ProxyPass / http://192.168.1.1:3001
ProxyPassReverse / http://192.168.1.1:3001
the correct code is:
ProxyPass / http://192.168.1.1:3001/
ProxyPassReverse / http://192.168.1.1:3001/
Sample from apache conf:
#Default value is 2 minutes
**Timeout 600**
ProxyRequests off
ProxyPass /app balancer://MyApp stickysession=JSESSIONID lbmethod=bytraffic nofailover=On
ProxyPassReverse /app balancer://MyApp
ProxyTimeout 600
<Proxy balancer://MyApp>
BalancerMember http://node1:8080/ route=node1 retry=1 max=25 timeout=600
.........
</Proxy>
I'm guessing your using mod_proxy_http (or proxy balancer).
Look in your tomcat logs (localhost.log, or catalina.log) I suspect your seeing an exception in your web stack bubbling up and closing the socket that the tomcat worker is connected to.
You can avoid global timeouts or having to virtual hosts by specifying the proxy timeouts in the ProxyPass directive as follows:
ProxyPass /svc http://example.com/svc timeout=600
ProxyPassReverse /svc http://example.com/svc timeout=600
Notice timeout=600 seconds.
However this does not always work when you have load balancer. In that case you must add the timeouts in both the places (tested in Apache 2.2.31)
Load Balancer example:
<Proxy "balancer://mycluster">
BalancerMember "http://member1:8080/svc" timeout=600
BalancerMember "http://member2:8080/svc" timeout=600
</Proxy>
ProxyPass /svc "balancer://mycluster" timeout=600
ProxyPassReverse /svc "balancer://mycluster" timeout=600
A side note: the timeout=600 on ProxyPass was not required when Chrome was the client (I don;t know why) but without this timeout on ProxyPass Internet Explorer (11) aborts saying connection reset by server.
My theory is that the :
ProxyPass timeout is used between the client(browser) and the Apache.
BalancerMember timeout is used between the Apache and the backend.
To those who use Tomcat or other backed you may also want to pay attention to the HTTP Connector timeouts.
you should be able to get this problem resolved through a timeout and proxyTimeout parameter set to 600 seconds. It worked for me after battling for a while.
Most likely you should increase Timeout parameter in apache conf (default value 120 sec)
If you want to handle your webapp's timeout with an apache load balancer, you first have to understand the different meaning of timeout.
I try to condense the discussion I found here: http://apache-http-server.18135.x6.nabble.com/mod-proxy-When-does-a-backend-be-considered-as-failed-td5031316.html :
It appears that mod_proxy considers a backend as failed only when the
transport layer connection to that backend fails. Unless failonstatus/failontimeout is used. ...
So, setting failontimeout is necessary for apache to consider a timeout of the webapp (e.g. served by tomcat) as a fail (and consecutively switch to the hot spare server). For the proper configuration, note the following misconfiguration:
ProxyPass / balancer://localbalance/ failontimeout=on timeout=10 failonstatus=50
This is a misconfiguration because:
You are defining a balancer here, so the timeout parameter relates to
the balancer (like the two others).
However for a balancer, the timeout parameter is not a connection
timeout (like the one used with BalancerMember), but the maximum time
to wait for a free worker/member (e.g. when all the workers are busy
or in error state, the default being to not wait).
So, a proper configuration is done like this
set timeout at the BalanceMember level:
<Proxy balancer://mycluster>
BalancerMember http://member1:8080/svc timeout=6
... more BalanceMembers here
</Proxy>
set the failontimeout on the balancer
ProxyPass /svc balancer://mycluster failontimeout=on
Restart apache.