I want to keep track of the cpm_usd value of all requests that arrive on my GAE frontend instances. I see the cpm_usd headers in the logs of my application. But is there a way to access those numbers at runtime in order to graph them? I want to create a near real-time cost metric for every endpoint.
/rest/foo1: $0.000011
/rest/foo2: $0.000013
/rest/bar1: $0.000014
/rest/bar2: $0.000016
Is there a trusted tester program to do this? If not, is there another way to do this which does not involve parsing my log files? Or can I only get those numbers by parsing the logs in the background?
Updates
As described here, I tried adding a sitebricks request filter which intercepts the {add,set}Header() calls.
public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
chain.doFilter(request, new HttpServletResponseWrapper((HttpServletResponse) response) {
#Override
public void setHeader(String name, String value) {
if ("X-AppEngine-Estimated-CPM-US-Dollars".equals(name)) {
// log request costs
}
super.setHeader(name, value);
}
#Override
public void addHeader(String name, String value) {
if ("X-AppEngine-Estimated-CPM-US-Dollars".equals(name)) {
// log request costs
}
super.addHeader(name, value);
}
});
}
I assume that the header either has a different name or GAE sets headers differently. In either case, I never see the cost header being caught.
The header is added by the application server that sits in front of your instance so you cannot access this header from your code.
You will have to parse the log files, and can consider using log2bq to do this.
Related
I wish to find a better way of handling responses than the method I was taught by the company that I worked for.
I was taught to use a generic HttpClient, that used volley to send the requests. The client had a static method that would be given a generic listener, ResponseListener, which would make the callback to the context when a volley response came through. The ResponseListener would keep track of all the request "type"s . That is the code given to the client so that the context can differentiate between requests.
Is there any other way of keeping track of all the request codes without having to keep one big directory type interface file? It becomes quite overwhelming to look at when you get past 100 requests. Then you write wrapper functions for the codes in the client, and it too gets messy.
HttpClient
public static void doRequestString(final ResponseListener listener, final int type, final String url, final JSONObject postData) {
// Request a string response
StringRequest request = new StringRequest(Request.Method.GET, url,
new Response.Listener<String>() {
#Override
public void onResponse(String response) {
// Result handling
listener.onRequestDone(type, response);
}
}, new Response.ErrorListener() {
#Override
public void onErrorResponse(VolleyError error) {
// Error handling
System.out.println("Something went wrong!");
error.printStackTrace();
}
});
request.setTag(context);
VolleyClient.getInstance(context).getRequestQueue().add(request);
}
Listener
public interface ResponseListener
{
int HELLO_REQUEST = 0;
int GOODBYE_REQUEST = 1;
// every other request numbered here, so they don't conflict
void onRequestDone(int type, String response);
}
Context
public void onRequestDone(int type, String response)
{
switch(type) {
case Response.Listener.HELLO_REQUEST:
handleHello();
break;
case Response.Listener.GOODBYE_REQUEST:
handleGoodbye();
break;
}
}
Well, there are not that many options, to be honest. You are dealing with your responses in a centralised manner right now. Another option would be stripping the request type as having a listener per request. The main disadvantage here is that you will get your code full of listeners.
I would suggest you try and combine the two approaches in a way that suites your use case. Maybe create an intermediate service layer, divide your functionality by some of their property (for example logically - all user requests grouped together, etc), expose a single listener per service and manage the request codes there. This way you can have a bit of modularity.
First of all, I'm using command pattern with this interface:
public CommandResponse execute(HttpServletRequest req)
throws CommandException;
And this is my service method:
public void service(HttpServletRequest req, HttpServletResponse res)
throws ServletException, IOException {
try {
Command cmd = lookupCommand(req.getParameter("cmd"));
CommandResponse next = cmd.execute(req, res);
CommandToken.set(req);
if (next != null) {
if (!req.getHeader("X-Requested-With").equals("XMLHttpRequest")) {
if (next.isRedirect()) {
redirect(next, res);
} else if (next.isForward()) {
forward(next, req, res);
}
} else {
ajaxCall(next, res);
}
}
} catch (CommandException e) {
res.sendError(HttpServletResponse.SC_BAD_REQUEST, e.toString());
}
}
As you can see my service method treats the logic of redirection and forward and my interface does not have access to the HttpServletResponse object. But for some actions of my servlet I need handle the response to add some content like a xml or a cookie. The more obvious way to achieve that is modify my interface to:
public CommandResponse execute(HttpServletRequest req, HttpServletResponse res)
throws CommandException;
Now the questions:
1-) In my opinion the solution above will break the elengacy and safety of the code since my actions now have access to the response and can redirect and forward, and even add content to the response and let the service method use sendRedirect wich we know would have no effect. Am I right? To solve that I do this:
public class CommandResponse {
private boolean redirect;
private boolean forward;
private String page;
private String contentType;
private String outContent;
private HashMap headers;
private List<Cookie> cookies;
...
public void mapToResponse(HttpServletResponse res) {
...
}
}
That is, I create a faked response and when I come back of the action execute method I map this faked response to the real http response (just if I have a forward or a ajax call).
2-) That is good? It makes sense? Should I not use it?
3-) There is a better way?
Thanks.
Why not just pass a strong buffer to capture any content that your subsequent chain may want to add? This way you are not confusing the responsibility of the components. Normally, httpresponse is expected to behave as any one would expect.
However, I would like to hear the reason behind this design.
Yes, there is a better way, take a look to Frontman, an elegant implementation of the Command pattern in servlets
http://www.bibeault.org/frontman/
H.e.l.l.o community, i hope someone can help me ... i am using apache tomcat 8.0.0-RC5 and JSR-356 web socket API ...
I have 2 questions:
1) Is it possible to get the client ip on #OnOpen method ??
2) Is it possible to get the origin of the connection ???
I followed the websocket example which comes with the distribution of tomcat and i was not able to find the answers .... My java class is basically as follow
#ServerEndpoint(value = "/data.socket")
public class MyWebSocket {
#OnOpen
public void onOpen(Session session) {
// Here is where i need the origin and remote client address
}
#OnClose
public void onClose() {
// disconnection handling
}
#OnMessage
public void onMessage(String message) {
// message handling
}
#OnError
public void onError(Session session, Throwable throwable) {
// Error handling
}
}
Repeating the answer I already gave you on the Tomcat users mailing list...
Client IP. No. Generally this type of information is available at the handshake
which occurs before OnOpen but client IP is not one of the pieces of
information exposed. You might be better blocking these earlier e.g. with iptables or similar.
Origin. ServerEndpointConfig.Configurator.checkOrigin(String) You'll need a custom Configurator. Keep in mind that a malicious client can forge the origin header.
I know this question is old, but just in case someone else finds it in a web search:
Yes there is an easy workaround. A Servlet can receive and forward a WebSocket upgrade request. The trick is to get the client IP address and expose it as a parameter.
Here's your servlet code:
#WebServlet("/myExternalEntryPoint")
public class WebSocketServlet extends HttpServlet {
protected void service(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
var dispatcher = getServletContext().getRequestDispatcher("/myInternalEntryPoint");
var requestWrapper = new MyRequestWrapper(request);
dispatcher.forward(requestWrapper, response);
}
}
And here's MyRequestWrapper:
class MyRequestWrapper extends HttpServletRequestWrapper {
public RequestWrapper(HttpServletRequest request) {
super(request);
}
public Map<String, String[]> getParameterMap() {
return Collections.singletonMap("remoteAddr", new String[] {getRequest().getRemoteAddr()});
}
}
Now in your WebSocket implementation, you'll be able to get remoteAddr via javax.websocket.Session.getRequestParameterMap().
Naturally, if your original request has parameters that you care about, you'll need to create a map that includes those as well. Also I recommend you append a separate, secret parameter and check for it in your WebSocket code to prevent anyone from hitting the internal entry point directly.
I figured out this was possible because of this thoughtful comment in the Tomcat source code (WsFilter.java):
// No endpoint registered for the requested path. Let the
// application handle it (it might redirect or forward for example)
(I'm not sure exactly how to phrase the title here, and because of that I'm not really sure how to go about searching for the answer either.)
I have a Java servlet engine that handles requests. Say we have a doGet() request:
public void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
//set up user data
//do whatever the user requested
SomeClass c = new SomeClass();
c.doSomething();
}
Now in doSomething(), I want to be able to access which user made the request. Right now I'm doing it by creating a Java object within the method and passing it to wherever I need it:
public void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
//set up user data
MyUserObj userObj = new MyUserObj();
userObj.setId('123');
//do whatever the user requested
SomeClass c = new SomeClass(userObj);
c.doSomething();
}
By doing this, I have access to the instance of MyUserObj, and it can be further passed along in the application as needed.
I know in ASP.NET MVC3 I can acheive this by storing items/attributes for the current thread like this: HttpContext.Current.Items.Add("myId", "123"). HttpContext is then available in other functions without explicitly having to pass around an object.
Is there a way in Java to set some variables per request (or even set the MyUserObject to be accessed later) without passing the object through as a parameter?
There isn't in the servlet API, but you can make your own pretty easily. (Some frameworks like spring-mvc, struts provide such functionality)
Just use a public static ThreadLocal to store and retrieve the object. You can even store the HttpServletRequest itself in the threadlocal and use its setAttribute()/getAttribute() methods, or you can store a threadlocal Map, to be agnostic of the servlet API. An important note is that you should clean the threadlocal after the request (with a Filter, for example).
Also note that passing the object as parameter is considered a better practice, because you usually pass it from the web layer to a service layer, which should not be dependent on web-related object, like a HttpContext.
If you decide that it is fine to store them in a thread-local, rather than passing them around:
public class RequestContext {
private static ThreadLocal<Map<Object, Object>> attributes = new ThreadLocal<>();
public static void initialize() {
attributes.set(new HashMap<Map<Object, Object>>());
}
public static void cleanup() {
attributes.set(null);
}
public static <T> T getAttribute(Object key) {
return (T) attributes.get().get(key);
}
public static void setAttribute(Object key, Object value) {
attributes.get().put(key, value);
}
}
And a necessary filter:
#WebFilter(urlPatterns="/")
public class RequestContextFilter implements Filter {
public void doFilter(..) {
RequestContext.initialize();
try {
chain.doFilter(request, response);
} finally {
RequestContext.cleanup();
}
}
}
You can attach an object to the current request with setAttribute. This API is primarily used for internal routing, but it's safe to use for your own purposes too, as long as you use a proper namespace for your attribute names.
Update: The issue is with the setting used for the MaxAge. Setting it to zero will cause the cookie to be deleted, hence it was shown in the response header but was then deleted and did not show in the request. Setting it to -1 so that it will not be stored persistently
I am doing some work on a filter in which I want it to set a cookie to indicate that the user qualifies to take a survey and that the next time he comes back to the site a survey popup window will be displayed.
I am not seeing any errors in the logs but the cookie never gets set.
Can a filter be used this way to add a cookie? Is there where a HttpServletResponseWrapper comes into play?
The thought here is that when the user comes to the the site there is a check to see if the cookie is present. If it is not then a cookie is created and added to the response. As the user navigates the site, the cookie check method is called to make sure that the hit counter is not increased for that given user.
The cookie check method never sees the cookie. Using web developer plugin for firefox, the cookie in question is not present.
Below is the filter class with the relevant methods.
public class HitCounterFilter extends TemplateFilter {
public void doMainProcessing(ServletRequest pRequest, ServletResponse pResponse, FilterChain pChain) {
HttpServletRequest httpRequest = (HttpServletRequest) pRequest;
HttpServletResponse httpResponse = (HttpServletResponse) pResponse;
// prevent thread interference and memory consistency errors
synchronized (lock) {
int hitCounter = this.readFile(localFile);
// if user has not been counted
if (!this.checkForCookie(httpRequest, "gtgo_visitor")) {
this.writeFile(localFile, ++hitCounter);
this.createCookie(httpRequest, httpResponse, String.valueOf(hitCounter), "gtgo_visitor");
}
}
}
private void createCookie(HttpServletRequest pHttpRequest, HttpServletResponse pHttpResponse, String pCookieValue, String pCookieName) {
try {
Cookie cookie = new Cookie(pCookieName, pCookieValue);
URL url = new URL(pHttpRequest.getRequestURL().toString());
cookie.setDomain(url.getHost());
cookie.setPath(this.getCookiePath(pHttpRequest));
cookie.setComment("user is not eligible to take the survey this time");
cookie.setMaxAge(0);
pHttpResponse.addCookie(cookie);
} catch (MalformedURLException e) {
e.printStackTrace();
}
}
private boolean checkForCookie(HttpServletRequest pHttpRequest, String pCookieName) {
for (Cookie cookie : pHttpRequest.getCookies()) {
if (StringUtils.equalsIgnoreCase(cookie.getName(), pCookieName)) {
return true;
}
}
return false;
}
}
The issue is with the setting used for the MaxAge. Setting it to zero will cause the cookie to be deleted, hence it was shown in the response header but was then deleted and did not show in the request. Setting it to -1 so that it will not be stored persistently - it will be present as long as the session is active (removed once the session is closed)
I haven't worked with the SCAM project before, but it looks like that is what you're using. I'm not sure if the super class's implementation of doFilter calls your implementation's doMainProcessing method before calling the FilterChain's doFilter method or not.
Because the framework passes the FilterChain into your doMainProcessing method, it is likely that it expects that your implementation will call pChain.doFilter(pRequest, pResponse, pChain).
I might be wrong about this, but if the filter chain is abandoned, its possible that the headers written to the ServletResponse object, which include the cookie you have attached, will not be returned to the client.
In any case, unless you are specifically blocking access to the requested resource, its wise to propagate the request using the FilterChain.