We have a very distributed system. A user's request on the site may involve calls to several services. For example, when a user log onto the site, calls are made to ads service, personalization service, related news service, etc. to construct the data needed for display up on login. High-level design: A request to a URL is mapped to a Spring MVC Controller and that controller makes the call (most of them using HttpClient) to different services.
We are implementing a centralized log solution using Logstash, ElasticSearch, Kibana, Log4j/SLF4J. When a issue is reported on the site, we want to be able to change log level to debug and see the log messages for a particular request across all services. We are populating request id in Log4j MDC, so we are able to identify log messages for that particular request on the webapp server. How do I go about correlating messages from the calls made to other services?
Flow:
User log in --> request mapped to Spring MVC Controller which logs messages by populating request id in Log4j MDC --> http client calls to service1, service2, service 3
How to correlate messages from service1, service2, service3 to the messages logged by MVC controller. One solution is to pass the request id in http client calls. There are lot of applications that follow this paradigm so changing code everywhere is not an ideal solution.
UPDATE1:
I don't know much about jvm agents, but I'm wondering if a custom agent can be developed to intercept network calls and add a parameter. The custome agent on the receiving side will detect the parameter add it to a ThreadLocal variable. Dynatrace PurePath technology somehow correlates calls across JVMs - they require injecting their jvm agent, so I'm guessing they are intercepting calls in the agent. Check out this video
You're going to have to bite the bullet and add the request ID to the HTTP client calls. If you don't want to pollute your APIs, add it as a custom HTTP header, then extract it using some kind of HTTP interceptor on the service side (depends on what web service stack you're using), and re-add it to the MDC.
Related
I'm working on an application with a large number of Remote EJB service methods, and I'd like to have some useful information about the client calling the methods (other than very basic information such as IP address...).
I found this question but it's a bit dated :
How can I identify the client or caller of an EJB within the request?
Is there some kind of custom client context / work area in which I could put the caller details and receive them on server side inside a thread local ?
Basically do I have another option than adding a parameter to every single method of every service ?
With security enabled you have the possibility to retrieve the current user. This is pretty simple, but probably won't fit all needs.
In general, there is nothing you can use out-of-the-box. Adding some custom parameter is probably the worst option.
JBoss and Wildfly are offering the possibility to use EJB client- and server-side container interceptors. Usage and implementation details depend on the version of your application server.
I implemented this by utilizing the MDC (mapped diagnostic context) of our logging framework to enhance server side logging with caller information. You can think about this like using a ThreadLocal. Of course, you need something like a caller context on the client side holding the specific information. Global remote client data (ip address, ...) can be set within the client interceptor, too.
A coarse overview what I did:
Configure client- and server-side logging to use additional MDC data
Enhance client side MDC with key/value data
Client-side interceptor enxtracts the data from the MDC and puts it on the invocation context
Server-side interceptor extracts the data from the invocation context and enhances server side MDC
This approach is working, but depending on your application complexity (e.g. with server2server calls, bean2bean calls on local asynch EJBs, ...) complexity increases. Don't forget to think about cleaning up e.g. your ThreadLocal data to avoid possible memory leaks.
I have a Java miroservice, which will be deployed as multiple service instances. The microservice works statelessly in a simple way: fetch a request according to predefined criteria from DB, process the request, fetch next and process it, and so on.
I consider now to add failover to the service. I might need add extra information(like processor_id) to the request if it's processed by some instance. So if the instance is determined without response, the request can be taken by other instances. I also need add heartbeat to the microservice. Maybe I can leverage Apache Zookeeper or Curator to achieve this.
But I don't know how to make different pieces work together. It'll be better if there are examples in Java.
What i have seen in my career is that there is a load balancer to distribuite the requests across the microservices. The load balancer knows the availability of the microservices invoking a status end point which returns an http status. If there is no response, or the response is negative, the load balancer excludes the microservice form the group.
Usually in spring boot applications, we can use jpa audit to do the tracking.
Spring Boot Jpa Auditing
While in microservices architecture, I'd try to avoid involving security in core microservice. Instead, we can do authentication/authorization at api gateway.
While, if the core service didn't get the current login user, we have to find an way to pass the current operator to core services. It could be an user identifier header on the request. Or Maybe we can pass token to core services to let it fetch the login user from auth server.
I am wondering if anyone has handled such case and give out some suggestion.
If I understand the question correctly ...
You have an API gateway in which authentication/authorisation is implemented
On successful negotiation though the API gateway the call is passed on to a core service
The core services perform some auditing of 'who does what'
In order to perform this auditing the core services need the identity of the calling user
I think the possible approaches here are:
Implement auditing in the API gateway. I suspect this is not a runner because the auditing is likely to be more fine grained than can be implemented in the API gateway. I suspect the most you could audit in the API getway is something like User A invoked Endpoint B whereas you probably want to audit something like User A inserted item {...} at time {...} and this could only be done within a core service.
Pass the original caller's credentials through to the core service and let it authenticate again. This will ensure that no unauthenticated calls can reach the core service and would also have the side effect of providing the user identity to the core service which it can then use for auditing. However, if your API gateway is the only entrypoint for the core services then authenticating again within a core service only serves to provide the user identity in which case it could be deemed overkill.
Pass the authenticated user identity from the API gateway to the core service and let the core service use this in its auditing. If your API gateway is the only entrypoint for the core services then there is no need to re-authenticate on the core service and the provision of the authenticated user identity could be deemed part of the core service's API. As for how this identity should be propagated from the API gateway to the core service, there are several options depending on the nature of the interop between API gateway and core service. It sounds like these are HTTP calls, if so then a request header would make sense since this state is request scoped. It's possible that you are already propagating some 'horizontal state' (i.e. state which is related to the call but is not a caller supplied parameter) such as a correlationId (which allows you to trace the call though the API getway down into the core service and back again), if so then the authenticated user identify could be added to that state and provided to the core service in the same way.
DO you have code Example. I have tried to pass token from zuul to other module. But I always got null in header in another module.
I have a service that is dependent on several resources.
This service has a initialization logic that checks if these resources are up and running, and starts / stops the service accordingly.
The problem is, that other services are addressing my services via REST while it is loading. It then tries to reply (in a different thread), and when it does, it tries to connects to one of the not-yet-available resources and crashes.
Is there a way to 'lock' the service while it is loading so that any request coming to it from the outside will return 'Service not available' while it is in its loading process?
There are many requests to the service and I don't wan't to add a 'check resource status' logic to every method that handles an HTTP request. I want to be able to block all requests and the unblock them when all resources are up.
Any help will be great. (I'm not very familiar with Spring yet).
Assuming these resources get initialised asynchronously, you can write a Filter or an Interceptor to filter the request and retuen 503 if the resources are not loaded.
Here's an example of how to configure a filter. You can even write an Interceptor if you want to do handle/intercept the requests at resource level (example here), however, as you want to filter all the incoming requests, I would recommend going ahead with Filter.
I have multiple Java web applications deployed on the same server (Wildfly).
They all should use a single WebSocket implementation to send messages (object, not plain text) to the user.
Edit: WebApp1-3 are the applications with the business logic. The only purpose of WebApp4 is to update a Primefaces panel in the browser based on the messages generated by the other WebApps. Sorry for the missleading illustration.
WebApp1
WebApp2 --> ??? --> WebApp4 (WebSocket-Server) --> JS/Browser
WebApp3
Which is the best way/pattern/implementation to make WebApp4 available to the other applications? (RMI, JMS, WebSocket, WebService, ....?)
My advice, for a general way of exposing services, is to expose REST services since they are simpler than SOAP web service and easily allow interoperability (if in the future a PHP or a RUBY webapp needs to consume your services it's much easier with a REST interface than with one base on RMI or JMS). The content of the REST service may vary, I suggest you to look at XML or JSON as a way of transmitting information over http REST services.
If all webapps are in the same server, you should forward requests from one to another. From the point of view of webapps 1-3, they would not need to be aware of whether their incoming requests were coming from webapp 4 or from outside (to which it appears that they are not connected). Of course, you are free to alter requests before forwarding them - or to drop them altogether, for example if authentication fails.
To do this in tomcat: https://stackoverflow.com/a/8951090/15472
When forwarding requests, the external client is completely unaware of the existence of webapps 1-3 -- as far as the client is concerned, it has sent a request to webapp 4, and it will think it is receiving a response from that same server.
You may need to configure your web server to allow these kinds of calls, but I am unfamiliar with WildFly.