We Keep Coding

Track users activities on a Spring web app

I have a web application written in Java using the Spring framework.
I would like to store the users activities like, page visits, actions, interactions etc.
I read that usually this is done by creating a table for each tracked aspect. I was wondering if there is a better way to do it using Spring framework, like a way to intercept all the requests and trigger some actions.
What kind of technology do you recommend to store all these information? Right know I’m using a MySql database interacting with it through JPA. But, since I’m really really new to these kind of things I don’t know if I should go with a NoSql database or stay with my already existing MySql database. This wonder comes from the idea that this kind of data flow will be much bigger than a normal data flow coming from more traditional actions such as signin, creation, deletion etc.
Hope to have explained myself... if not just tell me and I’ll try to add more details.
[EDIT 1]
The web app is an e-commerce. So far it does not have So many users but it will (in the order of thousands).
The goal of the user tracking it’s just to profile them in order to give them a better and more custom service. For instance, if a see that a user is taking a look to a lot of items of a precise category I can show him more items of that kind.
I do no care that much about the performance, I mean, it does not have to be so fast.
Right know I have just one database and everything is stored inside it. I don’t know if charging it with this kind of data flow would slow down its performance.
The application is running on AWS ElasticBeanstalk and the database is on AWS RDS.

In general its a very broad topic.
The following considerations come to my mind:
How many requests pass to the microservice per some period of time? If its a small number of users (which translates to the number of records to the database) - then its ok to go with the MySQL approach - the table won't be large. Note however, that sometimes it should be cleaned anyway
Is the latency important? Sometimes requests have to be served very quickly, adding a hop to the database to save the user preference can be problematic
How do you want to consume this kind of information? Are you planning to use dashboards (in this case micrometer + Prometheus / InfluxDB and Grafana can be a good solution). Are you planning to actually charge the users per number of requests with an ability to send the monthly bill to their email in PDF or provide a web access to such an information (like AWS does for example)?
How about Rate limiter? Are you planning to deny some requests if they're frequent and coming from the same user?
How many instance will "add" this kind of information? What if you have thousands of microservices that now will have to write to MySQL - it might not survive such a load (in addition to the regular load its set up for)?
The range of solutions can vary.
You can Batch the requests per user in memory and send once in while a message into Kafka and then use kafka streams to provide aggregations on it. With this approach you'll minimize the impact of the added processing on the existing solution and will deploy some other service that will be able to process this pretty large amount of data.
Another option: maybe you can create an asynchronously populated log file and store the information there. Then you might want to add some "agent" / side-car container like logstash and stream the data into some storage. Yet Another project that might be relevant in this field is Apache Flume which will allow you to construct a pipeline.
For billing you might use specialized systems AFAIK spring doesn't have anything like this usually these are ready products that you can integrate with.
For Rate Limiting you might consider: Resilience4j or solve it with redis

Yeah , That's possible , Here below are the three approaches with some sample snippets which would help you in the implementation , Moreover it depends on the data you store when you log the activity and when do you consider the activity data as obsolete and there are many factors which can decides your data store.
Approach 1: You can keep track of the login user using Spring-Security
You can write a HTTPSessionBindingListener and track the actions something like this
#Component
public class LoggedUser implements HttpSessionBindingListener {
private String username;
private ActiveUserStore activeUserStore;
public LoggedUser(String username, ActiveUserStore activeUserStore) {
this.username = username;
this.activeUserStore = activeUserStore;
}
public LoggedUser() {}
#Override
public void valueBound(HttpSessionBindingEvent event) {
List<String> users = activeUserStore.getUsers();
LoggedUser user = (LoggedUser) event.getValue();
if (!users.contains(user.getUsername())) {
users.add(user.getUsername());
}
}
#Override
public void valueUnbound(HttpSessionBindingEvent event) {
List<String> users = activeUserStore.getUsers();
LoggedUser user = (LoggedUser) event.getValue();
if (users.contains(user.getUsername())) {
users.remove(user.getUsername());
}
}
// standard getter and setter
}
and for login and logout you can track using AuthenticationSuccessHandler
#Component("myAuthenticationSuccessHandler")
public class MySimpleUrlAuthenticationSuccessHandler implements AuthenticationSuccessHandler {
#Autowired
ActiveUserStore activeUserStore;
#Override
public void onAuthenticationSuccess(HttpServletRequest request,
HttpServletResponse response, Authentication authentication)
throws IOException {
HttpSession session = request.getSession(false);
if (session != null) {
LoggedUser user = new LoggedUser(authentication.getName(), activeUserStore);
session.setAttribute("user", user);
}
}
}
Approach 2 : The other method if you want to make it very simple is that you can write a OncePerRequestFilter
#Component
#Ordered(Ordered.LOWEST_PRECEDENCE)
public class LogFilter extends OncePerRequestFilter {
#Override
protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain)
throws ServletException, IOException {
// Log the info you need
// ...
filterChain.doFilter(request, response);
}
}
Approach 3 : Implement using Spring AOP.
#Aspect
#Component
public class WebMethodAuditor {
protected final Log logger = LogFactory.getLog(getClass());
public static final String DATE_FORMAT_NOW = "yyyy-MM-dd HH:mm:ss";
#Autowired
AuditRecordDAO auditRecordDAO;
#Before("execution(* com.mycontrollers.*.*(..))")
public void beforeWebMethodExecution(JoinPoint joinPoint) {
Object[] args = joinPoint.getArgs();
String methodName = joinPoint.getSignature().getName();
User principal = (User)SecurityContextHolder.getContext().getAuthentication().getPrincipal();
Timestamp timestamp = new Timestamp(new java.util.Date().getTime());
// only log those methods called by an end user
if(principal.getUsername() != null) {
for(Object o : args) {
Boolean doInspect = true;
if(o instanceof ServletRequestDataBinder) doInspect = false;
if(o instanceof ExtendedModelMap) doInspect = false;
if(doInspect) {
if(o instanceof BaseForm ) {
// only show form objects
AuditRecord ar = new AuditRecord();
ar.setUsername(principal.getUsername());
ar.setClazz(o.getClass().getCanonicalName());
ar.setMethod(methodName);
ar.setAsString(o.toString());
ar.setAudit_timestamp(timestamp);
auditRecordDAO.save(ar);
}
}
}
}
}
}
Source and More details :
https://www.baeldung.com/spring-security-track-logged-in-users
Spring / AOP: Best way to implement an activities log in the database
What is the best way to log Activity in Spring Boot with Thymeleaf?

Safe way to find if we are within a thread bound request without checking for an Exception

Problem
So we have a Request Interceptor (Feign) that checks an autowired HttpServletRequest for headers, and then propagates/copies these to the outgoing request. Our interceptor's job is to propogate headers from microservice to microservice so that even the last micro-service in the graph has the information about who initiated the request (e.g. tenant).
Sometimes we call feign as a result of an HTTP request thread, and sometimes we call it on start-up or from a scheduled thread.
In the case of a scheduled thread, we would like to be able to detect whether a request exists without needing to do a try/catch. This is the case where we are the initiating party and we don't need to copy anything.
I was expecting the following to work, but we get a proxy object that throws an exception:
The following check fails because this.request is not null:
this.request!=null && this.request.getHeader("X-Application")
With the following error:
No thread-bound request found: Are you referring to request attributes outside of an actual web request, or processing a request outside of the originally receiving thread? If you are actually operating within a web request and still receive this message, your code is probably running outside of DispatcherServlet/DispatcherPortlet: In this case, use RequestContextListener or RequestContextFilter to expose the current request.
I understand the error. I would like to avoid doing the obvious workaround which would be something like:
Current Workaround - clumsy and bad
//TODO: Review this
boolean requestExists = true;
try{
request.getHeader(APPLICATION_HEADER);
}catch (IllegalStateException e ){
requestExists = false;
}
Current Code Causing the Problem
public class ServiceNameFeignInterceptor implements RequestInterceptor {
private static final Logger log = LoggerFactory.getLogger(ServiceNameFeignInterceptor.class);
final TenantIdResolver tenantResolver;
final ApplicationNameResolver appResolver;
private final String APPLICATION_HEADER = "X-Application";
private final String TENANT_ID = "X-Tenant-Id";
...
#Autowired
HttpServletRequest request;
public void apply(RequestTemplate requestTemplate) {
...
if (this.request!=null && this.request.getHeader("X-Application") != null) {
log.info("Application header found in the request !!!");
requestTemplate.header("X-Application", new String[]{this.request.getHeader("X-Application")});
requestTemplate.header("X-Tenant-Id", new String[]{this.request.getHeader("X-Tenant-Id")});
} else {
log.info("Setting {} as {} for URL {} ", new Object[]{"X-Application", appName, requestTemplate.url()});
requestTemplate.header("X-Application", new String[]{appName});
requestTemplate.header("X-Tenant-Id", new String[]{appName});
}
}
Current Options
Please correct me on the following points or propose better options if possible.
I currently have three options in mind:
Use the try/catch solution ( least favored)
Check threadlocal variables for the existence of the request
Pass our own additional thread local variable that will be a flag ( that we are not within a request context ).
Problems
I don't like 1, because catching exceptions are expensive, and because they would possibly mask any real errors.
I don't like 2, because if the spring implementation changed maybe implementation details would change (e.g. key) and our implementation in our starter would break. But in any case whenever upgrading spring boot various minor or major things need to be fixed.
Option 3 I like because it is a conscious action to set a flag before call our feign client. So there is no risk of errors going un-noticed.
Opinions, options, solutions?
Update
One of the team members suggests we use: new NamedThreadLocal("Request attributes");
They suggest this because of the implementation at:
https://github.com/spring-projects/spring-framework/blob/master/spring-web/src/main/java/org/springframework/web/context/request/RequestContextHolder.java#L50
https://github.com/spring-projects/spring-framework/blob/master/spring-web/src/main/java/org/springframework/web/context/request/RequestContextHolder.java#L107
So we would use something like:
ThreadLocal<RequestAttributes> requestAttributesHolder = new NamedThreadLocal("Request attributes");
RequestAttributes attributes = (RequestAttributes)requestAttributesHolder.get();
boolean requestExists = attributes != null;
But this is quite dependent on the internals of spring and them continuing to use "Request attributes".

I have a similar problem and I use RequestContextHolder to check if the request is bound to the thread. According to the doc, getRequestAttributes returns null if no RequestAttributes bound to the thread.
RequestAttributes attrs = RequestContextHolder.getRequestAttributes();
if (attrs == null) {
return;
}

Soteria HttpMessageContext.setRegisterSession() not working as expected?

I have implemented a custom HttpAuthenticationMechanism subclass to provide authentication using the Soteria/Java EE Security framework. I've got authentication working just fine. However, I've noticed that when I call HttpMessageContext.setRegisterSession(String, Set<String>) to create a Java EE Session, the behavior is not what I would expect. I am expecting that the authenticated identity be associated with the web Session, and my AuthenticationMechanism's validateRequest(HttpServletRequest req, HttpServletResponse res, HttpMessageContext ctx) method would not be called on subsequent requests. What I am observing, however, is that validateRequest() is called on every request, even if the user has already authenticated successfully.
I am able to get the behavior I want using the #AutoApplySession annotation on my AuthenticationMechanism class, but that is not the behavior I want. I'd like to choose whether or not to create a session based on the type of credential provided.
Is my understanding of the setRegisterSession() method incorrect? Or is this a bug within Soteria?

#AutoApplySession is the new way to do this in Soteria (JSR 375). If it does not suit your needs (as you need to either remember the authenticated identity or re-authenticate for all requests during the same HTTP session based on some other credential info), validateRequest method will still be called, regardless of whether you call the HttpMessageContext'ssetRegisterSession method or not. HttpMessageContext.setRegisterSession will make the container remember the credentials but will not reuse them automatically, you still need to make the container reuse the authentication identity by doing the same thing Soteria's AutoApplySessionInterceptor does. So in your class which implements HttpAuthenticationMechanism you should add the following code before your actual authentication logic is performed in the validateRequest method:
Principal userPrincipal = request.getUserPrincipal();
if (userPrincipal != null) {
httpMessageContext.getHandler().handle(new Callback[] {
new CallerPrincipalCallback(httpMessageContext.getClientSubject(), userPrincipal) }
);
return AuthenticationStatus.SUCCESS;
}
Also, see this answer by Arjan Tijms. Although it's about JASPIC not Soteria, but in this case I think it's relevant.
Hope this helps.

I thing you're following the incorrect source you've look at this IMPLEMENTATION.
/* (non-Javadoc)
* #see javax.security.authenticationmechanism.http.HttpMessageContext#setRegisterSession(java.lang.String, java.util.Set)
*/
#Override
public void setRegisterSession(String username, Set<String> groups) {
Jaspic.setRegisterSession(messageInfo, username, groups);
}
Under the library location:
import org.glassfish.soteria.mechanisms.jaspic.Jaspic;
From the mechanisms.

How do you to detect the end of a thread via ThreadId?

Talking Java Servlets here... I'm working on creating my own "Per Request Context" and I was looking to tie the "Per Request Context" object to the Thread.currentThread().getId() value.
Instead of passing around this context object everywhere I was planning on checking the current threadid when a user calls a function that is Per Request based and automatically getting the Context object out of a hashtable for that threadId.
I would use the code this like..
public void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException
{
MyFramework.EnterContext();
try {
// do stuff here that leads to other classes on the same thread
// Access current context via static MyFramework.getCurrentContext()
}
finally { MyFramework.ExitContext(); }
}
However I would like to protect my application automatically from any potential user that does not call ExitContext(). In C# there is an event handler on the thread object for onexit...(think I wrong on this) is there some way to detect or poll when a thread exits? I'm currently storing only the threadId (long).
Any ideas?

unfortunatelly, there is no such feature built in for threads in Java. Besides, thread id is only guaranteed to be unique at any one time, but may be reused eventually when the thread dies (from the docs). however, the servlet framework that you are using may be implementing such feature (just a speculation).
i would recommend you implement a servlet filter, and tell your users to include it in their web.xml. with this you can be sure the client code always gets correctly wraped in your thread context.

A ThreadLocal seems to fit your use perfectly. A ThreadLocal object can provide a way to store a variable per thread. The internal workings of this class are very much of what you describe, it uses a map to give thread-local variables.
Something like this should do the trick:
private static final ThreadLocal<UserContext> userContext = new ThreadLocal<UserContext>();
public void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
MyFramework.EnterContext();
try {
UserContext context = userContext.get();
//if you used the set method in this thread earlier
//a thread local context would be returned using get
}
finally { MyFramework.ExitContext(); }
}
As for your other problem, you can use an observer pattern and notify when the thread completes its task.

What would be the best way to match up two object instances between two different applications in a J2EE server?

I have a J2ee application where I basically want two objects, created by two separate servlets to communicate directly and I need these intances to be stable, i.e. to "know" each other during the session.
The sequence is roughly:
Client sends a request to Servlet #1, which creates object A
Client sends a second request (after the first returns) to servlet #2 which creates object B.
Object B finds A, using JNDI, and the two objects interact.
The client now continues to send requests to object A which needs to find B again.
How do I make sure that these two instances know each throughout the session?
Binding them to JNDI doesn't entirely solve the problem, since object B needs to communicate with its original servlet (servlet #2), which is not kept stable across requests.
Any ideas?
Thanks in advance.
Yes, I admit the problem description is a bit vague. But it's not a very simple application.
Still, I will try to ask it better:
My end goal is to create a sort of a "semantic debugger" for my application that, as opposed to a java debugger which simply debugs the java statements.
The application debugged is basically a servlet. which my tool connects to.
The tool maintains a connection to the application through another servlet which controls the debugging process.
These two servlets need to communicate with each other constantly and directly.
My current thought is to set up a stateful session bean that will facilitate this communication (never done it, still struggling with setting it up).
But I would appreciate any thoughts on how to achieve this better.

And what stops you from using the Session? You don't need JNDI, just place your object into session under a predefined name. If the communication object is application-wide, use Singleton.
P.S. It looks to me you're doing something weird, while the solution could in fact be simpler. Can you describe the task, not the proposed implementation? What is a "semantic debugger" anyway?

To be honest: I don't fully understand what you are trying to achieve.
Can you perhaps try to explain the problem you are trying to solve instead of the solution?
What do these objects depend on? Are they user specific? Then put them into the session and you can retrieve them from the session again (request.getSession().getAttribute("A")).
Are they global for all users? In that case put them into a spring configuration and retrieve them from there.
Never store any information inside the servlet.
EDIT:
ok, so from what I understand storing the values in the session is imho the best way to solve this problem (in "Java-Pseudo-Code"):
public class BusinessServlet {
protected void doGet(HttpServletRequest req, HttpServletResponse res) {
HttpSession session = req.getSession(true);
BusinessCode business = session.getAttribute("A");
if (business == null) {
business = new BusinessCode();
session.setAttribute("A", business);
}
DebugObject debug = session.getAttribute("B");
if (debug == null) {
debug = new DebugObject();
session.setAttribute("B", debug);
}
business.doSomeWork(debug);
}
}
public class DebugServlet {
protected void doGet(HttpServletRequest req, HttpServletResponse res) {
HttpSession session = req.getSession(true);
DebugObject debug = session.getAttribute("B");
if (debug != null) {
debug.printDebugMessages(res);
}
}
}
Does this help?