I have a web application written in Java using the Spring framework.
I would like to store the users activities like, page visits, actions, interactions etc.
I read that usually this is done by creating a table for each tracked aspect. I was wondering if there is a better way to do it using Spring framework, like a way to intercept all the requests and trigger some actions.
What kind of technology do you recommend to store all these information? Right know I’m using a MySql database interacting with it through JPA. But, since I’m really really new to these kind of things I don’t know if I should go with a NoSql database or stay with my already existing MySql database. This wonder comes from the idea that this kind of data flow will be much bigger than a normal data flow coming from more traditional actions such as signin, creation, deletion etc.
Hope to have explained myself... if not just tell me and I’ll try to add more details.
[EDIT 1]
The web app is an e-commerce. So far it does not have So many users but it will (in the order of thousands).
The goal of the user tracking it’s just to profile them in order to give them a better and more custom service. For instance, if a see that a user is taking a look to a lot of items of a precise category I can show him more items of that kind.
I do no care that much about the performance, I mean, it does not have to be so fast.
Right know I have just one database and everything is stored inside it. I don’t know if charging it with this kind of data flow would slow down its performance.
The application is running on AWS ElasticBeanstalk and the database is on AWS RDS.
In general its a very broad topic.
The following considerations come to my mind:
How many requests pass to the microservice per some period of time? If its a small number of users (which translates to the number of records to the database) - then its ok to go with the MySQL approach - the table won't be large. Note however, that sometimes it should be cleaned anyway
Is the latency important? Sometimes requests have to be served very quickly, adding a hop to the database to save the user preference can be problematic
How do you want to consume this kind of information? Are you planning to use dashboards (in this case micrometer + Prometheus / InfluxDB and Grafana can be a good solution). Are you planning to actually charge the users per number of requests with an ability to send the monthly bill to their email in PDF or provide a web access to such an information (like AWS does for example)?
How about Rate limiter? Are you planning to deny some requests if they're frequent and coming from the same user?
How many instance will "add" this kind of information? What if you have thousands of microservices that now will have to write to MySQL - it might not survive such a load (in addition to the regular load its set up for)?
The range of solutions can vary.
You can Batch the requests per user in memory and send once in while a message into Kafka and then use kafka streams to provide aggregations on it. With this approach you'll minimize the impact of the added processing on the existing solution and will deploy some other service that will be able to process this pretty large amount of data.
Another option: maybe you can create an asynchronously populated log file and store the information there. Then you might want to add some "agent" / side-car container like logstash and stream the data into some storage. Yet Another project that might be relevant in this field is Apache Flume which will allow you to construct a pipeline.
For billing you might use specialized systems AFAIK spring doesn't have anything like this usually these are ready products that you can integrate with.
For Rate Limiting you might consider: Resilience4j or solve it with redis
Yeah , That's possible , Here below are the three approaches with some sample snippets which would help you in the implementation , Moreover it depends on the data you store when you log the activity and when do you consider the activity data as obsolete and there are many factors which can decides your data store.
Approach 1: You can keep track of the login user using Spring-Security
You can write a HTTPSessionBindingListener and track the actions something like this
#Component
public class LoggedUser implements HttpSessionBindingListener {
private String username;
private ActiveUserStore activeUserStore;
public LoggedUser(String username, ActiveUserStore activeUserStore) {
this.username = username;
this.activeUserStore = activeUserStore;
}
public LoggedUser() {}
#Override
public void valueBound(HttpSessionBindingEvent event) {
List<String> users = activeUserStore.getUsers();
LoggedUser user = (LoggedUser) event.getValue();
if (!users.contains(user.getUsername())) {
users.add(user.getUsername());
}
}
#Override
public void valueUnbound(HttpSessionBindingEvent event) {
List<String> users = activeUserStore.getUsers();
LoggedUser user = (LoggedUser) event.getValue();
if (users.contains(user.getUsername())) {
users.remove(user.getUsername());
}
}
// standard getter and setter
}
and for login and logout you can track using AuthenticationSuccessHandler
#Component("myAuthenticationSuccessHandler")
public class MySimpleUrlAuthenticationSuccessHandler implements AuthenticationSuccessHandler {
#Autowired
ActiveUserStore activeUserStore;
#Override
public void onAuthenticationSuccess(HttpServletRequest request,
HttpServletResponse response, Authentication authentication)
throws IOException {
HttpSession session = request.getSession(false);
if (session != null) {
LoggedUser user = new LoggedUser(authentication.getName(), activeUserStore);
session.setAttribute("user", user);
}
}
}
Approach 2 : The other method if you want to make it very simple is that you can write a OncePerRequestFilter
#Component
#Ordered(Ordered.LOWEST_PRECEDENCE)
public class LogFilter extends OncePerRequestFilter {
#Override
protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain)
throws ServletException, IOException {
// Log the info you need
// ...
filterChain.doFilter(request, response);
}
}
Approach 3 : Implement using Spring AOP.
#Aspect
#Component
public class WebMethodAuditor {
protected final Log logger = LogFactory.getLog(getClass());
public static final String DATE_FORMAT_NOW = "yyyy-MM-dd HH:mm:ss";
#Autowired
AuditRecordDAO auditRecordDAO;
#Before("execution(* com.mycontrollers.*.*(..))")
public void beforeWebMethodExecution(JoinPoint joinPoint) {
Object[] args = joinPoint.getArgs();
String methodName = joinPoint.getSignature().getName();
User principal = (User)SecurityContextHolder.getContext().getAuthentication().getPrincipal();
Timestamp timestamp = new Timestamp(new java.util.Date().getTime());
// only log those methods called by an end user
if(principal.getUsername() != null) {
for(Object o : args) {
Boolean doInspect = true;
if(o instanceof ServletRequestDataBinder) doInspect = false;
if(o instanceof ExtendedModelMap) doInspect = false;
if(doInspect) {
if(o instanceof BaseForm ) {
// only show form objects
AuditRecord ar = new AuditRecord();
ar.setUsername(principal.getUsername());
ar.setClazz(o.getClass().getCanonicalName());
ar.setMethod(methodName);
ar.setAsString(o.toString());
ar.setAudit_timestamp(timestamp);
auditRecordDAO.save(ar);
}
}
}
}
}
}
Source and More details :
https://www.baeldung.com/spring-security-track-logged-in-users
Spring / AOP: Best way to implement an activities log in the database
What is the best way to log Activity in Spring Boot with Thymeleaf?
Related
I'm trying to figure out a good way to organize a javax.websocket multiplayer card game I'm working on.
I want to split up my code into multiple classes which are each a ServerEndpoint. My problem is that I need an effective way of sharing session data between them.
I have an index at "/index", which is where I'm currently creating Player objects for clients. I'm setting these like so:
#ServerEndpoint("/index")
public class IndexEndpoint {
#OnOpen
public void openConnection(Session session) {
session.getUserProperties().put("player", new Player());
}
}
And that works; I can access the Player objects elsewhere throughout IndexEndpoint.
But when I try to access the user properties from another endpoint (after having established a connection with IndexEndpoint in JavaScript, waiting 5 seconds and then opening up an additional connection to LobbyEndpoint on the same page), I get null.
#ServerEndpoint("/lobby")
public class LobbyEndpoint {
#OnOpen
public void openConnection(Session session) {
System.out.println(session.getUserProperties().get("player")); // prints null
}
}
Which leads me to imply that session data is unfortunately not shared across endpoints.
Is there any good way for me to share websocket session data across multiple classes?
I guess one solution would be to just have one über endpoint for all users to connect to, and which can handle any message type. Then that endpoint delegates message data to other parts of my application.
However I feel like this design would be very messy and restrictive. I'd like to have endpoints dedicated to specific rooms, using annotations like #ServerEndpoint("/rooms/{room-id}") and #PathParam("room-id"). So if I was using a monolithic Endpoint then I couldn't do that; I wouldn't be able to know if a connecting user is a user who access to the room.
what about a singleton EJB containing all player and game data?
#Stateless
#ServerEndpoint("/lobby/{sessionKey}")
public class IndexEndpoint {
#EJB
GameData data;
#OnOpen
public void onOpen(Session session, EndpointConfig config,
#PathParam("sessionKey") int id) {
Player player = data.getPlayer(id);
session.getUserProperties().put("player", player);
}
}
The first message from the IndexEndpoint could provide the client with its id and then the client could provide its id to the LobbyEndpoint in the ws URL (or in a header with a Configurator), that is how I plan to do it.
I have a design problem as follows: I want to execute several soap webservices, where each response depends on the former.
When all responses are obtained, I want to validate all obtained data, then build some output based on it, and also issue from DB update.
Therefore I created a TemplateFacade method that wrapps all webservices that are to be executed. Problem: I obviously have to persist the responses between the method calls. Which will be problematic as autowired services should by definition be stateless and are singletons.
So how can I use injection with services that have to maintain some kind of state (at least until the Executor.execute() has terminated)?
Could you recommend a better design approach?
#Component
class Executor {
#Autowired
TemplateFacade template;
public void execute() {
template.run();
template.validate();
template.buildOutput();
template.updateDatabase();
}
}
#Service
class TemplateFacade {
//service classes wrapping webservice soap execution logic
#Autowired
PersonSoap personSoap;
#Autowired
CarSsoap carSoap;
#Autowired
ServiceDao dao;
private WebserviceRsp personRsp, carRsp;
void run() {
personRsp = personSoap.invoke();
//process response and prepare CarSoapXML accordingly, then send
carRsp = carSoap.invoke();
}
//the following methods will all
void validate() {
//validate both responses
}
void buildOutput() {
//create system out based on responses
}
void updateDatabase() {
dao.update(..);
}
}
To share state between multiple web services, you could keep track using a PersonState in the session which is tied to the user. I recommend encryption or hashing to secure the information.
When the validate completes, you could keep a PersonState in the session. When the buildOutput starts, you could get the PersonState object and continue with your logic and so on.
It is important that, you keep the PersonState to have a smaller memory footprint. Incase of a lot of data, you could just create a stateObject that will have the necessary state for the next step. e.g. at the end of validate you could create, BuildState object and put it in the session. build will get the object from the session and continue.
But I am not sure if it is really necessary to keep track of state and do it in 2 web services calls. The better solution would be to move all the logic part to another layer, and use the web services as just a window to your business/process layer.
Edit:
One more solution that could work you, is that the response of each step could contain the necessary state that is required for the next step. e.g. validateResponse contains personState and that could somehow be passed for the build.
We want to implement a public RESTful API integrated in our software (written in java) that might be used by various clients to build small e-commerce apps (e.g. for Android or iPhone). This API includes getting a list of products, categories, shopping cart support, etc.
We need to provide an API that will allow user registration and couple of other sensitive functions. How should we protect this API against spam and bruteforcing? In the standard product we use reCAPTCHA. Any alternative for the REST counterpart?
First, think of separation of concerns. What is the purpose of REST API?
A REST API should do offer a service to the client. Client sends a request via REST protocol, and gets a response for its request. In code, this looks something like:
#GET
public Response getClientInfo(#QueryParam("clientId") Integer clientId) {
ClientDTO clientDTO = database.getClientInfo(clientId);
return ResponseWrapper.wrap(clientDTO);
}
Now, you want your REST method doing ONLY this and nothing else. Otherwise, you would put block-bruteforce-and-spam-logic in your REST method and you would get a mess of the code that is not extensible, hard to version, etc. If you want to change your, e.g. blacklisting policy you would have to change each and every REST method, and it's bulky. If you want to check the calls before the make it to REST methods, then take a look at Filters. Every request and response pass through a chain of filters and could be check for misuse of the server.
I don't know what is your technology stack is, but I would suggest looking into these:
JBoss AS7.
DeltaSpike (enables you powerful Interceptors that will check user rights and execution rights before the execution of the REST method).
for example:
#LoggedInUser
#GET
public Response getClientInfo(...) {
...
}
This security annotation #LoggedInUser (which, by the way, you define) will give sign to an Interceptor to check this security constraint, e.g.
#Secures (built in annotation)
#LoggedInUser
public boolean hasRight(Identity identity) {
return identity.isLoggedIn(); //or if he is in certain group of users
}
Context and Dependency Injection context (used in DeltaSpike).
JBoss Filters (a filter chain where you can create your own filter that, for example, checks if some IP is trying to send multiple calls within a very short period ~ 10 lines of code).
An example of the Filter
#Startup
#ApplicationScoped
#Filter(around= "org.jboss.seam.web.ajax4jsfFilter")
public class IPTrackerFilter extends AbstractFilter {
//IPTracker is your #ApplicationScoped bean that remembers all IP addresses accessing the application.
#Inject
private IPTracker fIPTracker;
#Override
public void doFilter(ServletRequest req, ServletResponse res, FilterChain chain) throws IOException, ServletException {
if (!(req instanceof HttpServletRequest)) {
chain.doFilter(req, res);
return;
}
final String ipAddress= ((HttpServletRequest)req).getRemoteAddr();
if (fIPTracker.isBlackListed(ipAddress)) {
//implement error message here
sendErrorMessage(response);
return;
} else {
//all good, continue
chain.doFilter(req, res);
}
}
}
PS. I gave you the link for DeltaSpike, for others is really easy to find. Also, if you find DeltaSpike to obscure, try with JBoss Seam Security Framework.
I'm having a problem where Exceptions are popping up in my production system but I really don't have good information about who is causing them. The person's username is stored as a variable in their tomcat session, which I have access to in my doPost or doGet method obviously, but unless I pass that information down as a parameter to each of my business objects, I don't have access to the session. For obvious reasons, I'd like to tack the username into the logging message so I have an idea of what is going on.
So my solution is to do something like this
public class ExceptionUtil {
private ExceptionUtil() { } // no instantiation
private static final ThreadLocal<String> local = new ThreadLocal<String>();
public static void set(String user) { local.set(user); }
public static String get() { return local.get(); }
}
Then in my posts/gets, I can do this
String username = request.getSession().getAttribute("username");
ExceptionUtil.set(username);
Then in my exceptions, I might do this (contrived, bad practice example)
catch(SQLException e) {
logger.error(ExceptionUtil.get() + " did something dumb in sql", e);
throw e;
}
The only problem I'm concerned about is how Tomcat will manage my threads. What if they keep the threads? Will they persist? Will the ThreadLocal values also persist? If I was storing the entire Session in the ThreadLocal instead of just a String, that would be a serious memory leak potential. It also means if someone forgot to re-set (or forgets to clear when done) the username/session on a thread that persisted for multiple requests, there might be stale data in there.
Call my cynical, but I don't want to have to rely on programmers (even, especially myself!) not forgetting to do things for a program's correctness. If I can idiot-proof my code, I'd like to. And that means getting a better understanding of how Tomcat will use the threads.
So, the question in a single-sentence form:
If I use ThreadLocal in a webapp running on Tomcat (7.0.27), do I run
the risk of a Thread being used for multiple requests, and with it
data from a previous request being persisted?
I should note that even though they don't answer the exact question of "Tomcat/ThreadLocal shenanigans", I am open to alternative solutions that allow me to elegantly access session variables for logging purposes. I am also open to commentary about potential pitfalls of my solution. I have a business problem to solve, and I'm not married to any one solution. I just want to know who keeps causing the exceptions on my prod system :)
Yes, tomcat uses the ThreadPool concept , that means the threads are being reused and hence as you suggested "Your Thread Local retains the values" ,
alternatives what i would suggest could be
clean up threads after you are done, somewhere in the view controller
Write a Request Filter and on start of filter Clean up everything and push new values,
and assign this to every url pattern on ur server.
for the approach you are following instead of saving certain values in classes,
Store the request in Thread Local and then use the request to pull values out of session using a homemade util class, that takes request and then returns you desired value, that way you save yourself of saving session in Thread and get the value, but please ensure that u add fresh every time and clean up request after you are done(use 2nd option for that ) .
You don't need to reinvent the wheel, the log system does it for you.
If logback/log4j is your logger implementation, then Mapped Diagnostic Context(MDC) is definitely your answer.
MDC is logically like ThreadLocal, but it's better:
MDC handle thread-safe and synchronization transparently
A child thread automatically inherits a copy of the mapped diagnostic context of its parent. So even you using multi-thread to process request, it's still ok.
So set MDC in servlet filter like this, to achieve your goal:
public void doFilter(ServletRequest request, ServletResponse response,
FilterChain chain) throws IOException, ServletException {
boolean successfulRegistration = false;
HttpServletRequest req = (HttpServletRequest) request;
Principal principal = req.getUserPrincipal();
// Please note that we could have also used a cookie to
// retrieve the user name
if (principal != null) {
String username = principal.getName();
successfulRegistration = registerUsername(username);
}
try {
chain.doFilter(request, response);
} finally {
if (successfulRegistration) {
MDC.remove(USER_KEY);
}
}
}
private boolean registerUsername(String username) {
if (username != null && username.trim().length() > 0) {
MDC.put(USER_KEY, username);
return true;
}
return false;
}
Then in your log configuration, add %X{USER_KEY} in your pattern layout to use the value you set in MDC.
In logback, there are out-of-box filter MDCInsertingServletFilter can log more information like remoteHost/requestUrl and etc, very useful information for logging.
Check the logback document on MDC http://logback.qos.ch/manual/mdc.html
I have a web based application that uses userName and password for login.
now how can i check on certain time which all users are logged in at that very time.
i am using session management and no DB is used in application everything is on filesystem
Edit: 1 more silly doubt.. how to define a variable with application scope.. is this something of this sort?
<env-entry>
<env-entry-name>test/MyEnv2</env-entry-name>
<env-entry-type>java.lang.Boolean</env-entry-type>
<env-entry-value>true</env-entry-value>
</env-entry>
Just collect all logged in users in a Set in the application scope. If your application is well designed, you should have a javabean User which represents the logged-in user. Let it implement HttpSessionBindingListener and add/remove the user from the Set when it's about to be bound/unbound in the session.
Kickoff example:
public class User implements HttpSessionBindingListener {
#Override
public void valueBound(HttpSessionBindingEvent event) {
Set<User> logins = (Set<User>) event.getSession().getServletContext().getAttribute("logins");
logins.add(this);
}
#Override
public void valueUnbound(HttpSessionBindingEvent event) {
Set<User> logins = (Set<User>) event.getSession().getServletContext().getAttribute("logins");
logins.remove(this);
}
// #Override equals() and hashCode() as well!
}
Note that you need to prepare the Set in the application scope so that it doesn't return null in above methods. You could do that in the same methods by a nullcheck, or with help of ServletContextListener#contextInitialized().
Then, anywhere in your application where you've access to the ServletContext, like in a servlet, you can just access the logged-in users as follows:
Set<User> logins = (Set<User>) getServletContext().getAttribute("logins");
Update
BalusC's approch is more suitable, here by this approach you will get no of session not logged in Users. to do that you need to track HttpSessionBindingListener .
You can implement HttpSessionListener and can track the logged in users
public void sessionCreated(HttpSessionEvent se) {
// adding logging in user to some application data map or inserting it to DB
}
public void sessionDestroyed(HttpSessionEvent se) {
// remove logged in user to some application data map or inserting it to DB
}
A more scaleable solution would be to add a column like "loggedIn" to your users table in the database and set it to true when you log a user in. This would take bloat off the application server and also support a distributed environment if your application needs to run more than one box in the future.