This problem has been in my head for a while and I can't seem to understand the logic when I find solutions.
Heres the deal:
I'm currently working on a simple application that has been split up into 2 microservices.
The application is very similar to the task of managing software Trello.
There's a microservice solely used for storing User information (User service)
Another microservice is responsible for the Boards (Boards hold lists and lists hold tasks but that's not relevant to my question). Both microservices use their own database.
All of this will be hosted on AWS, the code is written in Java and I use Hibernate to generate the databases.
My problem:
How do I make use of the User service and have a Board be used by multiple User entities?
I understand the practice of using a many-to-many table which stores the BoardId's together with the UserId's, but what would happen if I were to remove a user that's connected to a board. There's no logical connection between the User that's in the user database and the userId that's stored by the board.
(A user can be signed to multiple boards and vice-versa)
My questions in short: How does this look in the database?
How does the Boards service access the User service and save Users to Boards?
What happens when I delete a User on the User microservice?
This seems to be a typical Microservice Communication related problem. As per my knowledge and experience, there is a solution i.e. to introduce a message broker in the architecture which can perform actions on events like say USER_DELETED event can be passed to board microservice which will handle board related db changes.
Implementing this can feel scary for you at the start, but nowadays it is not that difficult with all the very supportive internet communities helping and coming up with new solutions every day.
If you are using Java for microservices with spring boot, then there are solutions like Apache Kafka, Hazelcast(quite cost effective).
If you are using NodeJS for micorservices, then NATS is one of the libraries which I have used recently.
Related
In the context of a university project, we have to develop a Java distributed application with these requirements:
The application will follow the classic client-server schema, with
multiple clients connecting to a central server on a different
machine, which also hosts a rdbms to which the server connects
The relational database we must use is postgresql (latest version)
Both client and server must be written in Java
We must use native JDBC to access the database (we can't use frameworks like Spring)
DISCLAIMER:please understand we are just a group of students and this is our first big project involving all these aspects and we aren't experts by any means, so please be patient with us :) (also English is not our first language, sorry for any mistake you might find)
We are currently in the design phase of the application (class diagrams, sequence diagrams etc) and we're stuck with a possible concurrency problem with the database:
ideally our server would listen for any requests and for each client that logs in the application, the server launches a dedicated thread that provides implemented services to the user (implementing pattern proxy-skeleton with basic socket programming). Each of these service providers (threads), upon completing the requested task should update/insert/delete data to the database. Here is the problem: how should we manage the concurrency here?
We tried to search the internet for this kind of issue and we found some things but we're still very confused:
Since we actually interact with the database from one single central
server (with one admin profile) we could implement a queue sistem for
the various transactions coming from the different threads we
launched
We manage concurrency at database level with some well-known mechanism such as MVCC, which is apparently a lot more complicated
Ideally we would like that read requests don't block other reads or writes, and writes only block other writes (which seems to be the case with MVCC). Which alternative would be best? Are there any other options that we could implement with the restrictions above mentioned? Thanks in advance for any suggestion
We have 13 years old monolithic java application using
Struts 2 for handling UI calls
JDBC/Spring JDBC Template for db calls
Spring DI
Tiles/JSP/Jquery for UI
Two deployables are created out of this single source code.
WAR for online application
JAR for running back-end jobs
The current UI is pretty old. Our goal is to redesign the application using microservices. We have identified modules which can run as separate microservice.
We have following questions in our mind
Which UI framework should we go for (Angular/React or a home grown one). Angular seems to be very slow and we need better performance as far as page loading is concerned.
Should UI/Javascript make call to backend web services directly or should there be a spring controller proxy in deployed WAR which kind of forwards UI calls to APIs. This will also help if a single UI calls requires getting/updating data from different microservice.
How should we cover microservice security aspect
Which load balancer should we go for if we want to have multiple instance of same microservice.
Since its a banking application, our organization does not allow using Elastic Search/Lucene for searching. So need suggestion for reporting using Oracle alone.
How should we run backend jobs?
There will also be a main payment microservice which will create payments. Since payments volume is huge hence it will require multiple instances. How will we manage user logged-in session. Should we go for in-memory distributed session store (may be memcache)
This is a very broad question. You need to get a consultant architect to understand your application in depth, because it is unlikely you will get meaningful in-depth answers here.
However as a rough guideline here are some brief answers:
Which UI framework should we go for (Angular/React or a home grown one). Angular seems to be very slow and we need better performance as far as page loading is concerned.
That depends on what the application actually needs to do. Angular is one of the leading frameworks, and is usually not slow at all. You might be doing something wrong (are you doing too many granular calls? is your backend slow?). React is also a strong contender, but seems to be losing popularity, although that is just a subjective opinion and could be wrong. Angular is a more feature complete framework, while React is more of a combination of tools. You would be just crazy if you think you can do a home grown one and bring it to the same maturity of these ready made tools.
Should UI/Javascript make call to backend web services directly or
should there be a spring controller proxy in deployed WAR which kind
of forwards UI calls to APIs. This will also help if a single UI calls
requires getting/updating data from different microservice.
A lot of larger microservice architectures often involve an API gateway. Then again it depends on your use case. You might also have an issue with CORS, so centralising calls through a proxy / API gateway, even if it is a simple reverse proxy (you don't need to develop it) might be a good idea.
How should we cover microservice security aspect.
Again no idea what your setup looks like. JWT is a common approach. I presume the authentication process itself uses some centralised LDAP / Exchange or similar process. Once you authenticate you can sign a token which you give to the client, which is then passed to the respective micro services in the HTTP authorization headers.
Which load balancer should we go for if we want to have multiple
instance of same microservice.
Depends on what you want. Are you deploying on a cloud based solution like AWS (in which case load balancing is provided by the infrastructure)? Are you going to deploy on a Kubernetes setup where load balancing and scaling is handled as part of its deployment fabric? Do you want client-side load balancing (comes part of Spring Cloud)?
Since its a banking application, our organization does not allow using
Elastic Search/Lucene for searching. So need suggestion for reporting
using Oracle alone.
Without knowledge of how the data on Oracle looks like and what the reporting requirements are, all solutions are possible.
How should we run backend jobs?
Depends on the infrastructure you choose. Everything is possible, from simple cron jobs, to cloud scheduling services, or integrated Java scheduling mechanisms like Quartz.
There will also be a main payment microservice which will create
payments. Since payments volume is huge hence it will require
multiple instances. How will we manage user logged-in session. Should
we go for in-memory distributed session store (may be memcache)
Not really. It will defeat the whole purpose of microservices. JWT tokens will be managed by the client's browser and expire automatically. You don't need to manage user logged-in session in such architectures.
As you have mentioned it's a banking site so security will be first priory. Here I have few suggestions for FE and BE.
FE : You better go with preactjs it's a react like library but much lighter and fast as compare to react. For ui you can go with styled components instead of using some heavy third party lib. This will also enhance performance and obviously CDNs for images and big files.
BE : As per your need you better go with hybrid solution node could be a good option.e.g. for sessions.
Setup an auth server and get you services validate user from there and it will be used in future for any kinda service .e.g. you will expose some kinda client API's.
User case for Auth : you can use redis for session info get user validated from auth server and add info to redis later check if user is logged in from redis this will reduce load from auth server. (I have used same strategy for a crypto exchange and went pretty well)
Load balancer : Don't have good familiarity with java but for node JS PM2 will do that for you not a big deal just one command and it will start multiple instances and will balance on it's own.
In case you have enormous traffic then you better go with some messaging service like rabbitmq this will reduce cost of servers by preventing you from scaling your servers.
BE Jobs : I have done that with node for extensive tasks and went quite well there you can use forking or spanning this will start a new instance for particular job and will be killed after completing it and you can easily generate logs along with that.
For further clarification I'm here :)
Background
I am trying to make a public facing API that is gated behind an OAuth2 workflow. I've found example database designs using JPA Repositories/Spring-OAuth2, which is the framework that I'm using. I basically have this application, which is currently using an InMemory authentication, which I'm trying to convert to pull the data from a database using JPA Repositories and a relational database design.
I did find this guy, but the problem is that it doesn't account for how the data is related to one another.
IE I was hoping that I could split the scopes into a separate table so it'd be as simple as adding additional scopes that are available for apps later on. I was also thinking of adding support for having types of OAuth clients, such as bots, website, desktop app, mobile app, etc.
I'm assuming there is some sort of a relationship between oauth_approvals, oauth_refresh_token, oauth_client_token and oauth_client_details. I want to also make sure that it's possible to map an oauth client to a user. Ultimately I'd like them to be able to have multiple clients.
For my current use case it's definitely overkill to have this flow, but I am using this application to learn about different technologies.
What I currently have
I'm sure this can be done better, if so how?
Questions
How do I properly make the relationship between the different components of an oauth client?
How can I make this so that I properly allow a user to have many clients?
How would I be able to dynamically get a client from the database, or is this handled through ClientDetailsServiceConfigurer.jdbc()?
How would I be able to add types of clients to the flow?
If you have additional questions let me know and I'll be more than happy to update my question.
I need to implement a proof of concept application Swing application where there's a server having a list of users and several clients which connect to the server and do CRUD operations on the database and hence on the list of users.
I have an obvious synchronization dilemma of keeping all clients lists updated so that if one client removed a user another one who still has it in his list cannot change its name.
Now I know a protocol in which before updating a user the client asks the server whether it still exists would work.
However this is just a simple example but in the real application I might have junction tables and complex references between objects which need always to be kept consistent and I don't want to reinvent the wheel.
What I'm wondering if there's some ready made solutions or some library which does this job which doesn't require me to change database or load extremely complex dependencies.
I did some research on Google but nothing seems to fit and the most similar example of client server synchronization I found was "chat programs", however chat programs are inherently simple because a message is never modified or deleted and all you have to keep consistent is the chronological order. I would need something more involved than that or some useful hints on the subject.
What you need is some sort of messaging between server and clients. Clients will either long-poll the server, asking for the updates, or subscribe to some streaming endpoint on your server.
You can take a look on Comet model, long-polling itself, websockets, etc.
Alternatively - there are couple of data management servies - BlazeDS, GraniteDS or any similar purpose solution. You can integrate one of those in your application and use for complete data management cycle.
I've been tasked with making an enterprise application multi-tenant. It has a Java/Glassfish BLL using SOAP web services and a PostgreSQL backend. Each tenant has its own database, so (in my case at least) "multi-tenant" means supporting multiple databases per application server.
The current single-tenant appserver initializes a C3P0 connection pool with a connection string that it gets from a config file. My thinking is that now there will need to be one connection pool per client/database serviced by the appserver.
Once a user is logged in, I can map it to the right connection pool by looking up its tenant. My main issue is how to get this far - when a user is first logged in, the backend's User table is queried and the corresponding User object is served up. It seems I will need to know which database to use with only a username to work with.
My only decent idea is that there will need to be a "config" database - a centralized database for managing tenant information such as connection strings. The BLL can query this database for enough information to initialize the necessary connection pools. But since I only have a username to work with, it seems I would need a centralized username lookup as well, in other words a UserName table with a foreign key to the Tenant table.
This is where my design plan starts to smell, giving me doubts. Now I would have user information in two separate databases, which would need to be maintained synchronously (user additions, updates, and deletions). Additionally, usernames would now have to be globally unique, whereas before they only needed to be unique per tenant.
I strongly suspect I'm reinventing the wheel, or that there is at least a better architecture possible. I have never done this kind of thing before, nor has anyone on my team, hence our ignorance. Unfortunately the application makes little use of existing technologies (the ORM was home-rolled for example), so our path may be a hard one.
I'm asking for the following:
Criticism of my existing design plan, and suggestions for improving or reworking the architecture.
Recommendations of existing technologies that provide a solution to this issue. I'm hoping for something that can be easily plugged in late in the game, though this may be unrealistic. I've read about jspirit, but have found little information on it - any feedback on it or other frameworks will be helpful.
UPDATE: The solution has been successfully implemented and deployed, and has passed initial testing. Thanks to #mikera for his helpful and reassuring answer!
Some quick thoughts:
You will definitely need some form of shared user management index (otherwise you can't associate a client login with the right target database instance). However I would suggest making this very lightweight, and only using it for initial login. Your User object can still be pulled from the client-specific database once you have determined which database this is.
You can make the primary key [clientID, username] so that usernames don't need to be unique across clients.
Apart from this thin user index layer, I would keep the majority of the user information where it is in the client-specific databases. Refactoring this right now will probably be too disruptive, you should get the basic multi-tenant capability working first.
You will need to keep the shared index in sync with the individual client databases. But I don't think that should be too difficult. You can also "test" the synchronisation and correct any errors with an batch job, which can be run overnight or by your DBA on demand if anything ever gets out of sync. I'd treat the client databases as the master, and use this to rebuild the shared user index on demand.
Over time you can refactor towards a fully shared user management layer (and even in the end fully shared client databases if you like. But save this for a future iteration.....