Here're goals i'm trying to achieve:
Take the scheduled jobs out of microservice because it can and would harm timings/performance
Execute jobs in a separate computation cluster aka workers
Avoid code duplication: i want to keep all my business logic in one Service, all DB-related operations in one Dao, do not write additional services/daos for jobs
Avoid dependency management problems: different jobs may require different libs/versions/etc. For instance, job from ServiceA may use javax.annotation-api while job originated from ServiceB may use jakarta.annotation-api. Making a worker depend both on ServiceA and ServiceB will cause build or runtime problems.
Are there any approaches/libraries/solutions to achieve all the goals at the same time?
UPD:
Both Temporal.io and quartz are not quite what I need - they both require worker to depend on workflow tasks.
I can imagine that I’m approaching the issue I face in incorrect way, so architectural advises are also appreciated
From architectural perspective, expose service (business logic) via API.
Have schedulers run on separate instance or if you are using some of the popular cloud solutions have their FaaS (function as a service in your case scheduler) trigger service API via HTTP (any or dedicated instance).
Azure -> user azure functions
AWS -> lambda functions
Google Cloud -> Google Cloud Functions
All of the above have comprehensive guide how to create scheduled function aka trigger.
Hope this helps and I'm not off topic.
From my perspective you have the option to use one of three possible solutions:
Most straight forward - Ensure that service logic which is required in the jobs also implements a local API (programming API).
As such it can act as and be imported as a library and reused in jobs without code duplication.
If you have a larger development organization you also want to make sure that such libraries are correctly version managed and version releases are pre-planned, which allows the teams using the libraries to treat them like they would third party libraries.
Also there is no magic - You would have to work through any build/dependency problems if there would be conflicts. (Since your question sounds like this is a deal breaker, let's take a look at the other solutions.)
The second solution would be to provide a wrapper for each service logic that allows to access functionality via CLI. That means you don't have to import the libraries, but rather execute them as jars/executables through the CLI. This would allow you to use the same code but avoid dependency problems.
(You will still have to deal with version management and version upgrades, etc.)
In case you use containerized deployments/hosting you can also consider to bundle up multiple containers together just for your jobs, where each job gets its own private service container instances for use during the job. Kubernetes and Docker Compose for example have options to run such multi-container deployments/jobs.
That solution would allow you to reuse the same services as they run for other purposes, but you have to make sure that they are configurable enough to work in this scenario.
One problem that all of the approaches have is that you have to make sure there are no runtime conflicts between your jobs and the deployed regular services. (For example state conflicts)
In terms of how to execute jobs it will depend on your deployment scenario. Kubernetes has an option to run containers as jobs natively, which makes it easy to bundle multiple jars, etc. But it is always an option to deploy a dedicated scheduler or workflow tool like Apache Airflow to run your jobs.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
Is it a good practice to deploy web services separately or should they be part of the web application? For instance, I am developing a spring rest based web service. The function of this service is to, let's say, to get user data.
Each webapplication that queries this web service has it's user data in different schema. So, now the webservice will need to know who is calling it - is it Appilcation A or Application B? If it's AppA, then it should get data from Schema A, if it's AppB, then its another schema. Note, that AppA and AppB are just the same code packed into two different wars and the schema they are supposed to query is supplied from properties file.
In a situation like this, does it make sense to pack the webservice with the webapp code and deploy it under different contexts, so it becomes a duplciate service running in a different context. Or, should it be deployed separately and somehow the AppA and AppB are supposed to identify themselves to this web service?
I prefer below approach, which is in use for 50K concurrent users.
Make sure that each web service encapsulates both UI and Schema independently by executing required business use case. Each web service will have all three layers - Model, View and Controller for that business service. That means your App-A is one web service & App-B is other web service.
All web services will register and un-register with Master web service. Master web service is responsible to redirecting user request to appropriate web service like App-A OR App-B.
You should have cluster of Master web service & cluster of individual web services - App-A & App-B
In this approach, your schema can reside on different database instead of single database
Advantages of this approach:
Each web service can scale horizontally. Just add additional VM nodes if you want to increase the scale.
If you have different schemas on different databases in different locations, you are avoiding network performane bottlenecks in OLTP queries (Online transaction processing queries).
Disadvantages:
I see only one disadvantage since Master Web Service acts like a Facade and it should know the internals of individual web service. But it's not a drawback for the advantages it is offering if you consider the trade-off.
I have no idea about your business requirement to maintain different schemas for user data and going with webservice.
But instead of maintaining multiple wars with same code, i would suggest you to configure multiple datasources within the application and switch to datasource as per your requirement.
This link may help you to configure multiple DS
If you fallow aforementioned logic, you may end up with single deployable context.
Still want to stick with multiple wars as webservice, i would suggest you to have look at SpringBoot simple, container less deployable and scalable.
It is a matter of opinion, both choices are okay. You should take into account the usage of the service, scaling concerns etc.
You could look at Microservices as an idea, but it has to make sense from your standpoint.
About the two different apps: if the differences are only in configuration, try externalizing it (23. Externalized Configuration). This way you can have a single artifact being deployed twice.
Given that scenario, it is a good practice having only one web service, in this way you improve the maintainability of the system because you don’t have the same code twice. If you have in the future other new similar app you don’t have to implement a new service.
Approach 1:- (Preffered)
You should have a single web application in which will have the entire code for application UI and Repo/data interaction.
Based on the type of request dynamically switch the data source as needed. You can have at look at Spring Dynamic datasource routing here
Approach 2:-
In case your UI has a completely different type of interactions managed by different teams, it makes sense to have separate UI components and the backend web services maintained at a same location.
Again based on the type of request you can dynamically route the datasource.
Hope this helps :)
my inputs:
1) Any specific reasons to build 2 different wars for same code? Is it only because you have two different data sources for each of them?
Why cant you have single application deploy with some parameterized mechanism in each request to identify which schema to get data from?
2) Why do you need a web service in first place? Why not application hook directly to database it needed.
3) Is underlying database transactional DB or some historical data? How about merging both schemas in one as one-time effort OR using some sort of virtualized views which picks data from 2 schemas based on input parameters.
***** edited after Jay's inputs:
My suggestion will be to have web service deployed separately from 2 web apps because it provides single place to manage code in long run. I have following additional suggestions:
Define your own headers in SOAP XML Schema which can give you both appContext(application making call) as well as userContext(user). Give a good thought on this aspect keeping long term view.
Keep SOAP request-response stateless which will give you scalability. Dont maintain any state of SOAP request at server side.
I have in past used a data virtualization solution (CISCO Composite)..what benefits it provides: if there are two (or more) data sources containing similar type of data(entities), it can join,cleanse & merge it virtually and expose it as REST/SOAP based web service. Try evaluating this option as well.
What it can further help if in future you have other consumers to access your information using plain SQL/JDBC call, they will be able to do it...also data virtualization solutions support many other interfaces to consumers like Hadoop, OData etc...again it depends on budget and other constraints of project...I am not sure if there is any effective open source data virtualization solution available or not?
Personally, in my experience, it's a lot better to have them separated, it usually depends on how big and how critical your main project is.
But even if at the beginning your project isn't that big and there's only 1 person working on it, later on, as it continues to grow, if you have microservices for all the things your main project do, it will be a lot easier to maintain, rather than having many people working on the same code handling many versions of an unique project, handling many small projects is less confusing and errors are easier to find.
Plus if something fails, you can have 1 microservice down while your main still runs without interruption, it will only by denied of 1 service, instead of having everything down while you fix it.
High availability is very important in production, and having them separated helps with this.
Given your situation I'd advice going with ONE webapp (one "project") with some caveat and then consider one of the two solutions:
1) Given you are using spring, I'll assume (hope) you are using maven as well..
Make a different compilation goal and make it so that, based on the goal invoked to produce the war, the relevant properties file is different..
This way you have ONE webapp, and based on the compilation (or rather based on the properties file tied to that specific compilation) you will obtain a war tied to a specific environment&schema... You deploy an individual war for each webservice with a clean separation, though the root code is the very same and it's only one application... [CLEANER SOLUTION]
2) Make it so that you don't only get the json request but also the https certificate of the sender (thus you identify a specific "webapp" based on the https certificate exposed), and based on the certificate AND The source of the request, you ensure the source as "qualified" to receive data from schema X rather than schema Y.. You deploy ONE war only that will, at his own discretion, apply logic to reroute your "user data fetch query" to one database or the other [I DISCOURAGE THIS PRACTICE]
of course there are other approach as well, but I think these two are the most feasible..
It really depends on what you want to achieve.
If you want to encapsulate the database/schema/table, then it should really be one service for each application. The main advantage of doing this is that you could swap the database later on if there is some problem with the current one, it also simplifies caching and invalidation, etc etc.
If the database/schema/table is not encapsulated anyway, then the single service is much easier and better. Each web application just have to identify themselves, and each of them will get exactly what they need. This could be achieved by putting the query/schema information in property file, or creating db views with the same name as client, etc.
If we were to go for this approach, a question will pop up. Why bother having this layer at all? Couldn't each web application just query the db directly? If the answer is yes, then just remove the whole layer completely.
You are trying to implement a Data Provider, or DAO as a service.
To make it -
Simple
Scalable
Maintainence-friendly,
Adaptable
You can simply have a single webservice, deployed outside the WebApp(s) and driven off configuration. The configuration itself can be stored as property file, or from a DB. The identifier for the client should be being passed in the webservice request.
This is actually a pretty standard approach implemented to enable optimizations at the Data tier outside of DB, like caching (again driven of configuration), expiry, pooling, etc.
The other option, to include as a shared jar within the webapp, yes, has advantage of code-reuse (which you get with externally deployed service as well), but the following disadvantages outweigh the option.
Coupling
Employing optimizations are difficult
Release management (this also depends upon how your code is organized)
Versioning.
Hope it helps.
I would deploy to one instance. No matter what. Of course, there are circumstances where it may be necessary to deploy separately. From a best "coding" practice, one instance should be used to allow for "right once, use many".
Then...
Define different XSD's for each AppA, AppB, etc. Marshall accordingly.
Or, use Groovy to marshall appropriate objects as json or xml.
I am trying to build a customizable workflow or kind of state transition which user should be able to update through UI.
Basically it should have all the states and user should be able to choose the transition from one state to another and also hook any predefined action to it. The predefined action could be anything like user task, email task or script task.
It should be flexible enough to add more states / actions to it.
I tried evaluating Activiti but it doesn't look like a good fit here. In fact Workflow engine doesn't look like a right candidate.
Can someone suggest any existing framework or library which I can use.
You can try using JBPM. I have used it in my earlier project.
http://www.jbpm.org/[Jbpm Home][1].
Drawback
JBPM 5 doesnt support clustering. (Not sure if It is fixed in JBPM 6)
I have chosen Squirrel foundation as my backend state transition machine. Decorated with my custom palates for denoting different kind of state (user intervention, service , email etc). And using Viz.js for visualizing and creating the workflow in JS.
My main goal is providing a search application written in jquery that is based on solr. (For those who unfamiliar with solr, just assume its a rest api that can return search result.)
For this goal I wrote many small applications and servlets that each one does an ad-hoc task.
For example:
SearchApp - a jquery app in which an end user can perform searches.
SolrProxy - A java servlet that plays a proxy role between the SearchApp and solr. One of the things it does is logging the user request for later analysis.
StatsApp- a servlet that performs analysis of the user activity and returns a json with the data.
Indexer - a java application that indexes data to solr according to my requirements. in this process it also fetches an SQLServer DB, and then performs some update commands to the DB.
IndexerServlet - an asynchronous servlet that uses Indexer to provide an ability to execute index by http request.
Nutch - an open source project that indexes data to solr for other requirements that are not accomplished in Indexer(3).
(MAYBE) - some service that will perform scheduled Nutch running.
And more components might be added.
It seems a bit wrong to have multiple java projects that each one does a single task, instead of having one project that handles most of the components.
Any ideas and insights on this?
Should I combine all the java apps to a single project? should I use some kind of a fremework for this? or should I live it as it is now?
I don't think it's a bad idea that you have all these separate applications. They all seem to be doing one thing, and doing it well. What you can do, is expose them via a unified interface. So essentially you have a facade that sits in front of all these disparate services that presents an abstract and uniform interface. The consumers of this service will have no idea what sits behind that facade. This is just as well, because now you can discretely update and replace individual components without affecting others. If you had combined all of them into one, you would have to push a new release every time you modified one of the components.
We have an issue where a Database table has to be updated on the status for a particular entity. Presently, its all Java code with a lot of if conditions and an update to the status. I was thinking along lines of using a Workflow engine since there can be multiple flows in future. Is it an overkill to use a Workflow Engine here... where do you draw the line ?
It depends on the complexity of your use case.
In a simple use case, we have a database column updated by multiple consumers for each stage in an Order lifecycle. This is done by a web service calling into the database.
The simple lifecycle goes from ACKNOWLEDGED > ACCEPTED/REJECTED > FULFILLED > CLOSED. All of these are in the same table on the same column. This is executed in java classes with no workflow.
A workflow engine is suited in a more complex use case which involves actions on multiple data providers eg: database or Content Mgmt or Document Mgmt or search engine, multiple parallel processes, forking based on the success/failure of a previous step, sending an email at a certain step, offline error alerting.
You can look at Apache ODE to implement this.
We have an issue where a Database table has to be updated on the status for a particular entity. Presently, its all Java code with a lot of if conditions and an update to the status.
Sounds like something punctual, no need for orchestrating actions among workflow participants.
Maybe a rule engine is better suited for this. Drools could be a good candidate. When X then Y.
If you're using Spring, this is a good article on how to implement your requirement
http://www.javaworld.com/javaworld/jw-04-2005/jw-0411-spring.html
I think you should consider a workflow engine. Workflow should be separated from application logic.
Reasons:
Maintainable: Easier to modify, add new flows and even easier to replace by another workflow engine.
Business Process management: Workflows are mostly software representations of BPM. So it is usually designed by process designers (Non-tech people). So it is not a good idea to code inside the application. Instead BPM products such as ALBPM or JPBM should be used which support graphical workflow designs.
Monitoring business flows: They are often monitored by the Top level managers and used to make strategic decisions.
Easier for Data mining/Reports/Statistics.
ALBPM(Now Oracle BPM): is a commercial tool from Oracle suitable for large scope projects.
My recommendation is JBPM. Open source tool from JBOSS. Unlike ALBPM which requires separate DB and application server, it can be packaged with your application and runs as another module in your application. I think suitable for your project.