I need suggestions regarding creating a scheduled task in a spring boot application. Our application is an order management portal that helps users with details about an order or a list of orders.
Currently, there is a manual process that is as followed: when a new order comes into the database, the user picks up some information of the order, prepares a doc file of it, and sends it to an external portal through email. That portal sends users a response in the email and the user saves the response in the system through our application. Now we want to automate this process in which instead of going the email route we will make a SOAP call to the external portal(send the doc as an attachment in XML request) and get the response back. For this, I want to write a scheduled process (#Scheduled) which will pick up the new orders and make a SOAP call for them.
My question is should I create the scheduled process in the same application or create a new module(a spring boot app) for it. I appreciate your suggestions.
Spring boot scheduling is very handy, go with it if you can. However if you have multiple instances of the spring application, scheduling will be enabled on all of them making it hard to synchronize who calls what.
Now if you are running multiple instances and assuming you have some sort of load balancer in front of them, than create a separate cron job which will call some endpoint and your load balancer will rout the request to one particular instance.
However in the case with multiply instances, probably the cleanest way is to use some sort of messaging like queues. Then all of the application could subscribe to a topic and your cron can just push notifications to it.
Related
I have a requirement to create around 10 Spring Batch jobs, which will consists of a reader and a writer. All readers read data from some different Oracle DB and write into a different Oracle Db(Source and destination servers are different). And the Spring jobs are implemented using Spring Boot. Also all 10+ jobs would be packaged into a single Jar File. So far fine.
Now the client also wants some UI to monitor the job status and act as a job organizer. I gone through the Spring Data flow Server documentation for UI requirement. But I'm not sure whether it'll serve the purpose, or is there any other alternative option available for monitoring the job status, stop and start the jobs whenever required from the UI.
Also how could I separate the the 10+ jobs inside a single Jar in the Spring Data Flow Server if it's the only option for an UI.
Thanks in advance.
I don't have reputation to add a comment. So, I am posting answer here. Although I know this is not the way to share reference link as an answer.
This might help you:
spring-batch-job-monitoring-with-angular-front-end-real-time-progress-bar
Observability of spring batch jobs is given by data that are persisted by the framework in a relational database... instances..executions..timestamps...read count..write count....
You have different way to exploit these data. SQL client, JMX, spring batch api (JobExplorer, JobOperator), spring admin (deprecated in favor of cloud data flow server).
Data flow is an orchestrator allowing you to execute data pipelines with streams and tasks(finite and short lived/monitored services). For your jobs we can imagine wrap each jobs in tasks and create a multitask pipeline. Data flow gives you status of each executions.
You can also expose your monitoring data by pushing them as metrics in an influxDb for instance...
I am trying to create a web server that provides web service functions, communicates with android devices using GCM, uses a database and probably also has some background work to do.
I am currently unsure on the architecture of such a server.
I know how to create a simple web service, but have not found tutorials or descriptions that go beyond simple "Hello World" examples. As far as i know i can create a class with the #WebService annotation, and once deployed to e.G. Tomcat the server will create at least one instance of this class and provide the annoteted functions in this class as web services.
Now i wonder how to best implement database connection. From what i know the server would create an instance of this class for every request, wich will be garbage collected once the connection is closed. Since the web server needs the database for nearly every function it provides i think it would not be a good idea to create a database connection for every instance, but rather use a global connection to query the database.
From what i found out so far this could be achieved by injecting a class that handles the database connection as #Resource or #Singleton into the web service class.
But is this the proper way to do this ? Or am i worring too much and just creating a new database connection for each request is fine ?
Then i want to send GCM messages (the simple POST ones using the HTML google server), so i would probably create a controller or manager class to handle these requests. Would this also be injected as a #Resource or #Singleton into the web service class ?
And last but not least the server probably has to do some work periodicaly, wich would be some kind of background thread, that is independant of the requests the web service is recieving. Here i am at a loss on how to do this. A web service does not have a main() method i am aware of, so i am unsure on how to create this.
Can anyone give me a guick overview on how to design such a web server or can point me to documentation that describe on how to achieve this ?
All i found so far were simple examples that don't cover advanced stuff like this.
Ok, so you have 3 questions/problems.
How to create a Webservice
How to manage database connections
Execute batch process
All of this, in the same App. First of all, I should advice you to split into 2 app, one for consume (Web Services) and another for batch processing, including push notifications (IMHO). But lets go one step at a time.
1. Webservice: It's depends on the framework you choose. I usually made a choice between Spring-MVC or Jersey
2. How to manage database connections When you are querying a database, you usually don't want to open and close connections crazily. You want to use a connection pool. In a connection pool you'll ask for an open connection, will use it, and will free it when you are done. Normally, a connection pool is managed por the application server. If you want to manage the connections manually, you have to use a singleton to centralise acquiring and releasing.
3. Execute batch process You probably should use a singleton to manage batch process. This job manager will launch the job executions on other threads
Don't know if that answers your question/concern.. please let me know.
Ryu,
I found myself in a similar scenario. After going through the webs for over 2 days, I stumbled upon this solution of running a background thread which is triggered during the initialization of a servlet(init method).
Perhaps you can give this a shot and let me know if it works for you.
Here's the link which has an example to try - http://docstore.mik.ua/orelly/java-ent/servlet/ch03_05.htm
Cheers!
Here's what i need.. I have a UI where a user has the capability to upload a file and extract a report based on the inputted(uploaded) data. Since there is a huge data to be extracted, once the user uploads the data i would like to come out of the servlet control so that user doesn't have to wait in the same page and that the control to be passed on to a java stand alone program there by making it possible for the user to work on something else. So once the control goes on to the java standalone,it would invoke back-end sps and build an extract out of it and place it in a file path on the server.
The user how-over has a capability from UI to check if the extract is ready for them to download.
So the question here is, what is the best practice or possibility in achieving the same? Please let me know your valuable comments.
Thanks!
If you're running in a Java EE environment I would suggest having the servlet dispatch the task to a JMS queue and use a message driven bean to do the (async) processing.
As others suggest, it would be fairly trivial to have the upload servlet redirect the user to some ajax-enabled page that polls the backend for job completion.
If you're not in an EE environment, you could create a standalone (thread pooled) application to consume from the queue and provide signalling eg. through the database (I assume the result goes in a DB anyway). The Spring framework provides very capable and extensive facilities for binding it all together.
But really, there are several free/open source EE containers available, from light weight up to enterprise, so there's no need to build the necessary stuff yourself.
Cheers,
Its very easy.
Have one thread in your servlet class.
Run the thread (Thread will extract the data etc).
After running the thread redirect user to a page where you have auto-refresh or something to show how much extraction is done.(You mentioned that you have a way to find it)
If you can't use message driven beans, you could have your servlet upload the data to a location on the filesystem and record a row in a DB table to say there's a job to be processed.
Then you have your standalone program polling for jobs, processing the data and updating the DB row on completion (including reasons for failure etc.).
Finally, you can poll the status of the job from the UI using an ajax request.
Allows the user to build up a queue of data jobs to be processed while they're doing something else.
I'm building a web service that executes a database process (SQL code to run several queries , then move data between two really large tables), I'm assuming some processes might take 2 to 10 hours to execute.
What are the best practices for executing a long running database process from within a Java web service (it's actually REST-based using JAX-RS and Spring)? The process would be executed upon 1 web service call. It is expected that this execution would be done once a week.
Thanks in advance!
It's gotta be asynchronous.
Since your web service call is an RPC, best to have the implementation validate the request, put it on a queue for processing, and immediately send back a response that has a token or URL to check on progress.
Set up a JMS queue and register a listener that takes the message off the queue and persists it.
If this is really taking 2-10 hours, I'd recommend looking at your schema and queries to see if you can speed it up. There's an index missing somewhere, I'd bet.
Where I work, I am currently evaluating different strategies for this exact situation, only times are different.
With the times you state, you may be better served by using Publish/Subscribe message queuing (ActiveMQ).
In one of our applications we need to call the Yahoo Soap Webservice to Get Weather and other related info.
I used the wsdl2java tool from axis1.4 and generated th required stubs and wrote a client. I use jsp's use bean to include the client bean and call methods defined in the client which call the yahoo webservice inturn.
Now the problem: When users make calls to the jsp the response time of the webservice differs greatly, like for one user it took less then 10 seconds and the other in the same network took more than a minute.
I was just wondering if Axis1.4 queues the requests even though the jsps are multithreaded.
And finally is there an efficient way of calling the webservice(Yahoo weather). Typically i get around 200 simultaneous requests from my users.
Why don't you schedule one thread to get the weather every minute or so, and expose that to the JSP, in stead of letting each JSP get its own weather report?
That's a lot more efficient for both you and Yahoo, and JSP's only need to lookup a local object (almost instantaneous) in stead of connecting to a web service.
EDIT
Some new requirements in the comments of this answer suggest a different way of choosing solutions.
It seems that not only weather, which not only doesn't change that often but is also the same for every user, is requested by web service but also other data like flight data.
The requirements for flight data retrieval are very much different than for weather data. So I think you should define a few types of (remote) data and choose a different solution
for each category.
As basis for the requirements I'd use something simple:
Users like their information promptly, they do not like waiting
The amount of data stored on the web server is finite
Remote web services have an EULA of sorts and are probably not happy with 200 concurrent requests of the same data by the same source (you)
Fast data access to users is best achieved by having the data locally, be it transient (kept in a bean) or persistent (a local database). That can be done by periodically requesting data from the remote source, and using the cached data in the JSP. That would also keep you in the clear with the third point.
A finite amount of data stored on the web service means that not everything can be cached. Data which differs per user, or large data sets which can vary over small periods of time, cannot readily be cached. It's not really a good idea to load data on all flights of all airports in the US every minute or so. That kind of requests would be better served by running a specific web service query when necessary.
The trick is now to identify when caching data is feasible. If it is feasible, do that, otherwise run the web service query in the background. That can be done by presenting the JSP now and starting the web service query in the background. The JSP can have an AJAX script which queries your web server whether the data is ready, and insert that data in the page when ready.
I'd use Google tools to monitor how long the call to the web service is taking.
There are several things going on here:
Map Java beans to XML request.
Send XML request to web service.
Unmarshall XML request on web service side.
Web service processes request
Web service marshalles XML response
Web service sends XML response to Java client
Unmarshall XML response and display on client.
You can't see inside the Yahoo web service, but do break out what you can see on the client side to see where the time is spent.
Check memory as well. If Axis is generating .class files, maybe your perm space is being consumed. Visual VM is available to you with the JDK. Attach it to the PID on your client to see what's going on in memory on your app server.
Maybe this would be a good place for an AJAX call. This will be a good solution if you can get the weather in the background while users are doing other things.
I would recommend local caching and data pooling. Instead of sending out 200 separate requests for similar/same locations run a background thread which pulls the weather for only the locations your users are interested in and caches them locally, this cache updates every minute or so. When users request their personal preferences, the requests hit the cache and refetch if the location is new or the data in the cache is stale. This way the user will have a more seamless experience and you will not hit Yahoo throttles and get denied service.