I have a frontend web tool which interacts with REST API written in danjgo. Each API calls take long time to process the call and is also CPU/GPU intensive. So I am planning to run only 1 call at a time and putting rest of the calls in queue. Now I am not sure if Celery with redis can be helpful here or should I stick with job queue approach at the java side.
So, the tool would be used by multiple users and and so each user would have their jobs. So, I need to put the jobs in queue so that they can be processed one by one asynchronously. Would Celery be helpful here?
Related
I have one Rest service running in PCF.It might take 10-15mins to complete.PCF app is configured to run on multiple instances.
I want to make sure at a single point of time,that Rest service to run only in one instance. e.g: if the Rest service running in one instance and I try to call the same service multiple times then it should not be taken by any other instance, rather it should wait till it's completion and then it can run in any other or same instance.
Kindly suggest some best practices to this.Thanks in advance.
This is a prime use case for a message queue. Have any app accept the request and add it to the queue, then a free listener in any app can consume the message. If you are scaling a lot and it's a intensive task it might even be recommendable to split the produce and consumer into separate apps, so a intensive consumer doesn't make you api unresponsive
We are trying to migrate our legacy system to Micro service
With Paas environment, we have scheduler jobs to trigger and put messages in MQ one by one and we have MQ listener in our Microservice to get message and create request and send request to external party.
Here the problem comes our micro service is capable doing Asynchronous call to external service, but our external service is not able to handle Asynchronous call so it is returning wrong data.
For example, we are hitting external service with 40 to 60 request per minute and external service is capable to handle only 6 request per minute.
So how can I make the MQ listener to process slowly.
I have tried reducing setMaxConcurrenceConsumer to 1 and
Used observable.toblocking.single() to make the process to run in only one thread.
We use RxJava in our micro service.
It sounds like either your micro service or the external service is not following the use case for Request-Reply messaging.
(1) Is the external service setting the Reply's message Correlation ID with the Request message's Message ID?
(2) Is your micro service performing an MQGET with the matching option of getting by Correlation ID.
You can blame the external service for the error but if your micro service is actually picking up the wrong message then it is your application's fault. i.e. Does your micro service simply get the "next" message on the queue?
Read this answer: How to match MQ Server reply messages to the correct request
Here's a explanation (looks like from the 90's but has good information): https://www.enterpriseintegrationpatterns.com/patterns/messaging/RequestReplyJmsExample.html
In long term approach we are planning to migrate the External service to as well.
In short time i have fixed it using the observable.toblocking.single() ,thread.sleep(), and setMaxConcurrenceConsumer() to 1 so only one thread will run at a time. which will avoid the Asynchronous call to external service.The sleep time will set dynamically with some analysis done on the external service.
I'm a bit new to web services, but I'm using Axis2 with Java.
My question is simple, is it possible to schedule a web service function to return String data every, say 10 minutes?
Having a webservice deliver something to you (push) is not possible, since it's an interface used for retrieving (pull) data.
Client side, you can build an application, either as a standalone application, using Thread.sleep, waiting for a given period of time, or you utilize JCron, Quartz or another scheduling API for querying at a given interval.
So, in summary, you cannot have a web service deliver "anything" at a given interval.
Currently I have a Java (and a half ported python version) app that runs in the background that has a queue of jobs (currently read out of a mysql database) which handles thread sleep/waking to share resources based on the job priority and running time. There is a front end php script that posts jobs to the database which are polled by the system every time interval.
This manner is somewhat inefficient (but nicer than locking issues using a job file) but I can't but wonder if there would be some way to simplify this.
My thoughts were java app (and or python app) sets up http service (jetty?) and has a web interface that directly pushes jobs to the queue without the middleman. Apache is serving other php sites so this would have to run in tandem.
I'm really after some other input as I'd prefer it to be a background service always running - having a cron execute jobs was painful (since some jobs run for 20+ hours so adding new ones was a pain with new php [ no threading] /java calls having to check if a service was running with outstanding jobs to add to instead of starting a new service) but also have a very simple web interface without too much resource wastage.
Thanks for your input.
Deploy a JSP using Tomcat (or similar) that allows the user to post job requests to a job scheduler web service using a webpage. On the backend, use Quartz Scheduler to manage your jobs and just have your web service add jobs to the Quartz queue.
I'm building a web service that executes a database process (SQL code to run several queries , then move data between two really large tables), I'm assuming some processes might take 2 to 10 hours to execute.
What are the best practices for executing a long running database process from within a Java web service (it's actually REST-based using JAX-RS and Spring)? The process would be executed upon 1 web service call. It is expected that this execution would be done once a week.
Thanks in advance!
It's gotta be asynchronous.
Since your web service call is an RPC, best to have the implementation validate the request, put it on a queue for processing, and immediately send back a response that has a token or URL to check on progress.
Set up a JMS queue and register a listener that takes the message off the queue and persists it.
If this is really taking 2-10 hours, I'd recommend looking at your schema and queries to see if you can speed it up. There's an index missing somewhere, I'd bet.
Where I work, I am currently evaluating different strategies for this exact situation, only times are different.
With the times you state, you may be better served by using Publish/Subscribe message queuing (ActiveMQ).