in following code I have asynchronous method which is executed by rest by authenticated user. In this method I execute loop in which is checking periodically cache of new data.
#Async
public CompletableFuture<List<Data>> pollData(Long previousMessageId, Long userId) throws InterruptedException {
// check db at first, if there are new data no need go to loop and waiting
List<Data> data = dataRepository.findByLastAndByUser(dataId, userId));
data not found so jump to loop for some time
if (data.size() == 0) {
short c = 0;
while (c < 100) {
// check if some new data added or not, if yes break loop
if (cache.getIfPresent(userId) != null) {
break;
}
c++;
Thread.sleep(1000);
System.out.println("SEQUENCE: " + c + " in " + Thread.currentThread().getName());
}
// check database on the end of loop or after break from loop
data = dataRepository.findByLastAndByUser(dataId, userId);
}
// clear data for that recipient and return result
cache.clear(userId);
return CompletableFuture.completedFuture(data);
}
and executor bean:
#Bean
public Executor asyncExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(2);
executor.setMaxPoolSize(2);
executor.setQueueCapacity(500);
executor.initialize();
return executor;
}
I execute this checking in separate thread for every request because these data are different for each user.
I need to optimize this code for many users (about 10k active users). In current state it doesn't work well because where there is more requests, these requests are waiting for some new free thread, and every another request takes very long time (5 min instead of 100 sec for example).
Can you help me improve it? Thanks in advice.
In case there are no other concurrent calls to the pollData method, it takes at most ~100s.
The parameter maxPoolSize defines the maximum number of thread that can run concurrently your #Asynch method.
So (number of users * execution time) / number of threads = 10K*100/2 = 500K[s].
I haven't completely understood the goal you want to reach with this method, but I suggest you to review the design of this functionality.
(For example take a look at spring cache, #evict,...)
(Notice that, in case you have multiple #async, you can bind the pool configuration with the #Asynch method by adding the name to the annotations #Bean("Pool1") and #Asynch("Pool1")).
I don't fully understand what you want to do.
But I think obviously it fill quickly your pool of thread.
I think you should try to use message broker or something like this.
Instead of trying to respond to request by waiting something new append, you should connect your clients with AMQP, WebSocket, Webhook... etc... On your server, when you detect new informations, you notify your clients.
So you don't need to occupy one thread by client.
Related
I have a Java program. The logic is as follow:
place order out (relying on Interactive Broker / Binance API)
Once the order is filled (there will be a callback from the API), immediately execute a method called "calculateSomething"
The order is placed using Interactive Broker / Binance API. Once the order is filled, the API callback method will return a message.
The problem is that I do not know how to write out the code to identify that the order has been filled, so i can immediately execute the "calculateSomething" method with minimal waiting time.
I can think of two ways:
while loop and thread.sleep
ReentrantLock.
Method 1 works, but it's not instantaneous. Hence, I am exploring ReentrantLock and I am not sure the code is correct. Nonetheless, which method is the most efficient and can immediately execute the "calculateSomething" once the order is completed If there is a more efficient approach, please give me some help, as I have been stuck in this problem for many days.
pseudocode below.
Method 1 - thread.sleep
placeOrder(); // place order to binance <- API method
while(order is not completed){
Thread.sleep(1000)
if(order is completed){
return
}
}
calculateSomething();
Method 2 - ReentrantLock
ReentrantLock lock = new ReentrantLock();
lock.lock();
System.out.println("1. Locked");
try {
while(lock.isLocked()) {
if(isOrderCompleted() == true){
lock.unlock();
}
}
} catch(Exception e){
e.printStackTrace();
}finally {
if(lock.isLocked()) {
lock.unlock();
}
}
calculateSomething();
You can have a blocking queue.
BlockingQueue<?> finishedOrders = new ArrayBlockingQueue<>(512);
Then you have a loop that processes finished orders.
public void processFinishedOrders() throws InterruptedException{
while(!Thread.interrupted()){
finishedOrders.take();
doSomethingRelevant();
}
}
I would also suggest populating finishedOrders with a meaningful class.
BlockingQueue<Order> finishedOrders;
Order fin = finishedOrders.take();
doSomethingRelevant( fin );
That way the thread waiting on the api call can create a an order and add it to the finished orders queue, and the processing thread will have the relevant information.
I am building a backend service whereby a REST call to my service creates a new thread. The thread waits for another REST call if it does not receive anything by say 5 minutes the thread will die.
To keep track of all the threads I have a collection that keeps track of all the currently running threads so that when the REST call finally comes in such as a user accepting or declining an action, I can then identify that thread using the userID. If its declined we will just remove that thread from the collection if its accepted the thread can carry on doing the next action. i have implemented this using a ConcurrentMap to avoid concurrency issues.
Since this is my first time working with threads I want to make sure that I am not overlooking any issues that may arise. Please have a look at my code and tell me if I could do it better or if there's any flaws.
public class UserAction extends Thread {
int userID;
boolean isAccepted = false;
boolean isDeclined = false;
long timeNow = System.currentTimeMillis();
long timeElapsed = timeNow + 50000;
public UserAction(int userID) {
this.userID = userID;
}
public void declineJob() {
this.isDeclined = true;
}
public void acceptJob() {
this.isAccepted = true;
}
public boolean waitForApproval(){
while (System.currentTimeMillis() < timeElapsed){
System.out.println("waiting for approval");
if (isAccepted) {
return true;
} else if (declined) {
return false;
}
}
return isAccepted;
}
#Override
public void run() {
if (!waitForApproval) {
// mustve timed out or user declined so remove from list and return thread immediately
tCollection.remove(userID);
// end the thread here
return;
}
// mustve been accepted so continue working
}
}
public class Controller {
public static ConcurrentHashMap<Integer, Thread> tCollection = new ConcurrentHashMap<>();
public static void main(String[] args) {
int barberID1 = 1;
int barberID2 = 2;
tCollection.put(barberID1, new UserAction(barberID1));
tCollection.put(barberID2, new UserAction(barberID2));
tCollection.get(barberID1).start();
tCollection.get(barberID2).start();
Thread.sleep(1000);
// simulate REST call accepting/declining job after 1 second. Usually this would be in a spring mvc RESTcontroller in a different class.
tCollection.get(barberID1).acceptJob();
tCollection.get(barberID2).declineJob();
}
}
You don't need (explicit) threads for this. Just a shared pool of task objects that are created on the first rest call.
When the second rest call comes, you already have a thread to use (the one that's handling the rest call). You just need to retrieve the task object according to the user id. You also need to get rid of expired tasks, which can be done with for example a DelayQueue.
Pseudocode:
public void rest1(User u) {
UserTask ut = new UserTask(u);
pool.put(u.getId(), ut);
delayPool.put(ut); // Assuming UserTask implements Delayed with a 5 minute delay
}
public void rest2(User u, Action a) {
UserTask ut = pool.get(u.getId());
if(!a.isAccepted() || ut == null)
pool.remove(u.getId());
else
process(ut);
// Clean up the pool from any expired tasks, can also be done in the beginning
// of the method, if you want to make sure that expired actions aren't performed
while((UserTask u = delayPool.poll()) != null)
pool.remove(u.getId());
}
There's a synchronization issue that you should make your flags isAccepted and isDeclined of class AtomicBoolean.
A critical concept is that you need to take steps to make sure changes to memory in one thread are communicated to other threads that need that data. They're called memory fences and they often occur implicitly between synchronization calls.
The idea of a (simple) Von Neumann architecture with a 'central memory' is false for most modern machines and you need to know data is being shared between caches/threads correctly.
Also as others suggest, creating a thread for each task is a poor model. It scales badly and leaves your application vulnerable to keeling over if too many tasks are submitted. There is some limit to memory so you can only have so many pending tasks at a time but the ceiling for threads will be much lower.
That will be made all the worse because you're spin waiting. Spin waiting puts a thread into a loop waiting for a condition. A better model would wait on a ConditionVariable so threads not doing anything (other than waiting) could be suspended by the operating system until notified that the thing they're waiting for is (or may be) ready.
There are often significant overheads in time and resources to creating and destroying threads. Given that most platforms can be simultaneously only executing a relatively small number of threads creating lots of 'expensive' threads to have them spend most of their time swapped out (suspended) doing nothing is very inefficient.
The right model launches a pool of a fixed number of threads (or relatively fixed number) and places tasks in a shared queue that the threads 'take' work from and process.
That model is known generically as a "Thread Pool".
The entry level implementation you should look at is ThreadPoolExecutor:
https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ThreadPoolExecutor.html
I'm trying to optimize my Futures management techniques.
Suppose we have this typical processing scenario: I run a query to fetch some records from a database:
SELECT * FROM mytable WHERE mycondition;
This query returns a lot of rows that I need to process with something like:
while (recordset have more results) {
MyRow row = recordset.getNextRow(); // Get the next row
processRow(row); // Process the row
}
Now suppose that all the rows are independent of each other, and the function processRow is slow because it performs some hard processing and queries on a C* cluster:
void processRow(MyRow row) {
// Fetch some useful data from the DB
int metadataid = row.getMetadataID();
Metadata metadata = getMetadataFromCassandra(metadataid);
// .... perform more processing on the row .....
// Store the processing result in the DB
ProcessingResult result = ....;
insertProcessingResultIntoCassandra(result);
}
A serial approach like this is expected to perform poorly, so a parallel execution is arguable.
With this basic processing structure in mind, here's some transformations I performed on the algorithm to get a major speed upgrade.
STEP 1: parallelize row processing
This is pretty straightforward. I created an Executor that gets the jobs done in parallel. Then I wait for all the jobs to finish. The code looks like:
ThreadPoolExecutor executor = (ThreadPoolExecutor)Executors.newCachedThreadPool();
int failedJobs = 0;
ArrayList<Future<Boolean>> futures = new ArrayList<>();
while (recordset have more results) {
final MyRow row = recordset.getNextRow(); // Get the next row
// Create the async job and send it to the executor
Callable<Boolean> c = new Callable<Boolean>() {
#Override
public Boolean call() {
try {
processRow(row);
} catch (Exception e) {
return false; // Job failed
}
return true; // Job is OK
}
};
futures.add(executor.submit(c));
}
// All jobs submitted. Wait for the completion.
while (futures.size() > 0) {
Future<Boolean> future = futures.remove(0);
Boolean result = false;
try {
result = future.get();
} catch (Exception e) {
e.printStackTrace();
}
failedJobs += (result ? 0 : 1);
}
STEP 2: limit the number of concurrent rows
So far so good, unless I have a low number of jobs this is expected to fail with an out of memory error, because the executor is backed by an unbound queue, and the main loop would submit the jobs all the way. I can solve this problem by controlling the maximum number of concurrent submitted jobs:
final const int MAX_JOBS = 1000;
while (recordset have more results) {
....
futures.add(executor.submit(c));
while (futures.size() >= MAX_JOBS) {
Future<Boolean> future = futures.remove(0);
Boolean result = false;
try {
result = future.get();
} catch (Exception e) {
e.printStackTrace();
}
failedJobs += (result ? 0 : 1);
}
}
Simply told, I wait for the first job of the list to be completed if we reached a certain threshold (1000 in this case). This works effectively, and this is a good speedup.
STEP 3: parallelize the single row processing
This is the step where I'd like to get a bit of help. I expect 1000 jobs will accumulate fast in the queue, due to the slowness of the IO. That is, I expect the JVM to fire 1000 threads to accomodate all the jobs. Now, 1000 threads when you only have an 8-core machine usually slow down everything, and I'm thinking that with a more tweaked parallelism this number could be lowered.
Currently, the getMetadataFromCassandra function is a wrapper around session.executeAsync, but manages retries:
public static ResultSet getMetadataFromCassandra(...) {
int retries = 0;
// Loop here
while (retries < MAX_RETRIES) {
// Execute the query
ResultSetFuture future = session.executeAsync(statement);
try {
// Try to get the result
return future.get(1000 * (int)Math.pow(2, retries), TimeUnit.MILLISECONDS);
} catch (Exception e) {
// Ooops. An error occurred. Cancel the future and schedule it again
future.cancel(true);
if (retries == MAX_RETRIES) {
e.printStackTrace();
String stackTrace = Throwables.getStackTraceAsString(e);
logToFile("Failed to execute query. Stack trace: " + stackTrace);
}
retries++;
}
}
return null;
}
As you can see, this is a blocking function because I .get() on the ResultSetFuture. That is, this call will block each thread waiting for the IO. So I'm getting an async approach, but I feel like I'm wasting a lot of hardware resources.
QUESTION
In my mind, I should be able to be notified when the .executeAsync results are available (or the timeout occurs), "freeing" the thread and allowing the same thread to perform other things.
Simply told, it seems to me that I'd need to transform the sequential structure of the processRow into a pipeline: the query is executed in an async way and, when the results are available, the remaining part of the processing is performed. And of course, I want the main loop to wait for the whole pipelined process to finish, not only the first part.
In other words,the main loop submits a job (let's call it jobJob) and I get a Future (let's call jobFuture) that I can .get() to wait for its completion. However, jobJob fires the "query" sub-job (let's call it queryJob), and queryJob is submitted async, so I get another Future (let's call it queryFuture) that should be used to fire the "process" sub-job (let's call processJob). At this point, I'm simply nesting Futures and blocking deep in the chain before completing the Future representing jobJob, and that means I'm back to the origin!!!
Before I go the hard route and implement this sort of pipeline as a Finite State Machine, I had a look at:
ForkJoinPool executor class
ListenableFuture from the Guava library
CompletableFuture class
None of them seem to satisfy my requirements of pipelining this process, or probably I didn't find a clear explanation on how to perform such apparent simple task. Can anyone simply enlighten me on this topic?
Any help is really appreciated.
I think I'm doing it wrong. I am creating threads that are suppose to crunch some data from a shared queue. My problem is the program is slow and a memory hog, I suspect that the queue may not be as shared as I hoped it would be. I suspect this because in my code I added a line that displayed the size of the queue and if I launch 2 threads then I get two outputs with completely different numbers and seem to increment on their own(I thought it could be the same number but maybe it was jumping from 100 to 2 and so on but after watching it shows 105 and 5 and goes at a different rate. If I have 4 threads then I see 4 different numbers).
Here's snippet of the relevant parts. I create a static class with the data I want in the queue at the top of the program
static class queue_class {
int number;
int[] data;
Context(int number, int[] data) {
this.number = number;
this.data = data;
}
}
Then I create the queue after sending some jobs to the callable..
static class process_threaded implements Callable<Void> {
// queue with contexts to process
private Queue<queue_class> queue;
process_threaded(queue_class request) {
queue = new ArrayDeque<queue_class>();
queue.add(request);
}
public Void call() {
while(!queue.isEmpty()) {
System.out.println("in contexts queue with a size of " + queue.size());
Context current = contexts.poll();
//get work and process it, if it work great then the solution goes elsewhere
//otherwise, depending on the data, its either discarded or parts of it is added back to queue
queue.add(new queue_class(k, data_list));
As you can see, there's 3 options for the data, get sent off if data is good, discard if its totally horrible or sent back to the queue. I think the queues are going when its getting sent back but I suspect because each thread is working on its own queue and not a shared one.
Is this guess correct and am I doing this wrong?
You are correct in your assessment that each thread is (probably) working with its own queue, since you are creating a queue in the constructor of your Callable. (It's actually very weird to have a Callable<Void> -- isn't that just a Runnable?)
There are other problems there, for example, the fact that you're working with a queue that isn't thread-safe, or the fact that your code won't compile as it is written.
The important question, though, is do you really need to explicitly create a queue in the first place? Why not have an ExecutorService to which you submit your Callables (or Runnables if you decide to make that switch): Pass a reference to the executor into your Callables, and they can add new Callables to the executor's queue of tasks to run. No need to reinvent the wheel.
For example:
static class process_threaded implements Runnable {
// Reference to an executor
private final ExecutorService exec;
// Reference to the job counter
private final AtomicInteger jobCounter;
// Request to process
private queue_class request;
process_threaded( ExecutorService exec, AtomicInteger counter, queue_class request) {
this.exec = exec;
this.jobCounter = counter;
this.jobCounter.incrementAndGet(); // Assuming that you will always
// submit the process_threaded to
// the executor if you create it.
this.request = request;
}
public run() {
//get work and process **request**, if it work great then the solution goes elsewhere
//otherwise, depending on the data, its either discarded or parts of are added back to the executor
exec.submit( new process_threaded( exec, new queue_class(k, data_list) ) );
// Can do some more work
// Always run before returning: counter update and notify the launcher
synchronized(jobCounter){
jobCounter.decrementAndGet();
jobCounter.notifyAll();
}
}
}
Edit:
To solve your problem of when to shut down the executor, I think the simplest solution is to have a job counter, and shutdown when it reaches 0. For thread-safety an AtomicInteger is probably the best choice. I added some code above to incorporate the change. Then your launching code would look something like this:
void theLauncher() {
AtomicInteger jobCounter = new AtomicInteger( 0 );
ExecutorService exec = Executors.newFixedThreadPool( Runtime.getRuntime().availableProcesses());
exec.submit( new process_threaded( exec, jobCounter, someProcessRequest ) );
// Can submit some other things here of course...
// Wait for jobs to complete:
for(;;jobCounter.get() > 0){
synchronized( jobCounter ){ // (I'm not sure if you have to have the synchronized block, but I think this is safer.
if( jobCounter.get() > 0 )
jobCounter.wait();
}
}
// Now you can shutdown:
exec.shutdown();
}
Don't reinvent the wheel! How about using ConcurrentLinkedQueue? From the javadocs:
An unbounded thread-safe queue based on linked nodes. This queue orders elements FIFO (first-in-first-out). The head of the queue is that element that has been on the queue the longest time. The tail of the queue is that element that has been on the queue the shortest time. New elements are inserted at the tail of the queue, and the queue retrieval operations obtain elements at the head of the queue. A ConcurrentLinkedQueue is an appropriate choice when many threads will share access to a common collection.
I want to force method run for specified time.
public Response run(Request req){
//method runImpl must run during specified interval of time(for example for 10 secs)
Response res = runImpl(req);
return response;
}
What is the best way to do this?Thanks!
Try to use this:
poolExecutor = new ScheduledThreadPoolExecutor(1);
poolExecutor.scheduleAtFixedRate(
new YourRunable(), startFrom/*10*/, startEvery/*5*/, TimeUnit.SECONDS);
runImpl should return a boolean for this code to work. You'll probably want this in a new Thread (basic tutorials abound) if you don't want your program stalled until it finishes.
public Response run(Request req){
long lasttime=Sys.getTime();
int i=0;
while(i<10){
if(Response res = runImpl(req);){
return response;
}
if((Sys.getTime-lasttime)>1000){
i++;
lasttime=Sys.getTime();
}
}
return null;
This runs every processor moment, if you want it to run on an interval AS WELL as running for 10 seconds, use:
public Response run(Request req){
long lasttime=Sys.getTime();
int i=0;
for(int i; i<(10000/yourchoiceinterval); i++){
if(Response res = runImpl(req);){
return response;
}
if((Sys.getTime-lasttime)>1000){
lasttime=Sys.getTime();
}
}
return null;
Best method would be to make your work in chunks of appropriate size, so each chunk can be done in, say, 1 second. This way at the end of each chunk you can check if time is elapsed.
If your work involves I/O work which can block for indefinite time, try to use interruptible methods (which can be interrupted by call to Thread.interrupt()) or methods which accept explicit max running time from caller.
Or you can delegate I/O work to some dedicated thread and check it's progress in separate thread, and act appropriately when running time is exceeded.
You could start a new thread that only calls your method and put it to sleep after a timer you create expires. That way though you can't guarantee that the method cycle has completed before suspending, but does the trick without having to modify the method. For you to achieve cycles, just put this logic of sleeping inside the thread's loop. Just remember to wake up the thread
There may be dozens of implementations since you didn't explained clear enough.