I have an use case, where I want to do a POST request to another Java Service(Service B) from my service (Service A) whose response I don't care about but I am concerned about not changing the existing latencies for functionality of Service A that executes the call.
So, basically I have set of checks I perform in Service A which run in parallel using a ThreadPool. An ExecutorService instance is used to spawn different threads. Each check has an CompletabaleFuture associated. These CompletableFuture objects are submitted to the ExecutorService, which then provides threads for execution. The ThreadPool has a limited size.
For one of the check that I perform parallely, I also want to additionally perform a delegation which a new class will handle and try to execute a POST request to Service B. If I do delegation within the same thread as the one used to perform a particular check, then because of the limited size of ThreadPool, the overall performance of the parallel checks will be affected, ultimately affecting my service.
Is there a way to restructure or make the calls to new service in an efficient manner ? I'm aware that in Kotlin, there's something called as Co-routines. Not sure if Java has something similar that could be useful.
Related
I understand that RestTempalte is thread-safe once constructed, which is also said in this post on StackOverflow Is RestTemplate thread safe?
But I'd like to be able to test that this is indeed thread-safe, I have no idea how to go about it seeing as the restTempate is going to be used to make the call to a service and I do not have control over the service it is going to call, so I cannot stall a request and then have another one sent in at the same time to test it. What are the other options I can look at?
I just would like to have 2 calls using the singleton restTemplate instance and be successful. The requirement to do this is because I want this to be demonstratable. Are there any inbuilt RestTemplate utility methods I can make use of to demonstrate this?
First of all it is not necessary to call the actual service. You can write your own service and call it simultaneously and stall it or whatever your plan is.
Second - it doesn't make sense to test code outside of your code base. RestTemplate is thread-safe - no need to prove that.
That being said - you can create a bunch of threads, inside of which you invoke the same calls to your single restTemplate instance. You can sleep the threads for some seconds to be "sure" that the restTemplates call will be fired simultaneously.
Go one abstraction level higher and use ExecutorService (e.g. Executors.newFixedThreadPool(N);) and submit callables to it (that call the same restTemplate instance).
Some external to JDK options are JMeter and concurrent-junit.
I am implementing something like a database where data manipulation statements (inserts, updates and deletes) get evaluated. Some statements can execute concurrently and others cannot (I compute that). I like the ease of use and convenience of RMI, however I need to have a deeper understanding of the RMI service implementation w.r.t multithreading. For example,
Can the multithreading be controlled in any way?
Is a thread created for each remote call (on server side) or are thread pools used?
More generally, using RMI, how can I ensure that some rmi calls wait for other calls to terminate?
Is there another non-RMI approach, with the same convenience and efficiency that would work better for this?
If I want multi-threading should I just create threads myself on the server side code? The concern is that if the RMI Service creates multiple threads than I would be adding additional unnecessary threads.
If, for example, a thread is created on each call, then I can use the java join method to order the statement execution. On the other hand, if thread pools are used then the join method won't work (since the threads don't terminate).
Overview
There seems to be a few questions within this post, so I will attempt to walk you through each portion in some detail.
Question 1 - Can the multi-threading be controlled in any way?
Yes! Your implementation of the multi-threading can be whatever you want it to be. A RMI implementation is only the communication between seperate JVMs with enough abstraction to feel like they exist on 1 JVM; thus has no effect on multi-threading as it is only the communication layer.
Question 2 - Is a thread created for each remote call (on the server side) or are thread-pools used?
See the documentation here. The short answer to if they are on separate threads is no.
A method dispatched by the RMI runtime to a remote object implementation may or may not execute in a separate thread. The RMI runtime makes no guarantees with respect to mapping remote object invocations to threads. Since remote method invocation on the same remote object may execute concurrently, a remote object implementation needs to make sure its implementation is thread-safe.
RMI using thread-pools depends on the implementation, but as a developer utilizing RMI this should be of no concern as it is encapsulated in the RMI connection layer.
Question 3 - Using RMI, how can I ensure that some RMI calls wait for other calls to terminate?
This is a rather vague question, but I think what your asking is how do you properly block when synchronizing in RMI. This comes with your design of the application. Lets take the scenario where you are trying to access the database and you must synchronize DB access. If the client attempts to invoke access through RMI, it will invoke the remote server's method that holds all the synchronization, thus wait for a lock if it must. Therefore, the Client will be waiting for its turn to access the DB via the server. So, with your current scenario, you want your synchronization of the DB to be present on the server-side.
Question 4 - Is there another non-RMI approach, with the same convenience and efficiency that would work better for this?
Absolutely. Below is a brief list of communication implementations that could be utilized for communication.
1) RESTful
2) RMI
3) Sockets
4) gRPC
My recommendation is to utilize RESTful as it is the most straight-forward and has plenty of implementations/documentation on the internet. Efficiency seems to be quite a high concern for you, but your operations are only manipulating a DB in a standard manner. Therefore, I believe a Restful implementation would provide more than enough efficiency.
Think of it like this; you have N number clients, a load-balancer, and M servers. There exists no constant connection between clients and servers thus reducing complexity and computation. As N clients grows, the load balancer creates more instances of servers and allocating the load appropriately. Note, the requests between clients and servers are actually quite small as they will have a payload and a request type. Additionally, servers will receive the requests and compute the operations as normal and in parallel. The optimization can be done on the server side via threadpools or frameworks such as spring.
What you are asking for is a way to coordinate execution of the tasks according to their dependencies. The fact that your tasks make RMI calls is insignificant. We can imagine a pure computational tasks which does not access remote machines and still are dependent on each other, for example, by providing values computed in one task as parameters for other tasks.
The coordination of dependent tasks is the central problem of asynchronous programming. The support of asynchronous programming in JDK is not full but sufficient for your problem. You need to use 2 things: CompletableFuture and an Executor. Note that an RMI call blocks the thread it runs on, so using an Executor with limited number of threads can lead to the deadlock of specific kind, named "thread starvation", when the computation cannot move on because all available threads are blocked. So use an executor with unlimited number of threads, the simplest is the one which creates new thread for each task:
Executor newThreadExecutor = (Runnable r)->new Thread(r).start();
Then, for each RMI call (or any other task), declare the task method. If the task does not depend on other tasks, then the method should have no parameters. If the task depends on the result(s) produced by other task(s), then declare one or two parameters (greater number of parameters is not supported by CompletableFuture directly). Let we have:
String m0a() {return "ok";} // no args
Integer m0b() {return 1;} // no args
Double m2(String arg, Integer arg2) {return arg2/2.0;} // 2 args
Let we want to compute the following result:
String r0a = m0a();
Integer r0b = m0b();
Double r2 = m2(r0a, r0b);
but asynchronously, so that calls to m0a and m0b are executed in parallel, and the call to m2 starts as soon as both m0a and m0b finished.
Then, wrap each task method with an instance of CompletableFuture Depending on the signature of the task method, different methods of CompletableFuture are used:
CompletableFuture<String> t0a = CompletableFuture.supplyAsync(this::m0a, newThreadExecutor)
CompletableFuture<Integer> t0b = CompletableFuture.supplyAsync(this::m0b, newThreadExecutor)
CompletableFuture<Double> t2 = t0a.thenCombineAsync(t0b, this::m2, newThreadExecutor)
The tasks start execution right after declaration, no call to special start method is required.
To get the final result from the last task t2, method get() of interface Future can be used:
Double res = t2.get();
I have an application that is quite complex (it's a command and control center spring + angular based application that is meant to be used by police and other emergency center controllers).
Main component of application (lets call it backbone [spring web app]) is communication with different applications/hardware. Most of that communication is done by using RabbitMQ messages (lets call them telegrams or in short TMs).
When one of that TMs is received by backbone new thread is created and in it some method/methods is/are executed.
The problem is that it can happen that backbone receives two or more TMs almost at the same time, and since they are executed in different threads it can happen that they are not finessed in same order as they arrived and hence wrong information is presented to the user.
Usually, problems like this I handle with Redis. I have a redis lock that basically looks like this
distributedRedisLocker.lock(() -> {
executeSomeMethod();
}, howLongIsLockKept, howLongDoWeWaitForItToFinnish);
But in this case I would like to avoid using redis, is there any other java/spring based solution to this?
I do not need it to be same as redis lock that I have, only what I want is that TMs are proccessed in order they arrive, and if one of them fails somewhere in method execution that it does not block the next one forever.
Reposting as an answer.
One way to solve your problem is to avoid concurrency. You could use Executors.newSingleThreadExecutor() which only uses one thread:
ExecutorService executor = Executors.newSingleThreadExecutor();
and then
executor.execute(...);
or
executor.submit(...);
This would help you avoid races: if some TM A is added to execution queue defined by this executor before some TM B, then A will be executed before B as a whole.
Also, no explicit locks are involved (apart from implicit ones that can be contained inside the executor implementation, but they are encapsulated and will not stay handing forever in case of an error).
There is a subtle moment: if two TMs arrive at the same time, it's impossible to predict which will be added earlier and which later.
We are designing an Orchestrator System in java that will host a web service and on a request to it will invoke a flow written in XML which are nothing but steps that are executed one after another but the XML tell the user what the flow is and he can also change the flow by changing the XML. He can add a new service and add it to the XML. But while designing I am now confused with things like.
Should I make a service a runnable with a blocking queue and keep it alive all the time by scheduling it to the executor so when the new request arrives I will push the request in the blocking queue and the service will process it. And create a Mailbox which will carry the message passing task between different services.
Instead of Making service runnable I should use spring IOC that will make the class singleton thus only one instance will be there and I will send a request directly to the service methods thus there will be no hassle that I have to do as there are no threads and also didn't need to implement any mailbox.
I read about how event driven system is faster like nodejs and ngnix so I was thinking to use disuptor framework to push all the incoming request to the ringbuffer and then write a handler that will emit the event to the orchestrator that will start processing the request and also send a callback with the request so that when the orchestrator is done it will send back the response back to the caller using callback. But as the request is not of the same type it so I would not be able to take advantage of disruptor object allocations.
The system needs to provide maximum throughput with low latency, the system will add new services/flows to XML in future so it should adopt the new flows without hitting the performance. I am allowed to only use java 7, no Scala so I am bounded.
Answer #1 is a terrible idea. You will effectively tie up a thread per service. If the number of services exceeds the number of threads backing the executor service you have an instant, automatic DOS. Plus, if the services are inter-dependent on each other... all the ways in which you can dead lock. Plus, the inefficiency of using N threads if only M (< N) are actually required.
Answer #2: if the proposed flow is Request Parsing -> Dispatch -> Service Processing -> Callbacks you rely on the actual 'services' not to foul up because that will prevent callbacks from being called and/or DOS the application. Essentially: what happens if an exception occurs in a service? Will that also impact future requests to the same service and/or other services?
Also the opportunities for parallelism are limited to the framework's way of handling incoming requests. Meaning if you have X requests and the framework inherently processes them serially, you get a backlog of X requests. Your latency requirements may be hard to meet in such a scenario.
Answer #3: an event driven system is indeed the better approach: have a scheduler farm out jobs to an executor service to allow the system to distribute the total load of all services dynamically and have a mechanism to generate and react on events to handle the control flow. This scales better if the number of services become very large and each 'job' is reasonably substantial (so the overhead of scheduling/dispatch is low compared to the actual work being performed).
I need to use memcached Java API in my Scala/Akka code. This API gives you both synchronous and asynchronous methods. The asynchronous ones return java.util.concurrent.Future. There was a question here about dealing with Java Futures in Scala here How do I wrap a java.util.concurrent.Future in an Akka Future?. However in my case I have two options:
Using synchronous API and wrapping blocking code in future and mark blocking:
Future {
blocking {
cache.get(key) //synchronous blocking call
}
}
Using asynchronous Java API and do polling every n ms on Java Future to check if the future completed (like described in one of the answers above in the linked question above).
Which one is better? I am leaning towards the first option because polling can dramatically impact response times. Shouldn't blocking { } block prevent from blocking the whole pool?
I always go with the first option. But i am doing it in a slightly different way. I don't use the blocking feature. (Actually i have not thought about it yet.) Instead i am providing a custom execution context to the Future that wraps the synchronous blocking call. So it looks basically like this:
val ecForBlockingMemcachedStuff = ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(100)) // whatever number you think is appropriate
// i create a separate ec for each blocking client/resource/api i use
Future {
cache.get(key) //synchronous blocking call
}(ecForBlockingMemcachedStuff) // or mark the execution context implicit. I like to mention it explicitly.
So all the blocking calls will use a dedicated execution context (= Threadpool). So it is separated from your main execution context responsible for non blocking stuff.
This approach is also explained in a online training video for Play/Akka provided by Typesafe. There is a video in lesson 4 about how to handle blocking calls. It is explained by Nilanjan Raychaudhuri (hope i spelled it correctly), who is a well known author for Scala books.
Update: I had a discussion with Nilanjan on twitter. He explained what the difference between the approach with blocking and a custom ExecutionContext is. The blocking feature just creates a special ExecutionContext. It provides a naive approach to the question how many threads you will need. It spawns a new thread every time, when all the other existing threads in the pool are busy. So it is actually an uncontrolled ExecutionContext. It could create lots of threads and lead to problems like an out of memory error. So the solution with the custom execution context is actually better, because it makes this problem obvious. Nilanjan also added that you need to consider circuit breaking for the case this pool gets overloaded with requests.
TLDR: Yeah, blocking calls suck. Use a custom/dedicated ExecutionContext for blocking calls. Also consider circuit breaking.
The Akka documentation provides a few suggestions on how to deal with blocking calls:
In some cases it is unavoidable to do blocking operations, i.e. to put
a thread to sleep for an indeterminate time, waiting for an external
event to occur. Examples are legacy RDBMS drivers or messaging APIs,
and the underlying reason is typically that (network) I/O occurs under
the covers. When facing this, you may be tempted to just wrap the
blocking call inside a Future and work with that instead, but this
strategy is too simple: you are quite likely to find bottlenecks or
run out of memory or threads when the application runs under increased
load.
The non-exhaustive list of adequate solutions to the “blocking
problem” includes the following suggestions:
Do the blocking call within an actor (or a set of actors managed by a router), making sure to configure a thread pool which is either
dedicated for this purpose or sufficiently sized.
Do the blocking call within a Future, ensuring an upper bound on the number of such calls at any point in time (submitting an unbounded
number of tasks of this nature will exhaust your memory or thread
limits).
Do the blocking call within a Future, providing a thread pool with an upper limit on the number of threads which is appropriate for the
hardware on which the application runs.
Dedicate a single thread to manage a set of blocking resources (e.g. a NIO selector driving multiple channels) and dispatch events as they
occur as actor messages.
The first possibility is especially well-suited for resources which
are single-threaded in nature, like database handles which
traditionally can only execute one outstanding query at a time and use
internal synchronization to ensure this. A common pattern is to create
a router for N actors, each of which wraps a single DB connection and
handles queries as sent to the router. The number N must then be tuned
for maximum throughput, which will vary depending on which DBMS is
deployed on what hardware.