Scenario
We are developing an API that will handle around 2-3 million hits per hour in a multi-threaded environment. The server is Apache Tomcat 7.0.64.
We have a custom object with lot of data let's call it XYZDataContext. When a new request comes in we associate XYZDataContext object to the request context. One XYZDataContext object per request. We will be spawning various threads in parallel to serve that request to collect/process data from/into XYZDataContext object. Our threads that will process things in parallel need access to this XYZDataContext object and
to avoid passing around of this object everywhere in the application, to various objects/methods/threads,
we are thinking to make it a threadlocal. Threads will use data from XYZDataContext object and will also update data in this object.
When the thread finishes we are planning to merge the data from the updated XYZDataContext object in the spawned child thread into the main thread's XYZDataContext object.
My questions:
Is this a good approach?
Threadpool risks - Tomcat server will maintain a threadpool and I read that using threadlocal with thread pools is a disaster because thread is not GCed per say and is reused so the references to the threadlocal objects will not get GCed and will result in storing huge objects in memory that we don't need anymore eventually resulting into OutOfMemory issues...
UNLESS they are referenced as weak references so that get GCed immediately.
We're using Java 1.7 open JDK. I saw the source code for ThreadLocal and the although the ThreadLocalMap.Entry is a weakreference it's not associated with a ReferenceQueue, and the comment for Entry constructor says "since reference queues are not used, stale entries are guaranteed to be removed only when the table starts running out of space."
I guess this works great in case of caches but is not the best thing in our case. I would like that the threadlocal XYZDataContext object be GCed immediately. Will the ThreadLocal.remove() method be effective here?
Is there any way to enforce emptying the space in the next GC run?
This is a right scenario to use ThreadLocal objects? Or are we abusing the threadlocal concept and using it where it shouldn't be used?
My gut feeling tells me you're on the wrong path. Since you already have a central context object (one for all threads) and you want to access it from multiple threads at the same time I would go with a Singleton hosting the context object and providing threadsafe methods to access it.
Instead of manipulating multiple properties of your context object, I would strongly suggest to do all manipulations at the same time. Best would be if you pass only one object containing all the properties you want to change in your context object.
e.g
Singleton.getInstance().adjustContext(ContextAdjuster contextAdjuster)
You might also want to consider using a threadsafe queue, filling it up with ContextAdjuster objects from your threads and finally processing it in the Context's thread.
Google for things like Concurrent, Blocking and Nonblocking Queue in Java. I am sure you'll find tons of example code.
Related
A Sonar rule available since Aug 21, 2019 (squid:S5164 / RSPEC-5164) mandates to clean up "ThreadLocal" variables when no longer used. So, let's take the following class (JDK6 compatible):
public class ThreadLocalExample {
private static final ThreadLocal<NumberFormat> formats = new ThreadLocal<NumberFormat>() {
#Override
protected NumberFormat initialValue() {
final NumberFormat nf = NumberFormat.getNumberInstance(Locale.US);
nf.setMinimumFractionDigits(2);
nf.setMaximumFractionDigits(2);
nf.setGroupingUsed(false);
return nf;
}
};
public static NumberFormat getFormatter() {
return formats.get();
}
}
Sonar reports a major bug on the ThreadLocal declaration, with the following explanation:
"ThreadLocal" variables should be cleaned up when no longer used
ThreadLocal variables are supposed to be garbage collected once the
holding thread is no longer alive. Memory leaks can occur when holding
threads are re-used which is the case on application servers using
pool of threads.
To avoid such problems, it is recommended to always clean up
ThreadLocal variables using the remove() method to remove the current
thread’s value for the ThreadLocal variable.
Now, I adopted the ThreadLocal approach in order to reuse NumberFormat instances as much as possible, avoiding the creation of one instance per call, so I think if I called remove() somewhere in the code, I would lose all the advantages of this solution. Am I missing something? Thanks a lot.
Sonar is right here.
Each thread will have its own ThreadLocal state and so its own instance of NumberFormat.
So in the general case it may be undesirable to not clear data from the state since the thread may be reused (recycled by the server) and the state valued for the previous client may be inconsistent for the current client.
For example some clients could have the format US, others the format FR, and so for...
Besides some threads could instantiate that ThreadLocal class, other no. But by not cleaning the state, the state will be still use memory for threads that may not need them.
Well, in your code, there is not variability of the ThreadLocal state since you set the state for any instance, so no inconsistency risk is likely, just memory "waste".
Now, I adopted the ThreadLocal approach in order to reuse NumberFormat
instances as much as possible, avoiding the creation of one instance
per call
You reuse the ThreadLocal state by a thread request basis.
So if you have 50 threads, you have 50 states.
In web applications, the server maps the client HTTP request to one thread.
So you don't create multiple instances of the formatter only in the scope of 1 http request. It means that If you use the formatter one or two time by request processing, the ThreadLocal cache doesn't bring a great value. But if you use it more, using it makes sense.
so I think if I called remove() somewhere in the code, I
would lose all the advantages of this solution
Calling remove() will not hurt performance if you do that when the request processing is done. You don't lose any advantage since you may use the formatter dozen of times in the scope of the request and it will be cleaned only at the end.
You have Request Listener in the servlet specification :
https://docs.oracle.com/javaee/7/api/javax/servlet/ServletRequestListener.html.
You could do that in void requestDestroyed(ServletRequestEvent sre).
You should not call #remove directly after you used the formatter. As you wrote that would defeat the purpose.
You only need to call #remove in the following case. Your web application is unloaded from the application server e.g. Tomcat. But the application server itself keeps on running.
In this case the application server probably keeps the threads it created for your application around and these threads will still have each an instance of NumberFormat associated with them. That's your memory leak.
So if you always restart the entire app server you probably don't need to care about this problem.
If you want to clean up the ThreadLocal properly you would want to call #remove once your application is starting to shut down. This way you reused the NumberFormat instance a maximum of times while still cleaning up properly.
The link you provided says the following:
Memory leaks can occur when holding threads are re-used which is the case on application servers using pool of threads.
For example you build a pool with 7 threads because your CPU has 8 cores. Then you submit a task to the pool and it would be solved by one of the threads. Afterwards the thread may do not have other tasks to do, but the thread still holds the ThreadLocal object. This would be a waste of memory and because ThreadLocal has a reference to the containing object the garbage collector can not remove the containing object (could result in a memory leak).
If you do not reuse the thread and cleanup all references to the thread , then there will not be a memory leak. If you reuse the thread later the threadLocal object will be in-memory until it is overwritten or cleaned up or until the thread is destroyed.
I have many threads adding result-like objects to an array, and would like to improve the performance of this area by removing synchronization.
To do this, I would like for each thread to instead post their results to a ThreadLocal array - then once processing is complete, I can combine the arrays for the following phase. Unfortunately, for this purpose ThreadLocal has a glaring issue: I cannot combine the collections at the end, as no thread has access the collection of another.
I can work around this by additionally adding each ThreadLocal array to a list next to the ThreadLocal as they are created, so I have all the lists available later on (this will require synchronization but only needs to happen once for each thread), however in order to avoid a memory leak I will have to somehow get all the threads to return at the end to clean up their ThreadLocal cache... I would much rather the simple process of adding a result be transparent, and not require any follow up work beyond simply adding the result.
Is there a programming pattern or existing ThreadLocal-like object which can solve this issue?
You're right, ThreadLocal objects are designed to be only accessible to the current thread. If you want to communicate across threads you cannot use ThreadLocal and should use a thread-safe data structure instead, such as ConcurrentHashMap or ConcurrentLinkedQueue.
For the use case you're describing it would be easy enough to share a ConcurrentLinkedQueue between your threads and have them all write to the queue as needed. Once they're all done (Thread.join() will wait for them to finish) you can read the queue into whatever other data structure you need.
I have a Java servlet that operates with a heavy-weight and thread-unsafe resource to handle user requests. The resource is an object that needs a long time to be instantiated (up to 10 seconds) and takes a large amount of memory.
But when the object is allocated, it takes a short time to run its method I need to process a request.
There can be several such resources, different from each other.
Each request comes with an ID, which points out on the certain resource.
I wish to implement a pool of such resources, so that requests with the same IDs will not instantiate a new object, but will pick one from the pool.
The scheme is following:
after the request has been received, servlet checks whether a resource with the requested ID is in the pool
if not, servlet creates one and provides it
if the resource is already instantiated, the request goes into a queue to be executed, doPost waits for it.
The operation over different resources must be concurrent, but synchronized within the same resource.
I am new to multithreading in Java, and the ThreadPoolExecutor does not seem to be usable as is, so I would be appreciated for an advice how to implement the above described scheme. Thanks.
You are correct - ThreadPoolExecutor is not what you want. It is simply a pool of threads to run tasks with, not a shared resource collection.
What you want is a cache. It needs to create a resource and return it to requesting threads to use, and reuse the things it returned previously. Also, the resource returned must be thread-safe (So if your underlying resources are not, you may need to write synchronized wrappers for them).
There are a number of thread-safe caches around, quite a few of them - opensource. Try those out, it shouldn't be too difficult to configure them for your use case (it seems fairly typical).
It is possible and not too difficult to implement a make-shift cache of your own, but you're far better off using a third-party solution if you are new to multithreading.
I would like to reuse instances of non-thread safe classes for performance reasons in a Servlet. I have two options,
use ThreadLocal where in Java takes care of doing the instance management per thread
use a static HashMap which uses Thread as the HashMap key and the instances are managed at this level
With the ThreadLocal approach there are potentials for memory leaks esp in Servlet enviornment. Because of this, I am thinking of using the 2nd option, I was wondering if anyone has experience in using this approach and any pitfalls of using the same?
Prefer the ThreadLocal approach because it is likely synchronized (or better yet, requires no synchronization) at the correct granularity and no larger.
If you roll your own solution using HashMap you'll have to acquire a lock over the HashMap every time you want to access any thread-local data. Why? Because a new thread could be created and threads can die. These are implicitly adding/removing items from a HashMap, which require synchronization on the full HashMap. You'll also have quite the time keeping object lifetimes straight because HashMap will keep alive all items it contains as long as it is referable from any thread. That is not how ThreadLocal store behaves.
The problem is not ThreadLocal itself, but the way it's being used. See here for a detailed explanation. So, your own implementation won't make a difference.
I need to use the Stanford Parser in a web service. As SentenceParser loads a big object, I will make sure it is a singleton, but in this case, is it thread safe (no according to http://nlp.stanford.edu/software/parser-faq.shtml). How else would it be done efficiently? One option is locking the object while being used.
Any idea how the people at Stanford are doing this for http://nlp.stanford.edu:8080/parser/ ?
If the contention is not a factor, locking (synchronization) would be one option as you mentioned, and it might be good enough.
If there are contentions, however, I see three general options.
(1) instantiating it every time
Just instantiate it as a local variable every time you perform parsing. Local variables are trivially safe. The instantiation is not free of course, but it may be acceptable depending on the specific situation.
(2) using threadlocals
If instantiation turns out to be costly, consider using threadlocals. Each thread would retain its own copy of the parser, and the parser instance would be reused on a given thread. Threadlocals are not without problems, however. Threadlocals may not be garbage collected without being set to null or until the holding thread goes away. So there is a memory concern if there are too many of them. Second, beware of the reuse. If these parsers are stateful, you need to ensure to clean up and restore the initial state so subsequent use of the threadlocal instance does not suffer from the side effect of previous use.
(3) pooling
Pooling is in general no longer recommended, but if the object sizes are truly large so that you need to have a hard limit on the number of instances you can allow, then using an object pool might be the best option.
I don't know how the people at Stanford have implemented their service but I would build such a service based on a message framework, such as http://www.rabbitmq.com/. So your front end service will receive documents and use a message queue to communicate (store documents and retrieve results) with several workers that execute NLP parsing. The workers -- after finishing processing -- will store results into a queue that is consumed by the front end service. This architecture will let you to dynamically add new workers in case of high load. Especially that NLP tagging takes some time - up several seconds per document.