We Keep Coding

Is there a way to solve this memory leak with my CompletableFuture code?

We have a server that does backpressure in http1.1 (yes, it turns off sockets and clients actually freeze up waiting if we are under load...it's quite nice). We do it in http2 as well without the http2 specification's doing backpressure in the http2 engine(ie. we don't need an backpressure there but we follow the spec and it also turns off sockets). When I say 'turn off', I only mean it deregisters it so we stop reading the nic and our nic buffer fills and then clients nic fills until he freezes.
Anyways, we ran into an interesting issue and have been trying to solve it without a memory leak. Our latest iteration was something like this
public CompletableFuture<Void> runLoop(List<T> newData, Session session, Processor<T> processFunction) {
//All the below futures must be chained with previous ones in case previous ones are not
//done which will serialize it all to be in sequence
CompletableFuture<Void> future = session.getProcessFuture();
for(T data : newData) {
//VERY IMPORTANT: Writing the code like this would slam through calling process N times
//BUT it doesn't give the clients a chance to set a flag between packets
//Mainly done for exceptions and streaming so you can log exc, set a boolean so you
//don't get 100 exceptions while something is happening like socket disconnect
//In these 2 lines of code, processCorrectly is CALLED N times RIGHT NOW
//The code below this only calls them right now IF AND ONLY IF the client returns
//a completed future each time!!!
//CompletableFuture<Void> messageFuture = processFunction.process(data);
//future = future.thenCompose( f -> messageFuture);
future = future.thenCompose( voidd -> processFunction.process(data));
}
session.setProcessFuturee(future);
return future;
}
As data comes in runLoop is called for N data chunks. runLoop is always called serially so we have no race conditions there. We call runLoop X times. The issue is that in the current code above processFunction.process MAY be called on a different thread (and in fact is quite frequently for our streaming endpoints only).
I am wondering how to cut the cord so to speak so that we stop chain. commenting out session.setProcessFuture DOES stop the chain but the issue there is incoming data that comes in 2nd can then beat the data that came in first(and has but rarely).
The test I am using which if we comment out setProcessFuture bounces between 500MB and 6MB of memory on and off is the link below. If I don't comment that out, it slowly uses up all 500MB.
https://github.com/deanhiller/webpieces/blob/master/core/core-util/src/test/java/org/webpieces/util/futures/TestLoopingChainMemory.java
Trying to not have the race condition and not have memory issues and also allow backpressure which is done using the future that we return from runLoop(). If there are too many unresolved, we stop feeding in traffic to runLoop.
FYI: thenCompose == scala's flatMap
EDIT: I had another idea and so I tried this but this failed as well. The List size is typically pretty small in production so I didn't mind stacking up the futures in the loop but trying to cut the chain outside the loop to prevent the memory from adding up over time and never being cleaned...
public CompletableFuture<Void> runLoop(List<T> newData, Session session, Processor<T> processFunction) {
//All the below futures must be chained with previous ones in case previous ones are not
//done which will serialize it all to be in sequence
CompletableFuture<Void> future = session.getProcessFuture();
CompletableFuture<Void> newFuture = new CompletableFuture<Void>();
for(T data : newData) {
//VERY IMPORTANT: Writing the code like this would slam through calling process N times
//BUT it doesn't give the clients a chance to seet a flag between packets
//Mainly done for exceptions and streaming so you can log exc, set a boolean so you
//don't get 100 exceptions while something is happening like socket disconnect
//In these 2 lines of code, processCorrectly is CALLED N times RIGHT NOW
//The code below this only calls them right now IF AND ONLY IF the client returns
//a completed future each time!!!
//This seems to have memory issues as well....
//CompletableFuture<Void> temp = processFunction.process(data);
//future = future.thenCompose(f -> temp);
future = future.thenCompose( voidd -> processFunction.process(data));
}
future.handle((voidd, t) -> {
if(t != null) {
newFuture.completeExceptionally(t);
return 0;
}
newFuture.complete(null);
return 0;
});
//comment this out and memory leak goes away of course.......
session.setProcessFuturee(newFuture);
return newFuture;
}
EDIT: ok, I discovered that this line helps BUT I had to put code in to wait much longer for memory to cleanup. It took a 'long' time to clean up and it memory goes down to 13MB. Now, I am wondering why cleanup is taking so long...perhaps objects made it through to older generations in the gc model
future = future.thenComposeAsync( voidd -> processFunction.process(data), executor );
THEN, I realized what if wait on my original code. This is where things got weird. It only went back down to 196MB and stayed there. I am not sure why or what the reference is. I really don't see anything in MAT(eclipse) or I am using that tool wrong.
Do I have a memory leak? I am confused on this last result. it should go down to < 20MB at least
EDIT(addressing Shadov's answer): I believe the future returned is not chained to where it came from. It is the 'tail' of the chain so to speak I think. It's the opposite though, the future that created it is tied to that so that when the future that created it is done, it can resolve that future. So in my mind, I am posting the tail of the list in the session each time(not creating a chain that is longer and longer). The real issue with futures is the thread that references it and whether it was resolved. This means the only one left referencing it is the anonymous Runnable that resolves it. Once that happens, it should be able to resolve.
CompletableFuture is not even herein memory (MAT eclipse)..
OMG, another major discovery.. If I comment out this line
future.complete(null);
then the program flies forever with memory bouncing between 180MB and 6MB.

Not really good at this, but you are building bigger and bigger future and never stopping - no matter the size of the list, it's only gonna consume more and more memory, never releasing it.
You will see if you do in your test:
chain.runLoop(list, s, p);
if(queue.size() == 0) {
System.out.println("queue empty");
s.setProcessFuturee(CompletableFuture.completedFuture(null));
rt.gc();
}
So in my opinion you need to use a different tool, CompletableFuture may be not strong enough for this. Maybe some full fledged reactive library, like rxjava or reactor?

Ok, I did a git push and git commit. There is no memory leak. What is happening is that we slam the main thread so hard, it all accumulates........when giving time stuff to resolve, memory goes back to 6MB. It just takes VERY long time but the logs helped see that easier AND I most importantly was missing the runtime.gc() call in a critical spot!!!!
soooo, with backpressure in place, we should have no issues since that makes all the futures resolves before adding load.

Background a task then end connection before task completion in Java (8)

I've spent a lot of time looking at this and there are a tonne of ways to background in Java (I'm specifically looking at Java 8 solutions, it should be noted).
Ok, so here is my (generic) situation - please note this is an example, so don't spend time over the way it works/what it's doing:
Someone requests something via an API call
The API retrieves some data from a datastore
However, I want to cache this aggregated response in some caching system
I need to call a cache API (via REST) to cache this response
I do not want to wait until this call is done before returning the response to the original API call
Some vague code structure:
#GET
# // api definitions
public Response myAPIMethod(){
// get data from datastore
Object o = getData();
// submit request to cache data, without blocking
saveDataToCache();
// return the response to the Client
return Response.ok(data).build();
}
What is the "best" (optimal, safest, standard) way to run saveDataToCache in the background without having to wait before returning data? Note that this caching should not occur too often (maybe a couple of times a second).
I attempted this a couple of ways, specifically with CompletableFutures but when I put in some logging it seemed that it always waited before returning the response (I did not call get).
Basically the connection from the client might close, before that caching call has finished - but I want it to have finished :) I'm not sure if the rules are the same as this is during the lifetime of a client connection.
Thanks in advance for any advice, let me know if anything is unclear... I tried to define it in a way understandable to those without the domain knowledge of what I'm trying to do (which I cannot disclose).

You could consider adding the objects to cache into a BlockingQueue and have a separate thread taking from the queue and storing into cache.

As per the comments, the cache API is already asynchronous (it actually returns a Future). I suppose it creates and manages an internal ExecutorService or receives one at startup.
My point is that there's no need to take care of the objects to cache, but of the returned Futures. Asynchronous behavior is actually provided by the cache client.
One option would be to just ignore the Future returned by this client. The problem with this approach is that you loose the chance to take a corrective action in case an error occurrs when attempting to store the object in the cache. In fact, you would never know that something went wrong.
Another option would be to take care of the returned Future. One way is with a Queue, as suggested in another answer, though I'd use a ConcurrentLinkedQueue instead, since it's unbounded and you have mentioned that adding objects to the cache would happen as much as twice a second. You could offer() the Future to the queue as soon as the cache client returns it and then, in another thread, that would be running an infinite loop, you could poll() the queue for a Future and, if a non null value is returned, invoke isDone() on it. (If the queue returns null it means it's empty, so you might want to sleep for a few milliseconds).
If isDone() returns true, you can safely invoke get() on the future, surrounded by a try/catch block that catches any ExecutionException and handles it as you wish. (You could retry the operation on the cache, log what happened, etc).
If isDone() returns false, you could simply offer() the Future to the queue again.
Now, here we're talking about handling errors from asynchronous operations of a cache. I wouldn't do anything and let the future returned by the cache client go in peace. If something goes wrong, the worst thing that may happen is that you'd have to go to the datastore again to retrieve the object.

Getter methods in java with asynchronous connection needing a delay

I'm making a series of connections asynchronously via MySQL, and I have a class which contains a bunch of easy-accesible static methods to update/remove/clear/get/etc data.
The issue I'm confronted with is that the getter methods won't return the proper value (practically ever) because they are returned before the async connection gets a chance to update the value to be returned.
Example:
public static int getSomething(final UUID user)
{
Connection c = StatsMain.getInstance().getSQL().getConnection();
PreparedStatement ps;
try
{
ps = c.prepareStatement("select something from stats where uuid=?");
ps.setString(1, user.toString());
ResultSet result = ps.executeQuery();
return result.getInt("something");
}
catch (SQLException e)
{
return false;
}
}
(Not copy & pasted, but pretty close)
I realize I can use a 'callback' effect by passing an interface to the method and doing such, but that becomes very tedious when the database stores 10 values for a key.

Sounds like you're looking for Futures since Java 6 or CompletableFuture, which is new in Java 8

Solution 1:
The best method I've come up with is have a thread with a loop in it that waits for MySQL to return values and responds to each value. This is rather like the callback in the get routine, but you only have the one loop. Of course, the loop has to know what to do with each possible returned piece of data.
This means rethinking a bit how your program works. Instead of: ask a question, get an answer, use the answer, you have two completely independent operations. The first is: ask a question, then forget about it. The second is: get an answer, then, knowing nothing about the question, use the answer. It's a completely different approach, and you need to get your head around it before using it.
(One possible further advantage of this approach is that MySQL end can now send data without being prompted. You have the option of feeding changes made by another user to your user in real time.)
Solution 2:
My other solution is simpler in some ways, but it can have you firing off lots of threads. Just have your getSomething method block until it has the answer and returns. To keep your program from hanging, just put the whole block of code that calls the method in its own thread.
Hybrid:
You can use both solutions together. The first one makes for cleaner code, but the second lets you answer a specific question when you get the reply. (If you get a "Customer Name" from the DB, and you have a dozen fields it could go in, it might help to know that you did ask for this field specifically, and that you asked because the user pushed a button to put the value in a specific text box on the screen.)
Lastly:
You can avoid a lot of multithreading headaches by using InvokeLater to put all changes to your data structures on your EventQueue. This can nicely limit the synchronization problems. (On the other hand, having 20 or 30 threads going at once can make good use of all your computer's cores, if you like to live dangerously.)
You may want to stick with synchronized calls, but if you do want to go asynchronous, this is how I'd do it. It's not too bad once you get some basic tools written and get your brain to stop thinking synchronously.

JavaEE6: using #Asynchronous to speed up the web application. When?

I really want to abuse #Asynchronous to speed up my web application, therefore I want to understand this a bit more in order to avoid incorrectly using this annotation.
So I know business logic inside this annotated method will be handled in a separated thread, so the user wont have to wait. So I have two method that persist data
public void persist(Object object) {
em.persist(object);
}
#Asynchronous
public void asynPersist(Object object) {
em.persist(object);
}
So I have couple scenario I want to ask which one of these scenario is not ok to do
1. B is not depend on A
A a = new A();
asynPersist(a); //Is it risky to `asynPersist(a) here?`
B b = new B();
persist(b);
//Cant asynPersist(B) here because I need the `b` to immediately
//reflect to the view, or should I `asynPersist(b)` as well?
2. Same as first scenario but B now depend on A. Should I `asynPersist(a)`
3. A and B are not related
A a = new A();
persist(a); //Since I need `a` content to reflect on the view
B b = new B();
asynPersist(b); //If I dont need content of b to immediately reflect on the view. Can I use asyn here?
EDIT: hi #arjan, thank you so much for your post, here is another scenario I want to ask your expertise. Please let me know if my case does not make any sense to u.
4. Assume User has an attribute call `count` type `int`
User user = null;
public void incVote(){
user = userDAO.getUserById(userId);
user.setCount(u.getCount() + 1);
userDAO.merge(user);
}
public User getUser(){ //Accessor method of user
return user;
}
If I understand you correctly, if my method getUserById use #Asynchronous, then the line u.setCount(u.getCount() + 1); will block until the result of u return, is it correct? So in this case, the use of #Asynchronous is useless, correct?
If the method merge (which merge all changes of u back to database) use #Asynchronous, and if in my JSF page, I have something like this
<p:commandButton value="Increment" actionListener="#{myBean.incVote}" update="cnt"/>
<h:outputText id="cnt" value="#{myBean.user.count}" />
So the button will invoke method incVote(), then send and ajaxical request to tell the outputText to update itself. Will this create an race condition (remember we make merge asynchronous)? So when the button tell the outputText to update itself, it invoke the accessor method getUser(), will the line return user; block to wait for the asynchronous userDAO.merge(user), or there might possible a race condition here (that count might not display the correct result) and therefore not recommend to do so?

There are quite a few places where you can take advantage of #Asynchronous. With this annotation, you can write your application as intended by the Java EE specification; don't use explicit multi-threading but let work being done by managed thread pools.
In the first place you can use this for "fire and forget" actions. E.g. sending an email to a user could be done in an #Asynchronous annotated method. The user does not need to wait for your code to contact the mail-server, negotiate the protocol, etc. It's a waste of everyone's time to let the main request processing thread wait for this.
Likewise, maybe you do some audit logging when a user logs in to your application and logs off again. Both these two persist actions are perfect candidates to put in asynchronous methods. It's senseless to let the user wait for such backend administration.
Then there is a class of situations where you need to fetch multiple data items that can't be (easily) fetched using a single query. For instance, I often see apps that do:
User user = userDAO.getByID(userID);
Invoice invoice = invoiceDAO.getByUserID(userID);
PaymentHistory paymentHistory = paymentDAO.getHistoryByuserID(userID);
List<Order> orders = orderDAO.getOpenOrdersByUserID(userID);
If you execute this as-is, your code will first go the DB and wait for the user to be fetched. It sits idle in between. Then it will go fetch the invoice and waits again. Etc etc.
This can be sped up by doing these individual calls asynchronously:
Future<User> futureUser = userDAO.getByID(userID);
Future<Invoice> futureInvoice = invoiceDAO.getByUserID(userID);
Future<PaymentHistory> futurePaymentHistory = paymentDAO.getHistoryByuserID(userID);
Future<List<Order>> futureOrders = orderDAO.getOpenOrdersByUserID(userID);
As soon as you actually need one of those objects, the code will automatically block if the result isn't there yet. This allows you to overlap fetching of individual items and even overlap other processing with fetching. For example, your JSF life cycle might already go through a few phases until you really need any of those objects.
The usual advice that multi threaded programming is hard to debug doesn't really apply here. You're not doing any explicit communication between threads and you're not even creating any threads yourself (which are the two main issues this historical advice is based upon).
For the following case, using asynchronous execution would be useless:
Future<user> futureUser = userDAO.getUserById(userId);
User user = futureUser.get(); // block happens here
user.setCount(user.getCount() + 1);
If you do something asynchronously and right thereafter wait for the result, the net effect is a sequential call.
will the line return user; block to wait for the asynchronous userDAO.merge(user)
I'm afraid you're not totally getting it yet. The return statement has no knowledge about any operation going on for the instance being processed in another context. This is not how Java works.
In my previous example, the getUserByID method returned a Future. The code automatically blocks on the get() operation.
So if you have something like:
public class SomeBean {
Future<User> futureUser;
public String doStuff() {
futureUser = dao.getByID(someID);
return "";
}
public getUser() {
return futureUser.get(); // blocks in case result is not there
}
}
Then in case of the button triggering an AJAX request and the outputText rendering itself with a binding to someBean.user, then there is no race condition. If the dao already did its thing, futureUser will immediately return an instance of type User. Otherwise it will automatically block until the User instance is available.
Regarding doing the merge() operation asynchronous in your example; this might run into race conditions. If your bean is in view scope and the user quickly presses the button again (e.g. perhaps having double clicked the first time) before the original merge is done, an increment might happen on the same instance that the first merge invocation is still persisting.
In this case you have to clone the User object first before sending it to the asynchronous merge operation.
The simple examples I started this answer with are pretty save, as they are about doing an isolated action or about doing reads with an immutable type (the userID, assume it is an int or a String) as input.
As soon as you start passing mutable data into asynchronous methods you'll have to be absolutely certain that there is no mutation being done to that data afterwards, otherwise stick to the simple rule to only pass in immutable data.

You should not use asynch this way if any process that follows the asynch piece depends on the outcome. If you persist data that a later thread needs, you'll have a race condition that will be a bad idea.
I think you should take a step back before you go this route. Write your app as recommended by Java EE: single threaded, with threads handled by the container. Profile your app and find out where the time is being spent. Make a change, reprofile, and see if it had the desired effect.
Multi-threaded apps are hard to write and debug. Don't do this unless you have a good reason and solid data to support your changes.

Java Concurrency - Web Application

I think I found more bugs in my web application. Normally, I do not worry about concurrency issues, but when you get a ConcurrentModificationException, you begin to rethink your design.
I am using JBoss Seam in conjuction with Hibernate and EHCache on Jetty. Right now, it is a single application server with multiple cores.
I briefly looked over my code and found a few places that haven't thrown an exception yet, but I am fairly sure they can.
The first servlet filter I have basically checks if there are messages to notify the user of an event that occurred in the background (from a job, or another user). The filter simply adds messages to the page in a modal popup. The messages are stored on the session context, so it is possible another request could pull the same messages off the session context.
Right now, it works fine, but I am not hitting a page with many concurrent requests. I am thinking that I might need to write some JMeter tests to ensure this doesn't happen.
The second servlet filter logs all incoming requests along with the session. This permits me to know where the client is coming from, what browser they're running, etc. The problem I am seeing more recently is on image gallery pages (where there are many requests at about the same time), I end up getting a concurrent modification exception because I'm adding a request to the session.
The session contains a list of requests, this list appears to be being hit by multiple threads.
#Entity
public class HttpSession
{
protected List<HttpRequest> httpRequests;
#Fetch(FetchMode.SUBSELECT)
#OneToMany(mappedBy = "httpSession")
public List<HttpRequest> getHttpRequests()
{return(httpRequests);}
...
}
#Entity
public class HttpRequest
{
protected HttpSession httpSession;
#ManyToOne(optional = false)
#JoinColumn(nullable = false)
public HttpSession getHttpSession()
{return(httpSession);}
...
}
In that second servlet filter, I am doing something of the sort:
httpSession.getHttpRequests().add(httpRequest);
session.saveOrUpdate(httpSession);
The part that errors out is when I do some comparison to see what changed from request to request:
for(HttpRequest httpRequest:httpSession.getHttpRequests())
That line there blows up with a concurrent modification exception.
Things to walk away with:
1. Will JMeter tests be useful here?
2. What books do you recommend for writing web applications that scale under concurrent load?
3. I tried placing synchronized around where I think I need it, ie on the method that loops through the requests, but it still fails. What else might I need to do?
I added some comments:
I had thought about making the logging of the http requests a background task. I can easily spawn a background task to save that information. I am trying to remember why I didn't evaluate that too much. I think there is some information that I would like to have access to on the spot.
If I made it asynchronous, that would speed up the throughput quite a bit - well I'd have to use JMeter to measure those differences.
I would still have to deal with the concurrency issue there.
Thanks,
Walter

A ConcurrentModificationException occurrs when any collection is modified while iterating over it. You can do it in a single thread, e.g.:
for( Object o : someList ) {
someList.add( new Object() );
}
Wrap your list with Collections.synchronizedList or return an unmodifiable copy of the list.

I'm not sure about scaling web applications in particular, but Java Concurrency in Practice is a fantastic book on concurrency in general.
The list should be replaced with a version that is threadsafe, or all access to it has to be synchronized (readers and writers) on the same object. It is not enough to synchronize just the method that reads from the list.

It's been caused because the list has been modified by another request while you're still iterating over it in one request. Replacing the List by ConcurrentLinkedQueue (click link to see javadoc) should fix the particular problem.
As to your other questions:
1: Will JMeter tests be useful here?
Yes, it is certainly useful to stress-test webapplications and spot concurrency bugs.
2: What books do you recommend for writing web applications that scale under concurrent load?
Not specific tied to webapplications, but more to concurrency in general, the book Concurrency in Practice is the most recommended one in that area. You can perfectly apply the learned things on webapplications as well, they are a perfect real world example of "heavy concurrent" applications.
3: I tried placing synchronized around where I think I need it, ie on the method that loops through the requests, but it still fails. What else might I need to do?
You basically need to synchronize any access to the list on the same lock. But just replacing by ConcurrentLinkedQueue is easier.

You're getting an exception on the iterator, because another thread is altering the collection backing the iterator while you're in mid-iteration.
You could wrap access to the list in synchronized access (both adding and iterating) but there are problems with this, as it could take significantly longer to iterate through a list, along with the processing that goes along with it, and you'd be holding the lock to the list for all of that time.
Another option would be to copy the list and pass out the copy for iteration, which might be a better idea if the objects are small, as you'd only be holding the lock while you make the copy, rather than while you're iterating through the list.
Store your values in a ConcurrentHashMap, which uses lock striping to minimize lock contention. You could then have your get method return a copied list of the keys you want, rather than the complete objects, and access them one at at time directly from the map.
As is mentioned in another answer here, Java Concurrency in Practice is a great book.

The other posters are correct in stating that you need to be writing to a threadsafe data structure. In doing so, you may slow down your response time due to thread contention. Since this essentially a logging operation that is a side effect of the request itself (Or am I not understanding you correctly?) you could spawn a new thread responsible for writing to the threadsafe data structure. That allows you to proceed with the actual response instead of burning response time on a logging operation. It might be worth investigating setting up a threadpool to reduce the time required to use the logging threads.
Any concurrency book by Doug Lea is worth reading.