We Keep Coding

Corda RPC: A hot observable returned from an RPC was never subscribed to

I build a CorDapp based on the Java Template. On top of that, I created a React front-end. Now, I want to start a flow from my front-end. To do so, I modified the template server, so that the controller starts my flow:
#GetMapping(value = "/templateendpoint", produces = "text/plain")
private String templateendpoint() {
proxy.startTrackedFlowDynamic(issueTokens.class, 30, "O=Bob, L=Berlin, C=DE");
return "The flow was started";
}
This operation does start the flow that issues 30 tokens to Bob. I can see that the flow was successful, by querying Bob's vault. However, I get the following error on the template server:
RPCClientProxyHandler.onRemoval - A hot observable returned from an RPC was never subscribed to.
This wastes server-side resources because it was queueing observations for retrieval.
It is being closed now, but please adjust your code to call .notUsed() on the observable to close it explicitly. (Java users: subscribe to it then unsubscribe).
If you aren't sure where the leak is coming from, set -Dnet.corda.client.rpc.trackRpcCallSites=true on the JVM command line and you will get a stack trace with this warning.
After this first transaction, I cannot start another flow. The .notUsed() method only works for Kotlin. However, I couldn't find a working way to subscribe and then unsubscribe from the observable.
Could anyone give me an example on how to implement this with the Corda flow? Moreover, what is the most practical way to pass information from the front-end to the controller class, in order to use that as flow arguments?

The reason that the error appears is that the Observable on the client-side gets garbage collected.
The solution is provided has been provided in the bracket-
(Java users: subscribe to it then unsubscribe)
So in your case, you can do something like this:
Subscription subs = updates.subscribe();
subs.unsubscribe();
Probably a more practical way is to keep the observable instance as a private attribute - such that it won't get garbage-collected. ie.
private Observable observable;
Ref: https://docs.corda.net/docs/corda-os/4.4/clientrpc.html#observables

Is running a long-running-process always bad on a servlet thread having a single HTTP request? [duplicate]

This question already has answers here:
How to run a background task in a servlet based web application?
(5 answers)
Closed 6 years ago.
I have a Java application (running on WAS 8.5) which acts as a kind of server inventory for the client. The application has a servlet that triggers a long running process.
The process: fetches data from a third party DB, executes Java logic, writes records back to the application's own DB (these DB connections are pooled ones) .
The servlet is not load-on-startup and is manually triggered only once a month by a single Operations guy (on some particular date based on the client's choice each month). The servlet had been historically using Timer and TimerTask in this way:
public class SyncMissingServlet extends HttpServlet implements Servlet{
public void doPost(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException
{
try{
SyncMissing.runSync();
}catch(Exception ex){
logger.error(new LogMessage("ERROR: "), ex);
this.sendReply(printWriter, "ERROR: " + ex.toString());
}
}
}
public class SyncMissing
{
public static void runSync() throws Exception
{
Timer t = new Timer(true);
SyncMissingTask task = new SyncMissingTask(); //SyncMissingTask is an instance of TimerTask
// Start the synchronization 5 secs from now, and run it every 30 days.
t.schedule(task, 5000, 2592000000l); //this 30 day timings never really worked out for the client,
//since the app server is restarted frequently for deployments.
}
}
There is no use of Timer.close() or TimerTask.close() in the current code.
Recently this Servlet seems to have got auto-trigerred, after a system reboot and restart of the WAS services on the system...and that's the worry.
While I couldn't explain the auto-trigger to my client, I proposed the following options:
1. drop off the use of Timer and TimerTask (the long-running process then runs on the servlet's thread itself)
2. instead of TimerTask, make it a regular Runnable and run it in a separate thread within the servlet thread.
3. make use of Java's Executor Service
4. migrate to Servlet 3.0 and turn the servlet into a Async servlet.
5. drop off the servlet altogether, and replace it with a batch job.
I understand that options 3 and 4 are really the recommended ones (or possibly option 5). But I have a feeling, that in my business scenario - Options 3 & 4 may be an overkill.
If the need is really a manual invocation of the servlet by only one user per month, are options 1 and 2 that bad?
(my client wants the quickest solution and would certainly not fund option 5)

Well, if the servlet is supposed to be run only once in a month and there is only one client triggering it, it is fine to run it in the servlet's thread itself or create a new thread inside the servlet and let that do the task. The question of load and response times arises when you have a lot of clients making simultaneous requests, at which point you might want to use an Executor service or an async servlet.

There is no need to activate a background task by invoking a servlet. Your web app has its own lifecycle. The Servlet spec provides hooks for your web app getting set-up and torn-down. Perfect place to launch and quit your background task without ever invoking a servlet by a client user.
No need to depend on a human user remembering to start the background task. Let your web app technology do the work for you.
Also, you may often hear/read "never launch threads from JSP or Servlet". This is worthy advice with regard to processing each incoming request for generating a web page. But background tasks (not directly related to a single servlet request) is a different animal; perfectly okay to have threads for background tasks as long as you handle them properly. By 'properly' I mean you explicitly end those threads appropriately, and you handle thread-safety issues. An example of a background task might be regularly polling a web service or database to refresh a cache of data.
ServletContextListener
If you want an automated task to be performed regularly within your web app, use a ServletContextListener.
That interface defines a pair of methods. One, contextInitialized, is called automatically when the web app launches, guaranteed to run before any HTTP requests are handled. The other method, contextDestroyed, runs when the web app is being torn down.
Tip: Marking your listener with a #WebListener annotation will cause your Servlet container to automatically notice it and instantiate when the web app is launched.
Beware of a nasty bug when doing development with NetBeans & Tomcat (development problem only, not a problem in deployment) where the web app does a double launch.
ScheduledExecutorService
In your custom class implementing that interface, in contextInitialized, establish a ScheduledExecutorService object to run your task repeatedly. In contextDestroyed, shutdown that executor. This is very important as the thread(s) of that executor will survive the shutdown of your web app and even the servlet container.
The ScheduledExecutorService technology supplants the Timer and TimerTask classes. These classes are especially not recommended for use in a Java Servlet environment.
You can store a reference to the executor in your listener object.
#WebListener
class MonthlyTaskRunner implements ServletContextListener {
private ScheduledExecutorService scheduledExecutorService;
void contextInitialized(ServletContextEvent see) {
// initialize your ScheduledExecutorService.
// The ScheduledExecutorService will use one or more threads for its work outside of this thread running now.
this.scheduledExecutorService = … ;
}
void contextInitialized(ServletContextEvent see) {
// Shutdown the executor along with its thread(s).
this.scheduledExecutorService.shutDown();
}
}
I and others have posted on this extensively here on Stack Overflow, such as this. So search Stack Overflow. I have posted extensive examples in the context of Vaadin web apps, but the principles apply to any servlet web app. And see the Oracle Tutorial on Executors.
Where to store a reference to your ScheduledExecutorService once instantiated? You could store in a member variable on your context listener. But a more accessible place would be as an "attribute" on the servlet context. I describe this in detail along with example code and a nifty diagram in my Answer to another Question: Start & Stop a ScheduledExecutorService in Java EE environment using servlet
YearMonth
In that executor task, get the year-month of the current date for the time zone of your business context. Compare that year-month to one recorded when the task was last performed. Record that year-month somewhere, in a file, in a database, someplace.
Schedule your ScheduledExecutorService to run more often than necessary. Rather than worry about scheduling out a month, just let it run everyday. The check to compare current YearMonth with stored year-month requires nearly no execution time. KISS.
Java includes a YearMonth class.
YearMonth ymThen = YearMonth.parse( "2016-11" ); // Retrieve that string from storage.
ZoneId z = ZoneId.of( "America/Montreal" );
YearMonth ymNow = YearMonth.now( z );
if( ymNow.isAfter( ymThen ) ) {
// … run the task
String ymOutput = ymNow.toString();
// … write that `ymOutput` string someplace in storage.
} // Else do nothing. Let the ScheduledExecutorService run again after its designated rest period.
Similar Questions
Background timer task in JSP/Servlet web application
How to run a background task in a servlet based web application?

How to handle bean lifecycle in an asynchronous request which spawns multiple threads

Question:
Is there a Scope for a Thread and it's spawned Threads which supports it's destruction as soon as all Threads accessing it are terminated? If not do I have to implement it myself or am I getting the concept of Scopes in Spring DI wrong?
Context:
I have a platform which has a REST-API on which processes can be started which are then running on the server. Some of these processes start multiple Threads of which some only terminate on system shutdown (e.g. listening on a stream and doing stuff with the data received).
I want to use Spring for dependency injection and now need to manage beans in a suitable scope.
Problem:
I want to take parameters from the request and provide these at multiple other locations. My approach is to take a container bean which is populated in the request handler and then used at all other occasions. The #Scope("request") is destroyed as soon as the response is send which happens instantly (since the handler only spawns a Thread) thus it's not applicable here.
I read about the ThreadScope implementation from springbyexample (http://www.springbyexample.org/examples/custom-thread-scope-module.html) and a way to modify the spring SimpleThreadScope to support inheritance within the hierarchy of spawned threads (https://stackoverflow.com/a/14987371/4502203). Both are only solving parts of my Problem.
What I need is a Scope which supports destruction callbacks (since I'm not keen on memory leak) and is inherited to Child-Threads.
Code Example:
#RequestMapping(value = "/myApi/{parameterA}/{parameterB}", method = RequestMethod.GET, produces = {MediaType.APPLICATION_JSON_VALUE})
public #ResponseBody void doFancyStuff(#PathVariable String parameterA, #PathVariable String parameterB) {
new Thread(() -> {
ParameterContainer parameterContainer = applicationContext.getBean(ParameterContainer.class);
parameterContainer.setParA(parameterA);
parameterContainer.setParB(parameterB);
/*
* spawn a couple of additional threads here which
* need to get access to the ParameterContainer.
*/
}
}

Continue processing after AsyncContext.complete()

First of all I want to say that English is not my first language, so excuse me if I make some obvious mistakes or something is not clear enough. The question:
Recently I've been moved to a new project where we are developing a Java EE application that provides some REST services via Jersey+Hibernate. Prior to this I only had experience with Java SE, but there was someone experienced already laying the foundations so I had my time to learn from his code and using Google and SO a lot.
The thing is, among those REST services there is one that can take a lot of time to complete, and the team decided to implement it in a non-blocking way. We will define two services: with the first one the client sends the data to process, then we return an acknowledgement and start the processing while the client can continue with other things; and the second one allows the client to check later if their job is done.
While investigating how to implement this the best way possible, my aforementioned colleague discovered the AsyncServlet feature of Servlets 3.0, and had a proof of concept implemented before I arrived, which later he evolved into a locally working (but very dirty) version of the service. He says he had to drop Jersey for this, since Servlets 3.0 weren't compatible with the version of Jersey we work with, and finally decided to implemented a plain servlet.
At the end, he got something like this (I don't have the code right now since I'm at home and writing by memory, but I'll try to write it as clear as possible and try to fix any big mistakes tomorrow morning):
A servlet which handles the new requests in doPost() and the checking in doGet():
#WebServlet(asyncSupported = true)
//...
void doGet(HttpServletRequest req, HttpServletResponse res) {
/*
* ...
* We query the previous "job request" here in the DB
* ...
*/
}
void doPost(HttpServletRequest req, HttpServletResponse res) {
/*
* ...
* We convert the JSON request to an entity and then start the asynchronous
* "worker" thread
* ...
*/
AsyncContext ctx = req.startAsync(req, res);
ctx.start(new WorkerThread(ctx, someOtherDataFromRequest);
}
And a worker thread that implements Runnable and the first thing it does is to call ctx.complete() on the AsyncContext that was sent to him in the constructor. My colleague reasoning is, if the worker notifies the parent right away he has completed, the parent can commit the response back to the client and then start his own processing with the other data passed to him in the constructor:
public class WorkerThread implements Runnable {
public WorkerThread(AsyncContext ctx, SomeOtherData data){
//...
}
public void run() {
ctx.complete();
// ... Now start doing the heavy processing with data
}
}
Well, as I said this works on his local test server (Tomcat 7), but a few days ago I was asked to clean his code and when ran on my machine which has a JBoss EAP 6.1, I found that it doesn't work as expected, since the parent servlet does not commit the response until the worker dies (we have different servers because the production machines are new and the higher ups haven't decided which server to install yet and changed their minds a bunch of times, bureaucracy...)
I did a bunch of tries and I'm pretty sure I didn't remove any key element of the async processing while cleaning, as my version compiles and runs fine. At the end I got a test case where the worker just sleeps 10 seconds, then writes at the log; in Tomcat the response reaches the client almost instantly then at 10s the log is written; while in JBoss the client has to wait the full 10s to receive the response.
Then, I started investigating the AsyncServlet feature, and I think he got the idea the wrong way, this feature seems to be directed to asynchronous internal processing and not as we want to use it, but I can't understand why it does work on his Tomcat. From the javadocs of the complete() method I understand the JBoss behavior is the correct one:
If this method is called before the container-initiated dispatch that called startAsync has returned to the container, then the call will not take effect (and any invocations of AsyncListener#onComplete(AsyncEvent) will be delayed) until after the container-initiated dispatch has returned to the container.
So, my question is if the AsyncServlet features are intended for our use case, and, if not, if there are any other cleaner ways to get our desired behavior (bonus points if they are compatible with Jersey). I'm thinking on just spawning a thread and not using the asyncontext at all, but sounds quite risky...
Thanks and sorry for the wall of text

This response may be a little too late but have you looked at using AsyncResource which comes with Jersey 2.0
https://jersey.java.net/documentation/latest/async.html
Also, you said "the parent servlet does not commit the response until the worker dies". How did you prove this ? Perhaps you have different settings for your Tomcat Server and Jboss server (Different threadpool size / different number of connections ?)
Also, from your description it looks like you have everything you need in SomeOtherData. So you could use an ExecutorService instead , add a runnable on to it that just takes SomeOtherData in its constructor and immediately call complete on the context in the doPost itself rather than doing it in the worker.

Calling System.exit() in Servlet's destroy() method

This is a follow up to my earlier question.
Tomcat 5.0.28 had a bug where the Servlet's destroy() method was not being invoked by the container on a shutdown. This is fixed in Tomcat 5.0.30, but if the Servlet's destroy() method had a System.exit(), it would result in the Tomcat windows service throwing the Error 1053 and refusing to shutdown gracefully (see above link for more details on this error)
Anybody has any idea on whether:
Calling System.exit() inside a Servlet's destroy() method to forcefully kill any non-daemon threads is a good idea?
Why does Tomcat 5.0.30 and (later versions including Tomcat 6.x.x) fail to shutdown properly if there's a System.exit() in the destroy() method of the Servlet.

Calling System.exit() inside a Servlet's destroy() method to forcefully kill any non-daemon threads is a good idea?
It is absolutely not a good idea - it is a horrible idea. The destroy() method is called when the servlet is taken out of service, which can happen for any number of reasons: the servlet/webapp has been stopped, the webapp is being undeployed, the webapp is being restarted etc.
System.exit() shuts down the entire JVM! Why would you want to forcibly shutdown the entire server simply because one servlet is being unloaded?
Why does Tomcat 5.0.30 and (later versions including Tomcat 6.x.x) fail to shutdown properly if there's a System.exit() in the destroy() method of the Servlet.
Probably to prevent such dangerous behavior like this.
You shouldn't write code that assumes that your code/application is the only thing running on the server.

You're asking two questions:
Question 1: Is calling System.exit() inside a Servlet's destroy() method to forcefully kill any non-daemon threads a good idea?
Calling System.exit() inside ANY servlet-related method is always 100% incorrect. Your code is not the only code running in the JVM - even if you are the only servlet running (the servlet container has resources it will need to cleanup when the JVM really exits.)
The correct way to handle this case is to clean up your threads in the destroy() method. This means starting them in a way that lets you gently stop them in a correct way. Here is an example (where MyThread is one of your threads, and extends ServletManagedThread):
public class MyServlet extends HttpServlet {
private List<ServletManagedThread> threads = new ArrayList<ServletManagedThread>();
// lots of irrelevant stuff left out for brevity
public void init() {
ServletManagedThread t = new MyThread();
threads.add(t);
t.start();
}
public void destroy() {
for(ServletManagedThread thread : threads) {
thread.stopExecuting();
}
}
}
public abstract class ServletManagedThread extends Thread {
private boolean keepGoing = true;
protected abstract void doSomeStuff();
protected abstract void probablySleepForABit();
protected abstract void cleanup();
public void stopExecuting() {
keepRunning = false;
}
public void run() {
while(keepGoing) {
doSomeStuff();
probablySleepForABit();
}
this.cleanup();
}
}
It's also worth noting that there are thread/concurrency libraries out there that can help with this - but if you really do have a handful of threads that are started at servlet initialization and should run until the servlet is destroyed, this is probably all you need.
Question 2: Why does Tomcat 5.0.30 and (later versions including Tomcat 6.x.x) fail to shutdown properly if there's a System.exit() in the destroy() method of the Servlet?
Without more analysis, it's hard to know for certain. Microsoft says that Error 1053 occurs when Windows asks a service to shutdown, but the request times out. That would make it seem like something happened internally to Tomcat that got it into a really bad state. I would certainly suspect that your call to System.exit() could be the culprit. Tomcat (specifically, Catalina) does register a shutdown hook with the VM (see org.apache.catalina.startup.Catalina.start(), at least in 5.0.30). That shutdown hook would get called by the JVM when you call System.exit(). The shutdown hook delegates to the running services, so each service could potentially be required to do alot of work.
If the shutdown hooks (triggered by your System.exit()) fail to execute (they deadlock or something like that,) then it is very easy to understand why the Error 1053 occurs, given the documentation of the Runtime.exit(int) method (which is called from System.exit()):
If this method is invoked after the
virtual machine has begun its shutdown
sequence then if shutdown hooks are
being run this method will block
indefinitely. If shutdown hooks have
already been run and on-exit
finalization has been enabled then
this method halts the virtual machine
with the given status code if the
status is nonzero; otherwise, it
blocks indefinitely.
This "indefinite blocking" behavior would definitely cause an Error 1053.
If you want a more complete answer than this, you can download the source and debug it yourself.
But, I would be willing to bet that if you properly handle your thread management issue (as outlined above,) your problems will go away.
In short, leave the System.exit() call to Tomcat - that's not your job.

Calling System.exit() inside a
Servlet's destroy() method to
forcefully kill any non-daemon threads
is a good idea?
Not a good idea. You will forcefully kill all threads, which might include part of Tomcat that is currently shutting down the system. This will cause Tomcat to un-gracefully shutdown. This can also prevent shutdown handlers from running which can lead to all sorts of problems.
Why does Tomcat 5.0.30 and (later
versions including Tomcat 6.x.x) fail
to shutdown properly if there's a
System.exit() in the destroy() method
of the Servlet.
A lot of code executes after a Servlet destory. The Context destroy and all of its other listeners for one... other servlets. Other applications. Tomcat itelf. By calling System.exit, you prevent all of that from running.
A better question is what are thse non-daemon threads, why are they running, and who starts them?

When writing thread shutdown code like Jared's, I normally make the "keepGoing" member and "stopExecuting()" method static so that all threads get the signal to go down with one shutdown call. Good idea or no?