I am working on a project where I need to get the native stack of the Java application. I am able to achieve this partially thanks to ptrace, multiprocessing, and signals.
On Linux, a normal Java application has, at a minimum, 14 threads. Out of these 14, I am interested in only the main thread of which I have to get the native stack. Considering this objective, I have started a separate process using fork() which is monitoring the native stack of the main thread. In short, I have 2 separate processes: one is being monitored and the other does the monitoring using ptrace and signal handling.
Steps in the monitoring process:
Get the main thread ID out of the 14 threads from the monitored process.
ptrace_attach on the main ID.
ptrace_cont on the main ID.
continuous loop starts
{
kill(main_ID, SIGSTOP)
nanosleep and check the status from the /proc/[pid]/stat directory.
ptrace_peekdata to read the stack and navigate.
ptrace_cont on the main ID.
nanosleep and check the status from the /proc/[pid]/stat directory.
}
ptrace_detach on the main ID.
This perfectly gives the native stack information continuously. However, sometimes I encounter an issue:
When I kill(main_ID, SIGSTOP) the main thread, the other threads from the process get into a finished or stoped state (T) and the entire process blocks. This is not the consistent behavior and sometimes entire process executes correctly. I cannot understand this behavior as i am only signaling the main thread. Why are the other threads affected?
Can someone help me analyze this problem?
I also tried sending SIGCONT and SIGSTOP to all of the threads of the process but the issue still occurs sometimes.
Thanks,
Sandeep
Assuming you are using Linux, you should be using tkill(2) or tgkill(2) instead of kill(2). On FreeBSD, you should use the SYS_thr_kill2 syscall. Per the tkill(2) manpage:
tgkill() sends the signal sig to the thread with the thread ID tid in
the thread group tgid. (By contrast, kill(2) can only be used to send
a signal to a process (i.e., thread group) as a whole, and the signal
will be delivered to an arbitrary thread within that process.)
Ignore the stuff about tkill(2) and friends being for internal thread library usage, it is commonly used by debuggers/tracers to send signals to specific threads.
Also, you should use waitpid(2) (or some variation of it) to wait for the thread to receive the SIGSTOP instead of polling on /proc/[pid]/stat. This approach will be more efficient and more responsive.
Finally, it appears that you are doing some sort of stack sampling. You may want to check out Google PerfTools as these tools include a CPU sampler that is doing stack sampling to obtain estimates of what functions are consuming the most CPU time. You could maybe reuse the work these tools have already done, as stack sampling can be tricky to make robust.
Related
I know this subject has been discussed here before, and we have utilized past conversations to attempt to resolve the DbMaxReadersExceededException that we are still experiencing. We are using version 2.5.1 of ObjectBox. We are also, heavily, using RxJava threads while manipulating our BoxStore DB. At any moment in time, potentially a handful of RxJava threads are running, accessing the DB. Threads are constantly spawning, executing and terminating.
This is a very "non-standard" use of Android. Our App is running on a non-cell phone device, that sits on a wall and is expected to be available 24x7. 95% of the RxJava threads that access the BoxStore DB are short lived, get in / get out threads, that retrieve information and present to the device user. We do have a few longer lived background RxJava threads, that talk to an external DB over the internet to keep the local DB up to date. But these threads to spawn, execute and terminate. Theses threads run in the background at regular intervals. These background threads are not associated with a Fragment nor Activity; therefore the common way of cleaning up, using a CompositeDisposable, is not utilized.
We are seeing that readers are accumulating, despite many attempts to resolve the situation. We have also noticed that threads, that have run to termination, marked as isAlive and appear to be part of the RxJava thread pool, also accumulate. We have observed this using Thread.getAllStackTraces() and printing out this information regularly. Separate issue I am not trying to resolve with this post (I am concentrating on the DbMaxReadersExceededException issue, but they may be related).
The readers accumulate as the result of .find() calls on a Query that is build; based upon analysis of when a reader occurs. That is not surprising, but sometimes a .find() causes a new reader and sometimes it does not. I do not understand this behavior, and I am not sure if that is a telling sign or not. But it does result in the accumlation of active readers everytime the RxJava thread that accessed a given Box is invoked.
Any help / assistance offered will be greatly appreciated. Please ask any questions about anything that I may have accidental left out.
Things that we have tried, based upon other posts that I have read, include:
Collect Disposables from RxJava background threads and dispose
We have tried collecting the Disposable generated by the .subscribe() from these background threads, and added a timer to .dispose() of them sometime (5 seconds) after the thread that was using this object terminates (run to completion).
Utilized BoxStore.diagnose()
We have written code to utilize BoxStore.diagnose() to be able to periodically watch the reader accumulation.
Tried BoxStore.closeThreadResources()
We have added BoxStore.closeThreadResources() calls when an RxJava thread terminates to cleanup any BoxStore resources that may be active.
Tried Box.closeThreadResources()
We have tried adding Box.closeThreadResources() calls closer to when the Box is accessed in order to access and then clean up ASAP.
Tried breaking down .method() sequence and added .close() calls to itermediate objects
We have tried breaking down the .method() call sequence that terminates with the .find() call and then .close() or .closeThreadResources() the intermediate objects along the way.
Tried combinations of the above
We have tried a combination of all of the above.
Wrote method to be able to monitor RxJava threads using Thread.getAllStackTraces() - RxJava threads seem to accumulate
We have written a method that helps us monitor RxJava threads using Thread.getAllStackTraces().
We have tried to manually invoke the Garbage Collector
We added code, after the .dispose(), mentioned above, to cause a manual Garbage Collection (System.gc()).
As far as I know, we have tried every suggestion that I have seen posted on this and other forms, regarding this issue. We are at a loss as to what to do or try next. I did see something about a package called RxObjectBox, but I have not pursued this any further.
Should we:
Look at restructuring our RxJava thread access?
Do we need to look closer at RxObjectBox?
Is there a known problem with ObjectBox 2.5.1 that we should be using a later version?
What haven't we tried that we should?
I am working on a design where I am processing all the exceptions caught in catch blocks to send to a server via webservice call.
The idea is not to block the main app at all while doing this job. I felt blocking queue pattern is appropriate for it. So, I created a blocking queue using array implementation with size 10. At the start of the main app I am initializing a consumer thread for this queue.
However, the producer side is little confusing to me. as per my understanding if, queue is full and if the main app hit an exception then doing a producer.put(object) would be blocked until queue has space and hence the main app will block too. is that correct understanding?
Yes, you are correct. Here is a very useful table of BlockingQueue methods
Usually it is good to have a bounded queue, but the limit should not be very low.
I think, you should write your exception to phone storage(SharedPreferences if Android) instead of keeping in main memory.
First, it will not block your main app.
And on network connected call back, start a thread which will read from shared preferences and send it to the server.
I built a small video frame analysis app with desktop Java 8. On each frame, I extract data (5 doubles now, but could expand to a 1920x1080x3 OpenCV Mat in the future). I would like to store this data into a database (Java DB, for example) to perform some time-series analysis, and periodically return the results to the user.
I am worried about hard-drive access times if I write to the database and run the app on a single thread, and the best solution that occured to me would be to implement the producer/consumer pattern with multithreading. The examples I found all implement 3 threads:
the main thread
the producer thread
the consumer thread
Is there an advantage in doing that compared to a 2 thread implementation?
main and producer thread
consumer thread
And is that the right way to handle real-time data with a database?
It's limiting to use a fixed number of threads. My PC has (only) 8 cores, your intensive sounding app is not going to use half of them, indeed probably only the consumer is the intensive one, so maybe 12.5%. You'll have to have several of each thread to get the most out of the CPU, and then you'll spend a lot of effort managing threads.
The alternative is to use one of various existing systems for executing work in the background. For example ThreadPoolExecutor With that you can just throw lots of work at it (Runnables) and it will queue work up, and execution can be scaled to suit the hardware it's running on by customizing the number of worker threads.
Or if you're using Swing, then SwingWorker. The advantage of this is you can do some work on a background thread and post the results on the foreground (main/UI) thread easily.
Your question is rather conceptional, so I think it belongs here: Programmers
But as one short hint from my experience, you separate the producer from the main because your main control may freeze if something goes wrong with the producer. Things like frozen forms, not responding controls etc. may be the result. Give your system a chance to reestablish by command.
How to set limit to the number of Thread that someone can create? What I do is running someone's code (something like ideone), and want to limit number of thread that he can spawn. How to do so? Some jvm setting or something else?
EDIT
I add more specified info because some people are not gettin my point.
Some random guy send me a code which my computer is going to execute
Code must be execute within using maximum of k threads
All must be automated - working like SPOJ, ideone, etc.
On Linux, you could run the program as a separate user and use the shell command ulimit -u nprocs to limit the number of threads (processes) for that user. If an attempt is made to exceed the limit, the JVM will throw an OutOfMemoryError.
But why do you want to do this? Are you worried that the program will consume all the CPU resources of the computer? If so, you might want to consider running the JVM at lower scheduling priority, using nice, so other processes will get preferential use of the CPU:
NPROCS=100 # for example
NICENESS=13 # for example
ulimit -u $NPROCS
nice -n $NICENESS java ...
Using nice in that manner should reduce the priority of all the threads, but it is not clear that it does so for Linux.
You can create your own subclass for thread that performs the desired checking in the constructor(s) or in the start method.
To ensure the code you are running uses your custom thread class, you must load the code with your own custom class loader and that class loader simply catches any request for the java.lang.Thread class and hands out your custom class instead (the concept can be extended to other classes as well).
Warning: Implementing this properly is not trivial.
AFAIK,Limit is purely depends on OS not on JVM.
And you can Monitor them by a Executor service
An Executor that provides methods to manage termination and methods that can produce a Future for tracking progress of one or more asynchronous tasks.
ExecutorService pool = Executors.newFixedThreadPool(n);
We have a C++ application with an embedded JVM (Sun's). Because we register our own signal handlers, it's recommended we do so before initializing the JVM since it installs its own handlers (see here).
From what I understood, the JVM knows internally if the signal originated from its own code and if not it passes it along the chain - to our handlers.
What we started seeing is that we're getting SIGPIPEs, with a call stack that looks roughly like this (the top entry is our signal handler):
/.../libos_independent_utilities.so(_ZN2os32smart_synchronous_signal_handlerEiP7siginfoPv+0x9) [0x2b124f7a3989]
/.../jvm/jre/lib/amd64/server/libjvm.so [0x2aaaab05dc6c]
/.../jvm/jre/lib/amd64/server/libjvm.so [0x2aaaab05bffb]
/.../jvm/jre/lib/amd64/server/libjvm.so(JVM_handle_linux_signal+0x718) [0x2aaaab05e878]
/.../jvm/jre/lib/amd64/server/libjvm.so [0x2aaaab05bf0e]
/lib64/libpthread.so.0 [0x3c2140e4c0]
/lib64/libpthread.so.0(send+0x91) [0x3c2140d841]
/.../jvm/jre/lib/amd64/libnet.so [0x2aaabd360269]
/.../jvm/jre/lib/amd64/libnet.so(Java_java_net_SocketOutputStream_socketWrite0+0xee) [0x2aaabd35cf4e]
[0x2aaaaeb3bf7f]
It seems like the JVM is deciding that the SIGPIPE that was raised from send should be passed along to our signal hander. Is it right when doing so?
Also, why is the call stack incomplete? I mean obviously it can't show me java code before socketWrite0 but why can't I see the stack before the java code?
The JVM can't tell whether the SIGPIPE came from it's own code, or your code. That information just isn't given by the signal. Because it doesn't want you to miss out on any possible events that you could be interested in, it has to pass you all SIGPIPEs, even the ones that it turns out were from its own code.
Unix signals come in two flavors -- "synchronous" and "asynchronous". A few exceptional conditions when just executing code can cause traps and result in "synchronous" signals. These are things such as unaligned memory access (SIGBUS), illegal memory access, often NULLs, (SIGSEGV), division by zero and other math errors (SIGFPE), undecodable instructions (SIGILL), and so forth. These have a precise execution context, and are delivered directly to the thread that caused them. The signal handler can look up the stack and see "hey I got an illegal memory access executing java code, and the pointer was a NULL. Let me go fix that up."
In contrast, the signals that interact with the outside world are the "asynchronous" variety, and include such things as SIGTERM, SIGQUIT, SIGUSR1, etc. These do not have a fixed execution context. For threaded programs they are delivered pretty much at random to any thread. Importantly, SIGPIPE is among these. Yes, in some sense, it is normally associated with one system call. But it is quite possible to (for instance) have two threads listening to two separate connections, both of which close before either thread is scheduled. The kernel just makes sure that there is a SIGPIPE pending (the usual implementation is as a bitmask of pending signals), and deals with it on rescheduling any of the threads in the process. This is only one of the simpler cases possible where the JVM might not have enough information to rule out your client code being interested in this signal.
(As to what happens to the read calls, they return "there was an error: EINTR" and continue on. At this point, the JVM can turn that into an exception, but the return happens after the signal delivery and the signal handler fires.)
The upshot is you'll just have to deal with false-positives. (And deal with getting only one signal where two might have been expected.)