I'm often need to organize something like critical section where multiple thread might access simultaneously but not more than X threads with the same key.
Simplified example:
thread pool to process the device commands from queue;
queue contains commands for different devices;
while one thread process device command other threads that process commands for the same device should wait their turn.
I know this could be achieved in different ways and you could say that there are architectural refactoring required and so on but this is the simplest example I could think.
Edit: I don't need the solution for this particular example, I need some library which allows to restrict access based on some key for example device IP/MAC string. For example: getLock().lock(deviceIP)
I have my own code to do this but I wounder did I miss something? Is there some standard Java SE 8 components or third-party libraries to do this?
Related
We are working on rewrite of an existing application, and need support for high number of read/write to database. For this, we are proceeding with sharding on MySQL. Since we are allowing bulk APIs for read/write, this would mean parallel execution of queries on different shards.
Can you suggest frameworks which would support the same in Java, mainly focussing on split-and-aggregate jobs. Basically I will define two interfaces ReadTask and WriteTask, and implementation of these tasks will be jobs and they would be submitted as a list for parallel execution.
I might not have termed this question in the right way, but I hope you got the context from the description. Let me know if there is any info needed for answer.
BLUF: This sounds like a common processing pattern in Akka.
This sounds like a Scatter-Gather patterned API.
If you have 1 job, you should first answer if that job will touch only one shard or more? If it will touch many shards you may choose to reject it (allowing only single-shard actions) or you may choose to break it up (scatter) it across other workers.
Akka gives you APIs, especially the Streaming API, that talk about this style of work. Akka is best expressed in Scala, but it has a Java API that gives you all the functionality of the Scala one. That you are talking about "mapping" and "reducing" (or "folding") data, these are functional operations and Scala gives you the functional idioms.
If you scatter it across other workers, you'll need to communicate the manifest of jobs to the gather side of the system.
Hope that's helpful.
You can use the ThreadPoolExecutor & Executors(factory) in Java to create Thread pools to which you can submit your read & write tasks. It allows for Runnable & Callable based on your situation.
I am having trouble developing a chat application including file sharing. It's a server and client based desktop application. I am able to develop chat and also file sharing individually.
Now I want merge two project and having problem. Is it possible to use multiple socket in single thread like main thread? Or do I have to use one for chatting and another for file sharing?
I think that is possible but in the same thread one operation with one socket have to wait all the operations from annother finish. if you want the two operations execute at the same time do it in different threads.
Hope that helps :)
Yes it is possible for a single thread to use multiple sockets. See the Selector API.
The basic idea is that the code in your thread needs to know about all of the selectable channels (sockets, open files, etc) that it needs to read from or write to. It registers the channels with the selector. Then you make a select call which will (for example) block until one of the registered channels is ready; e.g. a socket that has data that can be read now ... without blocking.
It is complicated, but rather than explaining it all here, I suggest that you read Baeldung's Introduction to the Java NIO Selector which includes an example.
But the bottom line is that using Selector will probably entail significant restructuring of your existing code. And if your code is using (for example) a 3rd-party library to do the I/O, then this approach probably won't work. Using multiple threads may well be a simpler option.
I'm used to C++/Qt's concept of signals (emit/listen) and now I'm doing a project in Java which requires some sort of data sending/receiving mechanism.
My needs are:
Emit an event (with some data) and let all threads listen/catch it.
Obviously, given the previous requirement, being able to listen/catch signals with attached data.
Is this possible in Java, and how? (I'll appreciate a small compilable example/link)
Java by default doesn't have a simple event handling mechanism such as .Net's events or Qt's Signals and Slots. It does have the notion of Listeners in various java GUI frameworks but I don't think that's what you're looking for.
You should consider a pub-sub library like Google Guava's EventBus framework.
If you don't want to use a third party lib then I suggest you start looking into using one of the sub-classes of BlockingQueue. See the FileCrawler example from page 62 of Java Concurrency in Practice to see how to use a BlockingQueue to send events/data to worker threads.
If you're looking for a more complicated solution for message/event notifications across the process boundary or the local machine boundary then you may want to look into:
RabbitMq
Redis
JMS
not sure if this will match your exact query but have you tried CountDownLatch or CyclicBarrier?
http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/CountDownLatch.html
http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/CyclicBarrier.html
this is a bit related to this question.
I'm using make to extract some information concerning some C programs. I'm wrapping the compilation using a bash script that runs my java program and then gcc. Basically, i'm doing:
make CC=~/my_script.sh
I would like to use several jobs (-j option with make). It's running several processes according to the dependency rules.
If i understood well, I would have as many instances of the jvm as jobs, right ?
The thing is that i'm using sqlite-jdb to collect some info. So the problem is how to avoid several processes trying to modify the db at the same time ?
It seems that the sqlite lock is jvm-dependant (i mean one lock can be "see" only inside the locking jvm), and that this is the same for RandomAccessFile.lock().
Do you have any idea how to do that ? (creating a tmp file and then looking if it exists or not seems to be one possibility but may be expensive. A locking table in the dB ? )
thanks
java.nio.channels.FileLock allows OS-level cross-process file locking.
However, using make to start a bash scripts that runs several JVMs in parallel before calling gcc sounds altogether too Rube-Goldbergian and brittle to me.
there are several solutions for this.
if your lock should be within the same machine, you can use a server socket to implement it (The process that manages to bind to the port first owns the lock, other processes waits for the port to become available).
if you need a lock that span across multiple machines you can use a memcached lock. this will require a memcached server running. I can paste some code if you are interested in this solution.
you can get Java library to connect to memcached here.
You may try Terracotta for sharing objects between various JVM instances. It may appear as a too heavy solution for your needs, but at least worth considering.
Without having the source code for a Java API, is there anyway to know if the API methods create multiple threads ? Are there any conventions to follow if you are writing Java APIs and they create multiple threads. This may be very fundamental question but it happened to spawn out of a discussion in which the crux question was - " How do you know which Java APIs create threads and which don't " ?
One way of determining which libraries create new threads is by disallowing Thread creation and ThreadGroup modification in the SecurityManager. See the java.lang.SecurityManager.checkAccess(Thread) method. By implementing your own SecurityManager, you are able to react on the creation of Threads.
To answer the other question: many libraries create new threads, even if you don't expect it. For example APIs for HTTP communication create Timers for Keep-Alives or session timeouts. Java 2D is creating a signalling thread. Java itself has multiple threads, e.g. the Finalizer thread; the AWT/Swing event dispatcher thread etc.
There's no way to tell. Actually, I don't think you normally would care that much unless you're in some kind of constrained environment. What's I've found is more relevant is to determine if a method is written with an expectation of being run on a particular thread (the AWT Event dispatch thread, in the case I've seen). There's not a way to do that either, unless the code is using some kind of naming convention, or it's documented.
In my experience, if you are looking at core java, not J2EE, the only time I can think that threads are created in core Java is with Swing.
I haven't seen any example of other threads being created by the core Java APIs, except for the Thread class, of course. :)
But, if you are using other libraries then it may be that they are creating threads, but, if you don't want to profile, you may want to use AspectJ to log whenever a new thread is created, and the stack track of what called it, so you can see what is creating the threads.
UPDATE:
Swing uses 4 threads, according to this post, but he also explains how you can go about killing off the threads, if needed.
http://www.herongyang.com/Swing/jframe_2.html
If you want to see active threads, just fire up the jvisualvm application (located in your $JDK/bin directory) to connect to any local java process. You'll be able to see a multitude of information about the process, including thread names, status, and history. Get more information here.