I've got the following problem to solve:
There are two jar files. These jar's start independently from each other.
Now, let's say the first jar A.jar calculates or computes something and has to commit the results to B.jar.
I tried to communicate via a central Singleton (enum Singleton and a Singleton that uses a own classloader as mentioned here: Singleton class with several different classloaders).
But it didn't seem to work for me. When i start the two jar's the hashcode of the instances are different.
Can anyone tell me what i'm doing wrong? Or other ideas how to solve my problem?
There are two jar files. These jar's start independently from each
other.
So they're separate processes. They won't share instances of classes, variables etc. You need some form of inter-process communication to communicate between them.
That would normally mean some form of network protocol (e.g. TCP/UDP sockets, HTTP etc.). You could also do something really simple like reading/writing a shared file (that's not particularly nice, but it is straightforward for simple cases)
If you are running 2 jar files separately, then each jar file runs its own JVM instance and nothing is shared between them. They are 2 seperate process. Full-Stop.
If you wish to communicate between them then here are alternatives:
It depends on what kind of data you wish to transfer.
If it is only Strings, then:
if number of process = 2 and if you are sure of it, then stdin &8 stdout is the best way forward. One process can start run another Jar file by creating a Process using ProcessBuilder and then using the streams to communicate. The other process can just do System.out to transfer message. This is preferred to Socket, because you dont have to handle graceful closing of socket etc. (In case it fails and the port is not un-binded successfully, it can be a big trouble)
if number of jar files > 2 (i.e. number of total processes) and less than say 10, you can probably use Sockets and communicate through Socket. This should work well, though extra effort goes in gracefully managing sockets.
if number of process is Large, then JMS should be used. It does a lot of things which you dont need to handle. Too big a task if the number of processes are less.
So in your case, process is the best way forward.
If the data you wish to transfer, can even be Objects. RMI can be used given the number of processes are less. If more, use JMS again.
Edit: Now for all the above, there is a lot of dirty work involved. For a change, if you are looking at something new & exciting, I would advice akka. It is a actor based model which communicate with each other using Messages. The beauty is, the actors can be on same JVM or another (very little config) and akka takes care the rest for you. I haven't seen a more cleaner way than doing this :)
Related
I have written a single Kafka consumer (using Spring Kafka), that reads from a single topic and is a part of a consumer group. Once a message is consumed, it will perform all downstream operations and move on to the next message offset. I have packaged this as a WAR file and my deployment pipeline pushes this out to a single instance. Using my deployment pipeline, I could potentially deploy this artifact to multiple instances in my deployment pool.
However, I am not able to understand the following, when I want multiple consumers as part of my infrastructure -
I can actually define multiple instances in my deployment pool and
have this WAR running on all those instances. This would mean, all of
them are listening to the same topic, are a part of the same consumer
group and will actually divide the partitions among themselves. The
downstream logic will work as is. This works perfectly fine for my
use case, however, I am not sure, if this is the optimal approach to
follow ?
Reading online, I came across resources here and here,
where people are defining a single consumer thread, but internally,
creating multiple worker threads. There are also examples where we
could define multiple consumer threads that do the downstream logic.
Thinking about these approaches and mapping them to deployment
environments, we could achieve the same result (as my theoretical
solution above could), but with less number of machines.
Personally, I think my solution is simple, scalable but might not be optimal, while the second approach might be optimal, but wanted to know your experiences, suggestions or any other metrics / constraints I should consider ? Also, I am thinking with my theoretical solution, I could actually employ bare bones simple machines as Kafka consumers.
While I know, I haven’t posted any code, please let me know if I need to move this question to another forum. If you need specific code examples, I can provide them too, but I didn’t think they are important, in the context of my question.
Your existing solution is best. Handing off to another thread will cause problems with offset management. Spring kafka allows you to run multiple threads in each instance, as long as you have enough partitions.
If your current approach works, just stick to it. It's the simple and elegant way to go.
You would only go to approach 2 in case you cannot for some reason increase the number of partitions but need higher level of parallelism. But then you have ordering and race conditions to worry about. If you ever need to go that route, I'd recommend the akka-stream-kafka library, which provides facilities to handle offset commits correctly and to do what you need in parallel and then merge back into a single stream preserving the original ordering, etc. Otherwise, these things are error-prone to do yourself.
There is an application that will need to have something like a look up table. This application can be started many times with different configurations. Is there a way to share a datastructure across JVMs. static would be valid within a JVM. Having a database would solve the issue. However, is there something simpler and fast?
You might use a file. Write the object to a file. There is no such thing as an object shared within JVMs because the life cycle of an Object is defined for and within a JVM.
File IO is usually faster than DB operations and simpler as well. But on the downside, ACID properties are not guaranteed by files and there could be inconsistencies if multiple processes try to read / write on the same file.
I have created a java application that frequently gathers data from a source and saves it to an sql database. I am also developing another application to operate on the data that is being saved to the database.
My problem is that I will be continuously modifying the operating classes but want to leave the gathering application up all the time so as not to miss any data, but the operating class needs to connect to the gathering class, and I am not sure exactly what the best way to do this is.
If possible I want the two as well connected as possible, just as if one was constructed by the other. What is the best way to do this in java? I am looking into RPC but not sure if it will be as fast as the above scenario.
Any advice would be greatly appreciated!
If you use an OSGi container like karaf you can update jars used in a running program. It has a web interface and command line interface to help you do that.
Another option is to have two programs where one talks to the other. This way you can restart one with a new version of code without stopping the other.
I started trying to learn tdd and wanted to use it on a real project. So I decided on programming a simple file synchronization with a client and a server in java.
After more or less finishing the client part, I got stuck while writing the server part.
In order to unit test the logic without accessing external resources, I put those in separate classes, so that I could mock them. So far, so good.
Now here's my question:
This image shows how I imagined everything could look like, where green parts are finished and the yellow parts are not yet implemented.
In my case I'd have to pass RemoteServer a ConnectionManager. The ConnectionManager would a need a FileAdapter and a Communicator in order to create a ClientConnectionHandler. That sounds like a little bit too much passing for me. Is this normal for tdd'ing or am I doing something wrong in order to keep everything testable?
edit: The class ClientConnectionHandler is only responsible for the file sync logic on the server side, which means: Following my own mini protocol to receive files from a client.
It's pretty normal. If you get one class that has many dependencies, it likely has too many responsibilities as well. But having one class that depends on another class that depends on another - that's normal.
This is why IoC Container libraries like Spring are popular. They make it easier to wire up (configure and resolve) all of the dependencies.
I have several Beans in my Application which getting updated regularly by the usual setter methods. I want to synchronize these beans with a remote application which has the same bean classes. In my case, bandwidth matters, so i have to keep the amount of transferred bytes as low as possible. My idea was to create deltas of the state changes and transfer them instead of the whole Objects. Currently, I want to write the protocol to transfer those changes by myself but I'm not bound to it and would prefer an existing solution.
Is there already a solution for this Problem out there? And if not, how could I easily monitor those state changes in an generalized way? AOP?
Edit: This problem is not caching related even it may first seem so. The data must be replicated from a central server to several clients (about 4 to 10) over the internet. The client is a standalone desktop application.
This sounds remarkably similar to JBossCache running in POJO mode.
This is a distributed, delta-based cache that breaks down java objects into a tree structure, and only transmits changes to the bits of the tree that changes.
Should be a perfect fit for you.
I like your idea of creating deltas and sending them.
A simple Map could handle the delta for one object. Serialization could simply get you the effective message send.
To reduce the number of messages that would kill your performance, you should group your deltas for all objects and send them as a whole. So you could have others collections or maps to contain this.
To monitor all changes to many beans, AOP seem like a good solution.
EDIT : see Skaffmann's answer.
Using an existing cache technology could be better.
Many problems could already have solutions implemented...