I have a system S of IP addresses that generate and receive data via JMS queues and topics. At the time being this system S is configured to connect to a broker with IP x.x.x.x.
I would like to use the same system S against two different brokers: in other terms I would like to use a sort of "network splitter" that receive the traffic from S and forward it to x.x.x.x AND also to another address y.y.y.y (a second JMS broker). The system S must be completely unaware of the splitting operation. The requirement is not to use networks of jms brokers, jms broker proxies or to play with brokers topology in general. The reason for that is to be general enough to manage protocols other than JMS as possible next steps. JMS connections uses TCP+TLS with mutual authentication.
The system S is made of both Linux and Windows servers.
I would like to do this at software level in order to cope with virtualization of both networks and servers.
What is the best strategy to achieve this? Is there specific software/library that you would recommend, preferably in java?
Thanks!
Related
I'm new-ish to networking, and I'm swimming (drowning) in semantics.
I have a VM which runs a Java application. Ideally, it would be fed inputs from the host through a RabbitMQ queue. The Java application would then place the results on another RabbitMQ queue on a different port where it will be used by the host application. After researching it for a bit, it seems like RabbitMQ only exists in the localhost space with listeners on different ports, am I correct in this?
Do I need 2 RabbitMQ servers running in tandem, then, (one on the VM and other on Host) each listening to the same port? Or do I just need one RabbitMQ server running while both applications are pointed to the same IP Address/Port?
Also, I have also read that you cannot connect as 'guest/guest' unless it is on localhost, which I understand, but how is RabbitMQ supposed to be configured/reachable to anything besides localhost?
I've been researching for several hours, but the documentation does not point to a direct answer/how-to guide. Perhaps it is my lack of network experience. If anyone could elaborate on these questions or point me to some articles/helpful guides, I would be much obliged.
P.S. -- I don't even know what code to display to give context. Let me know and I'll edit the code into the post.
RabbitMQ listens to TCP port 5672 on all network interfaces out-of-the-box. This includes the "loopback" interface (to allow fast connections to self) and interfaces visible to other remote hosts (including VMs).
For your use case, you probably need a single RabbitMQ instance for both directions. The application on the host will publish messages to one queue and the Java application in the VM will consume messages from that queue and push the result to a second queue. This second queue can be consumed by the application on the host.
For the user, you need to create a new user with the appropriate rights. This is documented in the access control article. To create the user, you can do it from the management web UI (after you enabled the management plugin) or using the rabbitmqctl command line tool.
The last part is networking between the host and the VM. It really depends on the technology you use. It may work out-of-the-box or you may have to configure how VMs are connected to the network. Refer to the documentation of your hypervisor.
I need to implement analytics system with server and terminals which in realtime.
I use library ZeroMq (pub|sub mode) to send messages to client (~40bytes).
if I connect with 1 client, messages come with delay (sometime more than 250ms).
if I connect with 100 clients a lot of clients lose uniformity of delivery (more than 750ms no one message, after that huge scope of data). It is so critical issue for me.
I have to publish to more than 6000 terminals...
Publish every 30ms, it is about 1700bytes to each client in the worst case (tcp)
Maybe I should use another technology to deliver messages in realtime?
As I said in the comment, Multicast is the way. The primary overriding concern is whether your terminals can join the group you are publishing on - irrespective of how far away they are.
You've not indicated how the terminals connect to your network - (for example vpn over internet, private line whatever..) You asked for a better technology - it's multicast.
Now there are some options if you are going to go down the tcp route:
Build a load-balacing infrastructure which sits in front of your
service. Meaning that your terminals don't connect to your
service, but to a set of load balancers which then connect to your
service. If you have 10 of these for example, each only has to deal
with 600 clients. Your problem is much smaller - you can scale this
way. Don't forget to use asynchronous io.
Buy better hardware - for example solace or tervela do hardware
message brokers which can scale to very large numbers concurrent tcp
connections - but this is not cheap.
Recently I started using ActiveMQ to act as a Message Middleware in my new project, this is the first time I try use ActiveMQ, the projects I had participated before used our previous company's inner message framework like Swallow. So before I begin implementing the system, I need to clear some design points.
Cases in our system will use ActiveMQ include sending mail, sending tasks to queue and doing tasks from queue, asynchronous request/response, so what kind of protocol and network connection is the right choice for our cases? I list some protocols and network connection options here:
ActiveMQ protocols:
MQTT
WS
Openwire
Stomp
Stomp
ActiveMQ Network connections:
VM
TCP
UDP
HTTP
Failover
Discovery
I will also consider the aspects of HA and cluster for my system, so can anybody gives me some ideas to decide how to choose the protocol and network connection?
Thanks a lot.
Openwire has historically been the default protocol the NIO transport can give performance improvements over TCP so if you use ActiveMQ as your only broker use one of these. However using AMQP means in the future you could possibly use RabbitMQ, another popular Message Broker. There are others, STOMP or MQTT are lightweight, VM is designed to be used when the application resides on the same machine as the broker so gets very high throughput.
As ActiveMQ can enable all protocols by default do some quick tests to gain an idea of throughput on the specific application you are building. Then consider the above points in making a decison.
Regarding UDP, TCP, HTTP I would choose TCP. UDP is unreliable and TCP is more than adequate in sending 1000's per second. HTTP could be useful if your company has awkward firewall rules.
I would wrap this in a failover transport. I have never used discovery but would argue this is more advanced and not required initially as it requires a discovery agent. Its only purpose is too discover the ActiveMQ broker dynamically (although you still have to know where the discover agent is).
is probably a stupid question but I can not find answer on the internet. I need to have two separate networks geographically and I would like to communicate with the nodes of a network with nodes of the other network through a broker MQTT. This protocol allows you to do this or are constrained to be on the same network?
thanks
First line on Wikipedia:
MQTT[1] (formerly Message Queue Telemetry Transport) is a publish-subscribe based "light weight" messaging protocol for use on top of the TCP/IP protocol. It is designed for connections with remote locations where a "small code footprint" is required and/or network bandwidth is limited.
With MQTT you can have remote brokers and clients connecting. It has one-to-may message distribution over TCP/IP with a small transport overhead, one advantage of MQTT is that it supports TLS and ACL.
A good introduction post is here, http://jpmens.net/2013/02/25/lots-of-messages-mqtt-pub-sub-and-the-mosquitto-broker/
I'm writting a java application, and I need to quickly discover any other running clients on any wired or wireless local networks in order to establish a TCP connection.
what's the best way of doing this? Are there libraries or code snippets that would do this?
Multicast UDP is a good way of doing this. It's used in a couple of technologies that support automatic discovery of networked devices over local IP networks (UPnP and ZeroConf).
Multicast UDP is not TCP, but it is still based on IP and, so, uses the same addressing mechanism i.e. IP addresses. Quite often it is compared to radio broadcasting i.e. a multicast sender only needs to send 1 message (i.e. it is like a broadcast) but only clients that are "tuned-in" to the multicast channel will receive it.
You can do a quick search on google or wikipedia for these as a starter, but the basic idea is as follows:
when a client starts, it sends out a multicast UDP "hello" message to some pre-specified multicast address and port (e.g. UPnP uses 239.255.255.250:1900)
existing clients are listening for incoming multicast "hello" messages on the specified address and port - when a client receives one, it sends a response to the sender
the client sending the "hello" message receives a response from each existing client on the network and is now aware of the IP address of each client
If you are looking for libraries to use, UPnP libraries can tend to be a bit heavyweight and a lot of folk generally don't like them, so ZeroConf might be a little more suitable. I don't know of any java implementations of such things but I'm sure you can find some with a little digging around.
A network scan can be very long, even longer on wireless networks. If you need them quickly thru Java you may implement a "meeting point" server on your network. This server listen to a predefined port, clients register on the server on startup and the server can distribute information about the clients on request.
HTH.
I guess you need to do a scan on your application's port on all IPs in your subnet.
Just what are the available IPs - or what is your subnet for that matter?
I'm afraid determining that could turn out to be impossible as the network is designed to be transparent to your application.
So, i'd use brute force: pick your IP and change the last byte. Might be too much, might be not enough though.
Or you send a broadcast (which usually would be targeted at x.x.x.255) and see who answers.
See Datagram Broadcasting and Multicasts. But i think that's not TCP/IP anymore.
There is a JGroups toolkit for reliable multicast communications. It allows automatic discovery of additional clients using Multicast techniques as described in other answers.
It also provides communication APIs on top of multicast sockets.
It is used in a number of projects such as JBoss, Tomcat and more to provide an infrastructure for distributed cache. See more here.