I need to develop a platform in Java to download tweets from Twitter (that was obvious). The idea is to have various computers downloading from the streaming API and a main controller to send tasks (keywords to download and other data) to each fetcher. My problem is related with the connection between this programs. Which is the best way to do this? Actually I'm using RMI to send commands like "stop", "start", "setTask" from the Controller (client) to each fetcher (servers) and a SSLSocket to make a quick validation, but I'm not sure if this is a good idea. I could use TCP sockets but maybe it's not a good idea to have permanent connections. What do you think? Is it a good idea to keep using RMI or should I take another point of view?
Thank you ;)
I propose you to use queue (and any queue protocol).
ActiveMQ, RabbitMQ, QPID, or one of many other tool.
I use ActiveMQ in prod and fine with it, but for very highroad RabbitMQ will be better.
you receive easy scaling for any count of workers and easiest way to share/split tasks between workers.
Also please look on ActiveMQ or RabbitMQ or ZeroMQ or
Related
I need to record from an unreliable web socket connection and stream into Kafka.
Our Kafka cluster is pretty reliable and we can make it highly available.
What is the best approach to make the web socket connection as reliable as possible? I would like to minimize data loss.
One solution would be to have multiple processes or web socket clients listening and streaming to multiple Kafka topics. Then do a filter with Kafka streams. This only works if each message that I obtain has a unique id, which is not always the case.
Another solution would be to monitor the web socket connection and restart or reset it. But then I might have data loss. Or rely on web socket heartbeats?
Or code my own error handlers?
What frameworks / libraries from the Java are in this space to do the best job? Currently, I use the web socket client from org.java-websocket.
I think this is a pretty standard use case in web socket and also in Kafka producer development, so I am sure I do not need to reinvent the wheel.
I am designing a Real Time backend chat application for mobile devices and to do this I am building everything over Java (to deal with incoming HTTP requests ) and Redis (Pub/Sub). Now I am looking for a Worker and already took a look at tools like Resque, Python-RQ and even Celery (also offers Redis integration), but maybe things grow and remain difficult to manage. I want to keep things as simple as possible. Has anyone tried to use Jedis (redis java client) to listening messages from a Redis channel and started a new Thread to each message received? Was the performace bad? what if a had hundreds of requests per second? It seems a poor solution (simple Thread as a Worker)
The flow is (for android example):
Android client send a message to chat
My Rest webservice (tomcat) receives the message and publish (jedis) a message to a Redis channel [quite simple]
The Worker (?) process the message and deliver it to all subscribers over Google Cloud Message (simple http request)
So, any suggestions or experiences about Redis workers implementations or Jedis library? What do you recommend? Thanks.
For those who wants a suggestion:
I opted for Python-RQ because of its simplicity. Too simple, well documented and solved my problem.
Regards.
In our new project we need to implement a server application. This server gets connection requests of 50,000(+) clients. Problem is these connections have to remain open and have to be managed somewhere. The application should work like a telephone exchange. So it can get requests of connected clients and connect them to other (maybe several) clients only if they are also connected. A proprietary protocol is used. My questions are:
How (and where) to manage the open sockets? Should I put them in a HashMap or something? This sounds curious to me. But I don't have experiences with so many open connections.
Are there any frameworks available which support this connection requirements?
Thank you for your help!
How (and where) to manage the open sockets? Should I put them in a HashMap or something?
Typically each socket will be managed by a thread that will be responsible for reading and writing to the socket. You would also have a master thread that is responsible for receiving all connection requests at a predefined network interface & port (using the ServerSocket API class), which may then hand off the actual processing work to the worker/slave threads. In this case, you ought to be looking at a thread pool for the worker threads, because creating 50k threads will most likely overwhelm your OS and the hardware.
Also, if you are indeed managing 50k concurrent sockets, using NIO API (java.nio.*) over the plain IO API of Java is highly recommended, although I haven't seen too many projects requiring more than 2-5k concurrent connections. There are atleast two known NIO based frameworks in the Java world - Apache MINA and JBoss Netty. I would however recommend reading the well written NIO tutorial, before heading onto use the NIO API or the NIO frameworks.
I'm writing a multiplayer/multiroom game (Hearts) in java, using RMI and a centralized server.
But there's a problem: RMI Callbacks will not work beacause clients are Natted and Firewalled. I basically need the server to push data updates to clients, possibly without using polling and without using sockets (I would code at an higher level)
In your opinion, what's the best solution for realizing this kind of architecture? Is an ajax application the only solution?
You say that you don't want polling, but AJAX is exactly that. You can look at Comet but it's hard to escape polling anyway (e.g. Comet itself uses polling underneath).
You could use a peer to peer framework such as JXTA.
I can suggest two main techniques.
The server has a method getUpdates, callable by clients. The method returns the control to the client when there is an update to show.
When Clients perform the registration, they give the server a callback remote object
Since this object is not registered in any rmi registry, there should no be any issue with natted clients.
I'm not sure how(if) ajax works for a non-browser-based app. You could just maintain your own pool of SocketConnections open for the duration of the application with a Thread per connection.
If you need to scale to a lot of concurrent connections, look to a non-blocking I/O framework like Apache Mina or Netty (related SO post: Netty vs Apache MINA).
I have to design a client/server system emulated on a website running Ruby on Rails that should work like this:
a page is requested by a web browser and once it's opened the server can push messages to it
I know this is not possible "naturally" but I was thinking of a sort of "java applet" that is running on that page, listening on a port for messages to be sent by the hosting server. This should be done opening a sort of a socket that listens on some port where the server can connect to send its messages.
Can this be done? Do I have to develop a java server thread or can I simply address the client applet via it's ip address and port and use any web service connection from the server?
thanks,
Luca
Comet is definately what you want. Depending on your needs, you can host your own comet server, or use a SaaS solution, such as WebSync On-Demand (disclaimer: I work there). Using the SaaS stuff, you get server-push capabilities without having to actually run your own comet server.
The easiest way to do that is to use Javascript to emulate the push mechanism. Polling in regular intervals using AJAX is sufficient in most cases. Have also a look at Comet.
An alternative to using a java applet may be to use a combination of javascript and an approach known as Comet. In a nutshell, Comet is a way to enable server push over HTTP. I'm not really a ruby on rails guy, but a quick google search for ruby on rails and comet nets a fair amount of useful information.
have you looked at juggernaut
If you want go the applet route, you need to make the connection from applet to the same server where the web page is served. The applet can't listen. Once the TCP connection is established, it's a 2-way channel, you can pull or push as long as your protocol allows it. This is how it's done with most Applet-based chat clients.
More and more people are simply using long polling in Javascript. It's pretty involved to get a reliable long polling system running, I would suggest you to use a framework. For example,
http://cometdproject.dojotoolkit.org/