I started recently studying Apache KAFKA, for IOT application I am working on. Right now, I am having devices which are already sending data to server using HTTP POST and GET.
After struggling a lot for at-least 5-6 hours. I am not able to figure out how we can send data using HTTP POST to Kafka server.
Also, are there any good real life examples present on web which shows detailed configuration of KAFKA cluster that will be really helpful for beginner like me.
First, you should know that sending messages with HTTP is suboptimal, compared to Kafka native TCP-based protocol.
But let's say that you know all that and still want to use HTTP.
You need to bring up the Kafka REST Server in order to receive messages over HTTP.
Given that you're still learning, maybe the best course of action would be to use Confluent's Kafka distribution that comes with the REST Proxy and has a great documentation.
BTW Confluent is for Kafka what Datastax is for Cassandra, or Hortonworks/Cloudera is for Hadoop. It's a young company founded by the very creators of Kafka from LinkedIn.
Related
I have a central server, to which many distributed servers need to transmit data in the form of somewhat large files, 500MB - 10GB+. The servers are not on the same physical network and can't be connected to one another via a VPN. While we're trying to get other ports opened, currently we can only talk over 443, HTTPS, which is great for our REST services but terrible for file transfer between servers.
I know this isn't as specific a question as one would like for Stackoverflow, but I would like to know: what methods might work better than the ones I've tried?
Server A -> generate file -> transfer over https -> DMZ -> proxy pass -> receive at Server B
Both servers use Java 1.8, Tomcat, and Spring 4.1.4.RELEASE. The DMZ is just Apache and pretty much out of our control.
Things I've tried...
Make RPC calls to a service using Spring's HttpInvokerProxyFactoryBean (this works fine for smaller sites, but the larger sites often drop connections while transferring data)
Multipart form post using Apache HttpPost (this also works, but we have to configure file limits in apache/tomcat, plus its connection is unreliable as well)
Using a library called RMIIO, which basically simulates RMI over HTTP if configured properly. This seemed promising as it requests a stream from the server and writes to the stream from the remote server. I haven't really gotten this to work over HTTPS yet, and the library was written in 2007 (with some updates up through July 2016), but it feels very dated, not highly maintained and I suspect there are better ways to do this sort of thing now-a-days (not that I can find them)
Looked at gRPC but realized it's just a binary protocol and I'd have to basically handle chunking the files if I wanted to get a streaming effect.
Read an article about Developing non-blocking REST services with Spring MVC, http://callistaenterprise.se/blogg/teknik/2014/04/22/c10k-developing-non-blocking-rest-services-with-spring-mvc/ again looked interesting if we were receiving a lot of files at the same time, but I don't see how it helps with a single file transfer.
I've looked at a lot of other things and tried a few more, but it all seems wrong. When I read about big data and Spark streams or any of the million streaming options that I see, I feel like there should be something similar for transferring a single file from one server to another without a broker in the middle. Maybe there are, just not over HTTPS.
It would be nice to know the progress of the transfer (on both ends) and be able to recover should there be connectivity issues or transfer errors.
But any direction or thoughts would be immensely helpful. Thanks for your time and input.
I have just been introduced to the WebSocket Application Messaging Protocol WAMP http://wamp.ws/ and on googling I found the Crossbar.io implementation of the protocol.
I installed it by following http://crossbar.io/docs/Quick-Start/ and http://crossbar.io/docs/Getting-started-with-Java/ .
My goal is to wire a Java server publishing real time messages to be displayed on a web browser. Could you point me in the right direction? Once I have that setup, I can proceed with structuring the data using Ext JS or similar JavaScript frameworks.
The Getting Started with Java has all you need. Write your server-side program using the former. This component will then connect to Crossbar.io which does all the hard work of dispatching events in real-time to clients.
I have this task that I'm undertaking where I would be reading data from a device and make it available over a web service. The data is read 4 times a second. I want the web clients to be have an open HTTP connection and get the device readings as a stream using chunked transfer as long as the client keeps the connection open.
As a proof of concept, I want to start with a service that constantly generates a random number, 4 times a second, wraps it in json and stream that to clients. I'm trying to model it loosely based on twitter streaming api.
I'm using restlet 2.1.2 to create that webservice but I'm not sure which Representation I should be using to achieve this. I tried searching for this but didn't find anything useful. Could someone point me in the right direction as to what I should be using and maybe some examples, perhaps.
Thanks
To achieve what you are trying to do, I'd use the WriterRepresentation (but see my answer to your other question), but I'm quite sure that you are going in the wrong architectural direction.
Indeed the following image from the documentation you linked
shows how even the Twitter streaming api is not intended to be connected by users, but by background processes that download messages in a store accessible by the HTTP. Users poll only the HTTP server, that reads the messages from the store and sends the back to the clients.
As a disconnected protocol, HTTP enable massive scalability that would not be possible otherwise. If each client establishes a persistent TCP connection backed by a dedicated server thread, you will rapidly exaust server resources! Moreover any HTTP proxy between the User Agent and the server could cause unexpected behaviours.
Thus, if you are bound to the HTTP protocol, the User Agent should poll. You can reduce the network load with headers like Last-Modified/If-Modified-Since or Etag/If-None-Match.
However, if you can adopt a different protocol, I strongly suggest to try a service bus over a connected TCP protocol.
Purpose:
I'm trying to make a real-time web service in Java. When there're changes in database, the data in clients(probably Android client) will be changed automatically without refreshing.
Platform:
I'm using Tomcat 7 container and Axis2.
Problems:
I'm stuck in the way to approach. I've realized that there were 2 ways:
1. When the data in server changes, notice to all the clients. Web-service allows us to do this, but I don't know what clients I have to inform!
2. Clients connect to the server, keep the connection alive and whenever there's a change, update data. However, if the client is the handheld, it will consume much power and data exchange over 3G network.
I know that's a complicated problem. So I just want to ask if there is an efficient way to approach? And if possible, anyone can give me one example of realtime web service like this?
I've searched over many pages, but nothing's helpful. Can anyone give me some suggestions? Tkx!
Is it a requirement to use webservices or Axis? Otherwise, you might be interested in WebSockets. These are connections over plain HTTP that provide two way communication. Your clients will connect to your server, and as soon as new data is available your server can push data to every connection it knows of, thus informing all clients that are connected.
I'm looking to establish a low latency 2way communication between a javascript interface (client) and a java server.
The client has to request data from the server (can ask for different set's of data, needs to be async, data are small sets of sensor data).
I was thinking of implementing this using websockets because of it's low latency. However I'm stuck at choosing a java websocket server implementation (I found jetty but there are so many, and there is also a case to be made for node.js and socket.io but there are not going to be a lot of clients in this case; just one client sending multiple requests so correct me if i'm wrong but there doesn't seem to be a reason for going the node.js path).
Last but not least; the server is running on a raspberry pi and is recieving it's sensor data over a special protocol; (but i don't think that's important for this question).
Is there anyone with some experience in this field and wants to share his/her toughts? Thx.
I've been using Kaazing (HTML5 edition) to proxy traffic received via a web-socket to a Java process listening on a traditional TCP server socket.
It's working well, latency is low and was consistently handling over 1000 messages/second (though we found our Java code was the limiting factor in that respect).
Kaazing also provides client APIs for Java, JavaScript and Flex, which allowed us to write an acceptance test suite using the familiar APIs (Concordion in my case).
I don't know how well it'd run on a Raspberry Pi, but given it's free to download there's a simple way to find out.
I've solved my problem by using Atmosphere which is a framework that provides compatibility to all major java servers and web browsers. (The Java Official Standard is still in the workings).
https://github.com/Atmosphere/atmosphere
I've got the chat demo up and running.