I am consuming a message feed using the below code from a third party message broker. We use the STOMP protocol and the code is developed using the activemq-all library. I have noticed that the connection is hanging occasionally (after every 1-2 weeks without any feed). So I wanted to use the heart-beat feature of STOMP protocol so I have added the heart-beat header for the stompConnection as shown below:
StompConnection stompConnection = new StompConnection();
stompConnection.open(new Socket("ABC", 1234));
HashMap<String, String> headers = new HashMap<>();
headers.put("login", "abcd");
headers.put("passcode", "defghij");
headers.put("heart-beat", "0,10000");//heart-beat header newly added
stompConnection.connect(headers);
stompConnection.subscribe("topic1", "auto");
while(true) {
StompFrame stompMessage = stompConnection.receive(10000);
String messageBody = stompMessage.getBody();
//process messageBody here
}
Now my question is that is there any way to find/trace that my application (above java client) is receiving the heart beats from the sender?
That library is a test support only tool with no support for issues other than those affecting ActiveMQ tests. Using it is risky and not recommended for any production level work. You'd be far better off using a full open source Stomp client with actual support. The track heart beats with this client you would need to drop to the socket level and track the incoming bytes directly.
Related
My first approach was to use the com.microsoft.azure:azure-eventhubs:3.2.0 dependency but I had the following problem: Client hangs when calling Azure Event Hub and facing connection error
So my 2nd approach can be
either to use another dependency (I need to use OAuth2 too) -> unfortunately I haven't found any
or don't use a dependency but code the message sending myself -> it might be a big task
Could you please recommend me library that supports OAuth2?
Or a sample code which supports message sending to Event Hub over AMQP with OAuth2?
Or a 3rd approach ...?
Thanks,
V.
Regarding your hang issue, I posted a new answer in the original thread Client hangs when calling Azure Event Hub and facing connection error. That should fix your issue.
Now coming back to this question, primarily you should follow the latest tutorial (uses Event Hub v5 sdk using Producer/Consumer pattern).
import com.azure.messaging.eventhubs.*;
public class Sender {
public static void main(String[] args) {
final String connectionString = "EVENT HUBS NAMESPACE CONNECTION STRING";
final String eventHubName = "EVENT HUB NAME";
// create a producer using the namespace connection string and event hub name
EventHubProducerClient producer = new EventHubClientBuilder()
.connectionString(connectionString, eventHubName)
.buildProducerClient();
// prepare a batch of events to send to the event hub
EventDataBatch batch = producer.createBatch();
batch.tryAdd(new EventData("First event"));
batch.tryAdd(new EventData("Second event"));
batch.tryAdd(new EventData("Third event"));
batch.tryAdd(new EventData("Fourth event"));
batch.tryAdd(new EventData("Fifth event"));
// send the batch of events to the event hub
producer.send(batch);
// close the producer
producer.close();
}
}
Regarding the question on OAuth2, it is very much possible to leverage AAD authentication approach instead of Shared Access Signature (the default connection string approach). You can follow this tutorial for Managed Identity scenario OR this tutorial for custom AAD application based auth.
Regarding the recommendation on OAuth2 library, the first party choice in the Azure world would be MSAL for seamless integration and support.
Regarding the question on writing the whole thing by custom code, while it's technically possible, but honestly it's super impractical to invest on this mammoth task due to following reasons:
Why to reinvent the wheel?
It will increase your maintenance headache for a hell lot of custom code which does only technical wiring, but hardly adds any value to the business.
You would be pretty much on your own to figure out any issue or future compatibility.
Adding all the low level AMQP and OAuth handling in your code would add unnecessary burden to your team to master those.
..and most importantly it would cost Money and Time :)
Using standard libraries not only saves you from lot of efforts and money, but ensures reusability, and support from both the creator and communities.
I'm quite new to Apache Camel and trying to bring some routes into action.
I have a TCP server which serves large JSON-Messages (up to ~30-50kB in size, where i do not have any control about the source size) that contain lots of measurement data which i want to process using certain additional routes that work fine.
I'm using camel 2.20 within spring-boot environment 1.5.7.
I faced the problem that if i commented out every other routes except the incoming reduced netty4 route (only from and to a counter), see below
#Bean
public RouteBuilder getRoute() {
String fromSource = String.format("netty4:tcp://%s:%d?clientMode=true&textline=true&receiveBufferSize=64000&decoderMaxLineLength=64000",sourceIp,sourcePort);
return new RouteBuilder() {
from(fromSource)
.to("metrics:counter:incomingCounter");
};
}
The route works nearly fine but consumes more and more heap-space (around 2MB every second, where there are messages served with a frequency of around 20-30Hz) until java throws java.lang.OutOfMemoryError: Java heap space.
Without any route no memory-leak was registered, as i can focus the problem to the netty-route
Any help will be appreciated.
Thanks in advance.
I found the resolution myself by debugging the code.
I forgot to set property sync=false in netty4-camel endpoint as i don't want to process message and send an answer back to the server after processing, just consuming - while sync=true (default settings) buffers all incoming data for later response which caused my "memory-leak".
The behavior of "sync" was not totally clear from the netty4-camel documentation (http://camel.apache.org/netty4.html) - i'll suggest an improvement of the documentation (will write a mail with a proposal) to make the usage a little more clearly.
Maybe this helps someone another having a similar problem.
Best
I would like to implement tracing in my microservices architecture. I am using Apache Kafka as message broker and I am not using Spring Framework. Tracing is a new concept for me. At first I wanted to create my own implementation, but now I would like to use existing libraries. Brave looks like the one I will want to use. I would like to know if there are some guides, examples or docs on how to do this. Documentation on Github page is minimal, and I find it hard to start using Brave. Or maybe there is better library with proper documentation, that is easier to use. I will be looking at Apache HTrace because it looks promising. Some getting started guides will be nice.
There are a bunch of ways to answer this, but I'll answer it from the "one-way" perspective. The short answer though, is I think you have to roll your own right now!
While Kafka can be used in many ways, it can be used as a transport for unidirectional single producer single consumer messages. This action is similar to normal one-way RPC, where you have a request, but no response.
In Zipkin, an RPC span is usually request-response. For example, you see timing of the client sending to the server, and also the way back to the client. One-way is where you leave out the other side. The span starts with a "cs" (client send) and ends with a "sr" (server received).
Mapping this to Kafka, you would mark client sent when you produce the message and server received when the consumer receives it.
The trick to Kafka is that there is no nice place to stuff the trace context. That's because unlike a lot of messaging systems, there are no headers in a Kafka message. Without a trace context, you don't know which trace (or span for that matter) you are completing!
The "hack" approach is to stuff trace identifiers as the message key. A less hacky way would be to coordinate a body wrapper which you can nest the trace context into.
Here's an example of the former:
https://gist.github.com/adriancole/76d94054b77e3be338bd75424ca8ba30
I meet the same problem too.Here is my solution, a less hacky way as above said.
ServerSpan serverSpan = brave.serverSpanThreadBinder().getCurrentServerSpan();
TraceHeader traceHeader = convert(serverSpan);
//in kafka producer,user KafkaTemplete to send
String wrapMsg = "wrap traceHeader with originMsg ";
kafkaTemplate.send(topic, wrapMsg).get(10, TimeUnit.SECONDS);// use synchronization
//then in kafka consumer
ConsumerRecords<String, String> records = consumer.poll(5000);
// for loop
for (ConsumerRecord<String, String> record : records) {
String topic = record.topic();
int partition = record.partition();
long offset = record.offset();
String val = record.value();
//parse val to json
Object warpObj = JSON.parseObject(val);
TraceHeader traceHeader = warpObj.getTraceHeader();
//then you can do something like this
MyRequest myRequest = new MyRequest(traceHeader, "/esb/consumer", "POST");
brave.serverRequestInterceptor().handle(new HttpServerRequestAdapter(new MyHttpServerRequest(myRequest), new DefaultSpanNameProvider()));
//then some httprequest within brave-apache-http-interceptors
//http.post(url,content)
}
you must implements MyHttpServerRequest and MyRequest.It is easy,you just return something a span need,such as uri,header,method.
This is a rough and ugly code example,just offer an idea.
Premise:
We have groovy scripts that execute every minute. I want one of those scripts to open an HTTP client, and poll a service bus queue / topic for messages. I have my rest client code working an getting messages from the service bus queue. I can do a "Get" every 5 seconds, and wireshark shows that it's reusing the same TCP connection which is better than I expected, but its still not ideal.
Goal:
I would like to make this http client do "long polling", for efficiency and to achieve actual real-time processing. It seems to be more complicated than I anticipated.
Problem:
When I do a "Delete" call to read message from a service bus queue, it immediately returns "HTTP/1.1 204 No Content", and the connection closes. I set a timeout on the client, but I don't think that matters.
Here's the article that shows service bus says it's supports long polling, which I imagine is the hard part. Azure Service Bus Queues
I feel that I don't understand something fundamental about how to implement long polling in code. My understanding is that when there is no data in the queue, it's supposed to delay the response until data exists, or until my client eventually times out waiting (which lets me set my own disconnect/reconnect interval). I don't even care about blocking/nonblocking etc, because the script execution is already spreadout into a threadpool, and will be terminated forcibly and all that.
Any help is greatly appreciated.
The correct and simple answer is that adding the following to the end of an Azure REST API URL (with service bus) is the way to implements long-polling with that service: ?timeout=60 , where 60 tells azure to wait 60 seconds before responding with no-data. So, your application can check for data every 60 seconds, with an internal timeout of 60 seconds on each HTTP request. This will hold the TCP connection open for that timeframe, waiting for an HTTP response.
For understanding long polling, I recommend you can learn the entry Comet of Wiki https://en.wikipedia.org/wiki/Comet_(programming). And there is an answered thread (Long polling in java) explained the mechanism of the HttpURLConnection Class support long polling in Java.
As I know, a simple way in Java Client instead of HttpURLConnection is using the client library of CometD. You can refer to the section Client Library of its offical document https://docs.cometd.org/current/reference/#_java_client to learn how to implement the long polling client in Java. You can download the library at https://download.cometd.org/.
The sample code from the offical document:
// Create (and eventually set up) Jetty's HttpClient:
HttpClient httpClient = new HttpClient();
httpClient.start();
// Prepare the transport
Map<String, Object> options = new HashMap<String, Object>();
ClientTransport transport = new LongPollingTransport(options, httpClient);
// Create the BayeuxClient
ClientSession client = new BayeuxClient("http://localhost:8080/cometd", transport);
// Here set up the BayeuxClient, for example:
// client.getChannel(Channel.META_CONNECT).addListener(new ClientSessionChannel.MessageListener() {
public void onMessage(ClientSessionChannel channel, Message message) {
if (message.isSuccessful()) {
// Here handshake is successful
}
}
});
client.handshake();
Note: There are two REST API of Azure Service Bus for getting messaging entity(s) Get Entity https://msdn.microsoft.com/en-us/library/azure/hh780754.aspx and Entities Discovery https://msdn.microsoft.com/en-us/library/azure/hh780782.aspx. You need to delete the used messaging entity manually thru the Delete Entity REST API. Requesting all of these REST API first require an access_token thru the post request the Request a Token from ACS API for secure access.
I'd like to use WebSocket with Java. Problem is, my server is separated from the client by a proxy that cannot be configured. I've been searching for implementations of WebSocket with fallbacks such as long-polling. I've found socket.io but don't know how the fallback works.
Under which case does it replace WebSocket and how?
Are there other libraries like socket.io with fallbacks implementations? I would like to find one in Java, but I only found Jetty.
EDIT: does the fallback only depend on the browser's compatibility with WebSocket? What if the cause of failure is a proxy badly configured, is socket.io going to detect it as a compatibilty failure and thus switch to long-polling (or another technique)?
Answer: since v1, socket.io includes engine.io, which brings the following features:
Socket.io is one of several implementations for the websockets protocol and its main selling point (IMO) is its ease of use: you don't need to code keep-alive mechanisms or decide which transport is best, it does it for you.
So, to make it clear, socket.io doesn't replace the websocket protocol, it's a package that implements it for you.
You mentioned long-polling. That is one of the transports used by socket.io. Long Polling is HTTP based and it's basically request --> wait --> response and the wait isn't very long, as it can be dropped by load balancers on EOF or stale connections. Nevertheless, it's still useful when the websockets protocol (TCP based) isn't available and socket.io automatically re-establishes the connection for you. Notice that websockets is a relatively new protocol, ratified in 2011, so older browsers don't support it. Well, socket.io detects that and then resorts to long polling, so you don't have to "worry" about it.
A websocket connection starts with HTTP, listening on the same port. For example, http://localhost:8080 (just a silly example). Then, when it's possible, socket.io switches to ws://localhost:8080 for you.
I never had problems with network topology challenges when using socket.io, as when the HTTP port is available and using long polling / websockets is possible, it just worked for me.
One of the libraries with fallback implementation, as you mentioned, is netty-socket.io. Notice how it configures the two transports:
public class Configuration {
private ExceptionListener exceptionListener = new DefaultExceptionListener();
private String context = "/socket.io";
private List<Transport> transports = Arrays.asList(Transport.WEBSOCKET, Transport.POLLING);
private int bossThreads = 0; // 0 = current_processors_amount * 2
private int workerThreads = 0; // 0 = current_processors_amount * 2
The complete code can be found here.
Node JS has also libraries for websockets, and I mention it here just to clarify that long polling and websockets aren't the only two available transports (might be the only ones in Java):
io.set('transports', [ // enable all transports (optional if you want flashsocket)
'websocket'
, 'flashsocket'
, 'htmlfile'
, 'xhr-polling'
, 'jsonp-polling'
]);
In a nutshell, socket.io attempts to make things as easy as possible for you, including not having to worry about what transports to use, as it's done under the hood for you, yet still configurable if you want.
I hope this brief explanation helps you!