There is a middleware in between of two other softwares. In the middleware I'm routing Apache ActiveMQ messages by Apache Camel.
the first software uses middleware to send message to the 3rd software and the 3rd one reply the message to the first(using middleware).
1stSoftware <<=>> Middleware <<=>> 3rdSoftware
Problem:
when with the first one I send message to the middleware, middleware sends that message directly to ActiveMQ.DLQ and the 3rd one can not consume it!(Interesting point is this: when I copy that message to the main queue with the Admin panel of ActiveMQ, software can consume it properly!)
What's the problem?! It was working until I changed the Linux date!!!!!!!
Middleware is like this:
#SuppressWarnings("deprecation")
public class MiddlewareDaemon {
private Main main;
public static void main(String[] args) throws Exception {
MiddlewareDaemon middlewareDaemon = new MiddlewareDaemon();
middlewareDaemon.boot();
}
public void boot() throws Exception {
main = new Main();
main.enableHangupSupport();
//?wireFormat.maxInactivityDuration=0
main.bind("activemq", activeMQComponent("tcp://localhost:61616")); //ToGet
main.bind("activemq2", activeMQComponent("tcp://192.168.10.103:61616")); //ToInOut
main.addRouteBuilder(new MyRouteBuilder());
System.out.println("Starting Camel(MiddlewareDaemon). Use ctrl + c to terminate the JVM.\n");
main.run();
}
private static class MyRouteBuilder extends RouteBuilder {
#Override
public void configure() throws Exception {
intercept().to("log:Midlleware?level=INFO&showHeaders=true&showException=true&showCaughtException=true&showStackTrace=true");
from("activemq:queue:Q.Midlleware")
.process(new Processor() {
public void process(Exchange exchange) {
Map<String, Object> header = null;
try {
Message in = exchange.getIn();
header = in.getHeaders();
} catch (Exception e) {
log.error("Exception:", e);
header.put("Midlleware_Exception", e.getMessage() + " - " + e);
}
}
})
.inOut("activemq2:queue:Q.Comp2")
}
}
}
And the 3rd software(Replier): (this is a daemon like above, i just copied the RouteBuilder part)
private static class MyRouteBuilder extends RouteBuilder {
#Override
public void configure() {
intercept().to("log:Comp2?level=INFO&showHeaders=true&showException=true&showCaughtException=true&showStackTrace=true");
from("activemq:queue:Q.Comp2")
.process(new Processor() {
public void process(Exchange exchange) {
Message in = exchange.getIn();
Map<String, Object> headers = null;
try {
headers = in.getHeaders();
in.setBody(ZipUtil.compress(/*somResults*/));
} catch (Exception e) {
log.error("Exception", e);
in.setBody(ZipUtil.compress("[]"));
in.getHeaders().put("Comp2_Exception", e.getMessage() + " - " + e);
}
}
})
;
}
}
If the only thing you changed is the time on one of the servers, then this might well be the problem.
For communications over MQ to work properly, it is essential that all involved host systems have their clock set in sync. In the case of ActiveMQ there is a default message time-to-live (30 seconds, I think) for response queues. If the responding system is more then 30 seconds in the future relative to the host running ActiveMQ, then ActiveMQ will immediately expire the message and move it to the DLQ.
Related
I'm trying to create a Netty (4.1) POC which can forward h2c (HTTP2 without TLS) frames onto a h2c server - i.e. essentially creating a Netty h2c proxy service. Wireshark shows Netty sending the frames out, and the h2c server replying (for example with the response header and data), although I'm then having a few issues receiving/processing the response HTTP frames within Netty itself.
As a starting point, I've adapted the multiplex.server example (io.netty.example.http2.helloworld.multiplex.server) so that in HelloWorldHttp2Handler, instead of responding with dummy messages, I connect to a remote node:
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
Channel remoteChannel = null;
// create or retrieve the remote channel (one to one mapping) associated with this incoming (client) channel
synchronized (lock) {
if (!ctx.channel().hasAttr(remoteChannelKey)) {
remoteChannel = this.connectToRemoteBlocking(ctx.channel());
ctx.channel().attr(remoteChannelKey).set(remoteChannel);
} else {
remoteChannel = ctx.channel().attr(remoteChannelKey).get();
}
}
if (msg instanceof Http2HeadersFrame) {
onHeadersRead(remoteChannel, (Http2HeadersFrame) msg);
} else if (msg instanceof Http2DataFrame) {
final Http2DataFrame data = (Http2DataFrame) msg;
onDataRead(remoteChannel, (Http2DataFrame) msg);
send(ctx.channel(), new DefaultHttp2WindowUpdateFrame(data.initialFlowControlledBytes()).stream(data.stream()));
} else {
super.channelRead(ctx, msg);
}
}
private void send(Channel remoteChannel, Http2Frame frame) {
remoteChannel.writeAndFlush(frame).addListener(new GenericFutureListener() {
#Override
public void operationComplete(Future future) throws Exception {
if (!future.isSuccess()) {
future.cause().printStackTrace();
}
}
});
}
/**
* If receive a frame with end-of-stream set, send a pre-canned response.
*/
private void onDataRead(Channel remoteChannel, Http2DataFrame data) throws Exception {
if (data.isEndStream()) {
send(remoteChannel, data);
} else {
// We do not send back the response to the remote-peer, so we need to release it.
data.release();
}
}
/**
* If receive a frame with end-of-stream set, send a pre-canned response.
*/
private void onHeadersRead(Channel remoteChannel, Http2HeadersFrame headers)
throws Exception {
if (headers.isEndStream()) {
send(remoteChannel, headers);
}
}
private Channel connectToRemoteBlocking(Channel clientChannel) {
try {
Bootstrap b = new Bootstrap();
b.group(new NioEventLoopGroup());
b.channel(NioSocketChannel.class);
b.option(ChannelOption.SO_KEEPALIVE, true);
b.remoteAddress("localhost", H2C_SERVER_PORT);
b.handler(new Http2ClientInitializer());
final Channel channel = b.connect().syncUninterruptibly().channel();
channel.config().setAutoRead(true);
channel.attr(clientChannelKey).set(clientChannel);
return channel;
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
When initializing the channel pipeline (in Http2ClientInitializer), if I do something like:
#Override
public void initChannel(SocketChannel ch) throws Exception {
ch.pipeline().addLast(Http2MultiplexCodecBuilder.forClient(new Http2OutboundClientHandler()).frameLogger(TESTLOGGER).build());
ch.pipeline().addLast(new UserEventLogger());
}
Then I can see the frames being forwarded correctly in Wireshark and the h2c server replies with the header and frame data, but Netty replies with a GOAWAY [INTERNAL_ERROR] due to:
14:23:09.324 [nioEventLoopGroup-3-1] WARN
i.n.channel.DefaultChannelPipeline - An exceptionCaught() event was
fired, and it reached at the tail of the pipeline. It usually means
the last handler in the pipeline did not handle the exception.
java.lang.IllegalStateException: Stream object required for
identifier: 1 at
io.netty.handler.codec.http2.Http2FrameCodec$FrameListener.requireStream(Http2FrameCodec.java:587)
at
io.netty.handler.codec.http2.Http2FrameCodec$FrameListener.onHeadersRead(Http2FrameCodec.java:550)
at
io.netty.handler.codec.http2.Http2FrameCodec$FrameListener.onHeadersRead(Http2FrameCodec.java:543)...
If I instead try making it have the pipeline configuration from the http2 client example, e.g.:
#Override
public void initChannel(SocketChannel ch) throws Exception {
final Http2Connection connection = new DefaultHttp2Connection(false);
ch.pipeline().addLast(
new Http2ConnectionHandlerBuilder()
.connection(connection)
.frameLogger(TESTLOGGER)
.frameListener(new DelegatingDecompressorFrameListener(connection, new InboundHttp2ToHttpAdapterBuilder(connection)
.maxContentLength(maxContentLength)
.propagateSettings(true)
.build() ))
.build());
}
Then I instead get:
java.lang.UnsupportedOperationException: unsupported message type:
DefaultHttp2HeadersFrame (expected: ByteBuf, FileRegion) at
io.netty.channel.nio.AbstractNioByteChannel.filterOutboundMessage(AbstractNioByteChannel.java:283)
at
io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:882)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1365)
If I then add in a HTTP2 frame codec (Http2MultiplexCodec or Http2FrameCodec):
#Override
public void initChannel(SocketChannel ch) throws Exception {
final Http2Connection connection = new DefaultHttp2Connection(false);
ch.pipeline().addLast(
new Http2ConnectionHandlerBuilder()
.connection(connection)
.frameLogger(TESTLOGGER)
.frameListener(new DelegatingDecompressorFrameListener(connection, new InboundHttp2ToHttpAdapterBuilder(connection)
.maxContentLength(maxContentLength)
.propagateSettings(true)
.build() ))
.build());
ch.pipeline().addLast(Http2MultiplexCodecBuilder.forClient(new Http2OutboundClientHandler()).frameLogger(TESTLOGGER).build());
}
Then Netty sends two connection preface frames, resulting in the h2c server rejecting with GOAWAY [PROTOCOL_ERROR]:
So that is where I am having issues - i.e. configuring the remote channel pipeline such that it will send the Http2Frame objects without error, but also then receive/process them back within Netty when the response is received.
Does anyone have any ideas/suggestions please?
I ended up getting this working; the following Github issues contain some useful code/info:
Generating a Http2StreamChannel, from a Channel
A Http2Client with Http2MultiplexCode
I need to investigate a few caveats further, although the gist of the approach is that you need to wrap your channel in a Http2StreamChannel, meaning that my connectToRemoteBlocking() method ends up as:
private Http2StreamChannel connectToRemoteBlocking(Channel clientChannel) {
try {
Bootstrap b = new Bootstrap();
b.group(new NioEventLoopGroup()); // TODO reuse existing event loop
b.channel(NioSocketChannel.class);
b.option(ChannelOption.SO_KEEPALIVE, true);
b.remoteAddress("localhost", H2C_SERVER_PORT);
b.handler(new Http2ClientInitializer());
final Channel channel = b.connect().syncUninterruptibly().channel();
channel.config().setAutoRead(true);
channel.attr(clientChannelKey).set(clientChannel);
// TODO make more robust, see example at https://github.com/netty/netty/issues/8692
final Http2StreamChannelBootstrap bs = new Http2StreamChannelBootstrap(channel);
final Http2StreamChannel http2Stream = bs.open().syncUninterruptibly().get();
http2Stream.attr(clientChannelKey).set(clientChannel);
http2Stream.pipeline().addLast(new Http2OutboundClientHandler()); // will read: DefaultHttp2HeadersFrame, DefaultHttp2DataFrame
return http2Stream;
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
Then to prevent the "Stream object required for identifier: 1" error (which is essentially saying: 'This (client) HTTP2 request is new, so why do we have this specific stream?' - since we were implicitly reusing the stream object from the originally received 'server' request), we need to change to use the remote channel's stream when forwarding our data on:
private void onHeadersRead(Http2StreamChannel remoteChannel, Http2HeadersFrame headers) throws Exception {
if (headers.isEndStream()) {
headers.stream(remoteChannel.stream());
send(remoteChannel, headers);
}
}
Then the configured channel inbound handler (which I've called Http2OutboundClientHandler due to its usage) will receive the incoming HTTP2 frames in the normal way:
#Sharable
public class Http2OutboundClientHandler extends SimpleChannelInboundHandler<Http2Frame> {
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
super.exceptionCaught(ctx, cause);
cause.printStackTrace();
ctx.close();
}
#Override
public void channelRead0(ChannelHandlerContext ctx, Http2Frame msg) throws Exception {
System.out.println("Http2OutboundClientHandler Http2Frame Type: " + msg.getClass().toString());
}
}
The client is subscribed to a x / # topic. There is the possibility of receiving message in the topics x / start and x / stop, and depending on the topic, it performs an action. I wonder how I can identify if it's coming up in the start or stop topic.
In the current code, I send an "action" key in the JSON: "start" or "stop". I want to delete this key and use the format that said above, identifying the topic.
Any further information they deem necessary, please request that I edit the post!
JDK 8
The code:
private MqttCallback callback = new MqttCallback() {
public void connectionLost(Throwable throwable) {
try {
connect();
} catch (MqttException e) {
e.printStackTrace();
}
}
public void messageArrived(String s, MqttMessage mqttMessage) throws Exception {
String messageReceived = new String(mqttMessage.getPayload());
actionPerformed(messageReceived);
}
public void deliveryComplete(IMqttDeliveryToken iMqttDeliveryToken) {
}
};
private void actionPerformed(String message) throws IOException {
ClientDTO clientDTO = new ObjectMapper().readValue(message, ClientDTO.class);
if (clientDTO.getAction().equalsIgnoreCase("start")) {
startView(clientDTO);
} else if (clientDTO.getAction().equalsIgnoreCase("stop")) {
stopView();
}
}
public void connect() throws MqttException {
MqttConnectOptions options = new MqttConnectOptions();
options.setUserName("a_nice_username");
options.setPassword("a_cool_password".toCharArray());
options.setAutomaticReconnect(true);
MqttClient client = new MqttClient("someaddress", MqttClient.generateClientId());
client.setCallback(callback);
try {
client.connect(options);
client.subscribe(topic);
TaskbarIcon.alteraIconeOnline();
} catch (Exception e) {
TaskbarIcon.alteraIconeOffline();
}
}
public void tipoConexao(int tipoConex) throws IOException {
switch (tipoConex) {
case 0:
topic += "/operador/" + getIdReceived() + "/#";
System.out.println(topic);
break;
//etc
}
The s in this method is the topic: public void messageArrived(String s, MqttMessage mqttMessage)
As is very well documented here:
messageArrived
void messageArrived(java.lang.String topic, MqttMessage message) throws java.lang.Exception
This method is called when a message arrives from the server.
This method is invoked synchronously by the MQTT client. An acknowledgment is not sent back to the server until this method
returns cleanly.
If an implementation of this method throws an Exception, then the client will be shut down. When the client is next re-connected, any
QoS 1 or 2 messages will be redelivered by the server.
Any additional messages which arrive while an implementation of this method is running, will build up in memory, and will then back up
on the network.
If an application needs to persist data, then it should ensure the data is persisted prior to returning from this method, as after
returning from this method, the message is considered to have been
delivered, and will not be reproducible.
It is possible to send a new message within an implementation of this callback (for example, a response to this message), but the
implementation must not disconnect the client, as it will be
impossible to send an acknowledgment for the message being processed,
and a deadlock will occur.
Parameters:
topic - name of the topic on the message was published to
message - the actual message.
Throws:
java.lang.Exception - if a terminal error has occurred, and the client should be shut down.
I want to start both TCP echo server and client in one app, client after server.
What I do is:
Start a client in a ChannelFutureListener returned by server.bind().sync() like this:
public void runClientAndServer() {
server.run().addListener((ChannelFutureListener) future -> {
// client.run(); //(1) this doesn't work!
new Thread(()->client.run()).start(); //(2) this works!
});
}
and server.run() is like this:
public ChannelFuture run() {
ServerBootstrap b = new ServerBootstrap();
//doing channel config stuff
return b.bind(6666).sync();
}
and client.run() is like this:
public void run() {
Bootstrap b = new Bootstrap();
//do some config stuff
f = b.connect(host, port).sync(); //wait till connected to server
}
What happens is:
In the statement (2) that works just fine; While in the statement (1), the client sent message, that can be observed in Wireshark, and the server replies TCP ACK segment, but no channelRead() method in server side ChannelInboundHandlerAdapter is invoked, nor any attempt to write message to socket can be observed, like this capture:
wireshark capture
I guess there must be something wrong with Netty threads, but I just cannot figure out
I have prepared an example based on the newest netty version (4.1.16.Final) and the code you posted. This works fine without an extra thread maybe you did something wrong initializing your server or client.
private static final NioEventLoopGroup EVENT_LOOP_GROUP = new NioEventLoopGroup();
private static ChannelFuture runServer() throws Exception {
return new ServerBootstrap()
.group(EVENT_LOOP_GROUP)
.channel(NioServerSocketChannel.class)
.childHandler(new ChannelInitializer<Channel>() {
#Override
protected void initChannel(Channel channel) throws Exception {
System.out.println("S - Channel connected: " + channel);
channel.pipeline().addLast(new SimpleChannelInboundHandler<ByteBuf>() {
#Override
protected void channelRead0(ChannelHandlerContext ctx, ByteBuf msg) throws Exception {
System.out.println("S - read: " + msg.toString(StandardCharsets.UTF_8));
}
});
}
})
.bind(6666).sync();
}
private static void runClient() throws Exception {
new Bootstrap()
.group(EVENT_LOOP_GROUP)
.channel(NioSocketChannel.class)
.handler(new ChannelInitializer<Channel>() {
#Override
protected void initChannel(Channel channel) throws Exception {
System.out.println("C - Initilized client");
channel.pipeline().addLast(new SimpleChannelInboundHandler<ByteBuf>() {
#Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
super.channelActive(ctx);
System.out.println("C - write: Hey this is a test message enjoy!");
ctx.writeAndFlush(Unpooled.copiedBuffer("Hey this is a test message enjoy!".getBytes(StandardCharsets.UTF_8)));
}
#Override
protected void channelRead0(ChannelHandlerContext ctx, ByteBuf msg) throws Exception { }
});
}
})
.connect("localhost", 6666).sync();
}
public static void main(String[] args) throws Exception {
runServer().addListener(future -> {
runClient();
});
}
That's what the output should look like:
C - Initilized client
C - write: Hey this is a test message enjoy!
S - Channel connected: [id: 0x1d676489, L:/127.0.0.1:6666 - R:/127.0.0.1:5079]
S - read: Hey this is a test message enjoy!
I also tried this example with a single threaded eventloopgroup which also worked fine but throw me an BlockingOperationException which did not affect the functionality of the program. If this code should work fine you should probably check your code and try to orientate your code on this example (Please don't create inline ChannelHandler for your real project like I did in this example).
We have a data processing application that runs on Karaf 2.4.3 with Camel 2.15.3.
In this application, we have a bunch of routes that import data. We have a management view that lists these routes and where each route can be started. Those routes do not directly import data, but call other routes (some of them in other bundles, called via direct-vm), sometimes directly and sometimes in a splitter.
Is there a way to also completely stop a route/therefore stopping the entire exchange from being further processed?
When simply using the stopRoute function like this:
route.getRouteContext().getCamelContext().stopRoute(route.getId());
I eventually get a success message with Graceful shutdown of 1 routes completed in 10 seconds - the exchange is still being processed though...
So I tried to mimic the behaviour of the StopProcessor by setting the stop property, but that also didn't help:
public void stopRoute(Route route) {
try {
Collection<InflightExchange> browse = route.getRouteContext().getCamelContext().getInflightRepository()
.browse();
for (InflightExchange inflightExchange : browse) {
String exchangeRouteId = inflightExchange.getRouteId();
if ((exchangeRouteId != null) && exchangeRouteId.equals(route.getId())) {
this.stopExchange(inflightExchange.getExchange());
}
}
} catch (Exception e) {
Notification.show("Error while trying to stop route", Type.ERROR_MESSAGE);
LOGGER.error(e, e);
}
}
public void stopExchange(Exchange exchange) throws Exception {
AsyncProcessorHelper.process(new AsyncProcessor() {
#Override
public void process(Exchange exchange) throws Exception {
AsyncProcessorHelper.process(this, exchange);
}
#Override
public boolean process(Exchange exchange, AsyncCallback callback) {
exchange.setProperty(Exchange.ROUTE_STOP, Boolean.TRUE);
callback.done(true);
return true;
}
}, exchange);
}
Is there any way to completely stop an exchange from being processed from outside the route?
Can you get an exchange?
I use exchange.setProperty(Exchange.ROUTE_STOP, true);
Route stop flow and doesn't go to next route.
I would like to be aware of the connection state in a camel/netty environment.
To do so, I tried something like this:
specified my camel route
from("direct:in").marshal().serialization()
.to("netty:tcp://localhost:42123?clientPipelineFactory=#cpf&sync=false");
implemented my pipeline factory
public class ConnectionStatusPipelineFactory extends ClientPipelineFactory {
#Override
public ChannelPipeline getPipeline() throws Exception {
ChannelPipeline cp = Channels.pipeline();
cp.addLast("statusHandler", new ConnectionStatusHandler());
return cp;
}
#Override
public ClientPipelineFactory createPipelineFactory(NettyProducer producer) {
return new ConnectionStatusPipelineFactory();
}
}
implemented my connection status handler
public class ConnectionStatusHandler extends SimpleChannelUpstreamHandler {
#Override
public void channelConnected(ChannelHandlerContext ctx, ChannelStateEvent e)
throws Exception {
System.out.println("Event: " + e);
super.channelConnected(ctx, e);
}
#Override
public void channelDisconnected(ChannelHandlerContext ctx,
ChannelStateEvent e) throws Exception {
System.out.println("Event: " + e);
super.channelDisconnected(ctx, e);
}
}
And finally bound "ConnectionStatusPipelineFactory" to "cpf" in my camel registry.
But the following exception occured:
java.lang.IllegalArgumentException: unsupported message type: class [B
Remarks:
"channelConnected" and "channelDisconnected" methods are called as expected.
When I disable this, everything works (message marshalling, connection, remote process...).
Questions are:
what's wrong with that ?
is it the good way to know the connection status (connected or not) ?
Try using the decoder option instead and not the entire client pipeline factory.
eg use option decoder=#myConnectionStatusHandler. And then register your ConnectionStatusHandler in the registry with the name myConnectionStatusHandler.
If you use the pipeline factory then you need to add all the others that Camel add out of the box.