we have on-premise Service Bus installation. I can publish and subscribe/read messages with QPID AMQP 1.0 0.24 client. However queue browsing doens§t work, the call to hasMoreElements() hangs indefinitely when there are no more messages in the queue. The stack trace is:
Thread [main] (Suspended)
waiting for: ConnectionEndpoint (id=19)
Object.wait(long) line: not available [native method]
ConnectionEndpoint(Object).wait() line: 503
Receiver.drainWait() line: 533
QueueBrowserImpl$MessageEnumeration.hasMoreElements() line: 154
Qpid.testBrowseTopic(Connection, Context) line: 209
Qpid.runTest(Qpid$Options) line: 93
Qpid.main(String[]) line: 63
The code:
ConnectionFactory connectionFactory = (ConnectionFactory) context.lookup("MS_SERVICE_BUS");
connection = connectionFactory.createConnection();
session = connection.createSession(false/*transacted*/, Session.AUTO_ACKNOWLEDGE);
Queue queue = (Queue) context.lookup("MY_QUEUE");
browser = session.createBrowser(queue);
Enumeration<Message> msgs = browser.getEnumeration();
while (msgs.hasMoreElements()) {// hangs when there are no more messages
Message message = msgs.nextElement();
//printMessage(message);
}
The same behaviour for QPID 0.22. Is this bug in QPID client or Service Bus?
Thanks, Jan
There are couple things going on here:
1) Service Bus doesn't support message browse over AMQP at this time. Because the session was created with AUTO_ACKNOWLEDGE, every message you get from the Enumeration is immediately removed from the queue.
2) I have reproduced the hang in hasMoreElements() with QPid 0.25. It appears that hasMoreElements() is waiting for more messages to arrive in the queue, at least at first. If I send more messages, the loop will continue and some of the newly-arrived messages will be returned, but it stops fairly quickly. I am still investigating to determine what's going on there.
Related
I'm using kafka + redis in my project.
I get message from Kafka, process and save to redis, but it is giving error like below when my code runs after some time my code
io.smallrye.mutiny.TimeoutException
at io.smallrye.mutiny.operators.uni.UniBlockingAwait.await(UniBlockingAwait.java:64)
at io.smallrye.mutiny.groups.UniAwait.atMost(UniAwait.java:65)
at io.quarkus.redis.client.runtime.RedisClientImpl.await(RedisClientImpl.java:1046)
at io.quarkus.redis.client.runtime.RedisClientImpl.set(RedisClientImpl.java:687)
at worker.redis.process.implementation.ProductImplementation.refresh(ProductImplementation.java:34)
at worker.redis.Worker.refresh(Worker.java:51)
at
kafka.InComingProductKafkaConsume.lambda$consume$0(InComingProductKafkaConsume.java:38)
at business.core.hpithead.ThreadStart.doRun(ThreadStart.java:34)
at business.core.hpithead.core.NotifyingThread.run(NotifyingThread.java:27)
at java.base/java.lang.Thread.run(Thread.java:833)
The record 51761 from topic-partition 'mer-outgoing-master-item-0' has waited for 153 seconds to be acknowledged. This waiting time is greater than the configured threshold (150000 ms). At the moment 2 messages from this partition are awaiting acknowledgement. The last committed offset for this partition was 51760. This error is due to a potential issue in the application which does not acknowledged the records in a timely fashion. The connector cannot commit as a record processing has not completed.
#Incoming("mer_product")
#Blocking
public CompletionStage<Void> consume2(Message<String> payload) {
var objectDto = configThreadLocal.mapper.readValue(payload.getPayload(), new TypeReference<KafkaPayload<ItemKO>>(){});
worker.refresh(objectDto.payload.castDto());
return payload.ack();
}
I have a grpc (1.13.x) server on java that isn't performing any computation or I/O intensive task. The intention is to check the number of requests this server can support per second on 80 core machine.
Server:
ExecutorService executor = new ThreadPoolExecutor(160, Integer.MAX_VALUE,
60L, TimeUnit.SECONDS,
new SynchronousQueue<Runnable>(),
new ThreadFactoryBuilder()
.setDaemon(true)
.setNameFormat("Glowroot-IT-Harness-GRPC-Executor-%d")
.build());
Server server = NettyServerBuilder.forPort(50051)
.addService(new MyService())
.executor(executor)
.build()
.start();
Service:
#Override
public void verify(Request request, StreamObserver<Result> responseObserver) {
Result result = Result.newBuilder()
.setMessage("hello")
.build();
responseObserver.onNext(result);
responseObserver.onCompleted();
}
I am using ghz client to perform a load test. Server is able to handle 40k requests per second but RPS count is not able to exceed more than 40k even on increase in number of concurrent clients with incoming requests rate 100k. GRPC server is able to handle just 40K requests per second and it queues all other requests. CPU is underutilized (7%). About 90% of grpc threads (with prefix grpc-default-executor) were in waiting state, despite no I/O operation. More than 25k threads are in waiting state.
Stacktrace of threads in waiting:
grpc-default-executor-4605
PRIORITY :5
THREAD ID :0X00007F15A4440D80
NATIVE ID :
stackTrace:
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base#15.0.1/Native Method)
- parking to wait for <0x00007f1df161ae20> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base#15.0.1/LockSupport.java:252)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base#15.0.1/SynchronousQueue.java:462)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base#15.0.1/SynchronousQueue.java:361)
at java.util.concurrent.SynchronousQueue.poll(java.base#15.0.1/SynchronousQueue.java:937)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base#15.0.1/ThreadPoolExecutor.java:1055)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base#15.0.1/ThreadPoolExecutor.java:1116)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base#15.0.1/ThreadPoolExecutor.java:630)
at java.lang.Thread.run(java.base#15.0.1/Thread.java:832)
Locked ownable synchronizers:
- None
How can I configure the server to support 100K+ requests?
Nothing in the gRPC stack seems to cause this limit. What's the average response time on the server side? It looks like you are limited by the ephemeral ports or TCP connection limit and you may want to tweak your kernel as described at here https://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-1 or here https://blog.box.com/ephemeral-port-exhaustion-and-web-services-at-scale
Jms outbound-channel-adapter works perfectly fine but I see intermittently this error in the log, however, MQ message still gets delivered.
2019-06-07 10:16:22 [JMSCCThreadPoolWorker-5] INFO o.s.j.c.CachingConnectionFactory - Encountered a JMSException - resetting the underlying JMS Connection
com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ1107: A problem with this connection has occurred.
at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:578)
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:214)
at com.ibm.msg.client.wmq.internal.WMQConnection.consumer(WMQConnection.java:794)
at com.ibm.mq.jmqi.remote.api.RemoteHconn.callEventHandler(RemoteHconn.java:2903)
at com.ibm.mq.jmqi.remote.api.RemoteHconn.driveEventsEH(RemoteHconn.java:628)
at com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.processHconn(RemoteDispatchThread.java:691)
at com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.run(RemoteDispatchThread.java:233)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:263)
at com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.java:99)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:284)
at com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.runWorkQueueItem(WorkQueueManager.java:312)
at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementation.java:1214)
Caused by: com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').
...This is errorChannel configuration:
<int:header-enricher id="errorMsg.HeaderEnricher"
input-channel="errorChannel"
output-channel="omniAlertsJmsErrorChannel">
...and jms outbound-channel-adapter configured:
<int-jms:outbound-channel-adapter
id="jmsOutToNE" channel="umpAlertNotificationJMSChannel"
destination="senderTopic"
jms-template="jmsQueueTemplate"
>
I am expecting omniAlertsJmsErrorChannel to receive the MessageHandlingException which is not happening from jmsOutToNE adapter. All other channel/flow errors are being routed to omniAlertsJmsErrorChannel.
Also, wondering if there jms outbound-channel-adapter is retrying internally when com.ibm.mq.MQException happens and it gets successful in the subsequent try?
The errorChannel is out of use in any MessageHandler. Their exception are just thrown to the caller. To handle those exception you need to consider adding a request-handler-advice-chain with the ExpressionEvaluatingRequestHandlerAdvice. Or if we talk about a retry, you also can add to that chain a RequestHandlerRetryAdvice.
See more info in the Reference Manual.
Not sure though why your message is still delivered to the MQ. There is no any retry out-of-the-box in the <int-jms:outbound-channel-adapter. It might a behavior of the MQ Connection Factory adapter in the IBM library.
I have a frequent Channel shutdown: connection error issues (under 24.133.241:5671 thread, name is truncated) in RabbitMQ Java client (my producer and consumer are far apart). Most of the time consumer is automatically restarted as I have enabled heartbeat (15 seconds). However, there were some instances only Channel shutdown: connection error but no Consumer raised exception and no Restarting Consumer (under cTaskExecutor-4 thread).
My current workaround is to restart my application. Anyone can shed some light on this matter?
2017-03-20 12:42:38.856 ERROR 24245 --- [24.133.241:5671] o.s.a.r.c.CachingConnectionFactory
: Channel shutdown: connection error
2017-03-20 12:42:39.642 WARN 24245 --- [cTaskExecutor-4] o.s.a.r.l.SimpleMessageListenerCont
ainer : Consumer raised exception, processing can restart if the connection factory supports
it
...
2017-03-20 12:42:39.642 INFO 24245 --- [cTaskExecutor-4] o.s.a.r.l.SimpleMessageListenerCont
ainer : Restarting Consumer: tags=[{amq.ctag-4CqrRsUP8plDpLQdNcOjDw=21-05060179}], channel=Ca
ched Rabbit Channel: AMQChannel(amqp://21-05060179#10.24.133.241:5671/,1), conn: Proxy#7ec317
54 Shared Rabbit Connection: SimpleConnection#44bac9ec [delegate=amqp://21-05060179#10.24.133
.241:5671/], acknowledgeMode=NONE local queue size=0
Generally, this is due to the consumer thread being "stuck" in user code somewhere, so it can't react to the broken connection.
If you have network issues, perhaps it's stuck reading or writing to a socket; make sure you have timeouts set for any I/O operations.
Next time it happens take a thread dump to see what the consumer threads are doing.
I've been pounding my head on this for a while, I hope someone can help?
I have some JavaPNS code which pushes fine to my device when running from my local machine, however, when I copy that code over to my server, everything runs fine, no errors, but I never get the alert on my device?
After examining the logs from my server compared to my local box, I noticed I never get the flushing message on my server, I am using the JavaPNS queue, with 30 threads. In both cases, local box and server, I am sending less than 30 alerts.
public class PushWorker
{
private PushQueue queue = null;
public PushWorker(File keystore, String password, boolean production) throws KeystoreException
{
BasicConfigurator.configure();
int threads = 30;
this.queue = Push.queue(keystore, password, production, threads);
queue.start();
}
public void push(String message, String sound, String token, String eventId) throws JSONException
{
BasicDevice bd = new BasicDevice();
bd.setToken(token);
PushNotificationPayload payload = PushNotificationPayload.complex();
payload.addAlert(message);
payload.addSound(sound);
payload.addCustomDictionary("eid", eventId);
push(payload, bd);
}
private void push(Payload payload, Device device)
{
queue.add(payload, device);
}
}
----BELOW is flush message from my local box --------------
4872 [JavaPNS grouped notification thread in QUEUE mode] DEBUG javapns.notification.PushNotificationManager - Flushing
4872 [JavaPNS grouped notification thread in QUEUE mode] DEBUG javapns.notification.PushNotificationManager - At this point, the entire 139-bytes message has been streamed out successfully through the SSL connection
4872 [JavaPNS grouped notification thread in QUEUE mode] DEBUG javapns.notification.PushNotificationManager - Notification sent on first attempt
Can I force the flush somehow from the queue?
----------------------BELOW is the server JavaPNS logging-----------------------
0 [JavaPNS grouped notification thread in QUEUE mode] DEBUG javapns.communication.ConnectionToAppleServer - Creating SSLSocketFactory
16 [JavaPNS grouped notification thread in QUEUE mode] DEBUG javapns.communication.ConnectionToAppleServer - Creating SSLSocketFactory
49 [JavaPNS grouped notification thread in QUEUE mode] DEBUG javapns.communication.ConnectionToAppleServer - Creating SSLSocket to gateway.sandbox.push.apple.com:2195
49 [JavaPNS grouped notification thread in QUEUE mode] DEBUG javapns.communication.ConnectionToAppleServer - Creating SSLSocket to gateway.sandbox.push.apple.com:2195
177 [JavaPNS grouped notification thread in QUEUE mode] DEBUG javapns.notification.PushNotificationManager - Initialized Connection to Host: [gateway.sandbox.push.apple.com] Port: [2195]: 5117f31e[SSL_NULL_WITH_NULL_NULL: Socket[addr=gateway.sandbox.push.apple.com/17.172.233.65,port=2195,localport=56015]]
...
...
DEBUG javapns.notification.Payload - Adding alert [blah, blah my alert]
14767 [main] DEBUG javapns.notification.Payload - Adding sound [default]
14767 [main] DEBUG javapns.notification.Payload - Adding custom Dictionary [eid] = [193790]
14776 [JavaPNS grouped notification thread in QUEUE mode] DEBUG javapns.notification.PushNotificationManager - Building Raw message from deviceToken and payload
14776 [JavaPNS grouped notification thread in QUEUE mode] DEBUG javapns.notification.PushNotificationManager - Built raw message ID 16777217 of total length 135
14777 [JavaPNS grouped notification thread in QUEUE mode] DEBUG javapns.notification.PushNotificationManager - Attempting to send notification: {"aps":{"sound":"default","alert":"blah, blah my alert"},"eid":"193790"}
14777 [JavaPNS grouped notification thread in QUEUE mode] DEBUG javapns.notification.PushNotificationManager - to device: [my device number]
And that's it, no flush...
Do you have iptables configured on the server or some other firewall device?
Maybe port 2195 needs to be allowed.
Try using telnet to test:
$ telnet gateway.sandbox.push.apple.com 2195
Do you have SELinux enabled? check the following setting:
$ getsebool -a | grep httpd
httpd_can_network_connect --> ?
make sure it is on.
Did you check the documentation on the javapns repo site?
Let us know how it goes.
Bill