I have used Spring Integration in my current successfully for some of the needs. Awesome..
There is some weird behavior observed on a heavy load where-in the same message seems to be processed more than once. I can confirm that because there are multiple rows in the database which is typically the last command on the chain that is configured over the channel.
Digging into the manual further, it looks seems like load-balancing is done automatically by spring. The manual says that the message is balanced between multiple message handlers.
Question is:
How many handlers are present on a channel by default? The spring XML that gets loaded does not seem to have that configuration. All i do is this (per the recommendation in the manual):
<int:channel id="SwPath.Channel"/>
<int:chain id="SwPath.chain" input-channel="SwPath.Channel">
</int:chain>
I can disable the fail-over but I am curious to know how many are present by default.
It's been a while since I worked on those load balancers, but I remember that the default number of threads in the thread pool was somewhere between 2 and 10.
It is possible that you have found a concurrency bug.
If you turn on TRACE logging the load balancer will give you a lot of information, but that could easily hide the problem.
If you would create a JIRA issue with a JUnit test case, I'm sure it would be much easier to figure out what happens exactly.
Related
Unfortunately Spring Integration metrics are registered when they are first used.
https://stackoverflow.com/a/63619360/13657000
this means that prometheus query functions like increase give incorrect calculations.
Is there a way to get all channel names in Spring integration so that I can initialize metrics for each channel?
I'm using dsl.
I looked at this https://gist.github.com/flopezluis/2964429 but they find their channel names using XML.
Any help is appreciated, thanks.
I'm not sure why you need names of channels since the story is really about calling a MessageChannel.send(), so you perhaps just need an ApplicationContext.getBeansOfType(MessageChannel.class). Pay attention though, that sending a message to the channel will trigger not only metrics registration but also their consumption on the other side. Therefore you might need to think about filtering these "initial" messages somehow before they reach your real consumer.
On the other hand I wonder if there is some Micrometer option to make those timer metrics to be registered eagerly even if we don't produce messages yet. Just call registry.timer() as early as possible?
In mule I have a flow that receive an InputStream with a 500mb xml file.
I want to split the xml file following certain rules.
The result is 390000 messages that I need to send to an ActiveMQ queue.
ActiveMQ will give an outofmemory exception if I send all messages in one transaction.
If I don't use transaction it will succeed but it will be much slower.
What's the best way to send the messages in batches of 1000?
Can I use standard components?
I am using ActiveMQ 5.13 and Mule 3.7
Thanks
I think what you are looking for batch commits doc here.
This component will execute the internal flow components once he collected the specified numbers of records.
Example:
<batch:commit size="100" doc:name="Batch Commit">
<!-- Put here all the message processor you want for example the active mq one -->
</batch:commit>
Please note that transaction are allowed only at step level and therefore also in the commit block that I think this is exactly what are you looking for.
Footnote: Batch commit processor is allowed only inside a batch step.
Hope this helps
Regards
In our project we use Activemq (jms template) - to publish many events from one webapp to another.
we use logging aspect (spring aop) as well - mainly we log errors and entering\quitting methods.
Now, sometimes we face racing conditions on the flow of the systems. i.e. an entity is being created on one web app, an event is fired to notify another webapp, but the handling of the other webapp requires a different event to be handled first, so if such scenario happens, the handling fails (for example, an id is missing ) and immediately retried (jms re-delivery), on the 2nd time of the retry its usually works (never more then 3 retries are required).
So basically, we have exceptions as part of our day to day flow, but:
our log files are huge and messy because of the exceptions thrown by such scenarios, any idea how can we not log the first few retries exceptions and only on a later exception we will log? Maybe another approach you can recommend?
Thanks.
You can use JMSXDeliveryCount property of Message to get the redelivery count. See http://activemq.apache.org/activemq-message-properties.html
I'm looking for a way to define transitive log message routing. Let's say we have an application called poly with these packages:
com.mycompany.server-common
com.mycompany.communication
com.mycompany.webservice
server-common is used by both of the 2 others. All 3 use org.hibernate as well.
Now, I like to have 1 logfile for the webservice component with all messages from com.mycompany.webservice and with those messages from com.mycompany.server-common and org.hibernate that were initiated by the webservice. And then, another coresponding file for the communication package.
My application is a war file running in tomcat, where all components run in 1 context (it comes in 1 war file). I already defined the multiple log files, but they naturally only log that what i defined statically, there is no transitive inclusion.
I would be very interested in ideas how I could achieve the desired behaviour. I already thought about using the MDC for that, but I'm not sure if that's a good idea.
Another idea was to separate the contexts, but I think in the current project state this will be hard and it does not offer the flexibility I hope for.
Any hints or discussions are appreciated.
If you set an MDC key when webservice starts serving a request and clear the MDC key at the end of the request, SiftingAppender will do what you are asking. Shout on the logback-user mailing list if you run into difficulties.
I'm working with core java and IBM Websphere MQ 6.0. We have a standalone module say DBcomponent that hits the database and fetches a resultset based on the runtime query. The query is passed to the application via MQ messaging medium. We have a trigger configured for the queue which invokes the DBComponent whenever a message is available in the queue. The DBComponent consumes the message, constructs the query and returns the resultset to another queue. In this overall process we use log4j to log statements on a log file for auditing.
The connection is pooled to the database using Apache pool. I am trying to check whether the log messages are logged correctly using a sample program. The program places the input message to the queue and checks for the logs in the log file. Its expected for the trigger method invocation to complete before i try to check for the message in log file, but every time my program to check for log message gets executed first leading my check to failure.
Even if i introduce a Thread.sleep(time) doesn't solves the case. How can i make it to keep my method execution waiting until the trigger operation completes?
Any suggestion will be helpful.
I suggest you go and read up about the concurrency primitives that Java offers you. http://tutorials.jenkov.com/java-concurrency/index.html seems to cover the bases, the Thread Signalling chapter in particular.
I would recommend against relying on log4j (or any logging functionality) even in a simple test program.
Have your test run as you would expect it to, putting debugging/tracing statements in the log as you see fit (be liberal about it, log4j is very fast!) Then, when it's done, check the log yourself.
Writing log parsing will only complicate your goals.
Write your test, view the result, view the logs. If you want automated testing, consider setting up a functional test. You can set up tests free using Selenium. (http://seleniumhq.org/) There's no need to write your own functional testing/parsing stuff when there's easy to configure, easy to use, easy to customize frameworks out there! :-)