In mule I have a flow that receive an InputStream with a 500mb xml file.
I want to split the xml file following certain rules.
The result is 390000 messages that I need to send to an ActiveMQ queue.
ActiveMQ will give an outofmemory exception if I send all messages in one transaction.
If I don't use transaction it will succeed but it will be much slower.
What's the best way to send the messages in batches of 1000?
Can I use standard components?
I am using ActiveMQ 5.13 and Mule 3.7
Thanks
I think what you are looking for batch commits doc here.
This component will execute the internal flow components once he collected the specified numbers of records.
Example:
<batch:commit size="100" doc:name="Batch Commit">
<!-- Put here all the message processor you want for example the active mq one -->
</batch:commit>
Please note that transaction are allowed only at step level and therefore also in the commit block that I think this is exactly what are you looking for.
Footnote: Batch commit processor is allowed only inside a batch step.
Hope this helps
Regards
Related
Unfortunately Spring Integration metrics are registered when they are first used.
https://stackoverflow.com/a/63619360/13657000
this means that prometheus query functions like increase give incorrect calculations.
Is there a way to get all channel names in Spring integration so that I can initialize metrics for each channel?
I'm using dsl.
I looked at this https://gist.github.com/flopezluis/2964429 but they find their channel names using XML.
Any help is appreciated, thanks.
I'm not sure why you need names of channels since the story is really about calling a MessageChannel.send(), so you perhaps just need an ApplicationContext.getBeansOfType(MessageChannel.class). Pay attention though, that sending a message to the channel will trigger not only metrics registration but also their consumption on the other side. Therefore you might need to think about filtering these "initial" messages somehow before they reach your real consumer.
On the other hand I wonder if there is some Micrometer option to make those timer metrics to be registered eagerly even if we don't produce messages yet. Just call registry.timer() as early as possible?
I was wondering on a Spring Boot method marked as #Transactional, when should a message appear on the queue? I’m asking because I’ve noticed 2 different behaviours on 2 different applications.
Both applications have the following in common:
Spring Boot 2.0 based
JMS message sending is using JmsTemplate, with setSessionTransacted set to true
No explicit transaction settings configured
There is a Mongo DB used (using Spring Data) and a record is being modified in the same method as the message is sent
The major difference between the two applications is:
One has a JPA data source (using Spring Dataj and a record is read and/or written in this method. The other application does not have this data source.
The difference in observed behaviour is that when the JPA source is present, the message is sent at the end of method. Without it, the message is sent immediately.
Is this the expected behaviour?
Can I configure the applications to behave the same way? Ideally I’d like the message to be sent at the end (so any Mongo changes that fail would cancel the message send and rollback any JPA changes made)? I realise that Mongo changes are not part of any transaction created.
Thanks
With JMS and a DB you have two resources.
To have a full transactional behavior you need distributed transactions support.
If you don't have this even when the message is sent as last operation if the sending fails the data is changed in the database anyway.
To configure distributed transaction you need JTA. This is described here:
With JMS and a DB you have two resources. To have a full transactional behavoir you need distributed transactions. We use Bistronix in our application and this works very well.
Have a look at the docs: https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-jta.html
The use case is following.
I am passing producer or consumer reference over many objects instances in Java code.
At some of them I would like to do some checks for the Kafka configuration.
It means I would like to get back, what effective configuration is stored in Kafka Producer/Consumer (including defaults).
I do not see anthing explicit in java docs:
KafkaProducer
KafkaConsumer
So, how to get back Kafka producer and consumer configuration?
Unfortunately it's not possible. I have to admit it could be a useful feature for showing the "core" configuration properties at least (avoiding the possibility to get the "secrets" for authentication stuff for example).
The only solution that I see today for you is to have a link between the consumer/producer instance and the related properties bag used for setting the client configuration. I understand it's a waste of memory because such configuration is internally in the client but you need to keep your properties bag for having it.
I have used Spring Integration in my current successfully for some of the needs. Awesome..
There is some weird behavior observed on a heavy load where-in the same message seems to be processed more than once. I can confirm that because there are multiple rows in the database which is typically the last command on the chain that is configured over the channel.
Digging into the manual further, it looks seems like load-balancing is done automatically by spring. The manual says that the message is balanced between multiple message handlers.
Question is:
How many handlers are present on a channel by default? The spring XML that gets loaded does not seem to have that configuration. All i do is this (per the recommendation in the manual):
<int:channel id="SwPath.Channel"/>
<int:chain id="SwPath.chain" input-channel="SwPath.Channel">
</int:chain>
I can disable the fail-over but I am curious to know how many are present by default.
It's been a while since I worked on those load balancers, but I remember that the default number of threads in the thread pool was somewhere between 2 and 10.
It is possible that you have found a concurrency bug.
If you turn on TRACE logging the load balancer will give you a lot of information, but that could easily hide the problem.
If you would create a JIRA issue with a JUnit test case, I'm sure it would be much easier to figure out what happens exactly.
I'm working with core java and IBM Websphere MQ 6.0. We have a standalone module say DBcomponent that hits the database and fetches a resultset based on the runtime query. The query is passed to the application via MQ messaging medium. We have a trigger configured for the queue which invokes the DBComponent whenever a message is available in the queue. The DBComponent consumes the message, constructs the query and returns the resultset to another queue. In this overall process we use log4j to log statements on a log file for auditing.
The connection is pooled to the database using Apache pool. I am trying to check whether the log messages are logged correctly using a sample program. The program places the input message to the queue and checks for the logs in the log file. Its expected for the trigger method invocation to complete before i try to check for the message in log file, but every time my program to check for log message gets executed first leading my check to failure.
Even if i introduce a Thread.sleep(time) doesn't solves the case. How can i make it to keep my method execution waiting until the trigger operation completes?
Any suggestion will be helpful.
I suggest you go and read up about the concurrency primitives that Java offers you. http://tutorials.jenkov.com/java-concurrency/index.html seems to cover the bases, the Thread Signalling chapter in particular.
I would recommend against relying on log4j (or any logging functionality) even in a simple test program.
Have your test run as you would expect it to, putting debugging/tracing statements in the log as you see fit (be liberal about it, log4j is very fast!) Then, when it's done, check the log yourself.
Writing log parsing will only complicate your goals.
Write your test, view the result, view the logs. If you want automated testing, consider setting up a functional test. You can set up tests free using Selenium. (http://seleniumhq.org/) There's no need to write your own functional testing/parsing stuff when there's easy to configure, easy to use, easy to customize frameworks out there! :-)