I want to list all of files on an FTP server using spring-integration and, for example, print them on screen. I've done something like this:
context:
<int:channel id="toSplitter">
<int:interceptors>
<int:wire-tap channel="logger"/>
</int:interceptors>
</int:channel>
<int:logging-channel-adapter id="logger" log-full-message="true"/>
<int:splitter id="splitter" input-channel="toSplitter" output-channel="getFtpChannel"/>
<int-ftp:outbound-gateway id="gatewayLS"
session-factory="ftpClientFactory"
request-channel="inbound"
command="ls"
expression="payload"
reply-channel="toSplitter"/>
<int:channel id="getFtpChannel">
<int:queue/>
</int:channel>
<bean id="ftpClientFactory"
class="org.springframework.integration.ftp.session.DefaultFtpSessionFactory">
<property name="host" value="${host}"/>
<property name="username" value="${user}"/>
<property name="password" value="${password}"/>
<property name="clientMode" value="0"/>
<property name="fileType" value="2"/>
<property name="bufferSize" value="10000000"/>
</bean>
Java code:
ConfigurableApplicationContext context =
new FileSystemXmlApplicationContext("/src/citrus/resources/citrus-context.xml");
final FtpFlowGateway ftpFlow = context.getBean(FtpFlowGateway.class);
ftpFlow.lsFiles("/");
PollableChannel channel = context.getBean("getFtpChannel", PollableChannel.class);
variable("tt", channel.receive().toString());
echo("${tt}");
output:
11:09:17,169 INFO port.LoggingReporter| Test action <echo>
11:09:17,169 INFO actions.EchoAction| [Payload=FileInfo [isDirectory=false, isLink=false, Size=3607, ModifiedTime=Tue Jul 15 14:18:00 CEST 2014, Filename=Smoke03_angart30_st40.exi, RemoteDirectory=/, Permiss
ions=-rw-r--r--]][Headers= {replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel#7829b776, sequenceNumber=1, errorChannel=org.springframework.integration.core.MessagingTempla
te$TemporaryReplyChannel#7829b776, file_remoteDirectory=/, sequenceSize=1, correlationId=49b57f2d-4dbf-4a89-b5b8-0dfb15bca2be, id=0a58ad65-74b4-4aae-87be- aa6034a41776, timestamp=1405501757060}]
11:09:17,169 INFO port.LoggingReporter| Test action <echo> done
11:09:17,169 INFO port.LoggingReporter| TEST STEP 1/1 done
The output is fine, but what should I do to print this information when I don't know how many files are stored on the FTP? (this code prints only one file). I've tried checking if channel.receive() is null but the test just freezes.
Since you send the result of LS to the <splitter>, your getFtpChannel will receive FileInfo<?> objects one by one.
To print them all you really should have an infinite loop:
while (true) {
variable("tt", channel.receive().toString());
echo("${tt}");
}
To stop the app you should provide some shoutDownHook or listen something from console input.
Another point, that it is bad to block your app with infinite receive().
There is na overloaded method, which applies a timeout param. The last one might be useful to determine the end of your loop:
while (true) {
Message<?> receive = channel.receive(10000);
if (receive == null) {
break;
}
variable("tt", receive.toString());
echo("${tt}");
}
#Configuration
public class FtpConfig {
#Bean
public DefaultFtpSessionFactory ftpSessionFactory() {
DefaultFtpSessionFactory ftpSessionFactory = new DefaultFtpSessionFactory();
ftpSessionFactory.setHost("localhost");
ftpSessionFactory.setPort(21);
ftpSessionFactory.setUsername("user");
ftpSessionFactory.setPassword("pass");
return ftpSessionFactory;
}
}
#Bean
public ApplicationRunner runner(DefaultFtpSessionFactory sf) {
return args -> {
FTPFile[] list = sf.getSession().list(".");
for (FTPFile file: list ) {
System.out.println("Result: " + file.getName());
}
};
}
Related
I have an activemq queue where messages are being sent to and the application uses Spring JmsTemplate receiveSelected(selector) to receive the messages synchronously. Message is processed before it is acknowledged. If the broker or the application shuts down before acknowledging the message while it is being processed the message needs to be resent or redelivered without getting lost. My understanding is with client_acknowledgement messages gets resent if not acknowledged also.
Configuration
<bean id="jmsConnection" class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="broker" value="tcp://localhost:61616" />
<property name="redeliveryPolicy" ref="redeliveryPolicy" />
</bean>
Redelivery Policy:
<bean id="redeliveryPolicy" class="org.apache.activemq.RedeliveryPolicy">
<property name="initialRedeliveryDelay" value="1000" />
<property name="redeliveryDelay" value="10000" />
<property name="maximumRedeliveries" value="2" />
<property name="useExponentialBackOff" value="true" />
<property name="backOffMultiplier" value="5" />
</bean>
JmsTemplate:
<bean id="jmstemplate" class="org.springframework.jms.core.JmsTemplate">
<property name="connectionFactory">
<bean class="org.springframework.jms.connection.CachingConnectionFactory">
<property name="targetConnectionFactory" ref="jmsConnection" />
</bean>
</property>
<property name="receiveTimeout" value="5000"/>
<property name="defaultDestinationName" value="messageQueue"/>
<property name="explicitQosEnabled" value="true"/>
<property name="sessionAcknowledgeMode" value="2"/>
<property name="sessionTransacted" value="false"/>
</bean>
Session call back to receive message:
Object getProcessedMessageObject {
return jmsTemplate.execute(session -> {
#Override
public Object doInJms (Session session) throws JMSException {
Object tmp = null;
MapMessage msg = (MapMessage) jmsTemplate.receiveSelected(selector);
try {
if (msg != null) {
MapMessage receivedMsg = msg;
tmp = processMsg(receivedMsg) if (tmp != null) {
msg.acknowledge();
}
}
} catch (JMSException) {
throw new RuntimeException();
} return tmp;
}
});
}
I am getting "Consumer is closed" when msg.acknowledge() is called. When I stop and restart my application the messages are not redelivered as they are not acknowledged. I'm trying to understand what I am missing and how to make it work.
I am using Spring SimpleMessageListenerConatiner where acknowledgement mode is 2 (client acknowledge) and Queue is Solace.
When I am throwing runtime exception from my unit test, means standalone spring config, messages are redelivering without any issue, but same code is not working when I am deploying my application in JBOSS.
public class MyListener implements MessageListener {
public void onMessage(Message message) {
try {
throw new ConnectionException("Error in Connection");
} catch (ConnectionException e) {
LOGGER.error("Throwing exception...");
throw new MyRuntimeException("Throwing exception");
} finally {
LOGGER.info("Done...");
}
}
Spring config is:
<bean id="solaceMessageListener" class="org.springframework.jms.listener.SimpleMessageListenerContainer">
<property name="connectionFactory" ref="solaceConnectionFactory"/>
<property name="destinationName" value="QueueName"/>
<property name="messageListener" ref="myListener"/>
<property name="concurrency" value="1"/>
<property name="destinationResolver" ref="destinationResolver" />
<property name="sessionAcknowledgeMode" value="2"/>
</bean>
Constaint:
1. I cannot use DefaultMessageListenerContainer
2. Session Transacted true is working but we have not to use it.
This is my main class
public class FtpInboundChannelAdapterSample {
private static final Logger LOGGER = LoggerFactory.getLogger(FtpInboundChannelAdapterSample.class);
public static void main(String[] args) throws Exception {
FtpInboundChannelAdapterSample ftp=new FtpInboundChannelAdapterSample();
ftp.ftpDownload();
}
public void ftpDownload() throws Exception {
ConfigurableApplicationContext ctx = new ClassPathXmlApplicationContext(
"META-INF/spring/integration/FtpInboundChannelAdapterSample-context.xml");
PollableChannel ftpChannel = ctx.getBean("ftpChannel", PollableChannel.class);
System.out.println("Files are transferred ");
ctx.close();
}
}
And my .xml is as follow
<bean id="ftpClientFactory" class="org.springframework.integration.ftp.session.DefaultFtpSessionFactory">
<property name="host" value="${host}"/>
<property name="port" value="${availableServerPort}"/>
<property name="username" value="${userid}"/>
<property name="password" value="${password}"/>
</bean>
<int-ftp:inbound-channel-adapter id="ftpInbound"
channel="ftpChannel"
session-factory="ftpClientFactory"
filename-pattern="*.*"
auto-create-local-directory="true"
delete-remote-files="false"
remote-directory="ftp://portal.********.com/foldername/"
local-directory="${localDir}">
<int:poller fixed-rate="1000"/>
</int-ftp:inbound-channel-adapter>
<int:channel id="ftpChannel">
<int:queue/>
</int:channel>
Is this a way to give the remote directory path?
remote-directory="ftp://portal.********.com/foldername/"
or
remote-directory="/ftpSample/"
I tried both but not downloading from server
If I'm using localhost with FileZilla server it was downloading..
The correct syntax is remote-directory="/ftpSample/" - you don't need a URL because the connection is specified by the session factory.
Are you sure you need the leading /, should it be remote-directory="ftpSample/" ?
Turn on DEBUG logging and check the server logs as well.
Try with this:
remote-directory="ftps://portal.********.com/foldername/" or
remote-directory="sftp://portal.********.com/foldername/"
I am trying to read data from cassandra using spring batch, where I have implemented ItemReader, ItemProcessor, and ItemWriter. I am able to read the data , process it and write back the data to the same table. I am creating xml file to execute the job:
xml:
<job id="LoadStatusIndicator" job-repository="jobRepository" restartable="false">
<step id="LoadStatus" next="">
<tasklet>
<chunk reader="StatusReader" processor="ItemProcessor" writer="ItemWriter"
commit-interval="10" />
</tasklet>
</step>
</job>
<beans:bean id="ItemWriter" scope="step"
class="com.batch.writer.ItemWriter">
</beans:bean>
<beans:bean id="ItemProcessor" scope="step"
class="com.batch.processor.ItemProcessor">
</beans:bean>
<beans:bean id="Reader" scope="step"
class="com.reader.ItemReader">
<beans:property name="dataSource" ref="CassandraSource" />
</beans:bean>
applicationcontext.xml:
<beans:bean id="CassandraSource" parent="DataSourceParent">
<beans:property name="url" value="jdbc:cassandra://${cassandra.hostName}:${cassandra.port}/${cassandra.keyspace}" />
<beans:property name="driverClassName" value="org.apache.cassandra.cql.jdbc.CassandraDriver" />
</beans:bean>
reader class:
public static final String query = "SELECT * FROM test_1 allow filtering;";
#Override
public List<Item> read() throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException
{
List<Item> results = new ArrayList<Item>();
try {
results = cassandraTemplate.select(query,Item.class);
} catch (Exception e) {
e.printStackTrace();
}
return results;
}
writer classs:
#Override
public void write(List<? extends Item> item) throws Exception {
try {
cassandraTemplate.insert(item);
}catch(Exception e){e.printStackTrace();}
But the problem is the whole job is getting executed multiple times , infact it is not stopping at all. I have to force stop the job execution. I have only 2 rows in the table.
I think it is because of the commit-interval defined in xml, but having commit-interval = 10, job executes more than 10 times
According to my understanding, when I run the xml file that means I am running the job only one time, it calls the reader once keeps the data in the run time memory (job repository), calls item processor once (I use list ) and the whole list is inserted at once
SOLVED
In reader class I wrote:
if (results.size!=0)
return results;
else
return null;
I have a problem the InOnly exchange pattern what I use with aciveMq.
I wrote a module what run in ServiceMix. It works correctly except that it send every message to dead letter queue (ActiveMQ.DLQ). If I check the message then dlqDeliveryFailureCause contains this message: java.lang.Throwable: Message Expired.
I checked the JMSExpiration = 0.
The route:
from("direct:" + reqOutQueue).id("reqInEnritch")
.log("Start dispatch")
.setExchangePattern(ExchangePattern.InOnly)
.recipientList().method(EsbDynamicRouter.class, "systemRoute").parallelProcessing();
The function, what gives back the endpoint list:
#RecipientList
public String[] systemRoute(#Body Object body) throws Exception
{
String[] result = null;
List<DispConfView> targetList;
int cikl = 0;
SystemQueueHelper systemInfo;
MessageHeaderHelper msgHeadHelp = new MessageHeaderHelper(body);
// The object contains the affected elements
targetList = dispHelp.getDispConfByMsgType(msgHeadHelp.getMsgType());
result = new String[targetList.size()];
for (DispConfView element : targetList)
{
// It builds the target andpoints
systemInfo = new SystemQueueHelper(element.getSystemCode(), null, msgHeadHelp.getDirection());
result[cikl] = systemInfo.getQueuName();
cikl++;
}
return result;
}
The list contains these values:
activemq:queue:ERP.req.in?exchangePattern=InOnly
activemq:queue:WF.req.in?exchangePattern=InOnly
As you see, I try to set the correct pattern, but every messages go to dead letter queue.
Please help, what I have to set up!
Thank you!
The solution:
It is settable in context file:
<!-- JMS configuration -->
<bean id="pooledJmsConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory" init-method="start" destroy-method="stop">
<property name="maxConnections" value="1" />
<property name="connectionFactory" ref="jmsConnectionFactory" />
</bean>
<bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="failover:(tcp://localhost:61616)" />
<property name="redeliveryPolicy">
<bean class="org.apache.activemq.RedeliveryPolicy">
<property name="maximumRedeliveries" value="5"/>
</bean>
</property>
</bean>
<bean id="jmsConfig" class="org.apache.camel.component.jms.JmsConfiguration">
<property name="connectionFactory" ref="pooledJmsConnectionFactory"/>
<property name="cacheLevelName" value="CACHE_CONSUMER" />
<property name="disableReplyTo" value="true" />
</bean>
The "jmsConfig bean ""diasbleReplayTo" property solve the problem.