We Keep Coding

camel pollEnrich is not working for the second time

I am reading and processing 2 files from 2 different file locations and comparing the content.
If 2nd file is not available , the rest of the process execute with 1st file. If 2nd file is available, comparison process should happen. For this I am using camel pollEnrich, but here the problem is that, camel is picking the 2nd file at first time only. Without restarting the camel route 2nd file is not getting picked up even if it is present there.
After restarting the camel route it is working fine, but after that its not picking the 2nd file.
I am moving the files to different locations after processing it.
Below is my piece of code,
from("sftp:" + firstFileLocation + "?privateKeyFile=" + ppkFileLocation + "&username=" + sftpUsername
+ "&readLock=changed&idempotent=true&move=" + firstFileArchiveLocation)
.pollEnrich("sftp:" + secondFileLocation + "?privateKeyFile=" + ppkFileLocation + "&username=" + sftpUsername
+ "&readLock=changed&idempotent=true&fileExist=Ignore&move="+ secondFileLocationArchive ,10000,new FileAggregationStrategy())
.routeId("READ_INPUT_FILE_ROUTE")
Need help.

You're setting idempotent=true in the sftp consumer, which means camel will not process the same file name twice. Since you're moving the files, it would make sense to set idempotent=false.
Quoted from camel documentation
Option to use the Idempotent Consumer EIP pattern to let Camel skip
already processed files. Will by default use a memory based LRUCache
that holds 1000 entries. If noop=true then idempotent will be enabled
as well to avoid consuming the same files over and over again.

I'm adding an alternative solution based on comments for the answer posted by Jeremy Ross. My answer is based on the following code example. I've only added the configure() method in the test route for brevity.
#Override
public void configure() throws Exception {
String firstFileLocation = "//127.0.0.1/Folder1";
String secondFileLocation = "//127.0.0.1/Folder2";
String ppkFileLocation = "./key.pem";
String sftpUsername = "user";
String sftpPassword = "xxxxxx";
String firstFileArchiveLocation = "./Archive1";
String secondFileLocationArchive = "./Archive2";
IdempotentRepository repository1 = MemoryIdempotentRepository.memoryIdempotentRepository(1000);
IdempotentRepository repository2 = MemoryIdempotentRepository.memoryIdempotentRepository(1000);
getCamelContext().getRegistry().bind("REPO1", repository1);
getCamelContext().getRegistry().bind("REPO2", repository2);
from("sftp:" + firstFileLocation
+ "?password=" + sftpPassword + "&username=" + sftpUsername
+ "&readLock=idempotent&idempotent=true&idempotentKey=\\${file:name}-\\${file:size}-\\${file:modified}" +
"&idempotentRepository=#REPO1&stepwise=true&download=true&delay=10&move=" + firstFileArchiveLocation)
.to("direct:combined");
from("sftp:" + secondFileLocation
+ "?password=" + sftpPassword + "&username=" + sftpUsername
+ "&readLock=idempotent&idempotent=true&idempotentKey=\\${file:name}-\\${file:size}-\\${file:modified}" +
"&idempotentRepository=#REPO2" +
"&stepwise=true&delay=10&move=" + secondFileLocationArchive)
.to("direct:combined");
from("direct:combined")
.aggregate(constant(true), (oldExchange, newExchange) -> {
if (oldExchange == null) {
oldExchange = newExchange;
}
String fileName = (String) newExchange.getIn().getHeaders().get("CamelFileName");
String filePath = (String) newExchange.getIn().getHeaders().get("CamelFileAbsolutePath");
if (filePath.contains("Folder1")) {
oldExchange.getIn().setHeader("File1", fileName);
} else {
oldExchange.getIn().setHeader("File2", fileName);
}
String file1Name = oldExchange.getIn().getHeader("File1", String.class);
String file2Name = oldExchange.getIn().getHeader("File2", String.class);
if (file1Name != null && file2Name != null) {
// Compare files
// Both files are available
oldExchange.getIn().setHeader("PROCEED", true);
} else if (file1Name != null) {
// No comparison, proceed with File 1
oldExchange.getIn().setHeader("PROCEED", true);
} else {
// Do not proceed, keep file 2 data and wait for File 1
oldExchange.getIn().setHeader("PROCEED", false);
}
String fileName1 = oldExchange.getIn().getHeader("File1", String.class);
String fileName2 = oldExchange.getIn().getHeader("File2", String.class);
oldExchange.getIn().setBody("File1: " + fileName1 + " File2: " + fileName2);
System.out.println(oldExchange);
return oldExchange;
}).completion(exchange -> {
if(exchange.getIn().getHeader("PROCEED", Boolean.class)) {
exchange.getIn().removeHeader("File1");
exchange.getIn().removeHeader("File2");
return true;
}
return false;
}).to("log:Test");
}
In this solution, two SFTP consumers were used, instead of pollEnrich, since we need to capture the file changes of both SFTP locations. I have used an idempotent repository and an idempotent key for ignoring duplicates. Further, I've used the same idempotent repository as the lock store assuming only camel routes are accessing the files.
After receiving the files from SFTP consumers, they are sent to the direct:combined producer, which then routes the exchange to an aggregator.
In the example aggregator strategy I have provided, you can see, that the file names are being stored in the exchange headers. According to the file information retrieved from the headers, the aggregator can decide how to process the file and whether or not to proceed with the exchange. (If only file2 is received, the exchange should not proceed to the next stages/routes)
Finally, the completion predicate expression decides whether or not to proceed with the exchange and log the exchange body, based on the headers set by the aggregator. I have added an example clean-up process in the predicate expression processor as well.
Hope you will get the basic idea of my suggestion to use an aggregator from this example.

Getting server 500 internal error in Postman but error not show in the spring suite tools Console

I am trying to execute a registration method having te API:
http://localhost:100/api/v1/registration/connect
When I run the front-end and the back-end and inspect the source code, I receive an internal server error 500
Internal Server Error 500
Regarding this, I went to check the Spring Tools Suite Console to check where the issue is but I found no error there. I need to see the error in the console to figure out the problem.
This is the method where I receive an error:
#Override
public String login(LoginRequest request) { //updated recently
if (userRepo.findByEmail(request.getEmail()).isPresent() && passwordEncoder
.matches(request.getPassword(), userRepo.findByEmail1(request.getEmail()).get().getPassword())) {
String token = UUID.randomUUID().toString();
ConfirmationToken confirmationToken = new ConfirmationToken(
token,
LocalDateTime.now(),
LocalDateTime.now().plusMinutes(5),
userRepo.findByEmail1(request.getEmail()).get()
);
confirmationTokenService.saveConfirmationToken(
confirmationToken);
t = confirmationToken.getExpiresAt();
emailSender.send(request.getEmail(), buildEmail(userRepo.findByEmail1(request.getEmail()).get().getFirstName(), token));
return "{" +
"\n\"text\" :\"Utilisateur connecté avec succès, voici le token de connexion\"," + token + "\"," +
"\n\"email\" :\"" + userRepo.findByEmail1(request.getEmail()).get().getEmail() + "\"," +
"\n\"Role\" :\"" + userRepo.findByEmail1(request.getEmail()).get().getRoles().toString() + "\"," +
"\n\"first_name\" : \"" + userRepo.findByEmail1(request.getEmail()).get().getFirstName() + "\"," +
"\n\"Enabled\" : " + userRepo.findByEmail1(request.getEmail()).get().getEnabled() +
"}";
}
throw new IllegalStateException(" Les identifiants de connexion sont eronnés ");
}
Here's the controller method:
#CrossOrigin(origins = "http://localhost:8081")
#PostMapping(path = "/connect")
public String login(#RequestBody LoginRequest request) {
return userService.login(request);
}
enter image description here

you can add logger/system.out.println/debug code/enable console to see error
make sure you request body has all valid data while requesting

Apache camel kafka aggregate before produce msg but lost header

I used apache camel Kafka with spring boot
<camel.version>3.14.2</camel.version>
I used default configuration on apache camel Kafka component
<dependency>
<groupId>org.apache.camel.springboot</groupId>
<artifactId>camel-kafka-starter</artifactId>
<version>${camel.version}</version>
</dependency>
My route camel - fileConsume have 6000 lines
from(fileConsume).split(body().tokenize()).setHeader("testHeader", "valueHeader").aggregate(new GroupedMessageAggregationStrategy())
.constant(true).completionTimeout(100L).to("kafka:topicTest");
All messages from file produced on Kafka very fast (less 2 secondes) but the header is not present.
When i remove aggregate
from(fileConsume).split(body().tokenize()).setHeader("testHeader", "valueHeader").to("kafka:topicTest");
All messages from file produced on Kafka very low (more 10 minutes) but the header is present.
I need some help to produce message with apache camel kafka component on speed way with header.

you must do this, in order to keep header when aggregation is doing.
from("sftp://xxxxxxx#localhost:"
+ "2222/data/in"
+ "?password="
+ "&preferredAuthentications=publickey"
+ "&knownHostsFile=~/.ssh/known_hosts"
+ "&privateKeyFile=xxxxxxx"
+ "&privateKeyPassphrase="
+ "&passiveMode=true"
+ "&fastExistsCheck=true"
+ "&download=true"
+ "&delete=true"
+ "&stepwise=false"
+ "&antInclude=*"
+ "&antExclude=**reject**"
+ "&recursive=false"
+ "&maxMessagesPerPoll=10"
+ "&initialDelay=0"
+ "&delay=0"
+ "&connectTimeout=10000"
+ "&soTimeout=300000"
+ "&timeout=30000"
+ "&shuffle=true"
+ "&eagerMaxMessagesPerPoll=false"
+ "&moveFailed=reject"
+ "&binary=true"
+ "&localWorkDirectory=/opt/camel_data/kafka/"
+ "&readLock=none"
+ "&readLockCheckInterval=1000"
+ "&readLockMinLength=1"
+ "&readLockLoggingLevel=INFO"
+ "&readLockIdempotentReleaseDelay=10000"
+ "&readLockRemoveOnCommit=false"
+ "&readLockRemoveOnRollback=true"
+ "&bulkRequests=1000"
+ "&charset=utf-8")
.routeId("Consume SFTP")
.id("Consume SFTP")
.setProperty("yoda_core_technical_id").header(Exchange.BREADCRUMB_ID)
.setProperty("x_filename_source").header(Exchange.FILE_NAME_ONLY)
.setProperty("x_filepath_source").header("CamelFileAbsolutePath")
.setProperty("x_correlation_id").header("CamelFileName")
.split(body().tokenize())
.setHeader("test",constant("test"))
.aggregate(new GroupedMessageAggregationStrategy())
.constant(true)
.completionTimeout(100L)
.to("direct:aggregate");
from("direct:aggregate")
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
System.out.println(exchange);
GenericFileMessage<String> message =(GenericFileMessage<String>) exchange.getMessage().getBody(List.class).get(0);
exchange.getMessage().setHeader("test",
message.getHeader("test"));
}
})
.to("mock:result");

How can I access the text of the SMS sent thru Twilio to my Spring Boot Controller?

I'm trying to use Twilio to receive SMS messages, and use the body of the SMS to perform various DB functions. The part where I'm stuck is parsing the message that I get from Twilio when they receive a text message.
Here's the controller:
#RequestMapping(
value = "/incomingSMS",
method = RequestMethod.POST)
public void getPhrase(#RequestBody String request) {
System.out.println("***********************************");
System.out.println(request);
System.out.println("***********************************");
}
And here's what is getting printed out (with new lines added for readability, and some numbers censored.):
ToCountry=US&
ToState=statecode&
SmsMessageSid=smsMessageSid&
NumMedia=0&
ToCity=city&
FromZip=zipCode&
SmsSid=SmsSid&
FromState=statecode&
SmsStatus=received&
FromCity=city&
Body=Hello+it%27s+John+&
FromCountry=US&
To=%2B1toPhoneNumber&
ToZip=55401&
NumSegments=1&
MessageSid=messageSid&
AccountSid=accountSid&
From=%2B1fromPhoneNumber&
ApiVersion=2010-04-01
I can see in "Body" my message is hiding. I also see the phone number. Is there any way I can just parse this into a Twilio object that I don't know about, so I can use methods like getBody(), getFrom()?

You can easily manipulate it by using the good old java.util.Properties class.
For the example bellow, I'm using the Apache Common IO lib to transform the String into an InputStream witch is required by Properties class. After that, all you have to do is use getProperty method to get what you need.
package com.pipocandobits.maven;
import org.apache.commons.io.IOUtils;
import java.io.IOException;
import java.util.Properties;
public class App {
public static void main( String[] args ) throws IOException {
System.out.println( "Hello World!" );
String source = "ToCountry=US&\n" +
"ToState=statecode&\n" +
"SmsMessageSid=smsMessageSid&\n" +
"NumMedia=0&\n" +
"ToCity=city&\n" +
"FromZip=zipCode&\n" +
"SmsSid=SmsSid&\n" +
"FromState=statecode&\n" +
"SmsStatus=received&\n" +
"FromCity=city&\n" +
"Body=Hello+it%27s+John+&\n" +
"FromCountry=US&\n" +
"To=%2B1toPhoneNumber&\n" +
"ToZip=55401&\n" +
"NumSegments=1&\n" +
"MessageSid=messageSid&\n" +
"AccountSid=accountSid&\n" +
"From=%2B1fromPhoneNumber&\n" +
"ApiVersion=2010-04-01";
Properties properties = new Properties();
properties.load(IOUtils.toInputStream(source, "UTF-8"));
System.out.println("Message body = " + properties.getProperty("Body"));
}
}
For more on how to use the java.util.Properties class check this link https://www.tutorialspoint.com/java/java_properties_class.htm

Strange output from FuseESB

I am working on parsing the request.
I developed route in Java for parsing incoming request.
I am using Camel 2.9 with FuseESB 7.0.1.0.84.
I used simple(“{body}”).getText() to fetch incoming request as per Camel Manual
So I am checking the incoming request by using the code as:
if (xmlStringToParse == null || xmlStringToParse.equals("") || xmlStringToParse.equals("${body}")) {
parsedXMLPath = "<error>Incoming request is as folows:"
+ "\nValue of xmlStringToParse: " + xmlStringToParse
+ "\n xmlStringToParse is empty: " + (xmlStringToParse.equals(""))
+ "\n xmlStringToParse equals ${body}: " + (xmlStringToParse.equals("${body}"))
+ "\nAgain checking incoming request:\n" + xmlStringToParse
+ "</error>";
}
Where xmlStringToParse = simple(“${body}”).getText()
The strange outcoming observed:
Value of xmlStringToParse is changed in just one line from soap request to "". Also “xmlStringToParse equals ${body} “ is printed as “xmlStringToParse equals” without printing ${body}. ${body} is not printed in logs.
You can find the log output follows:
<error>
Value of xmlStringToParse: <somesoapRequest>
xmlStringToParse is empty: false
xmlStringToParse equals : true
Again checking incoming request:
</error>
Can anyone tell me how to solve this issue and the reason for this strange behavior?

I used simple(“{body}”).getText() to fetch incoming request as per Camel Manual
Where did you see that? Do you have a link?
You should get the message body in another way than what you do, such as
String body = exchange.getIn().getBody(String.class);
Or if you use bean parameter binding, you can bind the message body and just declare the parameter to be of String type
public void foo(String body) {
...
}
See more details at the Camel docs such at: http://camel.apache.org/bean-binding.html