How to move a File in a FTP route to a different directory based on the error?
from("sftp://XXX#safsdf.de/dir/?delay=2s&move=done&moveFailed=failImport")
.split()
.body()
.process(e -> {
String fileName = (String) e.getIn().getHeader(Exchange.FILE_NAME);
// do some magic which could throw a exception
})
.log("Imported file ${file:name} completely.");
Check out the onException handler in Camel. It allows you to catch specific exceptions and then route accordingly. There is also try-catch-like syntax if you want the error handling to be more in-lined with your Camel route.
onException(BadThingHappenedException.class).to(file:///errorFolder)
Documentation:
onException
try-catch-finally
Custom file processing strategy
It my be archived with a onException, try-catch-finally or Custom file processing strategy but none of it is really handy.
I came up with this simple code:
#Override
public void configure() throws Exception {
errorHandler(loggingErrorHandler(logger));
from("sftp://XXX#safsdf.de/dir/?delay=2s&move=${in.header.outDirectory}&moveFailed=failImport")
.process(exchange -> exchange.setProperty(originalMessage, exchange.getIn()))
.split()
.body()
.process(e -> {
String fileName = (String) e.getIn().getHeader(Exchange.FILE_NAME);
Message origMsg = (Message) e.getProperty(originalMessage);
try {
// do what ever
origMsg.setHeader(outDirectory, "done/" + fileName);
} catch (KaboomException ex) {
origMsg.setHeader(outDirectory, "retry/" + fileName);
}
})
.log("Imported file ${file:name} completely.");
}
Its important to set the header on the original message.
Related
I want to add an inline image to the email, so I am using spring boot and their JavaMailSender mechanism.
Now it results in a NPE when I try to add an image via the addInline method
byte[] attachmentData = aMailConfig.getAttachments().get(messageAttach).getBinaryData();
LOG.debug("size of attachment is {0}", attachmentData.length);
Resource res = new InputStreamResource(new ByteArrayInputStream(attachmentData));
msg.addInline(messageAttach, res);
Here the stacktrace
java.lang.NullPointerException
at javax.activation.MimetypesFileTypeMap.getContentType(MimetypesFileTypeMap.java:299)
at org.springframework.mail.javamail.ConfigurableMimeFileTypeMap.getContentType(ConfigurableMimeFileTypeMap.java:184)
at org.springframework.mail.javamail.MimeMessageHelper.addInline(MimeMessageHelper.java:941)
The problem is, that there is call to the resource.getFilename() in the addInline method.
As I am generating the Resource with InputStreamResource, there is no filename set. Is there a way so that I can set this filename in my case or is it necessary to create a tempfile and generate the resource object by using this file?
Here the addLine mehod of the org.springframework.mail.javamail.MimeMessageHelper
public void addInline(String contentId, Resource resource) throws MessagingException {
Assert.notNull(resource, "Resource must not be null");
String contentType = getFileTypeMap().getContentType(resource.getFilename());
addInline(contentId, resource, contentType);
}
Straight forward:
Resource res = new InputStreamResource(new ByteArrayInputStream(attachmentData)) {
#Override
public String getFilename() {
return "dummy-filename.io"; // or something more complex/dynamic (but "file name like"!)
}
};
...(anonymous) override the "malicious method", with something that suits us better.
But BEST, we would know the MimeType of attachment (not guessing it from a fictive file name), use the "more appropriate not malicious method" (overload), and just set it like:
msg.addInline(
"contentId",
new InputStreamResource(new ByteArrayInputStream(attachmentData)),
"image/jpeg" // knowing it or guessing it.
);
javadoc.
So VERY BEST would be, when:
aMailConfig.getAttachments().get(messageAttach)
besides getBinaryData():byte[] also had a (correct) getContentType():String (..or even get[Original]FileName():String ;).
I'm trying to set up automatic cleanup of temp files, but some strange things happen when a Mono created from a CompletableFuture (from the S3 sdk) comes into play. The temp file is deleted too early in the chain. I am wondering whether this is a misunderstanding of reactor from my side, or a bug.
I've condensed the code to mimic the behavior I'm experiencing.
I would expect the output of this test to be:
uploaded file response
do other stuff
clean up file
However it is:
clean up file
uploaded file response
do other stuff
When not creating a Mono using Mono.fromFuture, everything works as expected.
#Test
void bla() throws InterruptedException {
CompletableFuture<String> future = CompletableFuture.supplyAsync(() -> {
try {
// "uploading"
Thread.sleep(500);
} catch (InterruptedException e) {
e.printStackTrace();
}
return "uploaded file response";
});
Mono<String> createTmpFile = Mono.just("create tempFile")
.doAfterTerminate(() -> System.out.println("clean up file"));
Mono<String> doPersistenceThings = Mono.fromFuture(future)
.doOnNext(System.out::println)
.then(Mono.just("do other stuff"));
Mono.just("data")
.flatMap(d -> createTmpFile)
.flatMap(d -> doPersistenceThings)
.subscribe(System.out::println);
Thread.sleep(2000);
}
I've got the following route:
public void configure() throws Exception {
from(ftpEndpoint)
.routeId("import-lib-files")
.log(INFO, "Processing file: '${headers.CamelFileName}' from Libri-FTP")
.choice()
.when(method(isFilenameAlreadyImported))
.log(DEBUG, "'${headers.CamelFileName}' is already imported.")
.endChoice()
.otherwise()
.bean(method(unzipLibFile))
.bean(method(persistFilename))
.log(DEBUG, "Import file '${headers.CamelFileName}'.")
.endChoice()
.end()
.end();
}
inside the unzipLibFile processor bean the file from the ftp gets uncompressed and is written to the HD.
I want to test (integration test) this route, like:
Copy file to ftp
Start the route
evaluate the 'outcome'
I like:
#Before
public void setUp() throws Exception {
// delete test-file from sftp
final String uploaded = ftpPath + "/" + destination + "/libri-testfile.zip";
final File uploadedFile = new File(uploaded);
uploadedFile.delete();
// delete unzipped test-file
final String unzippedFile = unzipped + "/libri-testfile.xml";
final File expectedFile = new File(unzippedFile);
expectedFile.delete();
// delete entries from db
importedLibFilenameRepository.deleteAll();
// copy file to ftp
final File source =
new ClassPathResource("vendors/references/lib.zip/libri-testfile.zip").getFile();
final String target = ftpPath + "/" + destination + "/libri-testfile.zip";
FileUtils.copyFile(new File(source.getAbsolutePath()), new File(target));
}
#Test
#Ignore
public void testStuff() throws Exception {
// Well here is a problem, I can't fix at the moment
// the Camel-Context within the SpringContext get started when the tests starts
// during this process the Camel-Routes are executed and because i copied the file to
// the ftp all is fine... but I don't want to have a sleep in a test, I want to start the
// route (like commented code beneath the sleep)
Thread.sleep(2000);
// final Map<String, Object> headers = Maps.newHashMap();
// headers.put("CamelFileName", "libri-testfile.zip");
//
// final File file =
// new ClassPathResource("vendors/references/lib.zip/libri-testfile.zip").getFile();
// final GenericFile<File> genericFile =
// FileConsumer.asGenericFile(file.getParent(), file, StandardCharsets.UTF_8.name(), false);
//
// final String uri = libFtpConfiguration.getFtpEndpoint();
// producer.sendBodyAndHeaders(uri, InOut, genericFile, headers);
// test if entry was made in the database
final List<ImportedLibFilename> filenames = importedLibFilenameRepository.findAll();
assertThat(filenames).usingElementComparatorIgnoringFields("id", "timestamp")
.containsExactly(expectedFilename("libri-testfile.zip"));
// test if content of unzipped file is valid
final String expected = unzipped + "/libri-testfile.xml";
final Path targetFile = Paths.get(expected);
final byte[] encoded = Files.readAllBytes(targetFile);
final String actualFileContent = new String(encoded, Charset.defaultCharset());
final String expectedFileContent = "This is my little test file for Libri import";
assertThat(actualFileContent).isEqualTo(expectedFileContent);
}
private ImportedLibFilename expectedFilename(final String filename) {
final ImportedLibFilename entity = new ImportedLibFilename();
entity.setFilename(filename);
return entity;
}
The problem is:
All camel route are started automaticly and because I copied the file to the FTP the test is green. But I've a #sleep inside my test, which I don't want. I want no camel route starting and start only the route I need.
My questions are:
How can I prevent the Camel-Routes from starting automaticly
Is the commented code (in the test method) the right way to start a route manually?
What are best practices to test a camel route with a ftp
Use .autoStartup(yourVariable) in your routes to make their startup configurable. Set the variable to true in normal environments and to falsein your test cases.
I don't see code to start a route?!?
Well, take a step back. Think about splitting your FTP route. For testing and more reasons:
For example split the route into an FTP and a processing route. The first does only the FTP transfer and then sends the received messages to the processing route (for example a direct: route).
Benefits:
SRP: Both routes do just one thing and you can concentrate on it.
Testability: You can test the processing route easily by sending messages to the direct: endpoint of the processing route. The tests can focus on one thing too.
Extensibility: Imagine there is a new input channel (JMS, HTTP, whatever). Then you just add another input route that also sends to your processing route. Done.
When you really want to test the whole process from FTP file drop until the end, think about using the Citrus test framework or similar tooling. Camel route tests are (in my opinion) a kind of "Unit tests for Camel routes", not full integration tests.
Thx to #burki...
His advise to split the routes (Single Responsibility) helped me to solve my problem:
Here is the route:
The "Main-Route" consuming from the sFTP:
#Override
public void configure() throws Exception {
// #formatter:off
from(endpoint)
.setHeader("Address", constant(address))
.log(INFO, "Import Libri changeset: Consuming from '${headers.Address}' the file '${headers.CamelFileName}'.")
.to("direct:import-new-file");
// #formatter:on
}
The first sub-route:
#Override
public void configure() throws Exception {
// #formatter:off
from("direct:import-new-file")
.choice()
.when(method(isFilenameAlreadyImported))
.log(TRACE, "'${headers.CamelFileName}' is already imported.")
.endChoice()
.otherwise()
.log(TRACE, "Import file '${headers.CamelFileName}'.")
.multicast()
.to("direct:persist-filename", "direct:unzip-file")
.endChoice()
.end()
.end();
// #formatter:on
}
The two multicasts:
#Override
public void configure() throws Exception {
// #formatter:off
from("direct:persist-filename")
.log(TRACE, "Try to write filename '${headers.CamelFileName}' to database.")
.bean(method(persistFilename))
.end();
// #formatter:on
}
and
#Override
public void configure() throws Exception {
// #formatter:off
from("direct:unzip-file")
.log(TRACE, "Try to unzip file '${headers.CamelFileName}'.")
.bean(method(unzipFile))
.end();
// #formatter:on
}
And with this setup I can write my tests like:
#Test
public void testRoute_validExtractedFile() throws Exception {
final File source = ZIP_FILE_RESOURCE.getFile();
producer.sendBodyAndHeaders(URI, InOut, source, headers());
final String actual = getFileContent(unzippedPath, FILENAME);
final String expected = "This is my little test file for Libri import";
assertThat(actual).isEqualTo(expected);
}
#Test
public void testRoute_databaseEntryExists() throws Exception {
final File source = ZIP_FILE_RESOURCE.getFile();
producer.sendBodyAndHeaders(URI, InOut, source, headers());
final List<ImportedFilename> actual = importedFilenameRepository.findAll();
final ImportedFilename expected = importedFilename(ZIPPED_FILENAME);
assertThat(actual).usingElementComparatorIgnoringFields("id", "timestamp")
.containsExactly(expected);
}
private String getFileContent(final String path, final String filename) throws IOException {
final String targetFile = path + "/" + filename;
final byte[] encodedFileContent = Files.readAllBytes(Paths.get(targetFile));
return new String(encodedFileContent, Charset.defaultCharset());
}
private Map<String, Object> headers() {
final Map<String, Object> headers = Maps.newHashMap();
headers.put("CamelFileName", ZIPPED_FILENAME);
return headers;
}
I can start the camel route with the ProducerTemplate (producer) and send a message to a direct endpoint (instead the ftp endpoint).
We have a legacy system that has a admim module that allows users to upload jar files. After the upload, the jar file is validated and if not compliant to internal rules, it is deleted.
The problem is that windows is throwing an exception telling that the file "is already being used by another process." (when I call Files.delete(tmpJar);). I'm not able to identify why the file is open. Seems to me that I have closed everything.
First, we are using primefaces (4.0) to upload the file. Primefaces relies on commons-fileupload (1.3.1). It call the following method:
public void handleFileUpload(FileUploadEvent event) {
Path tmpJar = null;
try {
tmpJar = Files.createFile(Paths.get(event.getFile().getFileName()));
Files.write(tmpJar, event.getFile().getContents());
} catch (IOException e) {
LOGGER.error(e.getMessage(), e);
}
if (tmpJar != null) {
try {
this.validateJar(tmpJar.toString());
Files.delete(tmpJar);
} catch (IOException e) {
LOGGER.error(e.getMessage(), e);
}
}
}
Before NIO Files.write, I was using "standard" java IO classes. The problem isn't related to the above code, because if I comment the call to validateJar, Files.delete(tmpJar) is executed without problems and the file is removed. So, the problem is related with the code below, but I can't find where...
Job is an internal class, basically a simple POJO. "jobAnnotation" is a custom annotation to identify Jobs. I have shortened the code, but the essencial parts are preserved.
private List<Job> validateJar(final String jarPath) throws IOException {
List<Job> jobs = new ArrayList<Job>();
try (JarFile jarFile = new JarFile(jarPath)) {
URL[] jars = { new URL("file:" + jarPath) };
ClassLoader jobClassLoader = URLClassLoader.newInstance(jars, this.getClass().getClassLoader());
Enumeration<JarEntry> jarEntries = jarFile.entries();
while (jarEntries.hasMoreElements()) {
JarEntry jarEntry = jarEntries.nextElement();
String className = jarEntry.getName();
Class<?> classToLoad;
try {
classToLoad = Class.forName(className, true, jobClassLoader);
} catch (Exception e1) {
LOGGER.error(e1.getMessage(), e1);
continue;
}
if (classToLoad.isAnnotationPresent(jobAnnotation)) {
String vlr = null;
try {
Class<?> jobClass = (Class<?>) Class.forName(classToLoad.getCanonicalName(), true, jobClassLoader);
Annotation annotation = jobClass.getAnnotation(jobAnnotation);
Method method = annotation.getClass().getMethod("getValue");
vlr = ((String) method.invoke(annotation, new Object[0]));
} catch (Exception e1) {
LOGGER.error(e1.getMessage(), e1);
}
Job job = new Job();
job.setEnabled(true);
job.setJarfile(jarPath);
job.setClassName(classToLoad.getName());
Parameter parameter = new Parameter();
parameter.setRequired(true);
parameter.setName("name");
parameter.setValue(vlr);
job.addParameter(parameter);
jobs.add(job);
}
}
} catch (IOException e) {
throw e;
}
return jobs;
}
Before using try-with-resources, I was using regular try-catch-finally to close the JarFile, thats the only thing that has a explicit close method. Probably is the classloading that is holding the file open, but I don't know how to close it.
I did some searches, and I found that I can't unload classes (Unloading classes in java?).
So, the problem is, how do I release it? Or how can I remove the file?
BTW, I'm using java 1.7.0_71, jboss 7.1.1, windows 7 (64).
The URLClassLoader class already has a close() method. The close() method will close any Jar file that are opened with the URLClassLoader. This should prevent the "file already in use" exception.
File is already being used by another process. says that it could be not your fault, maybe just another application is used that file. You can check this question to find a process which is used your file.
Some Virus scanner software take a long time in checking JARs. Try to disable the Virusscanner. Other candidates can be the Windows indexer process, or the explorer.exe itself. When you don't find any reason for the file lock, try a delay between the validation and the deletion. Maybe you need a loop with multiple tries.
We work on a Java (Java EE) application, and we generate XML files in order to send them to a remote .NET application with MSMQ reading on their side.
The XML file is generated by JDom, like so :
// add elements...
Document doc = new Document(root);
String XmlData = new XMLOutputter(Format.getPrettyFormat().setOmitEncoding(true)).outputString(doc);
try {
SendFile( XmlData, "title" , "path");
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
} catch (MessageQueueException e) {
e.printStackTrace();
}
Then we use this function, using the MsmqJava library to send the file :
private void SendFile(String data, String title, String outputPath) throws UnsupportedEncodingException, MessageQueueException{
String qname="name_of_the_queue";
String fullname= "server_path" + qname;
String body = data;
String label = title;
String correlationId= "L:none";
try {
Queue queue= new Queue(fullname);
Message msg= new Message(body, label, correlationId);
queue.send(msg);
} catch (MessageQueueException ex1) {
System.out.println("Put failure: " + ex1.toString());
}
}
They correctly receive the file, but they told us that the bodyType was set to "VT_EMPTY" while they wanted "VT_BSTR", and we haven't find a clue about how to fix this. If you know another lib who does the job, or a workaround to this one, we can change with no problem.
Thanks !
Looking at the documentation for the library you use, it is not possible using that library.
Jmsmqqueue also doesn't provide the functionality you need.
It seems sun also had an adapter: https://wikis.oracle.com/display/JavaCAPS/Sun+Adapter+for+MSMQ