We Keep Coding

Writing Junit Test case for merging files in sftp server

I am using apache commons VFS to connect to sftp server and write the content of files in /input directory into a single file in /output directory.The names of files in input directory is provided as a List. I am struggling to write Junit test case for it.My intention is that once the code gets executed, I will compare the contents of file in /input against content of file in /output
public void exportFile(List<String> fileNamesList){
for (String file : fileNamesList){
try(FileObject fileObject= //getsFileObject
OutputStream fileOutputStream= fileObject.resolveFile("/output/"+"exportfile.txt").getContent().getOutputStream(true);
)
fileObject.resolveFile("/input/"+file).getContent().getInputStream().transferTo(fileOutputStream);
}
}
I want to write Junit test case for the above. The below is my setup for test case
#BeforeAll
static void setUpSftpServer() throws IOException {
System.out.println("inside setup ssh");
sshd= SshServer.setUpDefaultServer();
sshd.setPort(1234);
sshd.setKeyPairProvider(new SimpleGeneratorHostKeyProvider());
sshd.setPublickeyAuthenticator(AcceptAllPublickeyAuthenticator.INSTANCE);
sshd.setSubsystemFactories(Arrays.asList(new SftpSubsystemFactory()));
sshd.start();
}
#Test
void exportFileTest() throws IOException, URISyntaxException {
System.out.println("Inside exportFile test");
FileObject fileObject=getFileObject();
when(sftpConfiguration.connect()).thenReturn(fileObject);
myobject.exportFile(Arrays.asList("a.txt"));
String actualContent=fileObject.resolveFile("/input/a.txt").getContentContent().getString("UTF-8");
String expectedContent=fileObject.resolveFile("/output/exportFile.txt").getContentContent().getString("UTF-8");
assertTrue(actualContent.equals(expectedContent));
}
static FileObject getFileObject() throws URISyntaxException, FileSystemException {
String userInfo = "uname" + ":" + "pwd";
SftpFileSystemConfigBuilder sftpConfigBuilder = SftpFileSystemConfigBuilder.getInstance();
FileSystemOptions options = new FileSystemOptions();
IdentityProvider identityInfo = new IdentityInfo(new File("/fake/path/to/key"), "test".getBytes());
sftpConfigBuilder.setIdentityProvider(options, identityInfo);
URI uri= new URI("sftp", userInfo, "127.0.0.1", Objects.requireNonNullElse(1234, -1), null, null, null);
FileObject fileObject= VFS.getManager().resolveFile(uri.toString(),options);
System.out.println("creating file object complete");
fileObject.resolveFile("/input").createFolder(); //create a folder in the path
fileObject.resolveFile("/output").createFolder();
//code to create a file called a.txt inside /input and write the string "abc" to the file
return fileObject;
}
But I am getting an exception like below
org.apache.commons.vfs2.FileSystemException: Unknown message with code "Could not get the user id of the current user (error code: -1)".
This exception I am getting at the line
FileObject fileObject= VFS.getManager().resolveFile(uri.toString(),options);
How do I write the unittest for this case correctly?

This is caused by the SftpFileSystem failing to run the command id -u which doesn't exist on some SSH connections such as Windows OpenSSH. It runs this command when attempting to detect the exec channel. Resolve this by adding the following SFTP configuration:
sftpConfigBuilder.setDisableDetectExecChannel(options, true);
See here.

Downloading file using Spring Webclient and AsynchronousFileChannel, file is empty

I want to download a few photos by URL using Webflux and AsynchronousFileChannel, and all files are created but empty.
Here is my code:
public void downloadFilesFromUrl() throws IOException {
List<Photo> notDownloadedFiles = //get photos with name and URL;
for (Photo photo : notDownloadedFiles) {
Path path = Paths.get(pathToFiles + File.separator + photo.getPhotoName());
WebClient client = WebClient.builder().baseUrl(photo.getLoadSource()).build();
Flux<DataBuffer> dataBufferFlux = client
.get().accept(MediaType.APPLICATION_OCTET_STREAM)
.retrieve()
.bodyToFlux(DataBuffer.class);
saveFileOnComputer(path, dataBufferFlux);
}
}
private void saveFileOnComputer(Path path, Flux<DataBuffer> dataBufferFlux) throws IOException {
AsynchronousFileChannel asynchronousFileChannel = AsynchronousFileChannel.open(path, CREATE, WRITE);
DataBufferUtils.write(dataBufferFlux, asynchronousFileChannel)
.doOnNext(DataBufferUtils.releaseConsumer())
.doAfterTerminate(() -> {
try {
asynchronousFileChannel.close();
} catch (IOException ignored) { }
}).then();
}
If I try to use
DataBufferUtils.write(dataBufferFlux, path, StandardOpenOption.CREATE).block();
instead of calling saveFileOnServer(..) method, everything is fine. But I want to use exactly AsynchronousFileChannel.

Okay, I think I fixed it.
private void saveFileOnServer(Path path, Flux<DataBuffer> dataBufferFlux) throws IOException {
AsynchronousFileChannel asynchronousFileChannel = AsynchronousFileChannel.open(path, CREATE, WRITE);
DataBufferUtils.write(dataBufferFlux, asynchronousFileChannel).subscribe();
}
The official documentation says "Note that the writing process does not start until the returned Flux is subscribed to".

How to write byte or stream to Apache Camel FTP to transfer file

In my code currently, I get data from the database and then I write a file out of the data. I have this kind of camel route and working solution:-
private static final String INPUT_FILE_DIRECTORY_URI = "file:" + System.getProperty("user.home")
+ "/data/cdr/?noop=false";
private static final String SFTP_SERVER = "sftp://" +System.getProperty("user.name")
+ "#sftp_server_url/data/cdr/?privateKeyFile=~/.ssh/id_rsa&passiveMode=true";
from(INPUT_FILE_DIRECTORY_URI)
.streamCaching()
.log("Sending file to local sftp")
.to(SFTP_SERVER);
I don't want to write a file in the local disk. Instead, I want to write file data directly to the SFTP server. I don't know how to do it? But I imagine it should be possible to do it. Can you tell me is it possible? If yes, how to do it?

I managed to solve this problem in another way. It is more suitable for my particular problem.
byte[] csvData = csvStringBuilder.toString().getBytes();
Routes.withProducer(producer)
.withHeader(Exchange.FILE_NAME, myCsvFile.csv)
.withBody(csvData)
.to(SFTP_SERVER).request(byte[].class);

You shouldn't use streamCaching unless you really using it. It store your file in memory, use it if you need to consume multiples times your input.
You can use Jpa component or a custom bean getting your data. Load it from database and then send it to your ftp server.
With Jpa :
#Entity
#NamedQuery(name = "data", query = "select x from Data x where x.id = 1")
public class Data { ... }
After that you can define a consumer uri like this one:
from("jpa://org.examples.Data?consumer.namedQuery=data")
.to("SFTP_SERVER");
EDIT : to convert a list to csv and send it to ftp :
from("jpa://org.examples.Data?consumer.namedQuery=data")
.marshal()
.csv()
.to("sftp://" +System.getProperty("user.name") +
"#sftp_server_url/data/cdr/myFile.csv?" +"privateKeyFile=~/.ssh/id_rsa&passiveMode=true");
See CSV component who convert a list to a csv file.

Yes it is possible :) To do this send the file inputStream in a camel DIRECT component and in the associated route make the copy to FTP. I use this case, to upload a file and directly copy it to ftp with from(directInputStreamName).to(yourFtpUri). This is an sample code :
Your service
#Service
public class FileService {
#Produce(uri = PfnumDownloadConstants.CAMEL_DIRECT_UPLOAD)
private ProducerTemplate producer;
public void sendFileToFtp(File fileToSend, String ftpDestinationUri) throws IOException {
Map<String, Object> headers = new HashMap<>();
//In this variable you can init the ftp destination uri or you can hard code it in route
headers.put("destinationUri", ftpDestinationUri);
//set filename to name your file in ftp
headers.put(Exchange.FILE_NAME_ONLY, file.getName());
InputStream targetStream = new FileInputStream(file);
//send stream as body and list of headers to direct
producer.sendBodyAndHeaders(targetStream, headers);
}
}
Your Camel route
#Component
public class FileUploadRoute extends RouteBuilder {
#Override
public void configure() throws Exception {
//Manage camel exception in a dedicated processor
onException(Exception.class).process(exceptionProcessor).log("error :: ${exception}");
from(CAMEL_DIRECT_UPLOAD)
.log("file copy to ftp '${header.CamelFileNameOnly}' in process")
.toD("file:/mnt?fileName=${header.CamelFileNameOnly}&delete=false")
.log("copy done");
}
}

Spring boot - Grabbing a file from the file system in a Get Request

I am trying to grab a file (in this case an image) from the file system and display it. I can do it from a resources subdirectory just fine, but when I try to go to the file system it is giving me a FileNotFound exception.
java.io.FileNotFoundException: file:\Y:\Kevin\downloads\pic_mountain.jpg (The filename, directory name, or volume label syntax is incorrect)
All the rest of my code is vanilla spring boot that was generated from the Initialize. Thanks.
#RestController
public class ImageProducerController {
#GetMapping("/get-text")
public #ResponseBody String getText() {
return "Hello World";
}
#GetMapping(value = "/get-jpg", produces = MediaType.IMAGE_JPEG_VALUE)
public void getImage(HttpServletResponse response) throws IOException {
FileSystemResource imgFile = new FileSystemResource("file:///Y:/Kevin/downloads/pic_mountain.jpg");
// ClassPathResource imgFile = new ClassPathResource("images/pic_mountain.jpg");
System.out.println(imgFile.getURL());
response.setContentType(MediaType.IMAGE_JPEG_VALUE);
StreamUtils.copy(imgFile.getInputStream(), response.getOutputStream());
}
}

from the docs:
public FileSystemResource(String path)
Create a new FileSystemResource from a file path
the constructor expects a path-part of the url, so in your case only Y:/Kevin/downloads/pic_mountain.jpg
so you should try to use it this way:
FileSystemResource imgFile = new FileSystemResource("Y:/Kevin/downloads/pic_mountain.jpg");
Btw. could it be, that you miss "Users" in your path? -> Y:/Users/Kevin/downloads/pic_mountain.jpg

Move files based on exception class to a different folder with camel

How to move a File in a FTP route to a different directory based on the error?
from("sftp://XXX#safsdf.de/dir/?delay=2s&move=done&moveFailed=failImport")
.split()
.body()
.process(e -> {
String fileName = (String) e.getIn().getHeader(Exchange.FILE_NAME);
// do some magic which could throw a exception
})
.log("Imported file ${file:name} completely.");

Check out the onException handler in Camel. It allows you to catch specific exceptions and then route accordingly. There is also try-catch-like syntax if you want the error handling to be more in-lined with your Camel route.
onException(BadThingHappenedException.class).to(file:///errorFolder)
Documentation:
onException
try-catch-finally
Custom file processing strategy

It my be archived with a onException, try-catch-finally or Custom file processing strategy but none of it is really handy.
I came up with this simple code:
#Override
public void configure() throws Exception {
errorHandler(loggingErrorHandler(logger));
from("sftp://XXX#safsdf.de/dir/?delay=2s&move=${in.header.outDirectory}&moveFailed=failImport")
.process(exchange -> exchange.setProperty(originalMessage, exchange.getIn()))
.split()
.body()
.process(e -> {
String fileName = (String) e.getIn().getHeader(Exchange.FILE_NAME);
Message origMsg = (Message) e.getProperty(originalMessage);
try {
// do what ever
origMsg.setHeader(outDirectory, "done/" + fileName);
} catch (KaboomException ex) {
origMsg.setHeader(outDirectory, "retry/" + fileName);
}
})
.log("Imported file ${file:name} completely.");
}
Its important to set the header on the original message.