We Keep Coding

camel pollEnrich is not working for the second time

I am reading and processing 2 files from 2 different file locations and comparing the content.
If 2nd file is not available , the rest of the process execute with 1st file. If 2nd file is available, comparison process should happen. For this I am using camel pollEnrich, but here the problem is that, camel is picking the 2nd file at first time only. Without restarting the camel route 2nd file is not getting picked up even if it is present there.
After restarting the camel route it is working fine, but after that its not picking the 2nd file.
I am moving the files to different locations after processing it.
Below is my piece of code,
from("sftp:" + firstFileLocation + "?privateKeyFile=" + ppkFileLocation + "&username=" + sftpUsername
+ "&readLock=changed&idempotent=true&move=" + firstFileArchiveLocation)
.pollEnrich("sftp:" + secondFileLocation + "?privateKeyFile=" + ppkFileLocation + "&username=" + sftpUsername
+ "&readLock=changed&idempotent=true&fileExist=Ignore&move="+ secondFileLocationArchive ,10000,new FileAggregationStrategy())
.routeId("READ_INPUT_FILE_ROUTE")
Need help.

You're setting idempotent=true in the sftp consumer, which means camel will not process the same file name twice. Since you're moving the files, it would make sense to set idempotent=false.
Quoted from camel documentation
Option to use the Idempotent Consumer EIP pattern to let Camel skip
already processed files. Will by default use a memory based LRUCache
that holds 1000 entries. If noop=true then idempotent will be enabled
as well to avoid consuming the same files over and over again.

I'm adding an alternative solution based on comments for the answer posted by Jeremy Ross. My answer is based on the following code example. I've only added the configure() method in the test route for brevity.
#Override
public void configure() throws Exception {
String firstFileLocation = "//127.0.0.1/Folder1";
String secondFileLocation = "//127.0.0.1/Folder2";
String ppkFileLocation = "./key.pem";
String sftpUsername = "user";
String sftpPassword = "xxxxxx";
String firstFileArchiveLocation = "./Archive1";
String secondFileLocationArchive = "./Archive2";
IdempotentRepository repository1 = MemoryIdempotentRepository.memoryIdempotentRepository(1000);
IdempotentRepository repository2 = MemoryIdempotentRepository.memoryIdempotentRepository(1000);
getCamelContext().getRegistry().bind("REPO1", repository1);
getCamelContext().getRegistry().bind("REPO2", repository2);
from("sftp:" + firstFileLocation
+ "?password=" + sftpPassword + "&username=" + sftpUsername
+ "&readLock=idempotent&idempotent=true&idempotentKey=\\${file:name}-\\${file:size}-\\${file:modified}" +
"&idempotentRepository=#REPO1&stepwise=true&download=true&delay=10&move=" + firstFileArchiveLocation)
.to("direct:combined");
from("sftp:" + secondFileLocation
+ "?password=" + sftpPassword + "&username=" + sftpUsername
+ "&readLock=idempotent&idempotent=true&idempotentKey=\\${file:name}-\\${file:size}-\\${file:modified}" +
"&idempotentRepository=#REPO2" +
"&stepwise=true&delay=10&move=" + secondFileLocationArchive)
.to("direct:combined");
from("direct:combined")
.aggregate(constant(true), (oldExchange, newExchange) -> {
if (oldExchange == null) {
oldExchange = newExchange;
}
String fileName = (String) newExchange.getIn().getHeaders().get("CamelFileName");
String filePath = (String) newExchange.getIn().getHeaders().get("CamelFileAbsolutePath");
if (filePath.contains("Folder1")) {
oldExchange.getIn().setHeader("File1", fileName);
} else {
oldExchange.getIn().setHeader("File2", fileName);
}
String file1Name = oldExchange.getIn().getHeader("File1", String.class);
String file2Name = oldExchange.getIn().getHeader("File2", String.class);
if (file1Name != null && file2Name != null) {
// Compare files
// Both files are available
oldExchange.getIn().setHeader("PROCEED", true);
} else if (file1Name != null) {
// No comparison, proceed with File 1
oldExchange.getIn().setHeader("PROCEED", true);
} else {
// Do not proceed, keep file 2 data and wait for File 1
oldExchange.getIn().setHeader("PROCEED", false);
}
String fileName1 = oldExchange.getIn().getHeader("File1", String.class);
String fileName2 = oldExchange.getIn().getHeader("File2", String.class);
oldExchange.getIn().setBody("File1: " + fileName1 + " File2: " + fileName2);
System.out.println(oldExchange);
return oldExchange;
}).completion(exchange -> {
if(exchange.getIn().getHeader("PROCEED", Boolean.class)) {
exchange.getIn().removeHeader("File1");
exchange.getIn().removeHeader("File2");
return true;
}
return false;
}).to("log:Test");
}
In this solution, two SFTP consumers were used, instead of pollEnrich, since we need to capture the file changes of both SFTP locations. I have used an idempotent repository and an idempotent key for ignoring duplicates. Further, I've used the same idempotent repository as the lock store assuming only camel routes are accessing the files.
After receiving the files from SFTP consumers, they are sent to the direct:combined producer, which then routes the exchange to an aggregator.
In the example aggregator strategy I have provided, you can see, that the file names are being stored in the exchange headers. According to the file information retrieved from the headers, the aggregator can decide how to process the file and whether or not to proceed with the exchange. (If only file2 is received, the exchange should not proceed to the next stages/routes)
Finally, the completion predicate expression decides whether or not to proceed with the exchange and log the exchange body, based on the headers set by the aggregator. I have added an example clean-up process in the predicate expression processor as well.
Hope you will get the basic idea of my suggestion to use an aggregator from this example.

bitcoinJ get transaction value

I downloaded a lot of blockchain data using https://bitcoin.org, I took some file and I try to analyse it with bitcoinj library.
I would like to get information from every transaction:
-who send bitcoins,
-how much,
-who receive bitcoins.
I use:
<dependency>
<groupId>org.bitcoinj</groupId>
<artifactId>bitcoinj-core</artifactId>
<version>0.15.10</version>
</dependency>
I have a code:
NetworkParameters np = new MainNetParams();
Context.getOrCreate(MainNetParams.get());
BlockFileLoader loader = new BlockFileLoader(np,List.of(new File("test/resources/blk00450.dat")));
for (Block block : loader) {
for (Transaction tx : block.getTransactions()) {
System.out.println("Transaction ID" + tx.getTxId().toString());
for (TransactionInput ti : tx.getInputs()) {
// how to get wallet addresses of inputs?
}
// this code works for 99% of transactions but for some throws exceptions
for (TransactionOutput to : tx.getOutputs()) {
// sometimes this line throws: org.bitcoinj.script.ScriptException: Cannot cast this script to an address
System.out.println("out address:" + to.getScriptPubKey().getToAddress(np));
System.out.println("out value:" + to.getValue().toString());
}
}
}
Can you share some snippet that will work for all transactions in the blockchain?

There are at least two type of transaction, P2PKH and P2SH.
Your code would work well with P2PKH, but wouldn not work with P2SH.
You can change the line from:
System.out.println("out address:" + to.getScriptPubKey().getToAddress(np));
to:
System.out.println("out address:" + to.getAddressFromP2PKHScript(np)!=null?to.getAddressFromP2PKHScript(np):to.getAddressFromP2SH(np));
The API of Bitcoin says the methods getAddressFromP2PKHScript() and getAddressFromP2SH() are deprecated, and I have not find suitable method.
However, P2SH means "Pay to Script Hash", which means it could contain two or more public keys to support multi-signature. Moreover, getAddressFromP2SH() returns only one address, perhaps this is the reason why it is deprecated.
I also wrote a convinient method to check the inputs and outputs of a block:
private void printCoinValueInOut(Block block) {
Coin blockInputSum = Coin.ZERO;
Coin blockOutputSum = Coin.ZERO;
System.out.println("--------------------Block["+block.getHashAsString()+"]------"+block.getPrevBlockHash()+"------------------------");
for(Transaction tx : block.getTransactions()) {
Coin txInputSum = tx.getOutputSum();
Coin txOutputSum = tx.getOutputSum();
blockInputSum = blockInputSum.add(txInputSum);
blockOutputSum = blockOutputSum.add(txOutputSum);
System.out.println("Tx["+tx.getTxId()+"]:\t" + txInputSum + "(satoshi) IN, " + txOutputSum + "(satoshi) OUT.");
}
System.out.println("Block total:\t" + blockInputSum + "(satoshi) IN, " + blockOutputSum + "(satoshi) OUT. \n");
}

Change album names using mp3agic

I wanted to loop through a folder containig .mp3 files and changing their album names (if they don't have one) to their title (e.g. Remix.mp3 with Title "Remix" gets the Album "Remix") using mp3agic.
This is my code so far:
if (mp3file.hasId3v1Tag()) {
ID3v1 id3v1Tag = mp3file.getId3v1Tag();
try {
if (id3v1Tag.getAlbum().equals("")) {
id3v1Tag.setAlbum(id3v1Tag.getTitle());
mp3file.save(SAVE_DIR + "\\" + child.getName());
System.out.println(SAVE_DIR + "/" + child.getName());
} else {
mp3file.save(SAVE_DIR + "/" + child.getName());
}
} catch (Exception e) {
mp3file.save(SAVE_DIR + "/" + child.getName());
}
}
I get the following error:
Exception in thread "main" com.mpatric.mp3agic.NotSupportedException: Packing Obselete frames is not supported
at com.mpatric.mp3agic.ID3v2ObseleteFrame.packFrame(ID3v2ObseleteFrame.java:32)
at com.mpatric.mp3agic.ID3v2Frame.toBytes(ID3v2Frame.java:83)
at com.mpatric.mp3agic.AbstractID3v2Tag.packSpecifiedFrames(AbstractID3v2Tag.java:275)
at com.mpatric.mp3agic.AbstractID3v2Tag.packFrames(AbstractID3v2Tag.java:261)
at com.mpatric.mp3agic.AbstractID3v2Tag.packTag(AbstractID3v2Tag.java:227)
at com.mpatric.mp3agic.AbstractID3v2Tag.toBytes(AbstractID3v2Tag.java:218)
at com.mpatric.mp3agic.Mp3File.save(Mp3File.java:450)
at de.thejetstream.main.Iterator.(Iterator.java:57)
at de.thejetstream.main.Main.main(Main.java:12)
at this file:
name: Feel Good in Black and Yellow.mp3
title: Feel Good in Black and Yellow (feat. Gorillaz & De La Soul)
album: Black and Yellow - Single
It crashes at line 57, which equals to the last save (in the catch).
What is the problem with this code? Is it just because the file uses an old kind of codec or something like this?

I found the solution:
The problem was that these files used ip3v2 tags instead of ip3v1. Simply checking which on it is and adjusting the code accordingly solved everything.

32-bit application works in Eclipse, but not after being deployed. (no errors..)

I need help. I am writing a Java FX application in Eclipse, with the use of e(fx)clipse. I am using it's generic build.fxbuild to generate the ant script needed to develop my .EXE file.
The application works perfectly in Eclipse IDE. And when packaged with a 64-bit JDK, it even works perfectly after being deployed as an .EXE.
My problem arises when I package it with a 32-bit JDK for a 32-bit install. With the 32-bit JDK, it still runs perfectly in Eclipse. When I create the .EXE, it seemingly runs fine. The problem is... the software is made to take an excel file of addresses, compare them to a sql database of addresses, and then append the excel file with recommendations of "ID"'s from the SQL database to give customer service a reference of which address (from excel) may exist in our database. The only thing the software doesn't do is the appending. But, it creates the XSSFWorkbook, and resaves the workbook and opens it. So it's getting the beginning code of this segment, as well as the end. But something is happening in the middle for 32-bit vs 64-bit.
Need help!
public static void writeMatchedRecords(XSSFWorkbook wb, HashMap<Integer, ExcelAddress> excelRecords,
HeaderTemplate template) {
if (Defaults.DEBUG) {
System.out.println("Writing " + excelRecords.size() + " excel records");
}
// Variable to allow writing to excel file
CreationHelper createHelper = wb.getCreationHelper();
// Iterate through every row of the excel sheet
for (Row row : wb.getSheetAt(0)) {
if (excelRecords.containsKey(row.getRowNum() + 1)) {
ExcelAddress excelTemp = excelRecords.get(row.getRowNum() + 1);
HashMap<Double, ArrayList<ShipTo>> matchedShipTos = excelTemp.getMatchedShipTos();
if (Defaults.DEBUG) {
System.out.print(row.getCell(template.getColName()) + " from Excel matches with " + excelTemp.getName() + " from HASH with " + matchedShipTos.size() + " matches.");
}
if (matchedShipTos.isEmpty() == false) {
if (Defaults.DEBUG) {
System.out.println(" (non-zero confirmed)");
}
// If Matched Ship contains 100% matches remove all other
// matches
if (matchedShipTos.containsKey(1d)) {
HashMap<Double, ArrayList<ShipTo>> tempHM = new HashMap<Double, ArrayList<ShipTo>>();
tempHM.put(1d, matchedShipTos.get(1d));
matchedShipTos.clear();
matchedShipTos.putAll(tempHM);
}
Map<Double, ArrayList<ShipTo>> sortedShipTos = new TreeMap<Double, ArrayList<ShipTo>>(matchedShipTos).descendingMap();
for (Map.Entry<Double, ArrayList<ShipTo>> entry : sortedShipTos.entrySet()) {
for (ShipTo shipTo : entry.getValue()) {
if (Defaults.DEBUG) {
System.out.print("Ship to Match: ");
System.out.print(shipTo.getName());
System.out.print(" P: " + entry.getKey() + "\n");
}
if (row.getLastCellNum() == wb.getSheetAt(0).getRow(0).getLastCellNum()) {
// Create additional headers
wb.getSheetAt(0).getRow(0).createCell(row.getLastCellNum())
.setCellValue(createHelper.createRichTextString("Probability"));
wb.getSheetAt(0).getRow(0).createCell(row.getLastCellNum() + 1)
.setCellValue(createHelper.createRichTextString("P21 - Ship to ID"));
wb.getSheetAt(0).getRow(0).createCell(row.getLastCellNum() + 2)
.setCellValue(createHelper.createRichTextString("P21 - Ship to Name"));
wb.getSheetAt(0).getRow(0).createCell(row.getLastCellNum() + 3).setCellValue(
createHelper.createRichTextString("P21 - Ship to Address Line 1"));
}
row.createCell(row.getLastCellNum()).setCellValue(entry.getKey());
row.createCell(row.getLastCellNum())
.setCellValue(createHelper.createRichTextString(Integer.toString(shipTo.getId())));
row.createCell(row.getLastCellNum())
.setCellValue(createHelper.createRichTextString(shipTo.getName()));
row.createCell(row.getLastCellNum())
.setCellValue(createHelper.createRichTextString(shipTo.getAddress1()));
}
}
}
}
}
Date date = new Date();
int rand = (int) (Math.random() * 10);
File file = new File(System.getProperty("user.home") + "/Desktop/"
+ String.format("%1$s %2$tF%3$s", template.getTemplateName(), date, " (" + rand + ").xlsx"));
try
{
FileOutputStream fileout = new FileOutputStream(file);
wb.write(fileout);
fileout.close();
Desktop.getDesktop().open(file);
} catch (
Exception e)
{
Alert alert = new Alert(AlertType.ERROR);
alert.setTitle("Error");
alert.setHeaderText("Could not save data");
alert.setContentText("Could not save data to file:\n" + file.getPath());
alert.showAndWait();
}
}

I got a similar problem with SWT
The general problem is when you need some native functions (like screen system), which depend on a particular jar.
related discussions about FX:
Creating 32 bit JavaFx Native Bundle in 64 bit machine
https://github.com/javafx-maven-plugin/javafx-maven-plugin/issues/81
Does JavaFX work in 32-bit Windows? (or with a 32-bit JVM)?
My way:
1 find the JAR
Finding javafx jar file for windows
2 embark the 2 jars in you app
3 at runtime, check 32/64
Properties prop=java.lang.System.getProperties();
String 32_64=prop.getProperty("sun.arch.data.model");
// => 32 or 64
4 load the "good" jar at runtime (check before is already loaded)
How should I load Jars dynamically at runtime?

Java Files.getLastModifiedTime Returning Unexpected Result

I'm writing a program that goes through files in a directory and checking each file's last modified date and comparing it with another variable. If the variable matches then I copy said file. I thought this was going to work like a charm but the last modified date being returned seems to be incorrect or there is a weird time zone thing happening.
I'm in the middle of a loop and the file currently being looked at is from 2014-08-18 and was actually last modified at 11:58 PM on that date but the getLastModifiedTime returns 2014-08-19T03:58:37.685611Z. So what gives???? Is this some kind of wacky time off set that I need to handle? This is important because if the last modified date is not accurate I won't know which file to copy....Anyone immediately know what's wrong? This is my first time using this way of iterating through files so I may be missing something.
//Creating a DirectoryStream inside a try-with-resource block
try (DirectoryStream<Path> ds =
Files.newDirectoryStream(FileSystems.getDefault().getPath(dir.getAbsolutePath()))) {for (Path p : ds) {
String lastMod = Files.getLastModifiedTime(p).toString();
String[] splitDte = lastMod.split("T");
if(dateSrc.equals(splitDte[0].toString()))
{
File fileToCopy = p.toFile();
copyFile(fileToCopy,
tempWorkingDir + "\\" + addLeadingZero(logM, 2) + ""
+ addLeadingZero(logDy, 2) + "\\" + fixedValue
+ "\\" + logType + "\\"
);
fileCountProcsd++;
}
}
}
catch (IOException e)
{
e.printStackTrace();
}

The Z indicates that the date is expressed in GMT.