I am reading and processing 2 files from 2 different file locations and comparing the content.
If 2nd file is not available , the rest of the process execute with 1st file. If 2nd file is available, comparison process should happen. For this I am using camel pollEnrich, but here the problem is that, camel is picking the 2nd file at first time only. Without restarting the camel route 2nd file is not getting picked up even if it is present there.
After restarting the camel route it is working fine, but after that its not picking the 2nd file.
I am moving the files to different locations after processing it.
Below is my piece of code,
from("sftp:" + firstFileLocation + "?privateKeyFile=" + ppkFileLocation + "&username=" + sftpUsername
+ "&readLock=changed&idempotent=true&move=" + firstFileArchiveLocation)
.pollEnrich("sftp:" + secondFileLocation + "?privateKeyFile=" + ppkFileLocation + "&username=" + sftpUsername
+ "&readLock=changed&idempotent=true&fileExist=Ignore&move="+ secondFileLocationArchive ,10000,new FileAggregationStrategy())
.routeId("READ_INPUT_FILE_ROUTE")
Need help.
You're setting idempotent=true in the sftp consumer, which means camel will not process the same file name twice. Since you're moving the files, it would make sense to set idempotent=false.
Quoted from camel documentation
Option to use the Idempotent Consumer EIP pattern to let Camel skip
already processed files. Will by default use a memory based LRUCache
that holds 1000 entries. If noop=true then idempotent will be enabled
as well to avoid consuming the same files over and over again.
I'm adding an alternative solution based on comments for the answer posted by Jeremy Ross. My answer is based on the following code example. I've only added the configure() method in the test route for brevity.
#Override
public void configure() throws Exception {
String firstFileLocation = "//127.0.0.1/Folder1";
String secondFileLocation = "//127.0.0.1/Folder2";
String ppkFileLocation = "./key.pem";
String sftpUsername = "user";
String sftpPassword = "xxxxxx";
String firstFileArchiveLocation = "./Archive1";
String secondFileLocationArchive = "./Archive2";
IdempotentRepository repository1 = MemoryIdempotentRepository.memoryIdempotentRepository(1000);
IdempotentRepository repository2 = MemoryIdempotentRepository.memoryIdempotentRepository(1000);
getCamelContext().getRegistry().bind("REPO1", repository1);
getCamelContext().getRegistry().bind("REPO2", repository2);
from("sftp:" + firstFileLocation
+ "?password=" + sftpPassword + "&username=" + sftpUsername
+ "&readLock=idempotent&idempotent=true&idempotentKey=\\${file:name}-\\${file:size}-\\${file:modified}" +
"&idempotentRepository=#REPO1&stepwise=true&download=true&delay=10&move=" + firstFileArchiveLocation)
.to("direct:combined");
from("sftp:" + secondFileLocation
+ "?password=" + sftpPassword + "&username=" + sftpUsername
+ "&readLock=idempotent&idempotent=true&idempotentKey=\\${file:name}-\\${file:size}-\\${file:modified}" +
"&idempotentRepository=#REPO2" +
"&stepwise=true&delay=10&move=" + secondFileLocationArchive)
.to("direct:combined");
from("direct:combined")
.aggregate(constant(true), (oldExchange, newExchange) -> {
if (oldExchange == null) {
oldExchange = newExchange;
}
String fileName = (String) newExchange.getIn().getHeaders().get("CamelFileName");
String filePath = (String) newExchange.getIn().getHeaders().get("CamelFileAbsolutePath");
if (filePath.contains("Folder1")) {
oldExchange.getIn().setHeader("File1", fileName);
} else {
oldExchange.getIn().setHeader("File2", fileName);
}
String file1Name = oldExchange.getIn().getHeader("File1", String.class);
String file2Name = oldExchange.getIn().getHeader("File2", String.class);
if (file1Name != null && file2Name != null) {
// Compare files
// Both files are available
oldExchange.getIn().setHeader("PROCEED", true);
} else if (file1Name != null) {
// No comparison, proceed with File 1
oldExchange.getIn().setHeader("PROCEED", true);
} else {
// Do not proceed, keep file 2 data and wait for File 1
oldExchange.getIn().setHeader("PROCEED", false);
}
String fileName1 = oldExchange.getIn().getHeader("File1", String.class);
String fileName2 = oldExchange.getIn().getHeader("File2", String.class);
oldExchange.getIn().setBody("File1: " + fileName1 + " File2: " + fileName2);
System.out.println(oldExchange);
return oldExchange;
}).completion(exchange -> {
if(exchange.getIn().getHeader("PROCEED", Boolean.class)) {
exchange.getIn().removeHeader("File1");
exchange.getIn().removeHeader("File2");
return true;
}
return false;
}).to("log:Test");
}
In this solution, two SFTP consumers were used, instead of pollEnrich, since we need to capture the file changes of both SFTP locations. I have used an idempotent repository and an idempotent key for ignoring duplicates. Further, I've used the same idempotent repository as the lock store assuming only camel routes are accessing the files.
After receiving the files from SFTP consumers, they are sent to the direct:combined producer, which then routes the exchange to an aggregator.
In the example aggregator strategy I have provided, you can see, that the file names are being stored in the exchange headers. According to the file information retrieved from the headers, the aggregator can decide how to process the file and whether or not to proceed with the exchange. (If only file2 is received, the exchange should not proceed to the next stages/routes)
Finally, the completion predicate expression decides whether or not to proceed with the exchange and log the exchange body, based on the headers set by the aggregator. I have added an example clean-up process in the predicate expression processor as well.
Hope you will get the basic idea of my suggestion to use an aggregator from this example.
Related
I downloaded a lot of blockchain data using https://bitcoin.org, I took some file and I try to analyse it with bitcoinj library.
I would like to get information from every transaction:
-who send bitcoins,
-how much,
-who receive bitcoins.
I use:
<dependency>
<groupId>org.bitcoinj</groupId>
<artifactId>bitcoinj-core</artifactId>
<version>0.15.10</version>
</dependency>
I have a code:
NetworkParameters np = new MainNetParams();
Context.getOrCreate(MainNetParams.get());
BlockFileLoader loader = new BlockFileLoader(np,List.of(new File("test/resources/blk00450.dat")));
for (Block block : loader) {
for (Transaction tx : block.getTransactions()) {
System.out.println("Transaction ID" + tx.getTxId().toString());
for (TransactionInput ti : tx.getInputs()) {
// how to get wallet addresses of inputs?
}
// this code works for 99% of transactions but for some throws exceptions
for (TransactionOutput to : tx.getOutputs()) {
// sometimes this line throws: org.bitcoinj.script.ScriptException: Cannot cast this script to an address
System.out.println("out address:" + to.getScriptPubKey().getToAddress(np));
System.out.println("out value:" + to.getValue().toString());
}
}
}
Can you share some snippet that will work for all transactions in the blockchain?
There are at least two type of transaction, P2PKH and P2SH.
Your code would work well with P2PKH, but wouldn not work with P2SH.
You can change the line from:
System.out.println("out address:" + to.getScriptPubKey().getToAddress(np));
to:
System.out.println("out address:" + to.getAddressFromP2PKHScript(np)!=null?to.getAddressFromP2PKHScript(np):to.getAddressFromP2SH(np));
The API of Bitcoin says the methods getAddressFromP2PKHScript() and getAddressFromP2SH() are deprecated, and I have not find suitable method.
However, P2SH means "Pay to Script Hash", which means it could contain two or more public keys to support multi-signature. Moreover, getAddressFromP2SH() returns only one address, perhaps this is the reason why it is deprecated.
I also wrote a convinient method to check the inputs and outputs of a block:
private void printCoinValueInOut(Block block) {
Coin blockInputSum = Coin.ZERO;
Coin blockOutputSum = Coin.ZERO;
System.out.println("--------------------Block["+block.getHashAsString()+"]------"+block.getPrevBlockHash()+"------------------------");
for(Transaction tx : block.getTransactions()) {
Coin txInputSum = tx.getOutputSum();
Coin txOutputSum = tx.getOutputSum();
blockInputSum = blockInputSum.add(txInputSum);
blockOutputSum = blockOutputSum.add(txOutputSum);
System.out.println("Tx["+tx.getTxId()+"]:\t" + txInputSum + "(satoshi) IN, " + txOutputSum + "(satoshi) OUT.");
}
System.out.println("Block total:\t" + blockInputSum + "(satoshi) IN, " + blockOutputSum + "(satoshi) OUT. \n");
}
Good Morning everyone
So I have this java code that parses into swagger documentation file (a JSON file) and split its:
{ Swagger swagger = new SwaggerParser().read("C:/Users/admin/Desktop/testdownload.txt");
Map<String, Path> paths = swagger.getPaths();
for (Map.Entry<String, Path> p : paths.entrySet()) {
Path path = p.getValue();
Map<HttpMethod, Operation> operations = path.getOperationMap();
for (java.util.Map.Entry<HttpMethod, Operation> o : operations.entrySet()) {
System.out.println("===");
System.out.println("PATH:" + p.getKey());
System.out.println("Http method:" + o.getKey());
System.out.println("Summary:" + o.getValue().getSummary());
System.out.println("Parameters number: " + o.getValue().getParameters().size());
for (Parameter parameter : o.getValue().getParameters()) {
System.out.println(" - " + parameter.getName());
}
System.out.println("Responses:");
for (Map.Entry<String, Response> r : o.getValue().getResponses().entrySet()) {
System.out.println(" - " + r.getKey() + ": " + r.getValue().getDescription());
}
System.out.println("");
}
}
}
And here is the input:
and the output is :
What I want to ask is: is it possible to send this output one path by one to apache Nifi ??
is there is any solution that Nifi extracts those outputs and put each one of them in a dependent processor??
You could start a HTTP lister service in NiFi. Use the HandleHttpRequest
Some time ago I did something like this. And was sending data from my Java application to this HandleHttpRequest. This Processor is designed to be used in conjunction with the HandleHttpResponse Processor in order to create a Web Service
You just have to post your data to this webservice and the webservice can consume it and you would already have your data in NiFi. From then on, you are manipulate and control you data as you please.
You can also look into ListenHTTP
//PRE-SET VARIABLES: symbolsToCheck, time
for (String s : symbolsToCheck) {
String fileName = "daylogs-" + time + "/" + s + ".txt";
File daylog = new File(fileName);
if (!daylog.exists()) {
if (!daylog.createNewFile()) {
System.out.println("ERROR creating day log for " + s);
} else {
System.out.println("Day log created: " + daylog.getCanonicalPath());
}
} else {
System.out.println("ERROR day log already exists for " + s);
}
}
Nothing is outputted from this, and I've confirmed that symbolsToCheck is populated (roughly a dozen strings). I can also confirm that time is set (integer timestamp) well before this code snippet is called. Been scratching my head for quite some time now, any ideas?
I've found the solution from a related post and Tom's suggestion, I've determined that the file creation is breaking due to my attempts to create a new folder and file at the same time, which does not work with createNewFile(). I followed the suggested in the related post and file creation works as expected now.
I am struggling for a couple of hours now on how to link a discid to a musicbrainz mbid.
So, using dietmar-steiner / JMBDiscId
JMBDiscId discId = new JMBDiscId();
if (discId.init(PropertyFinder.getProperty("libdiscid.path")))
{
String musicBrainzDiscID = discId.getDiscId(PropertyFinder.getProperty("cdrom.path"));
}
or musicbrainzws2-java
Disc controller = new Disc();
String drive = PropertyFinder.getProperty("cdrom.path");
try {
DiscWs2 disc =controller.lookUp(drive);
log.info("DISC: " + disc.getDiscId() + " match: " + disc.getReleases().size() + " releases");
....
I can extract a discid for freedb or musicbrainz easily (more or less), but I have not found a way on calculating the id I that I need to download cover art via the CoverArtArchiveClient from last.fm.
CoverArtArchiveClient client = new DefaultCoverArtArchiveClient();
try
{
UUID mbid = UUID.fromString("mbid to locate release");
fm.last.musicbrainz.coverart.CoverArt coverArt = client.getByMbid(mbid);
Theoretically, I assume, I could you the data collected by musicbrainzws2-java to trigger a search, and then use the mbid from the result ... but that cannot be the best option to do.
I am happy about any push into the right direction...
Cheers,
Ed.
You don't calculate the MBID. The MBID is attached on every entity you retrieve from MusicBrainz.
When getting releases by DiscID you get a list. Each entry is a release and has an MBID, accessible with getId():
for (ReleaseWs2 rel : disc.getReleases()){
log.info("MBID: " + rel.getId() + ", String: " + rel.toString());
}
You then probably want to try the CoverArtArchive (CAA) for every release and take the first cover art you get.
Unfortunately I don't know of any API documentation for musicbrainzws2 on the web. I recommend running javadoc on all source files.
I believe that most of you would be thinking that this is the same question you have heard multiple times (and answered ) about string concatenation in Java. But trust me, it is different. In fact, so different that I am even hesitant in posting it here. But anyways, here it is. I have some piece of code which goes like:
public void handleSuccess(String result)
{
result = result.trim();
MessageBox.alert("Information","Result after trimming: '" + result + "'");
result = result.substring(result.indexOf('\n') + 1);
MessageBox.alert("Information","Result after substring: '" + result + "'");
String returns = getReturns();
MessageBox.alert("Information","Returns: '" + returns + "'");
String action = getAction();
MessageBox.alert("Information","Action: '" + action + "'");
String finalResult = result + returns + action;
MessageBox.alert("Information","Final result: '" + finalResult + "'");
}
Now the situation here is that, all of these : getReturns(), result and getAction() return non blank values, and in fact the string finalResult contains the concatenated value after the last line is executed.
So, at Line 1, "result" contains "12/03/2013|04-AERTY|". The value of result remains same at end of line 1,2. getReturns() returns value 12.4724. So at end of line 3, finalResult contains "12/03/2013|04-AERTY|12.4724". getAction() returns "expt". So, at end of line 5, finalResult contains "12/03/2013|04-AERTY|12.4724|expt"
This is , when I debug or run the application in eclipse. As soon as build the same application on a UNIX system to generate a "war" file, and deploy the war on a tomcat server, the problem rears it's ugly head. When I run the application on the deployed war, the last line does not contain the concatenated value. So at the end of line 5, finalResult contains just "12/03/2013|04-AERTY|12.4724". I expected it to contain "12/03/2013|04-AERTY|12.4724|expt" as it does while running in eclipse.
I have tried stringbuffer, stringbuilder and the "+" operator as well, but nothing seems to work. I am not even getting an exception.
Can somebody help me in fixing this or at least enlightening me in what I might be doing wrong here?
Just to stress again, the code on eclipse(which is on a windows machine) and UNIX machine are exactly same. I have done a diff on them.
Here is what I get after putting the message-boxes:
Message-box 1: "Result after trimming: '12/03/2013|04-AERTY|'"
Message-box 2: "Result after substring: '12/03/2013|04-AERTY|'"
Message-box 3:"Returns: '12.4724'"
Message-box 4:"Action: '|expt'"
Message-box 5:"Final result: '12/03/2013|04-AERTY|12.4724|expt'"
Message-box 5 output is the one I receive when I execute code using eclipse
When running on deployed war, Message-box 1-4 have the same output as above, but Message-box 5 says: "Final result: '12/03/2013|04-AERTY|12.4724"
It's not clear where the extra "|" is meant to come from - if getAction() just returns expt, the result would be 12/03/2013|04-AERTY|12.4724|expt.
Anyway, I think it's safe to say that string concatenation will be working fine, and something else is wrong. You should add more diagnostics, logging everything:
public void handleSuccess(String result) {
result = result.trim();
log.info("Result after trimming: '" + result + "'");
result = result.substring(result.indexOf('\n') + 1);
log.info("Result after substring: '" + result + "'");
String returns = getReturns();
log.info("Returns: '" + returns + "'");
String action = getAction();
log.info("Action: '" + action + "'");
// It's not clear what this is meant to do. I suggest you remove it and
// use logging instead.
MessageBox.alert("Information", "The selected action is " + action, null);
String finalResult = result + returns + action;
log.info("Final result: '" + finalResult + "'");
I suspect you'll find that action is an empty string in the broken case.
Note that I've added quotes round each of the logged values, very deliberately. That means that if there's some unprintable character at the end of a string which causes problems, you should be able to detect that in the logging.
EDIT: As per the comment thread, when these were turned into message boxes (as it turns out this is running in GWT) it looks like there's something wrong with the early strings, as the closing ' isn't seen in diagnostics, in the broken case. The OP is going to investigate further.