We Keep Coding

How to get Log Aggregation in code by yarnClient

I'm following this example about create YarnApp by java API.
https://github.com/hortonworks/simple-yarn-app
Works fine, but, the log exists only execution, after it the log gone.
How I can caught this by code ? or maybe enable one option?

You can find logs using LogCliHelpers by application id after application had finished:
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.yarn.api.records.ApplicationId;
import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
import org.apache.hadoop.yarn.client.api.YarnClientApplication;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.exceptions.YarnException;
import org.apache.hadoop.yarn.logaggregation.LogCLIHelpers;
import java.io.IOException;
import java.io.PrintStream;
public static void getLogs(YarnConfiguration conf, YarnClientApplication app) throws IOException, YarnException {
ApplicationSubmissionContext appContext =
app.getApplicationSubmissionContext();
ApplicationId appId = appContext.getApplicationId();
LogCLIHelpers logCLIHelpers = new LogCLIHelpers();
logCLIHelpers.setConf(conf);
FileSystem fs = FileSystem.get(conf);
Path logFile = new Path("/path/to/log/file.log");
fs.create(logFile, false);
try (PrintStream printStream = new PrintStream(logFile.toString())) {
logCLIHelpers.dumpAllContainersLogs(appId, UserGroupInformation.getCurrentUser().getShortUserName(), printStream);
}
}

Where is "from" imported from camel in java?

I am trying to import csv file to xml file
i see apache has feature to do
from(in)
.to(out)
.split(body().tokenize("\n")).streaming()
.unmarshal().csv();
but i have a "cannot resolve method 'from(java.lang.String)' error
when I try to import then i cannot find any packages for camel
this one works:
import org.apache.camel.dataformat.bindy.csv.BindyCsvDataFormat;
what is the package to use "from" from org.apache.camel.???
this is my file:
import org.apache.camel.Exchange;
import org.apache.camel.dataformat.bindy.BindyAbstractDataFormat;
import org.apache.camel.dataformat.bindy.BindyAbstractFactory;
import org.apache.camel.dataformat.bindy.BindyFixedLengthFactory;
import org.apache.camel.dataformat.bindy.FormatFactory;
import org.apache.camel.dataformat.bindy.csv.BindyCsvDataFormat;
import org.apache.camel.dataformat.bindy.util.ConverterUtils;
import org.apache.camel.spi.DataFormat;
import org.apache.camel.util.IOHelper;
import org.apache.camel.util.ObjectHelper;
import org.apache.camel.CamelContext;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.Exchange;
import org.apache.camel.Message;
import org.apache.camel.Processor;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.impl.DefaultCamelContext;
public class Csvtoxml {
public static void convert(String in, String out) throws Exception {
DataFormat bindy = new BindyCsvDataFormat(Model.class);
from("myCsvFile.csv")
.to("myXmlFile.xml")
.split(body().tokenize("\n")).streaming()
.unmarshal().csv();
}
}

It is not imported from anywhere. In order to use it this way you have to inherit your class from org.apache.camel.builder.RouteBuilder

.keys() method not working on JSONObject eclipse

i am trying to use the .keys() method to get the name of JSON Object's
the code im using is;
Iterator<String> keys = JSONObject.keys();
.keys() is underelined as red on eclipse, i dont know why, can any one help, thanks! -
I have JSON simple as an external library and have imported it, not sure what else to do
EDIT:
Here is more code;
JSONParser parser = new JSONParser();
FileReader testfile = new FileReader("test2.txt");
Object obj = parser.parse(testfile);
JSONObject jsonObject = (JSONObject) obj;
JSONObject name = (JSONObject) jsonObject.get("txt");
String time = (String) name.get("name");
JSONObject example2 = (JSONObject) jsonObject.get("birth");
System.out.println(example2);
Iterator keys = example2 .keys(); <-- where the red line shows up
Second edit:
here are my imports.
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStream;
import java.net.MalformedURLException;
import java.net.URL;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Iterator;
import java.util.Set;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
import org.json.simple.parser.ParseException;

try:
Iterator<String> keys = example2.keySet().iterator();

With this artifact: https://mvnrepository.com/artifact/org.json/json/20170516
Use the following code:
JSONObject obj = new JSONObject("{\"key\":\"value\"}");
for(Object o : obj.keys()) { ... }
It will work.
Do not mix API.

Strange Behaviour in Eclipse 3.8.1

I'm a newbie here. I have a simple problem in ONE java source file: the row System.out.pritln(...) has been treated as an erroneous expression. Here's the code snippet:
package vk.gui;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.net.MalformedURLException;
import java.util.Properties;
import com.itextpdf.text.BadElementException;
import com.itextpdf.text.Chunk;
import com.itextpdf.text.Document;
import com.itextpdf.text.DocumentException;
import com.itextpdf.text.Font;
import com.itextpdf.text.Image;
import com.itextpdf.text.PageSize;
import com.itextpdf.text.Paragraph;
import com.itextpdf.text.Rectangle;
import com.itextpdf.text.pdf.BarcodeEAN;
import com.itextpdf.text.pdf.BaseFont;
import com.itextpdf.text.pdf.PdfContentByte;
import com.itextpdf.text.pdf.PdfPCell;
import com.itextpdf.text.pdf.PdfPCellEvent;
import com.itextpdf.text.pdf.PdfPTable;
import com.itextpdf.text.pdf.PdfWriter;
public class MatrixSheet1 {
Properties p;
File file;
Document document;
PdfWriter writer;
Image logo = null;
Image EANimg = null;
float mnoz = new Double(72/25.6).floatValue();
int IMG_WIDTH= new Double(35*mnoz).intValue();
int IMG_HEIGHT=new Double(35*mnoz).intValue();
String err=p.getProperty("cell.height");
System.out.println("Arrgh!"); ///-------------->ERROR!
float cell_Height = Float.parseFloat(p.getProperty("cell.height"))*mnoz;
float cell_Width = Float.parseFloat(p.getProperty("cell.width"))*mnoz;
The reported error is
Multiple markers at this line
Syntax error on token ""Arrgh!"", delete this token
Syntax error on token(s), misplaced construct(s)
The sout and sysout shortcuts do not work neither. In other existing source files of same package everything is OK, the shortcuts work and the expression does not trigger an error.
I tried to create another source file and copy/paste the content, but I got the same error. What and where went wrong?
I need the printing just for debugging, but this is a bit annoying symptom.
Thanks in advance.

This happens because you can use System.out.println() only inside methodes. If you would do something like this, it would work:
public class MatrixSheet1 {
Properties p;
File file;
Document document;
PdfWriter writer;
Image logo = null;
Image EANimg = null;
float mnoz = new Double(72/25.6).floatValue();
int IMG_WIDTH= new Double(35*mnoz).intValue();
int IMG_HEIGHT=new Double(35*mnoz).intValue();
String err=p.getProperty("cell.height");
systemMessage("Argh!");
float cell_Height = Float.parseFloat(p.getProperty("cell.height"))*mnoz;
float cell_Width = Float.parseFloat(p.getProperty("cell.width"))*mnoz;
private void systemMessage(String message){
System.out.println(message);
}
}

rfaces.get(0).getSecondObject() in OpenIMAJ face recognition always returns null

import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import javax.imageio.ImageIO;
import org.codehaus.jackson.map.introspect.Annotated;
import org.openimaj.data.dataset.GroupedDataset;
import org.openimaj.data.dataset.ListDataset;
import org.openimaj.data.dataset.VFSGroupDataset;
import org.openimaj.experiment.dataset.split.GroupedRandomSplitter;
import org.openimaj.experiment.dataset.util.DatasetAdaptors;
import org.openimaj.feature.DoubleFV;
import org.openimaj.feature.DoubleFVComparison;
import org.openimaj.image.DisplayUtilities;
import org.openimaj.image.FImage;
import org.openimaj.image.ImageUtilities;
import org.openimaj.image.model.EigenImages;
import org.openimaj.image.processing.face.alignment.RotateScaleAligner;
import org.openimaj.image.processing.face.detection.HaarCascadeDetector;
import org.openimaj.image.processing.face.detection.keypoints.FKEFaceDetector;
import org.openimaj.image.processing.face.detection.keypoints.KEDetectedFace;
import org.openimaj.image.processing.face.recognition.EigenFaceRecogniser;
import org.openimaj.image.processing.face.recognition.FaceRecognitionEngine;
import org.openimaj.ml.annotation.AnnotatedObject;
import org.openimaj.ml.annotation.ScoredAnnotation;
import org.openimaj.util.pair.IndependentPair;
public class Demo {
public static void main(String ... s) throws IOException {
FKEFaceDetector faceDetector = new FKEFaceDetector(new HaarCascadeDetector());
EigenFaceRecogniser<KEDetectedFace, Integer> faceRecognizer = EigenFaceRecogniser.create(20, new RotateScaleAligner(), 1, DoubleFVComparison.CORRELATION, 0.9f);
FaceRecognitionEngine<KEDetectedFace, Integer> faceEngine = FaceRecognitionEngine.create(faceDetector, faceRecognizer);
for(int i = 1; i < 15; i ++){
System.out.println(i);
FImage pamu = ImageUtilities.createFImage(ImageIO.read(new File("/home/nagarjuna/Pictures/Webcam/"+i+".jpg")));
List<KEDetectedFace> faces = faceEngine.getDetector().detectFaces(pamu);
if(faces.size() > 0)
faceEngine.train(faces.get(0), i);
}
FImage pamu = ImageUtilities.createFImage(ImageIO.read(new File("/home/nagarjuna/Pictures/Webcam/"+12+".jpg")));
List<KEDetectedFace> faces = faceEngine.getDetector().detectFaces(pamu);
List<IndependentPair<KEDetectedFace, ScoredAnnotation<Integer>>> rfaces = faceEngine.recogniseBest(faces.get(0).getFacePatch());
ScoredAnnotation<Integer> score = rfaces.get(0).getSecondObject();
DisplayUtilities.display(rfaces.get(0).getFirstObject().getFacePatch());
if(score != null) {
System.out.println("confidence: "+score.confidence);
System.out.println("annotation: "+score.annotation);
}else {
System.out.println("score is null");
}
}
}
Output:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
score is null

Unfortunately it appears a bug crept in with the way OpenIMAJ handles similarity measures (as opposed to distance measures) in the KNNAnnotator class. A fix has been committed (revision #2560), and a new snapshot (version 1.3-SNAPSHOT) incorporating it should be available in a few hours. For the time being, you should be able to change DoubleFVComparison.CORRELATION to DoubleFVComparison.EUCLIDEAN and it should then work.
I noticed a couple of other things in your code that should probably be fixed:
Instead of ImageUtilities.createFImage(ImageIO.read(new File("...")));, you should use ImageUtilities.readF(new File("...")); as this works around a number of documented bugs in ImageIO, especially with respect to jpeg reading, and you also get support for a number of other image file formats.
You don't need to manually perform the face detection on the query image yourself; the faceEngine.recogniseBest(FImage) method does that for you internally (together with alignment)