We Keep Coding

How can I serialize/deserialize a Proto object to a local storage in Android project?

How can I save Proto data in a file?
If my proto data looks like this, can I save this in a file or do I need to make some changes in my Proto file?
message GameHistory {
repeated Game game = 1;
}
message Game {
string title = 1;
oneof entity {
Easy easy = 2;
Hard hard = 3;
}
}
message Easy {
}
message Hard {
Bool isPaid =1;
int32 number_of_levels = 2;
}
Here is my list of games in Java code:
GameResponse.Game game1 = GameResponse.Game.newBuilder().setTitle("Asd").setEasy(GameResponse.Easy.getDefaultInstance()).build();
GameResponse.Game game2 = GameResponse.Game.newBuilder().setTitle("Bsd").setHard(GameResponse.Hard.getDefaultInstance()).build();
final GameResponse.GameHistory build = GameResponse.GameHistory.newBuilder().addGame(game1).addGame(game2).build();

Using com.google.protobuf the GameResponse.GameHistory class contains both methods of:
void writeTo(final OutputStream output)
static GameResponse.GameHistory parseFrom(InputStream input)
So, you will be able to use these two methods in order to write/read a protocol message to/from a file.
// to write:
File file = new File(context.getExternalFilesDir(null), "fileName.bin");
OutputStream fos = FileOutputStream(file);
gameHistory.writeTo(fos);
fos.flush();
fos.close();
// to read:
File file = new File(context.getExternalFilesDir(null), "fileName.bin");
InputStream fis = FileInputStream(file);
GameResponse.GameHistory gameHistory = GameResponse.GameHistory.parseFrom(fis);
fis.close();

Appending data to a file in HDFS using java failed , getting error

i create a new csv file in hdfs using java , and i am trying to append the data to that csv file , but failed to append with error
Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[192.168.1.25:9866,DS-b6d8a63b-357d-4d39-9f27-1ab76b8b6ccc,DISK]], original=[Dat
below is the code
csv file created and uplaoded to HDFS from java code , but not able append data to the existing file . but a newly uploaded csv from ui interface was able to appended data with java code , please help to resolve this issue.
private void appendFileToFile (String fileName) throws Exception{
long testTime1 = System.currentTimeMillis();
String hdfsHostDetails = new String("hdfs://192.168.1.25:9000");
Configuration conf = new Configuration();
conf.setBoolean("dfs.support.append", true);
FileSystem fs = FileSystem.get(URI.create(hdfsHostDetails), conf);
String dirpath = new String(hdfsHostDetails);
String targetfilepath = new String(dirpath+"/"+fileName);
int count = 0;
while (count < 2) {
int offset = 0;
int limit = 10000;
IgniteTable table = new IgniteTable(ignite, "nok_customer_demand");
String query = "SELECT * FROM nok_customer_demand OFFSET "+ offset +" ROWS FETCH NEXT "+ limit +" ROWS ONLY";
List<List<?>> lists = table._select(query);
List<String[]> rows = new ArrayList();
System.out.println(":::::::::::::::::: Data Ready for iteration ::::::::::::::"+ count);
// create a new file on each iteration
File file = new File("/home/tejatest1"+count+".csv");
FileWriter outputfile = new FileWriter(file);
CSVWriter writer = new CSVWriter(outputfile);
for (List eachlist : lists) {
String[] eachRowAsString = new String[eachlist.size()];
;
int i = 0;
for (Object eachcol : eachlist) {
eachRowAsString[i] = String.valueOf(eachcol);
rows.add(eachRowAsString);
i++;
}
writer.writeNext(eachRowAsString);
}
// on each iteration append the data in the file to hdfs
InputStream in = new BufferedInputStream(new FileInputStream(file));
FSDataOutputStream out =null;
if(!fs.exists(new Path(targetfilepath))) {
out = fs.create(new Path(targetfilepath));
} else{
out = fs.append(new Path(targetfilepath));
}
IOUtils.copyBytes(in, out, 4096, true);
writer.close();
out.close();
outputfile.close();
lists.clear();
in.close();
file.delete();
count++;
}
long testTime2 = System.currentTimeMillis();
System.out.println("-----total time taken for data fetch for all records in table using limit and offset:-------" + (testTime2 - testTime1) + " ms");
fs.close();
}

i resolve this issue with the below configuration
Configuration conf = new Configuration();
conf.set("fs.defaultFS",hdfsHostDetails);
conf.setInt("dfs.replication",1);
conf.setBoolean("dfs.client.block.write.replace-datanode-on-failure.enable",false);
conf.setBoolean("dfs.support.append", true);
FileSystem fs = FileSystem.get(URI.create(hdfsHostDetails), conf);

Need to Merge Avro Files using java application

I am having multiple avro files under a directory which reside on hadoop environment, I need to merge all these files and make it as a single avro file.
example
/abc->
x.avro
y.avro } => a.avro
z.avro
The file a.avro will contain contents of all x,y,z files, where x,y,z files having same schema. I need to create a java application. Any help appreciated.
Thanks.

There are few tools provided by the apache avro in order to deal with the avro file operations here. These tools include Merging/Concat tool which merge same schema avro file with non-reserved metadata, catTool to extract samples from an Avro data file, conversion tool which Converts an input file from Avro binary into JSON, recoveryTool which Recovers data from a corrupt Avro Data file etc(Find more on the github url mentioned).
I have extract the code from the same tools mentioned on github, here is the java application that does solve your purpose.
Path inPath = new Path("C:\\Users\\vaijnathp\\IdeaProjects\\MSExcel\\vaj");
Path outPath = new Path("getDestinationPath") ;
FileSystem fs = FileSystem.get(new Configuration());
FileStatus [] contents contents = fs.listStatus(inPath, new OutputLogFilter());
DataFileWriter<GenericRecord> writer = new DataFileWriter<>(new GenericDatumWriter<>());
Schema schema = null;
String inputCodec = null;
Map<String, byte[]> metadata = new TreeMap<>();
BufferedOutputStream output = new BufferedOutputStream(new BufferedOutputStream(fs.create(outPath)));
for (int i = 0; i < contents.length; i++) {
FileStatus folderContent = contents[i];
if (folderContent.isFile() && folderContent.getPath().getName().endsWith(".avro")) {
InputStream input = new BufferedInputStream(fs.open(folderContent.getPath()));
DataFileStream<GenericRecord> reader = new DataFileStream<>(input, new GenericDatumReader<GenericRecord>());
if (schema == null) {
schema = reader.getSchema();
//extract metadata for further check.
extractAvroFileMetadata(writer, metadata, reader);
inputCodec = reader.getMetaString(DataFileConstants.CODEC);
if (inputCodec == null) inputCodec = DataFileConstants.NULL_CODEC;
writer.setCodec(CodecFactory.fromString(inputCodec));
writer.create(schema, output);
} else {
if (!schema.equals(reader.getSchema())) reader.close();
//compare FileMetadata with previously extracted one
CompareAvroFileMetadata(metadata, reader, folderContent.getPath().getName());
String thisCodec = reader.getMetaString(DataFileConstants.CODEC);
if (thisCodec == null) thisCodec = DataFileConstants.NULL_CODEC;
if (!inputCodec.equals(thisCodec)) reader.close();
}
writer.appendAllFrom(reader, false);
reader.close();
}
}
writer.close();
}catch (Exception e){
e.printStackTrace();
}
I hope this code snippet will help you create your java application. Thanks.

Java - code to convert string to post script file using Ghostscript

I never worked with postscript before... And I need to replace a tool that:
Converts a string to postscript
Generates a pdf file based on a postscript file (done)
My issue is: I have no idea how to achieve step 1. By the way, I would preferably do in a similar way to the one I did on step (2).
I was wondering if I can just replace the parameters, but how? Could you please assist me?
The code to item (2) is below:
public byte[] convertPostScriptToPDF() throws IOException {
//get Ghostscript instance
Ghostscript gs = Ghostscript.getInstance();
File file= new File (this.getClass().getResource( "/resources/employer_report_last_page2.ps").getFile());//(Config.EMP_REPORT.REPORT_LAST_PAGE_STORE_PATH);
File pdfGenerated = File.createTempFile("output", "pdf");
System.out.println("Path for temp file -> " + pdfGenerated.getAbsolutePath());
//prepare Ghostscript interpreter parameters
//refer to Ghostscript documentation for parameter usage
String[] gsArgs = new String[10];
gsArgs[0] = "-ps2pdf";
gsArgs[1] = "-dNOPAUSE";
gsArgs[2] = "-dBATCH";
gsArgs[3] = "-dSAFER";
gsArgs[4] = "-sDEVICE=pdfwrite";
// gsArgs[5] = "-sOutputFile=output2.pdf";//output file name
gsArgs[5] = "-sOutputFile=" + pdfGenerated.getAbsolutePath();
// gsArgs[5] = "-sOutputFile=" + file.getAbsolutePath();
gsArgs[6] = "-c";
gsArgs[7] = ".setpdfwrite";
gsArgs[8] = "-f";
// gsArgs[9] = "input.ps";//input file name
gsArgs[9] = file.getAbsolutePath();//input file name
//execute and exit interpreter
try {
gs.initialize(gsArgs);
gs.exit();
} catch (GhostscriptException e) {
System.out.println("ERROR: " + e.getMessage());
}
FileInputStream fis = new FileInputStream(pdfGenerated);
return IOUtils.toByteArray(fis);
}
Thank you in advance!

I got no solution, so I realized I could try a workaround...
I changed the application logic. At the end it would convert everything to pdf, so instead of :
convert string to ps
add ps content other ps
convert ps to string
I did:
onvertd string to pdf
convert ps to pdf
merged both using the PDFMergerUtility

Trying to create file without succes - file appears elsewhere?

I tried to create 3 empty files in my home directory, using this:
this.mainpath = System.getenv("HOME"+"/");
this.single = new File(mainpath + "sin.r");
this.complete = new File (mainpath + "com.r");
this.ward = new File (mainpath+"w.r");
I was unter the impression that this would give me the files desired. However, if I search my home directory, or any other directory, for this files, none of them exists. What am I doing wrong?
Edit: I just find out: I do get a file, but not in my home directory, but the path to it would be /home/myname/NetBeansProjects/myorojectname/nullsin.r.
However, I specifically wanted to create the file in my home!
Well, my code now reads:
this.mainpath = System.getenv("user.home");
this.mainpath = this.mainpath + "/";
this.single = new File(mainpath + "sin.r");
this.single.createNewFile();
System.out.println(this.single.getAbsolutePath());
this.complete = new File (mainpath + "comp.r");
this.complete.createNewFile();
this.ward = new File (mainpath+"w.r");
this.ward.createNewFile();
The "success" of this, however, is that I get an IOException at the first createNeWFile(): File not found.
as for my code how I tried to write sth into those file, there it is:
FileWriter writer1 = null;
FileWriter writer2 = null;
FileWriter writer3 = null;
try {
writer1 = new FileWriter(single);
writer2 = new FileWriter(complete);
writer3 = new FileWriter(ward);
writer1.write("x = cbind(1,2,3)");
writer2.write("x = cbind(1,2,3)");
writer3.write("x = cbind(1,2,3)");
writer1.flush();
writer2.flush();
writer3.flush();
} catch (IOException ex) {
System.out.println(ex.getStackTrace());
} finally {
try {
writer1.close();
writer2.close();
writer3.close();
} catch (IOException ex) {
System.out.println(ex.getStackTrace());
}

You need to use getProperty() instead
System.getProperty("user.home");
Also, the / should be appended after getting the directory path.
this.mainpath = System.getProperty("user.home");
this.single = new File(mainpath + "/sin.r");
this.complete = new File (mainpath + "/com.r");
this.ward = new File (mainpath+"/w.r");

You can call the "createNewFile"-method for each of the objects you've declared to actually create them.