I have created a sequence file out of directory and then given index according to groups I want so that I can create groups using that index. This groups are then given one by one to my customized java class which gives information based on the file present in the group.
My problem is that some time it runs perfectly but some time gives different errors like null pointer exception, data type of field not found.
The problem is may be due to size of group. Because I am creating folder based group and then do the fetches the information from that folder inside my customized jar.
So how can I resolve this issue?
Below is my java class code:
public class OperateDirectory extends EvalFunc<DataBag>{
public TupleFactory tupleFactory = TupleFactory.getInstance();
public BagFactory bagFactory = BagFactory.getInstance();
public DataBag exec(Tuple input) throws IOException{
ArrayList<String> protoTuple = new ArrayList<>();
DataBag dataBag = bagFactory.newDefaultBag();
/* Create Directory */
if(input == null)
return dataBag;
if(input.size() != 2)
return dataBag;
long id = (long)input.get(0);
DataBag infoBag = (DataBag)input.get(1);
Iterator<Tuple> it = infoBag.iterator();
File dir = new File("/tmp/TestFolder"+id);
if(dir.exists())
{
FileUtils.cleanDirectory(dir);
}
else
{
dir.mkdir();
}
while(it.hasNext())
{
Tuple file_details = (Tuple)it.next();
if(file_details != null && file_details.size()==3)
{
String file_name = (String)file_details.get(1);
BytesWritable file_contents = (BytesWritable)file_details.get(2);
File f = new File(dir.getPath()+"/"+file_name);
f.deleteOnExit();
writeToFile(file_contents, f);
}
}
/* Perform operation here */
File f = new File("output"+id+".log");
ProcessBuilder performProcess1 = new ProcessBuilder("processes/processor", dir.getPath(),f.getPath());
Process process1 = performProcess1.start();
try
{
process1.waitFor();
if(f.exists() && f.length()>0)
{
ProcessBuilder performProcess2 = new ProcessBuilder("perl", "scripts/ParseFile.pl", f.getPath());
Process process2 = performProcess2.start();
InputStream is = process2.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line;
while ((line = br.readLine()) != null)
{
if(!line.isEmpty())
{
String [] tmpArray = line.split(",");
if(tmpArray.length == 2)
{
protoTuple.clear();
protoTuple.add(tmpArray[0]);
protoTuple.add(tmpArray[1]);
dataBag.add(tupleFactory.newTuple(protoTuple));
}
}
}
}
else
{
protoTuple.clear();
protoTuple.add("Error");
protoTuple.add("File "+f.getPath()+" does not exists ");
dataBag.add(tupleFactory.newTuple(protoTuple));
}
}
catch(Exception e)
{
protoTuple.clear();
protoTuple.add("Error ");
protoTuple.add(e.getMessage());
dataBag.add(tupleFactory.newTuple(protoTuple));
}
try
{
FileUtils.cleanDirectory(dir);
FileUtils.deleteDirectory(dir);
}
catch(Exception e)
{
}
return dataBag;
}
void writeToFile(BytesWritable value, File binaryFile) throws IOException{
FileOutputStream fileOut = new FileOutputStream(binaryFile);
fileOut.write(value.getBytes(), 0, value.getLength());
fileOut.close();
}
}
Related
I am having a spot of bother getting BufferedWriter to write output to a txt file I have.
When I compile the program, I do not get any errors but the txt file arrives in my project blank.
I have looked on here at similar questions and tried a few of the proposed solutions to resolve it without success, like closing the stream etc
public void storeToDoItemsOntoTxtFile () throws IOException {
Path path = Paths.get(fileName);
BufferedWriter buffyWriter = Files.newBufferedWriter(path);
try {
Iterator<ToDoItem> bigIterator = toDoItems.iterator();
while (bigIterator.hasNext()) {
ToDoItem item = bigIterator.next();
buffyWriter.write(String.format("%s\t%S\t%s", item.getShortDescription(), item.getDetails(), item.getDeadline().format(formatter)));
buffyWriter.newLine();
}
} finally {
if (buffyWriter !=null) {
buffyWriter.close();// when we are done working with the writer
}
}
}
BufferedReader
public void loadToDoItemsFromTxtFile() throws IOException {
toDoItems = FXCollections.observableArrayList();
Path path = Paths.get(fileName);
BufferedReader buffyReader = Files.newBufferedReader(path);
String buffyInput;
try {
while ((buffyInput = buffyReader.readLine()) !=null) {
String [] itemPieces = buffyInput.split("\t");
String shortDescription = itemPieces[0];
String details = itemPieces[1]; //
String dateString = itemPieces [2];
LocalDate date = LocalDate.parse(dateString, formatter);
ToDoItem toDoItem = new ToDoItem(shortDescription,details,date);
toDoItems.add(toDoItem);
}
} finally {
if (buffyReader != null) {// ie the buffy reader states that there is a toDoItem" in the txt file
buffyReader.close();
}
}
}
I want to use Java export Oracle database, and obtain the export status(indicates success or failure), if the operation failed, should return the reason why failed.
But I have trouble in this problem, the export is success,but the label I defined is always false and return [].
What should I do to get the true status or obtain the failure details.
public class DumpFile {
/**
* Default constructor
*/
public DumpFile() {
}
/**
* #return
* #throws InterruptedException
*/
public static boolean LoadToOracle(String Path) throws InterruptedException {
String importStr = "imp scott/tiger#orcl file="+Path+" full=y ignore=y";
Process process_oracle = null;
boolean flag = false;
List<String[]> processListOracle = new ArrayList<String[]>();
try {
process_oracle = Runtime.getRuntime().exec(importStr);
process_oracle.waitFor();
BufferedReader input = new BufferedReader(new InputStreamReader(
process_oracle.getInputStream(), "utf8"));
String line = "";
while ((line = input.readLine()) != null) {
System.out.println(line);
String[] content = line.split("\n");
processListOracle.add(content);
}
int exevalue = process_oracle.waitFor();
System.out.println("exevalue:"+exevalue);
input.close();
} catch (Exception e) {
e.printStackTrace();
}
for (String[] line : processListOracle)
for (String temp : line) {
if (temp.trim().equals("successfully"))
flag = true;
}
return flag;
}
public static String exportFromOracle(String FileName) throws InterruptedException {
String Path="/home/oracle/output/";
String exportStr = "exp scott/tiger#orcl file="+Path+FileName;
Process process_oracle = null;
boolean flag = false;
List<String[]> processListOracle = new ArrayList<String[]>();
try {
process_oracle = Runtime.getRuntime().exec(exportStr);
BufferedReader input = new BufferedReader(new InputStreamReader(
process_oracle.getInputStream(), "utf8"));
String line = "";
while ((line = input.readLine()) != null) {
System.out.println("line:"+line);
String[] content = line.split("\n");
processListOracle.add(content);
}
int exevalue = process_oracle.waitFor();
System.out.println("exevalue:"+exevalue);
input.close();
} catch (Exception e) {
e.printStackTrace();
}
for (String[] line : processListOracle)
for (String temp : line) {
if (temp.trim().equals("successfully"))
flag = true;
}
System.out.println("flag:"+flag);
if (flag==true)
return flag+"test"+Path;
else{
for (String[] line : processListOracle)
System.out.println(line);
}
return processListOracle.toString();
}
public static void main(String[] args) throws Exception {
String a=exportFromOracle("test.dmp");
System.out.println("a.isEmpty:"+a.isEmpty());
System.out.println(a);
}
}
OUTPUT:
exevalue:0
a.isEmpty:false
[]
I have solved this problem by "log file", I use imp export the database and output the log file, then I use Java to read the log file, to obtain the status.
I have written a Java code which I need to be written in Map Reduce.
I am relatively new to Map Reduce concepts.
As an input, I have a file containing multiple columns which are Pipe delimited.
Based on certain criteria which are given in my if condition of my code, I need to link the Row-Id's from the file.
for Example:
rowid|rollno|sbjctcode|subjct|dt_of_exam......|marks
2|1000|0101|PHY|02072015......|-060
9|1000|0101|PHY|02072015......|060
Desired Output:
2[9]
so that, I can get to know where is the error in the file, or for a case multiple entries etc.
Here's my Java Code:
import java.io.*;
import java.util.*;
public class MRCode {
public static void main(String[] args) {
BufferedReader in = null;
BufferedWriter out = null;
String in_line;
String PrevRollNo = "150";
ArrayList<Transaction> PCMList = new ArrayList<Transaction>();
ArrayList<Transaction> AdditionalList = new ArrayList<Transaction>();
ArrayList<Transaction> OtherList = new ArrayList<Transaction>();
try {
in = new BufferedReader(new FileReader(
"C:\\Users\\Desktop\\data.txt"));
File out_file = new File("C:\\Users\\Desktop\\op.txt");
if (!out_file.exists()) {
out_file.createNewFile();
}
FileWriter fw = new FileWriter(out_file);
out = new BufferedWriter(fw);
while ((in_line = in.readLine()) != null) {
Transaction transact = new Transaction(in_line);
if (transact.rollNo.equals(PrevRollNo)) {
if (transact.subjctCode.equals("PHY")
|| transact.subjctCode.equals("CHEM")
|| transact.subjctCode.equals("MATH")
&& transact.dt_of_exam == PrevDate) {
PCMList.add(transact);
break;
} else {
OtherList.add(transact);
}
if (transact.subjctCode.equals("0101")) {
Iterator<Transaction> pcm;
pcm = PCMList.iterator();
while (pcm.hasNext()) {
Transaction pcmtxn = pcm.next();
if (pcmtxn.subjctCode.equals("0102")
&& pcmtxn.dt_of_exam == transact.dt_of_exam
&& pcmtxn.subjct.equals(transact.subjct)
&& pcmtxn.marks == Math.abs(transact.marks)) {
pcmtxn.tgtfound = true;
transact.srcfound = true;
System.out.println(pcmtxn.row_id + "["
+ transact.row_id + "]");
}
}
}
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
static class Transaction {
public String rollNo, subjctCode, subjct, l1, l2;
public int row_id, dt_of_exam, dt_of_chking;
public double marks;
public boolean srcfound, tgtfound;
public Transaction(String in_line) {
String[] SplitData = in_line.split("\\|");
row_id = Integer.parseInt(SplitData[0]);
rollNo = SplitData[1];
subjctCode = SplitData[4];
subjct = SplitData[5];
l1 = SplitData[7];
l2 = SplitData[8];
dt_of_exam = Integer.parseInt(SplitData[2]);
dt_of_chking = Integer.parseInt(SplitData[3]);
marks = Double.parseDouble(SplitData[11]);
srcfound = false;
tgtfound = false;
}
}
}
Please help me with your thoughts on how can use Array List in my mapper class.
or how can I write this code better in Map Reduce.
Note: This is just a snapshot.
Do share your views.
I am able to load files in Fixed Format but unable to write a fixed format file using FixedFormat4j.
Any idea how to do that?
public class MainFormat {
public static void main(String[] args) {
new MainFormat().start();
}
private FixedFormatManager manager;
public void start(){
System.out.println("here");
manager = new FixedFormatManagerImpl();
try {
BufferedReader br = new BufferedReader(new FileReader("d:/myrecords.txt"));
System.out.println("here1");
String text;
MyRecord mr = null;
while ((text = br.readLine()) != null) {
mr = manager.load(MyRecord.class, text);
System.out.println(""+mr.getAddress() + " - "+mr.getName());
}
mr.setName("Rora");
manager.export(mr);
} catch (IOException | FixedFormatException ex) {
System.out.println(""+ex);
}
}
}
I have seen export method but don't understand how to use it? Nothing happens in above code
The export method returns a string representing the marshalled record.
In your code you would need to write out the result of the export command to a FileWriter. So:
before the while loop:
FileWriter fw = new FileWriter("d:/myrecords_modified.txt", true);
BufferedWriter bw = new BufferedWriter(fw);
after the while loop:
mr.setName("Rora");
String modifiedRecord = manager.export(mr);
bw.write(modifiedRecord);
I'm working on a homework assignment and have run into an odd "ArrayOutOfBoundsException" error - I know what the error means (essentially I'm trying to reference a location in an array that isn't there) but I'm not sure why it's throwing that error? I'm not sure what I'm missing, but obviously there must be some logic error somewhere that I'm not seeing.
PhoneDirectory.java
import java.util.HashMap;
import java.io.*;
class PhoneDirectory {
private HashMap<String, String> directoryMap;
File directory;
public PhoneDirectory() { //create file for phone-directory
directory = new File("phone-directory.txt");
directoryMap = new HashMap<String, String>();
try(BufferedReader buffer = new BufferedReader(new FileReader(directory))) {
String currentLine;
while((currentLine = buffer.readLine()) != null) { //set currentLine = buffer.readLine() and check if not null
String[] fileData = currentLine.split(","); //create array of values in text file - split by comma
directoryMap.put(fileData[0], fileData[1]); //add item to directoryMap
}
}
catch(IOException err) {
err.printStackTrace();
}
}
public PhoneDirectory(String phoneDirectoryFile) {
directory = new File(phoneDirectoryFile);
directoryMap = new HashMap<String, String>();
try(BufferedReader buffer = new BufferedReader(new FileReader(directory))) {
String currentLine;
while((currentLine = buffer.readLine()) != null) { //set currentLine = buffer.readLine() and check if not null
String[] fileData = currentLine.split(","); //create array of values in text file - split by comma
directoryMap.put(fileData[0], fileData[1]); //add item to directoryMap
}
}
catch(IOException err) {
err.printStackTrace();
}
}
public String Lookup(String personName) {
if(directoryMap.containsKey(personName))
return directoryMap.get(personName);
else
return "This person is not in the directory.";
}
public void AddOrChangeEntry(String name, String phoneNumber) {
//ASK IF "IF-ELSE" CHECK IS NECESSARY
if(directoryMap.containsKey(name))
directoryMap.put(name,phoneNumber); //if name is a key, update listing
else
directoryMap.put(name, phoneNumber); //otherwise - create new entry with name
}
public void DeleteEntry(String name) {
if(directoryMap.containsKey(name))
directoryMap.remove(name);
else
System.out.println("The person you are looking for is not in this directory.");
}
public void Write() {
try(BufferedWriter writeDestination = new BufferedWriter(new FileWriter(directory)))
{
for(String key : directoryMap.keySet())
{
writeDestination.write(key + ", " + directoryMap.get(key) + '\n');
writeDestination.newLine();
}
}
catch(IOException err) {
err.printStackTrace();
}
}
}
Driver.java
public class Driver {
PhoneDirectory list1;
public static void main(String[] args) {
PhoneDirectory list1 = new PhoneDirectory("test.txt");
list1.AddOrChangeEntry("Disney World","123-456-7890");
list1.Write();
}
}
Essentially I'm creating a file called "test.txt" and adding the line "Disney World, 123-456-7890" - what's weird is that the code still works - but it throws me that error anyway, so what's really happening? (For the record, I'm referring to the line(s): directoryMap.put(fileData[0], fileData[1]) - which would be line 14 and 28 respectively.)