MR implementation not working in Hadoop cluster - java

Decision tree is working perfectly in Eclipse Juno.
But when i tried to run that in my cluster it is showing error.
Folder "n " is created in my local disk /user/sree
When i tried hadoop fs -ls /user/sree/n
Nothing is in n and no "intermediate" files are created in my /user/sree/n
why is it so? It is working perfectly in Eclipse.
Any suggesions.
** UPDATE **
I updated my code to
1.Instead of
BufferedWriter bw = new BufferedWriter(new FileWriter(new File("n/intermediate"+id.current_index+".txt"), true));
in Reduce.java changed to
BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(fs.create(new Path("n/intermediate"+id.current_index+".txt"), true)));
2.Instead of
fstream = new FileInputStream("n/intermediate"+id.current_index+".txt");
DataInputStream in = new DataInputStream(fstream);
BufferedReader br = new BufferedReader(new InputStreamReader(in));
in GainRatio.java changed to
BufferedReader br = new BufferedReader(new InputStreamReader(fs.open(new Path("n/intermediate"+id.current_index+".txt"))));
It is executing,but it is not completely executed.
I am not able to get the final out.
Am i doing anything wrong.

Because
BufferedWriter bw = new BufferedWriter(new FileWriter(new File("C45/intermediate"+id.current_index+".txt"), true));
bw.write(text);
Writes to the local disk and not to HDFS. Thus you have to look for it in your local filesystem.

Related

How to properly read and write files in Gradle resources?

Java Gradle project
File saving in build/resources, but reading from src/resources
Does anyone know how to save and read from the same directory?
below is my code for writing:
File file = new File((this.getClass().getClassLoader().getResource("test.txt")).toURI());
FileWriter fw = new FileWriter(file, false);
BufferedWriter bw = new BufferedWriter(fw);
PrintWriter pw = new PrintWriter(bw);
pw.println("test");
pw.close();
below is my code for reading:
BufferedReader br = new BufferedReader(new FileReader(new File((this.getClass().getClassLoader().getResource("test.txt")).toURI())));
String amount = br.readLine();

Editing an already existing text file

I'm trying to edit an existing file that I had just created and so far I have no clue on how it's done.
Can anyone show me how and please explain line by line on what the code does?
import java.io.*;
public class Hey {
public static void main(String[] args)throws Exception{
BufferedReader br = new BufferedReader (new InputStreamReader(System.in));
System.out.println("Title");
String title = br.readLine();
File f = new File(title +".txt");
f.createNewFile();
FileWriter fw = new FileWriter(f);
BufferedWriter bw = new BufferedWriter(fw);
System.out.println("What you want to input in the text");
String text = br.readLine();
bw.write(text);
bw.flush();
bw.close();
}
}
BufferedReader br = new BufferedReader (new InputStreamReader(System.in));
Creates a read buffer from the standard input.
String title = br.readLine();
Reads from this buffer until there's a return character sequence found ('\n', '\r' or "\r\n"). The entire line excluding the return sequence will be saved as title.
File f = new File(title +".txt");
Creates a File object with the name read from the console.
f.createNewFile();
Creates the file if it doesn't exist yet.
FileWriter fw = new FileWriter(f);
BufferedWriter bw = new BufferedWriter(fw);
Creates a buffered writer to write into fw.
String text = br.readLine();
Again reads a line from the console.
bw.write(text);
Writes this line into the buffer.
bw.flush();
Ensures the whole buffer is flushed into the file (written into the file).
bw.close();
Closes the buffer of your buffered writer. You should also close the reader buffer br and the FileWriter fw.

How to close a file in java

This is my code :
FileWriter fw = new FileWriter(log,true);
BufferedWriter bw = new BufferedWriter(fw);
StringWriter sw = new StringWriter();
PrintWriter pw = new PrintWriter(sw);
fw.close();
bw.close();
sw.close();
pw.close();
I want to change it to something like this :
BufferedWriter bw = new BufferedWriter(new FileWriter(log,true));
PrintWriter pw = new PrintWriter(new StringWriter());
bw.close();
pw.close();
Will this be correct, or will the missing close() calls cause problems?
To be sure the close is not forgotten you could use the try-with-resource statement
try (BufferedWriter bw = new BufferedWriter(new FileWriter(log, true));
PrintWriter pw = new PrintWriter(new StringWriter())) {
// do your processing here
} catch (IOException ex) {
// do your exception handling
}
The compiler will add for your the appropriate code for the close().
When you close a BufferedWriter Object which takes a FileWriter Object in its constructor, then implicitly the FileWriter Object is also closed.
So to answer your question, both ways are fine and the same and it don't make any problem
Previous Answer
Best practice
Since Java 7 you can let java automatically close the resources for you. Take a look at the try with resources.
try (BufferedReader br = new BufferedReader(new FileReader(path))) {
return br.readLine();
}

Opening and appending files

I need to open an existing file for appending and create new file for appending if it doesn't exist.
I tried PrintWriter function but it always create a new file and deletes the old. So could you help me? What should I use for that?
UPD: That's what I already tried
writer = new PrintWriter(System.getProperty("db.file"), "UTF-8");
writer.println("The first line");
Try this
PrintWriter pw = new PrintWriter(new BufferedWriter(new FileWriter("your_file.txt", true)));
The true parameter of FileWriter indicates it has to append data.
To add specify encoding you can use
PrintWriter pw = new PrintWriter(new BufferedWriter(new OutputStreamWriter(
new FileOutputStream("your_file.txt", true), "UTF-8")));

Good practice in Java File I/O

I am trying to read integers from a file, apply some operation on them and writing those resulting integers to another file.
// Input
FileReader fr = new FileReader("test.txt");
BufferedReader br = new BufferedReader(fr);
Scanner s = new Scanner(br);
// Output
FileWriter fw = new FileWriter("out.txt");
BufferedWriter bw = new BufferedWriter(fw);
PrintWriter pw = new PrintWriter(bw);
int i;
while(s.hasNextInt())
{
i = s.nextInt();
pw.println(i+5);
}
I want to ask is it a good practice to wrap these input and output streams like this?
I am new to java and on internet, I saw lots of other ways of I/O in files. I want to stick to one approach so is above the best approach ?
- Well consider that you went shopping into a food mall, Now what you do usually, pick-up each item from the selves and then go to the billing counter then again go to the selves and back to billing counter ....?? Or Store all the item into a Cart then go to the billing counter.
- Its similar here in Java, Files deal with bytes, and Buffer deals with characters, so there is a conversion of bytes to characters and trust me it works well, there will not be any noticeable overhead.
So to Read the File:
File f = new File("Path");
FileReader fr = new FileReader(f);
BufferedReader br = new BufferedReader(fr);
So to Write the File:
File f = new File("Path");
FileWriter fw = new FileWriter(f);
BufferedWriter bw = new BufferedWriter(fw);
And when you use Scanner there is no need to use BufferedReader
Keep in mind that the design of those classes is based on the Decorator design pattern. A good practice is to close all instances of java.io.Closeable in a finally block. For example:
Reader r = null;
Scanner s = null;
try {
r = new FileReader("test.txt");
s = new Scanner(r);
// Do your stuff here.
} finally {
if (r != null)
r.close();
if (s != null)
s.close();
}
or, if you are using Java 7 or higher:
try (
Reader r = new FileReader("test.txt");
Scanner s = new Scanner(r)
) {
// Do your stuff here.
}
you dont really need BuffredWriter when you are using PrintWriter to write character data, printwriter has a constructor which takes filewriter as an argument. and dont need a scanner to read from a file you could acheive it using bufferedreader itself.
FileReader fr = new FileReader("test.txt");
BufferedReader br = new BufferedReader(fr);
while((line=br.readLine())!=null){
//do read operations here
}
FileWriter fw = new FileWriter("out.txt");
PrintWriter pw = new PrintWriter(fw);
pw.println("write some data to the file")
Scanner does not need the BufferedReader. You can wrap it over the FileReader.
Scanner s = new Scanner(new FileReader("test.txt"));
While using the scanner its better to assume that the source contains various content. Its good to close the scanner after using it.
while(s.hasNext()){
if(s.hasNextInt())
int i = s.nextInt();
s.next();
}
s.close();
I usually do this:
String inputFileLocation = "Write it here";
BufferedReader br = new BufferedReader(new FileReader(new File(fileLocation)));
while((line=br.readLine())!=null){
//Scanner operations here
}
String outputFileLocation = "Here";
PrintWriter pr = new PrintWriter(new FileWriter(new File(outputFileLocation)));

Categories