We Keep Coding

Processing and splitting large files with Java 8

I'm new to Java 8 and I have just started using the NIO package for file-handling. I need help in how to process large files--varying from 100,000 lines to 1,000,000 lines per file--by transforming each line into a specific format and writing the formatted lines to new files. The new file(s) generated must only contain a maximum of 100,000 lines per file. So:
if I have a 500,000-line file for processing, I must transform those
lines and distribute and print them on 5 new files.
if I have a 745,000-line file for processing, I must transform those
lines and print them on 8 new files.
I'm having a hard time figuring out an approach that will efficiently utilize the new features of Java 8. I've started out with determining the number of new files to be generated based on the line count of the large file, and then creating those new empty files:
Path largFile = Path.get("path\to\file");
long recordCount = Files.lines(file).count();
int maxRecordOfNewFiles = 100000;
int numberOfNewFiles = 1;
if (recordCount > maxRecordOfNewFiles) {
numberOfNewFiles = Math.toIntExact(recordCount / maxRecordOfNewFiles);
if (Math.toIntExact(recordCount % maxRecordOfNewFiles) > 0) {
numberOfNewFiles ++;
}
}
IntStream.rangeClosed(1, numberOfNewFiles).forEach((i)
-> {
try {
Path newFile = Paths.get("path\to\newFiles\newFile1.txt");
Files.createFile(cdpFile);
} catch (IOException iOex) {
}
});
But as I go through the the lines of the largeFile through the Files.lines(largeFile).forEach(()) capability, I got lost on how to proceed with formatting the first 100,000 lines and then determining the first of the new files and printing them on that file, and then the second batch of 100,000 to the second new file, and so on.
Any help will be appreciated. :)

When you start conceiving batch processes, I think you should consider using a framework specialized in that. You may want to handle restarts, scheduling... Spring Batch is very good for that and already provides what you want: MultiResourceItemWriter that writes to multiple files with max lines per file and FlatFileItemReader to read data from a file.
In this case, what you want is to loop over each line of an input file and write a transformation of each line in multiple output files.
One way to do that would be to create a Stream over the lines of the input file, map each line and send it to a custom writer. This custom writer would implement the logic of switching writer when it has reached the maximum number of lines per file.
In the following code MyWriter opens a BufferedWriter to a file. When the maxLines is reached (a multiple of it), this writer is closed and another one is opened, incrementing currentFile. This way, it is transparent for the reader that we're writing to multiple files.
public static void main(String[] args) throws IOException {
try (
MyWriter writer = new MyWriter(10);
Stream<String> lines = Files.lines(Paths.get("path/to/file"));
) {
lines.map(l -> /* do transformation here */ l).forEach(writer::write);
}
}
private static class MyWriter implements AutoCloseable {
private long count = 0, currentFile = 1, maxLines = 0;
private BufferedWriter bw = null;
public MyWriter(long maxLines) {
this.maxLines = maxLines;
}
public void write(String line) {
try {
if (count % maxLines == 0) {
close();
bw = Files.newBufferedWriter(Paths.get("path/to/newFiles/newFile" + currentFile++ + ".txt"));
}
bw.write(line);
bw.newLine();
count++;
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
#Override
public void close() throws IOException {
if (bw != null) bw.close();
}
}

From what I understand in question. A simple way can be:
BufferedReader buff = new BufferedReader(new FileReader(new File("H:\\Docs\\log.txt")));
Pair<Integer, BufferedWriter> ans = buff.lines().reduce(new Pair<Integer, BufferedWriter>(0, null), (count, line) -> {
try {
BufferedWriter w;
if (count.getKey() % 1000 == 0) {
if (count.getValue() != null) count.getValue().close();
w = new BufferedWriter(new FileWriter(new File("f" + count.getKey() + ".txt")));
} else w = count.getValue();
w.write(line + "\n"); //do something
return new Pair<>(count.getKey() + 1, w);
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}, (x, y) -> {
throw new RuntimeException("Not supproted");
});
ans.getValue().close();

Export to CSV JSF and PrimeFaces

Export to Excel JSF and PrimeFaces
I am trying to download CSV file which is created in runtime. This link is useful for excel and I need to do the same for CSV. HSSFWorkbook is used for excel but I am using FileWriter for CSV. I need a line to use rather than workbook.write(externalContext.getResponseOutputStream()); I cannot use writer.write(externalContext.getResponseOutputStream()); writer is FileWriter variable and does not accept outputStream as parameter.

It seems to me that you have two issues here :
You shouldn't have a FileWriter if you don't want to write to a file - you need to choose the right implementation of the Writer abstract class for your use case (here, you want to chose the one that writes to an OutputStream, not to a File).
You're trying to use Writer#write(...) like HSSFWorkbook#write(java.io.OutputStream), but they don't do the same thing at all. In HSSFWorkbook, the write method writes the workbook's content to some OutputStream; the parameter tells the method where you want to write. In Writer, the write method writes something to the writer itself; the parameter tells the method what you want to write.
Based on your link for writing from a HSSFWorkbook, writing a CSV in a similar way could look something like :
public void getReportData() throws IOException {
FacesContext facesContext = FacesContext.getCurrentInstance();
ExternalContext externalContext = facesContext.getExternalContext();
externalContext.setResponseContentType("text/csv");
externalContext.setResponseHeader("Content-Disposition", "attachment; filename=\"my.csv\"");
OutputStream out = externalContext.getResponseOutputStream());
Writer writer = new OutputStreamWriter(out);
// Let's write the CSV content
try {
writer.write("Line number,Col 1,Col 2");
writer.write("1,Value 1,Value 2");
writer.write("2,Value 3,Value4");
} finally {
if (writer != null {
// Closing the writer also flushes it, and does the same to the underlying OutputStream
writer.close();
}
}
facesContext.responseComplete();
}

Whole working copy that you may use ;
String csvFileName = "mydoc.csv";
FileWriter writer = new FileWriter(csvFileName);
int columnNameSize = activeTab.getColumnNames().size();
for (int i = 0; i < columnNameSize; i++) {
writer.append(activeTab.getColumnNames().get(i));
if (i != (columnNameSize - 1)) {
if (delimiterType.equalsIgnoreCase(TAB_DELIMITER_VALUE_NAME)) {
writer.append('\t');
} else {
writer.append(delimiterType);
}
}
}
writer.append("\n");
for (DBData[] temp : activeTab.getTabularData()) {
int tempSize = temp.length;
for (int k = 0; k < tempSize; k++) {
writer.append(temp[k].toFullString());
if (k != (tempSize - 1)) {
if (delimiterType.equalsIgnoreCase(TAB_DELIMITER_VALUE_NAME)) {
writer.append('\t');
} else {
writer.append(delimiterType);
}
}
}
writer.append("\n");
}
writer.flush();
writer.close();
InputStream stream = new BufferedInputStream(new FileInputStream(csvFileName));
exportFile = new DefaultStreamedContent(stream, "application/csv", csvFileName);

java split string[] array to multiple files

I'm having a problem figuring out how to split a string to multiple files. At the moment I should get two files both with JSON data. The code below writes to the first file but leaves the second empty. Any ideas why?
public void splitFile(List<String> results) throws IOException {
int name = 0;
for (int i=0; i<results.size(); i ++) {
write = new FileWriter("/home/tom/files/"+ name +".json");
out = new BufferedWriter(write);
out.write(results.get(i));
if (results.get(i).startsWith("}")) {
name++;
}
}
}
Edit: it splits at line starting with { because that denotes the end of a JSON document.

Enhance the cut-control
Get togher this:
write = new FileWriter("/home/tom/files/"+ name +".json");
out = new BufferedWriter(write);
and this:
name++;
Check for starting, not for end
Check for line starting with {, and execute those three lines to open the file.
Remember to close and flush
If it's not the first line (i > 0) then close the last writer (write.close();).
Close the last opened writer
if (!results.isEmpty())
out.close();
Result
It should look something like this:
public void splitFile(List<String> results) throws IOException {
int name = 0;
BufferedWriter out = null;
for (int i=0; i<results.size(); i ++) {
String line = results.get(i);
if (line.startsWith("{")) {
if (out != null) // it's not the first
out.close(); // tell buffered it's going to close, it makes it flush
FileWriter writer = new FileWriter("/home/tom/files/"+ name +".json");
out = new BufferedWriter(writer);
name++;
}
if (out == null)
throw new IllegalArgumentException("first line doesn't start with {");
out.write(line);
}
if (out != null) // there was at least one file
out.close();
}

I would close your buffered writer after each completed write sequence. i.e. after each iteration through the loop before you assign write to a new FileWriter().
Closing the BufferedWriter will close the underlying FileWriter, and consequently force a flush on the data written to the disk.
Note: If you're using a distinct FileWriter per loop then I'd scope that variable to that inner loop e.g.
FileWriter write = new FileWriter("/home/tom/files/"+ name +".json");
The same goes for the BufferedWriter. In fact you can write:
BufferedWriter outer = new BufferedWriter(new FileWriter(...
and just deal with outer.

Try the following code..
public void splitFile(List<String> results) throws IOException {
int name = 0;
for (int i = 0; i < results.size(); i++) {
write = new FileWriter("/home/tom/files/" + name + ".json");
out = new BufferedWriter(write);
out.write(results.get(i));
out.flush();
out.close(); // you have to close your stream every time in your case.
if (results.get(i).startsWith("}")) {
name++;
}
}
}

Java - Scanner not scanning after a certain number of lines

I'm doing some relatively simple I/O in Java. I have a .txt files that I'm reading from using a Scanner and a .txt file I'm writing to using a BufferedWriter. Another Scanner then reads that file and another BufferedWriter then creates another .txt file. I've provided the code below just in case, but I don't know if it will help too much, as I don't think the code is the issue here. The code compiles without any errors, but it's not doing what I expect it to. For some reason, charReader will only read about half of its file, then hasNext() will return false, even though the end of the file hasn't been reached. These aren't big text files - statsReader's file is 34 KB and charReader's file is 29 KB, which is even weirder, because statsReader reads its entire file fine, and it's bigger! Also, I do have that code surrounded in a try/catch, I just didn't include it.
From what I've looked up online, this may happen with very large files, but these are quite small, so I'm pretty lost.
My OS is Windows 7 64-bit.
Scanner statsReader = new Scanner(statsFile);
BufferedWriter statsWriter = new BufferedWriter(new FileWriter(outputFile));
while (statsReader.hasNext()) {
statsWriter.write(statsReader.next());
name = statsReader.nextLine();
temp = statsReader.nextLine();
if (temp.contains("form")) {
name += " " + temp;
temp = statsReader.next();
}
statsWriter.write(name);
statsWriter.newLine();
statsWriter.write(temp);
if (! (temp = statsReader.next()).equals("-"))
statsWriter.write("/" + temp);
statsWriter.write("\t");
statsWriter.write(statsReader.nextInt() + "\t");
statsWriter.write(statsReader.nextInt() + "\t");
statsWriter.write(statsReader.nextInt() + "\t");
statsWriter.write(statsReader.nextInt() + "\t");
statsWriter.write(statsReader.nextInt() + "\t");
statsWriter.write(statsReader.nextInt() + "");
statsWriter.newLine();
statsReader.nextInt();
}
Scanner charReader = new Scanner(charFile);
BufferedWriter codeWriter = new BufferedWriter(new FileWriter(codeFile));
while (charReader.hasNext()) {
color = charReader.next();
name = charReader.nextLine();
name = name.replaceAll("\t", "");
typing = pokeReader.next();
place = charReader.nextInt();
area = charReader.nextInt();
def = charReader.nextInt();
shape = charReader.nextInt();
size = charReader.nextInt();
spe = charReader.nextInt();
index = typing.indexOf('/');
if (index == -1) {
typeOne = determineType(typing);
typeTwo = '0';
}
else {
typeOne = determineType(typing.substring(0, index));
typeTwo = determineType(typing.substring(index+1, typing.length()));
}
}
SSCCE:
public class Tester {
public static void main(String[] args) {
File statsFile = new File("stats.txt");
File testFile = new File("test.txt");
try {
Scanner statsReader = new Scanner(statsFile);
BufferedWriter statsWriter = new BufferedWriter(new FileWriter(testFile));
while (statsReader.hasNext()) {
statsWriter.write(statsReader.nextLine());
statsWriter.newLine();
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}

This is a classic problem: You need to flush and close the output stream (in this case statsWriter) before reading the file.
Being buffered, it doesn't actually write to the file with ever call to write. Calling flush forces it to complete any pending write operations.
Here's the javadoc for OutputStream.flush():
Flushes this output stream and forces any buffered output bytes to be written out. The general contract of flush is that calling it is an indication that, if any bytes previously written have been buffered by the implementation of the output stream, such bytes should immediately be written to their intended destination.

After you have written your file with your statsWriter, you need to call:
statsWriter.flush();
statsWriter.close();
or simply:
statsWriter.close(); // this will call flush();
This is becuase your are using a Buffered Writer, it does not write everything out to the file as you call the write functions, but rather in buffered chunks. When you call flush() and close(), it empties all the content it still has in it's buffer out to the file, and closes the stream.
You will need to do the same for your second writer.

How to write console output to a txt file

I have tried to write the console output to a txt file using this code suggestion (http://www.daniweb.com/forums/thread23883.html#) however I was not successful. What's wrong?
try {
//create a buffered reader that connects to the console, we use it so we can read lines
BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
//read a line from the console
String lineFromInput = in.readLine();
//create an print writer for writing to a file
PrintWriter out = new PrintWriter(new FileWriter("output.txt"));
//output to the file a line
out.println(lineFromInput);
//close the file (VERY IMPORTANT!)
out.close();
}
catch(IOException e1) {
System.out.println("Error during reading/writing");
}

You need to do something like this:
PrintStream out = new PrintStream(new FileOutputStream("output.txt"));
System.setOut(out);
The second statement is the key. It changes the value of the supposedly "final" System.out attribute to be the supplied PrintStream value.
There are analogous methods (setIn and setErr) for changing the standard input and error streams; refer to the java.lang.System javadocs for details.
A more general version of the above is this:
PrintStream out = new PrintStream(
new FileOutputStream("output.txt", append), autoFlush);
System.setOut(out);
If append is true, the stream will append to an existing file instead of truncating it. If autoflush is true, the output buffer will be flushed whenever a byte array is written, one of the println methods is called, or a \n is written.
I'd just like to add that it is usually a better idea to use a logging subsystem like Log4j, Logback or the standard Java java.util.logging subsystem. These offer fine-grained logging control via runtime configuration files, support for rolling log files, feeds to system logging, and so on.
Alternatively, if you are not "logging" then consider the following:
With typical shells, you can redirecting standard output (or standard error) to a file on the command line; e.g.
$ java MyApp > output.txt
For more information, refer to a shell tutorial or manual entry.
You could change your application to use an out stream passed as a method parameter or via a singleton or dependency injection rather than writing to System.out.
Changing System.out may cause nasty surprises for other code in your JVM that is not expecting this to happen. (A properly designed Java library will avoid depending on System.out and System.err, but you could be unlucky.)

There is no need to write any code, just in cmd
on the console you can write:
javac myFile.java
java ClassName > a.txt
The output data is stored in the a.txt file.

to preserve the console output, that is, write to a file and also have it displayed on the console, you could use a class like:
public class TeePrintStream extends PrintStream {
private final PrintStream second;
public TeePrintStream(OutputStream main, PrintStream second) {
super(main);
this.second = second;
}
/**
* Closes the main stream.
* The second stream is just flushed but <b>not</b> closed.
* #see java.io.PrintStream#close()
*/
#Override
public void close() {
// just for documentation
super.close();
}
#Override
public void flush() {
super.flush();
second.flush();
}
#Override
public void write(byte[] buf, int off, int len) {
super.write(buf, off, len);
second.write(buf, off, len);
}
#Override
public void write(int b) {
super.write(b);
second.write(b);
}
#Override
public void write(byte[] b) throws IOException {
super.write(b);
second.write(b);
}
}
and used as in:
FileOutputStream file = new FileOutputStream("test.txt");
TeePrintStream tee = new TeePrintStream(file, System.out);
System.setOut(tee);
(just an idea, not complete)

Create the following method:
public class Logger {
public static void log(String message) {
PrintWriter out = new PrintWriter(new FileWriter("output.txt", true), true);
out.write(message);
out.close();
}
}
(I haven't included the proper IO handling in the above class, and it won't compile - do it yourself. Also consider configuring the file name. Note the "true" argument. This means the file will not be re-created each time you call the method)
Then instead of System.out.println(str) call Logger.log(str)
This manual approach is not preferable. Use a logging framework - slf4j, log4j, commons-logging, and many more

In addition to the several programatic approaches discussed, another option is to redirect standard output from the shell. Here are several Unix and DOS examples.

You can use System.setOut() at the start of your program to redirect all output via System.out to your own PrintStream.

This is my idea of what you are trying to do and it works fine:
public static void main(String[] args) throws IOException{
BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
BufferedWriter out = new BufferedWriter(new FileWriter("c://output.txt"));
try {
String inputLine = null;
do {
inputLine=in.readLine();
out.write(inputLine);
out.newLine();
} while (!inputLine.equalsIgnoreCase("eof"));
System.out.print("Write Successful");
} catch(IOException e1) {
System.out.println("Error during reading/writing");
} finally {
out.close();
in.close();
}
}

The easiest way to write console output to text file is
//create a file first
PrintWriter outputfile = new PrintWriter(filename);
//replace your System.out.print("your output");
outputfile.print("your output");
outputfile.close();

To write console output to a txt file
public static void main(String[] args) {
int i;
List<String> ls = new ArrayList<String>();
for (i = 1; i <= 100; i++) {
String str = null;
str = +i + ":- HOW TO WRITE A CONSOLE OUTPUT IN A TEXT FILE";
ls.add(str);
}
String listString = "";
for (String s : ls) {
listString += s + "\n";
}
FileWriter writer = null;
try {
writer = new FileWriter("final.txt");
writer.write(listString);
writer.close();
} catch (IOException e) {
e.printStackTrace();
}
}
If you want to generate the PDF rather then the text file, you use the dependency given below:
<dependency>
<groupId>com.itextpdf</groupId>
<artifactId>itextpdf</artifactId>
<version>5.0.6</version>
</dependency>
To generate a PDF, use this code:
public static void main(String[] args) {
int i;
List<String> ls = new ArrayList<String>();
for (i = 1; i <= 100; i++) {
String str = null;
str = +i + ":- HOW TO WRITE A CONSOLE OUTPUT IN A PDF";
ls.add(str);
}
String listString = "";
for (String s : ls) {
listString += s + "\n";
}
Document document = new Document();
try {
PdfWriter writer1 = PdfWriter
.getInstance(
document,
new FileOutputStream(
"final_pdf.pdf"));
document.open();
document.add(new Paragraph(listString));
document.close();
writer1.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (DocumentException e) {
e.printStackTrace();
}
}

PrintWriter out = null;
try {
out = new PrintWriter(new FileWriter("C:\\testing.txt"));
} catch (IOException e) {
e.printStackTrace();
}
out.println("output");
out.close();
I am using absolute path for the FileWriter. It is working for me like a charm. Also Make sure the file is present in the location. Else It will throw a FileNotFoundException. This method does not create a new file in the target location if the file is not found.

In netbeans, you can right click the mouse and then save as a .txt file. Then, based on the created .txt file, you can convert to the file in any format you want to get.