We Keep Coding

Processing Very Large Files (6Gig or greater) in Java

I was using the following to load file and process as String:
import java.nio.file.Files;
import java.nio.file.Paths;
.
.
.
.
readFile(String inputFile) throws IOException {
String content = "";
content = new
String(Files.readAllBytes(Paths.get(inputFile)));
return content;
}
parseAndOutputToNewFile() {
String string = readFile(inputFile);
dostuff(string);
}
If my files were larger than a couple of Gig, they I would get an output of memory exception within the readFile() and never got to doStuff(). What is the more appropriate way to process larger files? Thank you.

Instead of Files.readAllBytes(Paths.get(inputFile)), you should use Files.lines(Paths.get(inputFile)), and process the lines as they are streamed.
try (Stream<String> stream = Files.lines(Paths.get(inputFile))) {
stream. ... // process streamed lines of text here
}
Or the Java 7 for loop version:
try (BufferedReader in = Files.newBufferedReader(Paths.get(inputFile), StandardCharsets.UTF_8)) {
for (String line; (line = in.readLine(buf)) != null; ) {
// process `line` here
}
}
If you need blocks of text, instead of lines of text, you should use a BufferedReader, e.g. using Files.newBufferedReader(Paths.get(inputFile)), like this:
try (BufferedReader in = Files.newBufferedReader(Paths.get(inputFile))) {
char[] buf = new char[4096];
for (int len; (len = in.read(buf)) > 0; ) {
// process `len` chars from `buf` here
}
}
But, it all depends on what dostuff() needs to do, i.e. whether what it does can be done in a streaming fashion. Without knowing more about it, we can't give you definitive solutions.

You don't have sufficient RAM allocated to the JVM (heap) to read the whole file in a string.
In order to bypass this, you'll need to modify the doStuff method to use a Reader and process the input lazily.
try(Reader reader=new BufferedReader(new FileReader(inputFile)){
doStuff(reader);
}

How to piece together String using file reader and char array

Wrote a file in another class and now I'm trying to piece together the file into a JLabel, so I need to convert the name in the file into a string. Using FileReader and a char array to separate each character into an array to be put together in the JLabel.
I'm getting this error on NamePieces[x] = (char)nr;:
Exception in thread "AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException: 0
at clients.initialize(clients.java:197)
at clients.<init>(clients.java:72)
This is the code that I want to read the file:
try(FileReader nameReader = new FileReader(NamePath)) {
int nr = nameReader.read();
int x = 0;
while(nr != -1) {
namePieces[x] = (char)nr;
nr = nameReader.read();
x++;
}
}
catch (FileNotFoundException e) {}
catch (IOException e1) {}
String name = String.valueOf(namePieces[0]) + namePieces[1];
Doesn't work

Most likely, your problem occurs because namePieces is not initialized. As was already mentioned in the comments, you should not use char[] as a container for your characters (because in real world you won't know the length of the files' contents every time, so you will probably need to resize your container), it is way more better to use StringBuilder, provided by Java standard library. It will protect you from getting out of bounds.
StringBuilder namePieces = new StringBuilder();
File file = new File(filePath);
BufferedReader reader = new BufferedReader(
new InputStreamReader(new FileInputStream(file),
Charset.forName("UTF-8")));
int c;
while((c = reader.read()) != -1) {
namePieces.append((char) c);
}
String nameString = namePieces.toString(); // Use this string as a complete array of needed characters
As you see I changed an approach by using not only StringBuilder, but also BufferedReader. However, for your task you can leave FileReader as it is. Just consider appending characters to builder.

If your file just contains a String there is a straightforward way to read it:
public String readMyFile( String fileName) throws IOException {
Path path = Paths.get(fileName);
return Files.readAllLines(path).get(0);
}

File encoding : saved content is different than when read

I have a slight problem trying to save a file in java.
For some reason the content I get after saving my file is different from what I have when I read it.
I guess this is related to file encoding, but without being sure.
Here is test code I put together. The idea is basically to read a file, and save it again.
When I open both files, they are different.
package workspaceFun;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import org.apache.commons.codec.DecoderException;
public class FileSaveTest {
public static void main(String[] args) throws IOException, DecoderException{
String location = "test.location";
File locationFile = new File(location);
FileInputStream fis = new FileInputStream(locationFile);
InputStreamReader r = new InputStreamReader(fis, Charset.forName("UTF-8"));
System.out.println(r.getEncoding());
StringBuilder builder = new StringBuilder();
int ch;
while((ch = fis.read()) != -1){
builder.append((char)ch);
}
String fullLocationString = builder.toString();
//Now we want to save back
FileOutputStream fos = new FileOutputStream("C:/Users/me/Desktop/test");
byte[] b = fullLocationString.getBytes();
fos.write(b);
fos.close();
r.close();
}
}
An extract from the input file (opened as plain text using Sublime 2):
40b1 8b81 23bc 0014 1a25 96e7 a393 be1e
and from the output file :
40c2 b1c2 8bc2 8123 c2bc 0014 1a25 c296
The getEncoding method returns "UTF8". Trying to save the output file using the same charset doest not seem to solve the issue.
What puzzles me is that when I try to read the input file using Hex from apache.commons.codec like this :
String hexLocationString2 = Hex.encodeHexString(fullLocationString.getBytes("UTF-8"));
The String already looks like my output file, not the input.
Would you have any idea on what can go wrong?
Thanks
Extra info for those being interested, I am trying to read an eclipse .location file.
EDIT: I placed the file online so that you can test the code

I believe is the way you are reading the stream.
You are using FileInputStream directly to read the content instead of wrapping it in the InputStreamReader
By using the InputStreamReader you may determine which Charset to use.
Take in consideration that the Charset defined in the InputStream must be the same you expect as InputStream doesn't detect charsets, it just reads them in that specific format.
Try the following changes:
InputStreamReader r = new InputStreamReader(new FileInputStream(locationFile), StandardCharsets.UTF_8);
then instead of fos.read() use r.read()
Finally when writing the String get the bytes in the same Charset as your Reader
FileOutputStream fos = new FileOutputStream("C:/Users/me/Desktop/test");
fos.write(fullLocationString.getBytes(StandardCharsets.UTF_8));
fos.close()

Try to read and write back as below:
public class FileSaveTest {
public static void main(String[] args) throws IOException {
String location = "D:\\test.txt";
BufferedReader br = new BufferedReader(new FileReader(location));
StringBuilder sb = new StringBuilder();
try {
String line = br.readLine();
while (line != null) {
sb.append(line);
line = br.readLine();
if (line != null)
sb.append(System.lineSeparator());
}
} finally {
br.close();
}
FileOutputStream fos = new FileOutputStream("D:\\text_created.txt");
byte[] b = sb.toString().getBytes();
fos.write(b);
fos.close();
}
}
Test file contains both Cirillic and Latin characters.
SDFASDF
XXFsd1
12312
іва

Java: How to read a text file

I want to read a text file containing space separated values. Values are integers.
How can I read it and put it in an array list?
Here is an example of contents of the text file:
1 62 4 55 5 6 77
I want to have it in an arraylist as [1, 62, 4, 55, 5, 6, 77]. How can I do it in Java?

You can use Files#readAllLines() to get all lines of a text file into a List<String>.
for (String line : Files.readAllLines(Paths.get("/path/to/file.txt"))) {
// ...
}
Tutorial: Basic I/O > File I/O > Reading, Writing and Creating text files
You can use String#split() to split a String in parts based on a regular expression.
for (String part : line.split("\\s+")) {
// ...
}
Tutorial: Numbers and Strings > Strings > Manipulating Characters in a String
You can use Integer#valueOf() to convert a String into an Integer.
Integer i = Integer.valueOf(part);
Tutorial: Numbers and Strings > Strings > Converting between Numbers and Strings
You can use List#add() to add an element to a List.
numbers.add(i);
Tutorial: Interfaces > The List Interface
So, in a nutshell (assuming that the file doesn't have empty lines nor trailing/leading whitespace).
List<Integer> numbers = new ArrayList<>();
for (String line : Files.readAllLines(Paths.get("/path/to/file.txt"))) {
for (String part : line.split("\\s+")) {
Integer i = Integer.valueOf(part);
numbers.add(i);
}
}
If you happen to be at Java 8 already, then you can even use Stream API for this, starting with Files#lines().
List<Integer> numbers = Files.lines(Paths.get("/path/to/test.txt"))
.map(line -> line.split("\\s+")).flatMap(Arrays::stream)
.map(Integer::valueOf)
.collect(Collectors.toList());
Tutorial: Processing data with Java 8 streams

Java 1.5 introduced the Scanner class for handling input from file and streams.
It is used for getting integers from a file and would look something like this:
List<Integer> integers = new ArrayList<Integer>();
Scanner fileScanner = new Scanner(new File("c:\\file.txt"));
while (fileScanner.hasNextInt()){
integers.add(fileScanner.nextInt());
}
Check the API though. There are many more options for dealing with different types of input sources, differing delimiters, and differing data types.

This example code shows you how to read file in Java.
import java.io.*;
/**
* This example code shows you how to read file in Java
*
* IN MY CASE RAILWAY IS MY TEXT FILE WHICH I WANT TO DISPLAY YOU CHANGE WITH YOUR OWN
*/
public class ReadFileExample
{
public static void main(String[] args)
{
System.out.println("Reading File from Java code");
//Name of the file
String fileName="RAILWAY.txt";
try{
//Create object of FileReader
FileReader inputFile = new FileReader(fileName);
//Instantiate the BufferedReader Class
BufferedReader bufferReader = new BufferedReader(inputFile);
//Variable to hold the one line data
String line;
// Read file line by line and print on the console
while ((line = bufferReader.readLine()) != null) {
System.out.println(line);
}
//Close the buffer reader
bufferReader.close();
}catch(Exception e){
System.out.println("Error while reading file line by line:" + e.getMessage());
}
}
}

Look at this example, and try to do your own:
import java.io.*;
public class ReadFile {
public static void main(String[] args){
String string = "";
String file = "textFile.txt";
// Reading
try{
InputStream ips = new FileInputStream(file);
InputStreamReader ipsr = new InputStreamReader(ips);
BufferedReader br = new BufferedReader(ipsr);
String line;
while ((line = br.readLine()) != null){
System.out.println(line);
string += line + "\n";
}
br.close();
}
catch (Exception e){
System.out.println(e.toString());
}
// Writing
try {
FileWriter fw = new FileWriter (file);
BufferedWriter bw = new BufferedWriter (fw);
PrintWriter fileOut = new PrintWriter (bw);
fileOut.println (string+"\n test of read and write !!");
fileOut.close();
System.out.println("the file " + file + " is created!");
}
catch (Exception e){
System.out.println(e.toString());
}
}
}

Just for fun, here's what I'd probably do in a real project, where I'm already using all my favourite libraries (in this case Guava, formerly known as Google Collections).
String text = Files.toString(new File("textfile.txt"), Charsets.UTF_8);
List<Integer> list = Lists.newArrayList();
for (String s : text.split("\\s")) {
list.add(Integer.valueOf(s));
}
Benefit: Not much own code to maintain (contrast with e.g. this). Edit: Although it is worth noting that in this case tschaible's Scanner solution doesn't have any more code!
Drawback: you obviously may not want to add new library dependencies just for this. (Then again, you'd be silly not to make use of Guava in your projects. ;-)

Use Apache Commons (IO and Lang) for simple/common things like this.
Imports:
import org.apache.commons.io.FileUtils;
import org.apache.commons.lang3.ArrayUtils;
Code:
String contents = FileUtils.readFileToString(new File("path/to/your/file.txt"));
String[] array = ArrayUtils.toArray(contents.split(" "));
Done.

Using Java 7 to read files with NIO.2
Import these packages:
import java.nio.charset.Charset;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
This is the process to read a file:
Path file = Paths.get("C:\\Java\\file.txt");
if(Files.exists(file) && Files.isReadable(file)) {
try {
// File reader
BufferedReader reader = Files.newBufferedReader(file, Charset.defaultCharset());
String line;
// read each line
while((line = reader.readLine()) != null) {
System.out.println(line);
// tokenize each number
StringTokenizer tokenizer = new StringTokenizer(line, " ");
while (tokenizer.hasMoreElements()) {
// parse each integer in file
int element = Integer.parseInt(tokenizer.nextToken());
}
}
reader.close();
} catch (Exception e) {
e.printStackTrace();
}
}
To read all lines of a file at once:
Path file = Paths.get("C:\\Java\\file.txt");
List<String> lines = Files.readAllLines(file, StandardCharsets.UTF_8);

All the answers so far given involve reading the file line by line, taking the line in as a String, and then processing the String.
There is no question that this is the easiest approach to understand, and if the file is fairly short (say, tens of thousands of lines), it'll also be acceptable in terms of efficiency. But if the file is long, it's a very inefficient way to do it, for two reasons:
Every character gets processed twice, once in constructing the String, and once in processing it.
The garbage collector will not be your friend if there are lots of lines in the file. You're constructing a new String for each line, and then throwing it away when you move to the next line. The garbage collector will eventually have to dispose of all these String objects that you don't want any more. Someone's got to clean up after you.
If you care about speed, you are much better off reading a block of data and then processing it byte by byte rather than line by line. Every time you come to the end of a number, you add it to the List you're building.
It will come out something like this:
private List<Integer> readIntegers(File file) throws IOException {
List<Integer> result = new ArrayList<>();
RandomAccessFile raf = new RandomAccessFile(file, "r");
byte buf[] = new byte[16 * 1024];
final FileChannel ch = raf.getChannel();
int fileLength = (int) ch.size();
final MappedByteBuffer mb = ch.map(FileChannel.MapMode.READ_ONLY, 0,
fileLength);
int acc = 0;
while (mb.hasRemaining()) {
int len = Math.min(mb.remaining(), buf.length);
mb.get(buf, 0, len);
for (int i = 0; i < len; i++)
if ((buf[i] >= 48) && (buf[i] <= 57))
acc = acc * 10 + buf[i] - 48;
else {
result.add(acc);
acc = 0;
}
}
ch.close();
raf.close();
return result;
}
The code above assumes that this is ASCII (though it could be easily tweaked for other encodings), and that anything that isn't a digit (in particular, a space or a newline) represents a boundary between digits. It also assumes that the file ends with a non-digit (in practice, that the last line ends with a newline), though, again, it could be tweaked to deal with the case where it doesn't.
It's much, much faster than any of the String-based approaches also given as answers to this question. There is a detailed investigation of a very similar issue in this question. You'll see there that there's the possibility of improving it still further if you want to go down the multi-threaded line.

read the file and then do whatever you want
java8
Files.lines(Paths.get("c://lines.txt")).collect(Collectors.toList());

Capture data read from file into string stream Java

I'm coming from a C++ background, so be kind on my n00bish queries...
I'd like to read data from an input file and store it in a stringstream. I can accomplish this in an easy way in C++ using stringstreams. I'm a bit lost trying to do the same in Java.
Following is a crude code/way I've developed where I'm storing the data read line-by-line in a string array. I need to use a string stream to capture my data into (rather than use a string array).. Any help?
char dataCharArray[] = new char[2];
int marker=0;
String inputLine;
String temp_to_write_data[] = new String[100];
// Now, read from output_x into stringstream
FileInputStream fstream = new FileInputStream("output_" + dataCharArray[0]);
// Convert our input stream to a BufferedReader
BufferedReader in = new BufferedReader (new InputStreamReader(fstream));
// Continue to read lines while there are still some left to read
while ((inputLine = in.readLine()) != null )
{
// Print file line to screen
// System.out.println (inputLine);
temp_to_write_data[marker] = inputLine;
marker++;
}
EDIT:
I think what I really wanted was a StringBuffer.
I need to read data from a file (into a StringBuffer, probably) and write/transfer all the data back to another file.

In Java, first preference should always be given to buying code from the library houses:
http://commons.apache.org/io/api-1.4/org/apache/commons/io/IOUtils.html
http://commons.apache.org/io/api-1.4/org/apache/commons/io/FileUtils.html
In short, what you need is this:
FileUtils.readFileToString(File file)

StringBuffer is one answer, but if you're just writing it to another file, then you can just open an OutputStream and write it directly out to the other file. Holding a whole file in memory is probably not a good idea.

In you simply want to read a file and write another one:
BufferedInputStream in = new BufferedInputStream( new FileInputStream( "in.txt" ) );
BufferedOutputStream out = new BufferedOutputStream( new FileOutputStream( "out.txt" ) );
int b;
while ( (b = in.read()) != -1 ) {
out.write( b );
}
If you want to read a file into a string:
StringWriter out = new StringWriter();
BufferedReader in = new BufferedReader( new FileReader( "in.txt" ) );
int c;
while ( (c = in.read()) != -1 ) {
out.write( c );
}
StringBuffer buf = out.getBuffer();
This can be made more efficient if you read using byte arrays. But I recommend that you use the excellent apache common-io. IOUtils (http://commons.apache.org/io/api-1.4/org/apache/commons/io/IOUtils.html) will do the loop for you.
Also, you should remember to close the streams.

I also come from C++, and I was looking for a class similar to the C++ 'StringStreamReader', but I couldn't find it. In my case (which I think was very simple), I was trying to read a file line by line and then read a String and an Integer from each of these lines. My final solution was to use two objects of the class java.util.Scanner, so that I could use one of them to read the lines of the file directly to a String and use the second one to re-read the content of each line (now in the String) to the variables (a new String and a positive 'int'). Here's my code:
try {
//"path" is a String containing the path of the file we want to read
Scanner sc = new Scanner(new BufferedReader(new FileReader(new File(path))));
while (sc.hasNextLine()) { //while the file isn't over
Scanner scLine = new Scanner(sc.nextLine());
//sc.nextLine() returns the next line of the file into a String
//scLine will now proceed to scan (i.e. analyze) the content of the string
//and identify the string and the positive 'int' (what in C++ would be an 'unsigned int')
String s = scLine.next(); //this returns the string wanted
int x;
if (!scLine.hasNextInt() || (x = scLine.nextInt()) < 0) return false;
//scLine.hasNextInt() analyzes if the following pattern can be interpreted as an int
//scLine.nextInt() reads the int, and then we check if it is positive or not
//AT THIS POINT, WE ALREADY HAVE THE VARIABLES WANTED AND WE CAN DO
//WHATEVER WE WANT WITH THEM
//in my case, I put them into a HashMap called 'hm'
hm.put(s, x);
}
sc.close();
//we finally close the scanner to point out that we won't need it again 'till the next time
} catch (Exception e) {
return false;
}
return true;
Hope that helped.