We Keep Coding

java split large files into smaller files while splitting the multiline record without breaking the record in incomplete state

I have a record split into multiple lines in a file. Only way to identify the end of the record is when new record starts with ABC. Below is the sample. File size could be 5-10 GB and I am looking for a efficient java logic ONLY to split the files(no need of reading every line), but splitting logic should a check to start a new file with new record, which should start with "ABC" in this case.
Added few more details, I am just looking for splitting the file and while splitting the last record should be ended correctly in a file.
Can someone please suggest?
HDR
ABCline1goesonforrecord1 //first record
line2goesonForRecord1
line3goesonForRecord1
line4goesonForRecord1
ABCline2goesOnForRecord2 //second record
line2goesonForRecord2
line3goesonForRecord2
line4goesonForRecord2
line5goesonForRecord2
ABCline2goesOnForRecord3 //third record
line2goesonForRecord3
line3goesonForRecord3
line4goesonForRecord3
TRL

So, this is the code that you need. I tested on a 10Gb file and it takes 64 seconds to split the file
import java.io.BufferedWriter;
import java.io.IOException;
import java.io.UncheckedIOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardOpenOption;
import java.util.concurrent.TimeUnit;
public class FileSplitter {
private final Path filePath;
private BufferedWriter writer;
private int fileCounter = 1;
public static void main(String[] args) throws Exception {
long startTime = System.nanoTime();
new FileSplitter(Path.of("/tmp/bigfile.txt")).split();
System.out.println("Time to split " + TimeUnit.NANOSECONDS.toSeconds(System.nanoTime() - startTime));
}
private static void generateBigFile() throws Exception {
var writer = Files.newBufferedWriter(Path.of("/tmp/bigfile.txt"), StandardOpenOption.CREATE, StandardOpenOption.TRUNCATE_EXISTING);
for (int i = 0; i < 100_000; i++) {
writer.write(String.format("ABCline1goesonforrecord%d\n", i + 1));
for (int j = 0; j < 10_000; j++) {
writer.write(String.format("line%dgoesonForRecord%d\n", j + 2, i + 1));
}
}
writer.flush();
writer.close();
}
public FileSplitter(Path filePath) {
this.filePath = filePath;
}
void split() throws IOException {
try (var stream = Files.lines(filePath, StandardCharsets.UTF_8)) {
stream.forEach(line -> {
if (line.startsWith("ABC")) {
closeWriter();
openWriter();
}
writeLine(line);
});
}
closeWriter();
}
private void writeLine(String line) {
if (writer != null) {
try {
writer.write(line);
writer.write("\n");
} catch (IOException e) {
throw new UncheckedIOException("Failed to write line to file part", e);
}
}
}
private void openWriter() {
if (this.writer == null) {
var filePartName = filePath.getFileName().toString().replace(".", "_part" + fileCounter + ".");
try {
writer = Files.newBufferedWriter(Path.of("/tmp/split", filePartName), StandardOpenOption.CREATE, StandardOpenOption.TRUNCATE_EXISTING);
} catch (IOException e) {
throw new UncheckedIOException("Failed to write line to file", e);
}
fileCounter++;
}
}
private void closeWriter() {
if (writer != null) {
try {
writer.flush();
writer.close();
writer = null;
} catch (IOException e) {
throw new UncheckedIOException("Failed to close writer", e);
}
}
}
}
Btw, the solution with Scanner works too.
Regarding not reading all the lines, I don't see why don't you want this. If you choose not not read all the lines (it is possible) then, first you will overcomplicate the solution and second I'm pretty sure that you will loose from performance because of that logic that you have to incorporate in the splitting.

I didn't test this but something like this should work, you are not reading the whole file in memory just one line at a time so it should not be bad.
public void spiltRecords(String filename) {
/*
HDR
ABCline1goesonforrecord1 //first record
line2goesonForRecord1
line3goesonForRecord1
line4goesonForRecord1
ABCline2goesOnForRecord2 //second record
line2goesonForRecord2
line3goesonForRecord2
line4goesonForRecord2
line5goesonForRecord2
ABCline2goesOnForRecord3 //third record
line2goesonForRecord3
line3goesonForRecord3
line4goesonForRecord3
TRL
*/
try {
Scanner scanFile = new Scanner(new File(filename));
// now you do not want to edit the existing file in case things go wrong. one way is to get list of index
// where a new record starts.
LinkedList<Long> startOfRecordIndexes = new LinkedList<>();
long index = 0;
while (scanFile.hasNext()) {
if (scanFile.nextLine().startsWith("ABC")) {
startOfRecordIndexes.add(index);
}
index++;
}
// Once you have the starting index for all records you can iterate through the list and create new records
scanFile = scanFile.reset();
index = 0;
BufferedWriter writer = null;
while (scanFile.hasNext()) {
if (!startOfRecordIndexes.isEmpty() && index == startOfRecordIndexes.peek()) {
if(writer != null) {
writer.write("TRL");
writer.close();
}
writer = new BufferedWriter(new OutputStreamWriter(
new FileOutputStream("Give unique filename"), StandardCharsets.UTF_8));
writer.write("HDR");
writer.write(scanFile.nextLine());
startOfRecordIndexes.remove();
} else {
writer.write(scanFile.nextLine());
}
}
// Close the last record
if(writer != null) {
writer.write("TRL");
writer.close();
}
} catch (IOException e) {
// deal with exception
}
}

Create multiple files from one text file in java

I have one input.txt file which consist on let suppose 520 lines.
I have to make a code in java which will act like this.
Create first file named file-001.txt from first 200 lines. then create another file-002 from 201-400 lines. then file-003.txt from remaining lines.
I have coded this, it just write first 200 lines. What changes I need to make in order to update its working to above scenario.
public class DataMaker {
public static void main(String args[]) throws IOException{
DataMaker dm=new DataMaker();
String file= "D:\\input.txt";
int roll=1;
String rollnum ="file-00"+roll;
String outputfilename="D:\\output\\"+rollnum+".txt";
String urduwords;
String path;
ArrayList<String> where = new ArrayList<String>();
int temp=0;
try(BufferedReader br = new BufferedReader(new FileReader(file))) {
for(String line; (line = br.readLine()) != null; ) {
++temp;
if(temp<201){ //may be i need some changes here
dm.filewriter(line+" "+temp+")",outputfilename);
}
}
} catch (FileNotFoundException e) {
System.out.println("File not found");
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
void filewriter(String linetoline,String filename) throws IOException{
BufferedWriter fbw =null;
try{
OutputStreamWriter writer = new OutputStreamWriter(
new FileOutputStream(filename, true), "UTF-8");
fbw = new BufferedWriter(writer);
fbw.write(linetoline);
fbw.newLine();
}catch (Exception e) {
System.out.println("Error: " + e.getMessage());
}
finally {
fbw.close();
}
}
}
One way can be use of if else but I cant just use it because my actual file is 6000+ lines.
I want this code to work like I run the code and give me 30+ output files.

You can change the following bit:
if(temp<201){ //may be i need some changes here
dm.filewriter(line+" "+temp+")",outputfilename);
}
to this:
dm.filewriter(line, "D:\\output\\file-00" + ((temp/200)+1) + ".txt");
This will make sure first 200 lines go to first file, next 200 lines go to next file and so on.
Also, you might want to batch 200 lines together and write them in one go rather than creating a writer everytime and write to file.

You may have a method that creates the Writer to the current File, reads up to limit number of lines, closes the Writer to the current File, then returns true if it had enough to read , false if it couldn't read the limit number of lines (i.e, abort next call, don't attempt to read more lines or write next file).
Then you would call this in a loop , passing the Reader, the new file name, and the limit number.
Here is an example :
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.FileReader;
import java.io.IOException;
import java.io.OutputStreamWriter;
public class DataMaker {
public static void main(final String args[]) throws IOException {
DataMaker dm = new DataMaker();
String file = "D:\\input.txt";
int roll = 1;
String rollnum = null;
String outputfilename = null;
boolean shouldContinue = false;
try (BufferedReader br = new BufferedReader(new FileReader(file))) {
do {
rollnum = "file-00" + roll;
outputfilename = "D:\\output\\" + rollnum + ".txt";
shouldContinue = dm.fillFile(outputfilename, br, 200);
roll++;
} while (shouldContinue);
} catch (FileNotFoundException e) {
System.out.println("File not found");
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
private boolean fillFile(final String outputfilename, final BufferedReader reader, final int limit)
throws IOException {
boolean result = false;
String line = null;
BufferedWriter fbw = null;
int temp = 0;
try {
OutputStreamWriter writer = new OutputStreamWriter(
new FileOutputStream(outputfilename, true), "UTF-8");
fbw = new BufferedWriter(writer);
while (temp < limit && ((line = reader.readLine()) != null)) {
temp++;
fbw.write(line);
fbw.newLine();
}
// abort if we didn't manage to read the "limit" number of lines
result = (temp == limit);
} catch (Exception e) {
System.out.println("Error: " + e.getMessage());
} finally {
fbw.close();
}
return result;
}
}

How to read a file, reverse the order, and write reverse order

Like a similar project I made, this project is reading characters from a txt file, reversing the order of the string and rewriting it to another txt file. But it keeps outputting my exception of "Something went wrong". Can anyone help me fix what is going wrong?
import java.io.File;
import java.io.IOException;
import java.io.PrintWriter;
import java.util.Scanner;
public class ReverseFile
{
public static void main(String[] args) throws IOException
{
try{
String source = args[0];
String target = args[1];
File sourceFile=new File(source);
Scanner content=new Scanner(sourceFile);
PrintWriter pwriter =new PrintWriter(target);
while(content.hasNextLine())
{
String s=content.nextLine();
StringBuffer buffer = new StringBuffer(s);
buffer=buffer.reverse();
String rs=buffer.toString();
pwriter.println(rs);
}
content.close();
pwriter.close();
System.out.println("File is copied successful!");
}
catch(Exception e){
System.out.println("Something went wrong");
}
}
}
So here is the information from the stacktrace:
java.lang.ArrayIndexOutOfBoundsException: 0
at ReverseFile.main(ReverseFile.java:36)

i am not so sure about your environment, and how long the text might be. and i am also not so sure why you need a scanner?
anyway, here's my take on the problem, hope this helps you :)
import java.io.File;
import java.io.FileInputStream;
import java.io.InputStreamReader;
import java.io.RandomAccessFile;
import java.io.Reader;
public class Reverse {
public static void main(String[] args) {
FileInputStream fis = null;
RandomAccessFile raf = null;
// by default, let's use utf-8
String characterEncoding = "utf-8";
// but if you pass an optional 3rd parameter, we use that
if(args.length==3) {
characterEncoding = args[2];
}
try{
// input file
File in = new File(args[0]);
fis = new FileInputStream(in);
// a reader, because it respects character encoding etc
Reader r = new InputStreamReader(fis,characterEncoding);
// an outputfile
File out = new File(args[1]);
// and a random access file of the same size as the input, so we can write in reverse order
raf = new RandomAccessFile(out, "rw");
raf.setLength(in.length());
// a buffer for the chars we want to read
char[] buff = new char[1];
// keep track of the current position (we're going backwards, so we start at the end)
long position = in.length();
// Reader.read will return -1 when it reached the end.
while((r.read(buff))>-1) {
// turn the character into bytes according to the character encoding
Character c = buff[0];
String s = c+"";
byte[] bBuff = s.getBytes(characterEncoding);
// go to the proper position in the random access file
position = position-bBuff.length;
raf.seek(position);
// write one or more bytes for the character
raf.write(bBuff);
}
} catch (Exception e) {
e.printStackTrace();
} finally {
// clean up
try {
fis.close();
} catch (Exception e2) {
}
try {
raf.close();
} catch (Exception e2) {
}
}
}
}

You need to specify the filenames(source and target) on command-line, while running the program.
java ReverseFile source.txt target.txt
In your program, you try to read the name of files from command-line as
String source = args[0];
String target = args[1];
So if you do not specify those names there, java tries to access the array args at index 0 and 1 which are empty and you get ArrayIndexOutOfBoundsException.

here is ur error free solution to ur problem,u were using "Scanner" without importing "util"
package.here we go:-----------
import java.io.*;
import java.util.*;
public class ReverseFile
{
public static void main(String[] args) throws IOException
{
try{
File sourceFile=new File(args[0]);
Scanner content=new Scanner(sourceFile);
PrintWriter pwriter =new PrintWriter(args[1]);
while(content.hasNextLine())
{
String s=content.nextLine();
StringBuffer buffer = new StringBuffer(s);
buffer=buffer.reverse();
String rs=buffer.toString();
pwriter.println(rs);
}
content.close();
pwriter.close();
System.out.println("File is copied successful!");
}
catch(Exception e){
System.out.println("Something went wrong");
}
}
}

Just thought of a simple approach.
public class ReadFileReverse {
public int[] readByte(File _file) throws IOException {
FileInputStream source = new FileInputStream(_file);
int currentByte = source.available();
int readCount = 0;
int byteContainer[] = new int[currentByte];
while(readCount < currentByte){
byteContainer[readCount] = source.read();
readCount++;
}
source.close();
return byteContainer;
}
public void printReverse(int[] fileContent){
for(int byt=fileContent.length -1; byt >= 0 ; byt--){
System.out.print((char) fileContent[byt]);
}
}
public static void main(String[] args) throws IOException {
File fileToRead = new File("/README.txt");
ReadFileReverse demo = new ReadFileReverse ();
int[] readBytes = demo.readByte(fileToRead);
demo.printReverse(readBytes);
}
}

Here we are reading a file in string variable, then making a String Builder object to perform reverse operation efficiently, then printing
package com;
import java.io.FileReader;
public class Main {
public static void main(String[] args) {
try {
FileReader fr = new FileReader("D:\\newfile.txt");
String str = "";
int ch;
//reading characters in to string variable
while ((ch = fr.read()) != -1) {
str += Character.toString((char) ch);
}
System.out.println("Original String : " + str);
//converting string variable to String Builder object
StringBuilder sb = new StringBuilder(str);
//reversing the string and printing
System.out.println("Reverse order : " + sb.reverse());
fr.close();
} catch (Exception e) {
System.out.println("error");
}
}
}
Output:

Java reading long text file is very slow

I have a text file (XML created with XStream) which is 63000 lines (3.5 MB) long. I'm trying to read it using Buffered reader:
BufferedReader br = new BufferedReader(new FileReader(file));
try {
String s = "";
String tempString;
int i = 0;
while ((tempString = br.readLine()) != null) {
s = s.concat(tempString);
// s=s+tempString;
i = i + 1;
if (i % 1000 == 0) {
System.out.println(Integer.toString(i));
}
}
br.close();
Here you can see my attempts to measure reading speed. And it's very low. It takes seconds to read 1000 lines after 10000 line. I'm clearly doing something wrong, but can't understand what. Thanks in advance for your help.

#PaulGrime is right. You are copying the string each time the loop reads a line. Once the string gets big (say 10,000 lines big), it is doing a lot of work to do that copying.
Try this:
StringBuilder sb = new StringBuilder();
while (...reading lines..){
....
sb.append(tempString); //should add newline
...
}
s = sb.toString();
Note: read Paul's answer below on why stripping newlines makes this a bad way to read in a file. Also, as mentioned in the question comments, XStream provides a way to read the file and even if it had not, IOUtils.toString(reader) would be a safer way to read a file.

Some immediate improvements you can do:
Use a StringBuilder instead of concat and +. Using + and concat can really affect the performance, specially when used in loops.
Reduce the access to the disk. You can do it by using a large buffer:
BufferedReader br = new BufferedReader(new FileReader("someFile.txt"), SIZE);

You should use a StringBuilder as String concatenation is extremely slow for even small strings.
Further, try using NIO rather than a BufferedReader.
public static void main(String[] args) throws IOException {
final File file = //some file
try (final FileChannel fileChannel = new RandomAccessFile(file, "r").getChannel()) {
final StringBuilder stringBuilder = new StringBuilder();
final ByteBuffer byteBuffer = ByteBuffer.allocate(1024);
final CharsetDecoder charsetDecoder = Charset.forName("UTF-8").newDecoder();
while (fileChannel.read(byteBuffer) > 0) {
byteBuffer.flip();
stringBuilder.append(charsetDecoder.decode(byteBuffer));
byteBuffer.clear();
}
}
}
You can tune the buffer size if it's still too slow - it's heavily system dependent what buffer size works better. For me it makes very little difference if the buffer is 1K or 4K but on other systems I have know that change to increase speed by an order of magnitude.

In addition to what has already been said, depending on your use of the XML, your code is potentially incorrect as it discards line endings. For example, this code:
package temp.stackoverflow.q15849706;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URL;
import com.thoughtworks.xstream.XStream;
public class ReadXmlLines {
public String read1(BufferedReader br) throws IOException {
try {
String s = "";
String tempString;
int i = 0;
while ((tempString = br.readLine()) != null) {
s = s.concat(tempString);
// s=s+tempString;
i = i + 1;
if (i % 1000 == 0) {
System.out.println(Integer.toString(i));
}
}
return s;
} finally {
br.close();
}
}
public static void main(String[] args) throws IOException {
ReadXmlLines r = new ReadXmlLines();
URL url = ReadXmlLines.class.getResource("xml.xml");
String xmlStr = r.read1(new BufferedReader(new InputStreamReader(url
.openStream())));
Object ob = null;
XStream xs = new XStream();
xs.alias("root", Root.class);
// This is incorrectly read/parsed, as the line endings are not
// preserved.
System.out.println("----------1");
System.out.println(xmlStr);
ob = xs.fromXML(xmlStr);
System.out.println(ob);
// This is correctly read/parsed, when passing in the URL directly
ob = xs.fromXML(url);
System.out.println("----------2");
System.out.println(ob);
// This is correctly read/parsed, when passing in the InputStream
// directly
ob = xs.fromXML(url.openStream());
System.out.println("----------3");
System.out.println(ob);
}
public static class Root {
public String script;
public String toString() {
return script;
}
}
}
and this xml.xml file on the classpath (in the same package as the class):
<root>
<script>
<![CDATA[
// taken from http://www.w3schools.com/xml/xml_cdata.asp
function matchwo(a,b)
{
if (a < b && a < 0) then
{
return 1;
}
else
{
return 0;
}
}
]]>
</script>
</root>
produces the following output. The first two lines shows the line endings have been removed, and thus made the Javascript in the CDATA section invalid (as the first JS comment now comments out the whole JS, because the JS lines have been merged).
----------1
<root> <script><![CDATA[// taken from http://www.w3schools.com/xml/xml_cdata.aspfunction matchwo(a,b){if (a < b && a < 0) then { return 1; }else { return 0; }}]]> </script></root>
// taken from http://www.w3schools.com/xml/xml_cdata.aspfunction matchwo(a,b){if (a < b && a < 0) then { return 1; }else { return 0; }}
----------2
// taken from http://www.w3schools.com/xml/xml_cdata.asp
function matchwo(a,b)
{
if (a < b && a < 0) then
{
return 1;
}
else
{
return 0;
}
}
...

how to count the loc without block comments?

I am trying to get the number of lines of code from a java file. But I am having trouble counting them.
First I tried to skip them with ifs, but my idea does not work. Now I am counting the same lines with comments, my Java file has this header. Any ideas, I am stuck in how to count them.
My if is for getting the number of lines with the comments block. I trying to make a subtract.
/*
example
example
*/
int totalLoc = 0;
int difference = 0;
while((line =buff.readLine()) !=null){
if((line.trim().length() !=0 &&(!line.contains("/*") ||!line.contains("*/")) )){
if(line.startsWith("/*")){
difference++;
}else if(linea.startsWith("*/")){
difference++;
}else{
difference++;
}
}
}

If you want to count lines in any file write below method and pass the fileName as input to below method and it will return counts.
public int count(String filename) throws IOException
{
InputStream is = new BufferedInputStream(new FileInputStream(filename));
try
{
byte[] c = new byte[1024];
int count = 0;
int readChars = 0;
boolean empty = true;
while ((readChars = is.read(c)) != -1)
{
empty = false;
for (int i = 0; i < readChars; ++i)
{
if (c[i] == '\n')
++count;
}
}
return (count == 0 && !empty) ? 1 : count;
}
finally
{
is.close();
}
}

Got the solution try below code it will print all multiline comments as well as total lines of multiline comments found in a file.
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.charset.Charset;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public class LinesOfCode {
public static void main(String[] args) {
try {
String s = readFile("D:\\src\\SampleClass.java");
Pattern p = Pattern.compile("/\\*[\\s\\S]*?\\*/");
Matcher m = p.matcher(s);
int total = 0;
while (m.find()) {
String lines[] = m.group(0).split("\n");
for (String string : lines) {
System.out.println(string);
total++;
}
}
System.out.println("Total line for comments = " + total);
} catch (Exception e) {
e.printStackTrace();
}
}
private static String readFile(String path) throws IOException {
FileInputStream stream = new FileInputStream(new File(path));
try {
FileChannel fc = stream.getChannel();
MappedByteBuffer bb = fc.map(FileChannel.MapMode.READ_ONLY, 0,
fc.size());
/* Instead of using default, pass in a decoder. */
return Charset.defaultCharset().decode(bb).toString();
} finally {
stream.close();
}
}
}

http://ostermiller.org/findcomment.html
check out this link it will help you more.
and by using this expression => (/*([^]|[\r\n]|(*+([^/]|[\r\n])))*+/)|(//.)
you can count both comments single line and multi line !!