We Keep Coding

Java - Prevent PrintWriter to overwrite the entire file everytime

I have a simulation program that needs to write certain results to a csv file very frequently during execution. I have found that there is something wrong with the printwriter which dramatically slows down running my program, as the output file is getting larger in size (near to 1 million rows). I doublt it's overwriting the entire file each time from the beginning, wheras I just need to append a single line at the bottom each time when it's being called. below is the code related to the writing fuctions.
one of the writing fuctions:
public void printHubSummary(Hub hub, String filePath) {
try {
StringBuilder sb = new StringBuilder();
String h = hub.getHub_code();
String date = Integer.toString(hub.getGs().getDate());
String time = hub.getGs().getHHMMFromMinute(hub.getGs().getClock());
String wgt = Double.toString(hub.getIb_wgt());
sb.append(h+","+date+","+time+","+wgt);
// System.out.println("truck print line: " + sb);
FileWriter.writeFile(sb.toString(),filePath);
}
catch (Exception e) {
System.out.println("Something wrong when outputing truck summary file!");
e.printStackTrace();
}
}
the file writer code: (should be where the problem is!)
public static boolean writeFile(String newStr, String filename) throws IOException {
boolean flag = false;
String filein = newStr + "\r\n";
String temp = "";
FileInputStream fis = null;
InputStreamReader isr = null;
BufferedReader br = null;
FileOutputStream fos = null;
PrintWriter pw = null;
try {
File file = new File(filename);
fis = new FileInputStream(file);
isr = new InputStreamReader(fis);
br = new BufferedReader(isr);
StringBuffer buf = new StringBuffer();
for (int j = 1; (temp = br.readLine()) != null; j++) {
buf = buf.append(temp);
buf = buf.append(System.getProperty("line.separator"));
}
if (buf.length() > 0 && buf.charAt(0) == '\uFEFF') {
buf.deleteCharAt(0);
}
buf.append(filein);
fos = new FileOutputStream(file);
byte[] unicode = {(byte)0xEF, (byte)0xBB, (byte)0xBF};
fos.write(unicode);
pw = new PrintWriter(fos);
pw.write(buf.toString().toCharArray());
pw.flush();
flag = true;
} catch (IOException e1) {
throw e1;
} finally {
if (pw != null) {
pw.close();
}
if (fos != null) {
fos.close();
}
if (br != null) {
br.close();
}
if (isr != null) {
isr.close();
}
if (fis != null) {
fis.close();
}
}
return flag;
}
An update on code modification. I have freezed the operations of repeatitively overwrting the entire file. It appears to solve the problem, but writing for sometime it's slowed down as well. Is it the best arrangement for wrting very large file? what other modifications can be done to make it even more efficient?
public static boolean writeFile1(String newStr, String filename) throws IOException {
boolean flag = false;
String filein = newStr + "\r\n";
String temp = "";
FileInputStream fis = null;
InputStreamReader isr = null;
BufferedReader br = null;
FileOutputStream fos = null;
PrintWriter pw = null;
try {
File file = new File(filename);
fis = new FileInputStream(file);
isr = new InputStreamReader(fis);
br = new BufferedReader(isr);
StringBuffer buf = new StringBuffer();
// for (int j = 1; (temp = br.readLine()) != null; j++) {
// buf = buf.append(temp);
// buf = buf.append(System.getProperty("line.separator"));
// }
// if (buf.length() > 0 && buf.charAt(0) == '\uFEFF') {
// buf.deleteCharAt(0);
// }
buf.append(filein);
fos = new FileOutputStream(file,true);
byte[] unicode = {(byte)0xEF, (byte)0xBB, (byte)0xBF};
fos.write(unicode);
pw = new PrintWriter(fos);
pw.write(buf.toString().toCharArray());
pw.flush();
flag = true;
} catch (IOException e1) {
throw e1;
} finally {
if (pw != null) {
pw.close();
}
if (fos != null) {
fos.close();
}
if (br != null) {
br.close();
}
if (isr != null) {
isr.close();
}
if (fis != null) {
fis.close();
}
}
return flag;
}

Provide a second argument to the FileOutputStream constructor to specify whether or not to use append mode, which will add to the end of the file rather than overwriting it.
fos = new FileOutputStream(file, true);
Alternatively, you could create a single static PrintWriter in append mode, which will probably be faster as it reduces garbage collection.

Use the Files / Path / Java NIO2 which is richer: the code below would need Java 7 at least.
Path path = Paths.get(filename);
try (BufferedWriter bw = Files.newBufferedWriter(
path, StandardCharsets.UTF_8, StandardOpenOption.APPEND, StandardOpenOption.CREATE, StandardOpenOption.WRITE)) {
bw.append(filein);
bw.newLine();
}
Your cue here is the StandardOpenOption.
You will probably have to do some additional code before to write the Unicode part (and fix the StandardCharsets.UTF_8):
if (Files.notExists(path)) {
Files.write(path, new byte[] {(byte)0xEF, (byte)0xBB, (byte)0xBF});
}
Also, try to not use StringBuffer in a local method, use StringBuilder: you don't need synchronisation most of the time.

how to copy only a part of .CSV based on first column elements with java

copy part like this(from date to date) I am trying to copy only a part of .CSV file based on the first column (Start Date and Time) data looks like (2019-01-28 10:22:00 AM) but the user have to put it like this (2019/01/28 10:22:00)
this is for windows, java opencsv , this is what I found but dont do what I need exaclty :
like this:
int startLine = get value1 from column csv ;
int endLine = get value2 from column csv;
public static void showLines(String fileName, int startLine, int endLine) throws IOException {
String line = null;
int currentLineNo = 1;
// int startLine = 20056;//40930;
// int currentLineNo = 0;
File currentDirectory = new File(new File(".").getAbsolutePath());
String fromPath = currentDirectory.getCanonicalPath() + "\\Target\\part.csv";
PrintWriter pw = null;
pw = new PrintWriter(new FileOutputStream(fromPath), true);
//pw.close();
BufferedReader in = null;
try {
in = new BufferedReader (new FileReader(fileName));
//read to startLine
while(currentLineNo<startLine) {
if (in.readLine()==null) {
// oops, early end of file
throw new IOException("File too small");
}
currentLineNo++;
}
//read until endLine
while(currentLineNo<=endLine) {
line = in.readLine();
if (line==null) {
// here, we'll forgive a short file
// note finally still cleans up
return;
}
System.out.println(line);
currentLineNo++;
pw.println(line);
}
} catch (IOException ex) {
System.out.println("Problem reading file.\n" + ex.getMessage());
}finally {
try { if (in!=null) in.close();
pw.close();
} catch(IOException ignore) {}
}
}
public static void main(String[] args) throws FileNotFoundException {
int startLine = 17 ;
int endLine = 2222;
File currentDirectory = new File(new File(".").getAbsolutePath());
try {
showLines(currentDirectory.getCanonicalPath() + "\\Sources\\concat.csv", startLine, endLine);
} catch (IOException e) {
e.printStackTrace();
}
// pw.println();
}

Common CSV format uses a comma as a delimiter, with quotations used to escape any column entry that uses them within the data. Assuming that your column one data is consistent with the format you posted, and that I wouldn't have to bother with quotations marks therefor, you could read the columns as:
public static void main(String[] args) {
//This is the path to the file you are writing to
String targetPath = "";
//This is the path to the file you are reading from
String inputFilePath = "";
String line = null;
ArrayList<String> lines = new ArrayList<String>();
boolean add = false;
String startLine = "2019/01/28 10:22:00";
String endLine = "2019/01/28 10:30:00";
String addFlagSplit[] = startLine.replace("/", "-").split(" ");
String addFlag = addFlagSplit[0] + " " + addFlagSplit[1];
String endFlagSplit[] = endLine.replace("/", "-").split(" ");
String endFlag = endFlagSplit[0] + " " + endFlagSplit[1];
try(PrintWriter pw = new PrintWriter(new FileOutputStream(targetPath), true)){
try (BufferedReader input = new BufferedReader(new FileReader(inputFilePath))){
while((line = input.readLine()) != null) {
String date = line.split(",")[0];
if(date.contains(addFlag)) {
add = true;
}else if(date.contains(endFlag)) {
break;
}
if(add) {
lines.add(line);
}
}
}
for(String currentLine : lines) {
pw.append(currentLine + "\n");
}
}catch(FileNotFoundException e) {
e.printStackTrace();
}catch(IOException e) {
e.printStackTrace();
}catch(Exception e) {
e.printStackTrace();
}
}

public static void main(String[] args) throws IOException {
File currentDirectory = new File(new File(".").getAbsolutePath());
String targetPath = currentDirectory.getCanonicalPath() + "\\Target\\part.csv";
String inputFilePath = currentDirectory.getCanonicalPath() + "\\Sources\\concat.csv";
String line = null;
ArrayList<String> lines = new ArrayList<String>();
boolean add = false;
String startLine = "2019/01/28 10:22:00";
String endLine = "2019/04/06 10:30:00";
try(PrintWriter pw = new PrintWriter(new FileOutputStream(targetPath), true)){
try (BufferedReader input = new BufferedReader(new FileReader(inputFilePath))){
while((line = input.readLine()) != null) {
String date = line.split(",")[0];
if(date.contains(startLine)) {
add = true;
}else if(date.contains(endLine)) {
break;
}
if(add) {
lines.add(line);
}
}
}
for(String currentLine : lines) {
pw.append(currentLine + "\n");
}
}catch(FileNotFoundException e) {
e.printStackTrace();
}catch(IOException e) {
e.printStackTrace();
}catch(Exception e) {
e.printStackTrace();
}
}

How to remove blank lines from a text file in java

I am trying to duplicate the original into a new file. In the new file I want the exact same things as the original BUT no blank lines.
Note: I looked at other posts and tried with no success.
Currently:
1
2
3
How I want it to be: -- no blank lines
1
2
3
Here is my code so far:
inputFileName = "x.txt";
outputFileName = "y.txt";
inputFile = new BufferedReader(new FileReader(inputFileName));
outputFile = new PrintWriter(new FileWriter(outputFileName));
String lineOfText = inputFile.readLine();
while(lineOfText != null)
{
if (lineOfText.isEmpty())
{
outputFile.print("null");
}
outputFile.println(lineOfText);
lineOfText = inputFile.readLine();
}
inputFile.close();
outputFile.close();
}
Thank you for all who can possibly help. I assumed that print("null") would print out 'nothing' but it indeed prints out null, I do not know how to print out 'nothing'.

You need to skip the println in case the line is empty:
while(lineOfText != null)
{
if (!lineOfText.isEmpty()) {
outputFile.println(lineOfText);
}
lineOfText = inputFile.readLine();
}

You're on the right track, but this
while(lineOfText != null)
{
if (lineOfText.isEmpty())
{
outputFile.print("null");
}
outputFile.println(lineOfText);
lineOfText = inputFile.readLine();
}
shouldn't be writing null on empty lines. I think you wanted something like
while(lineOfText != null)
{
if (!lineOfText.isEmpty())
{
outputFile.println(lineOfText);
}
lineOfText = inputFile.readLine();
}
Also, I suggest you use a try-with-resources Statement instead of manually managing your close(s). It's probably a good idea to trim (as suggested in the comments) before your test, and you can simplify your loop and you should limit variable visibility. All together like,
String inputFileName = "x.txt";
String outputFileName = "y.txt";
try (BufferedReader inputFile = new BufferedReader(new FileReader(inputFileName));
PrintWriter outputFile = new PrintWriter(new FileWriter(outputFileName))) {
String lineOfText;
while ((lineOfText = inputFile.readLine()) != null) {
lineOfText = lineOfText.trim();
if (!lineOfText.isEmpty()) {
outputFile.println(lineOfText);
}
}
}

public static void main(String[] args) {
Scanner file;
PrintWriter writer;
try {
file = new Scanner(new File("src/data1.txt"));
writer = new PrintWriter("src/data2.txt");
while (file.hasNext()) {
String line = file.nextLine();
if (!line.isEmpty()) {
writer.write(line);
writer.write("\n");
}
}
file.close();
writer.close();
} catch (FileNotFoundException ex) {
Logger.getLogger(Test.class.getName()).log(Level.SEVERE, null, ex);
}
}

splitting of csv file based on column value in java

I want to split csv file into multiple csv files depending on column value.
Structure of csv file: Name,Id,Dept,Course
abc,1,CSE,Btech
fgj,2,EE,Btech
(Rows are not separated by ; at end)
If value of Dept is CSE or ME , write it to file1.csv, if value is ECE or EE write it to file2.csv and so on.
Can I use drools for this purpose? I don't know drools much.
Any help how it can be done?
This is what I have done yet:
public void run() {
String csvFile = "C:/csvFiles/file1.csv";
BufferedReader br = null;
BufferedWriter writer=null,writer2=null;
String line = "";
String cvsSplitBy = ",";
String FileName = "C:/csvFiles/file3.csv";
String FileName2 = "C:/csvFiles/file4.csv";
try {
writer = new BufferedWriter(new FileWriter(FileName));
writer2 = new BufferedWriter(new FileWriter(FileName2));
br = new BufferedReader(new FileReader(csvFile));
while ((line = br.readLine()) != null) {
String[] values=line.split(cvsSplitBy);
if(values[2].equals("CSE"))
{
writer.write(line);
}
else if(values[2].equals("ECE"))
{
writer2.write(line);
}
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
if (br != null) {
try {
br.close();
writer.flush();
writer.close();
writer2.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}

1) First find column index using header row or if header is not present then by index
2) Follow below algorithm which will result map of key value where key is column by which split is performed
global resultMap;
Method add(key,row) {
data = (resultMap.containsKey(key))? resultMap.get(key):new ArrayList<String>();
data.add(row);
resultMap.put(key, data );
}
Method getSplittedMap(List rows) {
for (String currentRow : rows) {
add(key, currentRow);
}
return resultMap;
}
hope this helps.

FileOutputStream f_ECE = new FileOutputStream("provideloaction&filenamehere");
FileOutputStream f_CSE_ME = new FileOutputStream("provideloaction&filenamehere");
FileInputputStream fin = new FileinputStream("provideloaction&filenamehere");
int size = fin.available(); // find the length of file
byte b[] = new byte[size];
fin.read(b);
String s = new String(b); // file copied into string
String s1[] = s.split("\n");
for (int i = 0; i < s1.length; i++) {
String s3[] = s1[i].split(",")
if (s3[2].equals("ECE"))
f_ECE.write(s1.getBytes());
if (s3[2].equals("CSE") || s3.equals("EEE"))
f_CSE_ME.write(payload.getBytes());
}

very long string as a response of web service

I am getting a really long string as the response of the web service I am collecting it in the using the StringBuilder but I am unable to obtain the full value I also used StringBuffer but had no success.
Here is the code I am using:
private static String read(InputStream in ) throws IOException {
//StringBuilder sb = new StringBuilder(1000);
StringBuffer sb = new StringBuffer();
String s = "";
BufferedReader r = new BufferedReader(new InputStreamReader( in ), 1000);
for (String line = r.readLine(); line != null; line = r.readLine()) {
sb.append(line);
s += line;
} in .close();
System.out.println("Response from Input Stream Reader >>>" + sb.toString());
System.out.println("Response from Input S >>>>>>>>>>>>" + s);
return sb.toString();
}
Any help is appreciated.

You can also split the string in array of strings in order to see all of them
String delimiter = "put a delimiter here e.g.: \n";
String[] datas=sb.toString().split(delimiter);
for(String string datas){
System.out.println("Response from Input S >>>>>>>>>>>>" + string);
}

The String may not print entirely to the console, but it is actually there. Save it to a file in order to see it.

I do not think that your input is too big for a String, but only not shown to the console because it doesn't accept too long lines. Anyways, here is the solution for a really huge input as characters:
private static String[] readHugeStream(InputStream in) throws IOException {
LinkedList<String> dataList = new LinkedList<>();
boolean finished = false;
//
BufferedReader r = new BufferedReader(new InputStreamReader(in), 0xFFFFFF);
String line = r.readLine();
while (!finished) {
int lengthRead = 0;
StringBuilder sb = new StringBuilder();
while (!finished) {
line = r.readLine();
if (line == null) {
finished = true;
} else {
lengthRead += line.length();
if (lengthRead == Integer.MAX_VALUE) {
break;
}
sb.append(line);
}
}
if (sb.length() != 0) {
dataList.add(sb.toString());
}
}
in.close();
String[] data = dataList.toArray(new String[]{});
///
return data;
}
public static void main(String[] args) {
try {
String[] data = readHugeStream(new FileInputStream("<big file>"));
} catch (IOException ex) {
Logger.getLogger(StackoverflowStringLong.class.getName()).log(Level.SEVERE, null, ex);
} catch (OutOfMemoryError ex) {
System.out.println("out of memory...");
}
}

System.out.println() does not print all the characters , it can display only limited number of characters in console. You can create a file in SD card and copy the string there as a text document to check your exact response.
try
{
File root = new File(Environment.getExternalStorageDirectory(), "Responsefromserver");
if (!root.exists())
{
root.mkdirs();
}
File gpxfile = new File(root, "response.txt");
FileWriter writer = new FileWriter(gpxfile);
writer.append(totalResponse);
writer.flush();
writer.close();
}
catch(IOException e)
{
System.out.println("Error:::::::::::::"+e.getMessage());
throw e;
}