We Keep Coding

Java - Prevent PrintWriter to overwrite the entire file everytime

I have a simulation program that needs to write certain results to a csv file very frequently during execution. I have found that there is something wrong with the printwriter which dramatically slows down running my program, as the output file is getting larger in size (near to 1 million rows). I doublt it's overwriting the entire file each time from the beginning, wheras I just need to append a single line at the bottom each time when it's being called. below is the code related to the writing fuctions.
one of the writing fuctions:
public void printHubSummary(Hub hub, String filePath) {
try {
StringBuilder sb = new StringBuilder();
String h = hub.getHub_code();
String date = Integer.toString(hub.getGs().getDate());
String time = hub.getGs().getHHMMFromMinute(hub.getGs().getClock());
String wgt = Double.toString(hub.getIb_wgt());
sb.append(h+","+date+","+time+","+wgt);
// System.out.println("truck print line: " + sb);
FileWriter.writeFile(sb.toString(),filePath);
}
catch (Exception e) {
System.out.println("Something wrong when outputing truck summary file!");
e.printStackTrace();
}
}
the file writer code: (should be where the problem is!)
public static boolean writeFile(String newStr, String filename) throws IOException {
boolean flag = false;
String filein = newStr + "\r\n";
String temp = "";
FileInputStream fis = null;
InputStreamReader isr = null;
BufferedReader br = null;
FileOutputStream fos = null;
PrintWriter pw = null;
try {
File file = new File(filename);
fis = new FileInputStream(file);
isr = new InputStreamReader(fis);
br = new BufferedReader(isr);
StringBuffer buf = new StringBuffer();
for (int j = 1; (temp = br.readLine()) != null; j++) {
buf = buf.append(temp);
buf = buf.append(System.getProperty("line.separator"));
}
if (buf.length() > 0 && buf.charAt(0) == '\uFEFF') {
buf.deleteCharAt(0);
}
buf.append(filein);
fos = new FileOutputStream(file);
byte[] unicode = {(byte)0xEF, (byte)0xBB, (byte)0xBF};
fos.write(unicode);
pw = new PrintWriter(fos);
pw.write(buf.toString().toCharArray());
pw.flush();
flag = true;
} catch (IOException e1) {
throw e1;
} finally {
if (pw != null) {
pw.close();
}
if (fos != null) {
fos.close();
}
if (br != null) {
br.close();
}
if (isr != null) {
isr.close();
}
if (fis != null) {
fis.close();
}
}
return flag;
}
An update on code modification. I have freezed the operations of repeatitively overwrting the entire file. It appears to solve the problem, but writing for sometime it's slowed down as well. Is it the best arrangement for wrting very large file? what other modifications can be done to make it even more efficient?
public static boolean writeFile1(String newStr, String filename) throws IOException {
boolean flag = false;
String filein = newStr + "\r\n";
String temp = "";
FileInputStream fis = null;
InputStreamReader isr = null;
BufferedReader br = null;
FileOutputStream fos = null;
PrintWriter pw = null;
try {
File file = new File(filename);
fis = new FileInputStream(file);
isr = new InputStreamReader(fis);
br = new BufferedReader(isr);
StringBuffer buf = new StringBuffer();
// for (int j = 1; (temp = br.readLine()) != null; j++) {
// buf = buf.append(temp);
// buf = buf.append(System.getProperty("line.separator"));
// }
// if (buf.length() > 0 && buf.charAt(0) == '\uFEFF') {
// buf.deleteCharAt(0);
// }
buf.append(filein);
fos = new FileOutputStream(file,true);
byte[] unicode = {(byte)0xEF, (byte)0xBB, (byte)0xBF};
fos.write(unicode);
pw = new PrintWriter(fos);
pw.write(buf.toString().toCharArray());
pw.flush();
flag = true;
} catch (IOException e1) {
throw e1;
} finally {
if (pw != null) {
pw.close();
}
if (fos != null) {
fos.close();
}
if (br != null) {
br.close();
}
if (isr != null) {
isr.close();
}
if (fis != null) {
fis.close();
}
}
return flag;
}

Provide a second argument to the FileOutputStream constructor to specify whether or not to use append mode, which will add to the end of the file rather than overwriting it.
fos = new FileOutputStream(file, true);
Alternatively, you could create a single static PrintWriter in append mode, which will probably be faster as it reduces garbage collection.

Use the Files / Path / Java NIO2 which is richer: the code below would need Java 7 at least.
Path path = Paths.get(filename);
try (BufferedWriter bw = Files.newBufferedWriter(
path, StandardCharsets.UTF_8, StandardOpenOption.APPEND, StandardOpenOption.CREATE, StandardOpenOption.WRITE)) {
bw.append(filein);
bw.newLine();
}
Your cue here is the StandardOpenOption.
You will probably have to do some additional code before to write the Unicode part (and fix the StandardCharsets.UTF_8):
if (Files.notExists(path)) {
Files.write(path, new byte[] {(byte)0xEF, (byte)0xBB, (byte)0xBF});
}
Also, try to not use StringBuffer in a local method, use StringBuilder: you don't need synchronisation most of the time.

Read complete file without looping in android

i need a solution for reading a text file which was stored in internal storage.
i don't want to read it line by line. without use looping how to read a complete text file and store it into a string.
BufferedReader br;
String line;
String data = "";
// String text="";
try {
br = new BufferedReader(new FileReader(new File(Environment.getExternalStorageDirectory(), "queue_mgr.txt")));
while ((line = br.readLine()) != null) {
sb.append(line);
}
br.close();
}

You can use a large byte buffer and gain some efficiency:
try
{
InputStream in = new FileInputStream (from);
OutputStream out = new FileOutputStream (to);
// Transfer bytes from in to out
byte[] buf = new byte[1024 * 10]; // 5MB would be about 500 iterations
int len;
while ((len = in.read (buf)) > 0)
out.write (buf, 0, len);
in.close ();
out.close ();
}
}
catch (FileNotFoundException e)
{
...
}
catch (IOException e)
{
...
}

How do I display UTF-8 characters in google app engine's logs?

I'm currently using Java and I print my string using System.out.println(myString);
However, when I look at my server logs on the google app engine dashboard, I see a bunch of question marks (???) in place of where special characters (in my particular case, emoticons) would be.
The string is obtained directly from the payload of the request.
The payload of the request is being read as:
StringBuilder stringBuilder = new StringBuilder();
BufferedReader bufferedReader = null;
try {
InputStream inputStream = request.getInputStream();
if (inputStream != null) {
bufferedReader = new BufferedReader(new InputStreamReader(inputStream, "UTF-8"));
char[] charBuffer = new char[128];
int bytesRead = -1;
while ((bytesRead = bufferedReader.read(charBuffer)) > 0) {
stringBuilder.append(charBuffer, 0, bytesRead);
}
} else {
stringBuilder.append("");
}
} catch (IOException e) {
e.printStackTrace();
} finally {
if (bufferedReader != null) {
try {
bufferedReader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
String body = stringBuilder.toString();

Create text file, tar the file and put it into an inputstream?

I want to put a textfile containing just one string into a tar archive and put this tar file into an inputstream... I want to do this "on the fly", without saving any temporary files to the disk...
I tried out this snippet, where I end up with a file:
FileOutputStream fos = null;
try {
fos = new FileOutputStream ("helloworld.txt");
} catch (FileNotFoundException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
ByteArrayOutputStream baos = new ByteArrayOutputStream();
String config = "HelloWorld!";
byte[] b = config.getBytes();
try {
baos.write(b);
baos.writeTo(fos);
fos.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
And I had a look at the jtar library:
https://code.google.com/p/jtar/
So I would like to put that textfile directly into that tar, but I also don't need that new FileOutputStream( "c:/test/test.tar" ), instead I want to put it into an inputstream directly.
I'm a bit confused with all those inputstreams/outputstreams and I'm not sure if it's even possible to move around a bunch of those streams instead of creating temporary files to do this.
THX & BR

Just get rid of the FileOutputStream and the bos.writeTo(), and do:
ByteArrayInputStream bais = new ByteArrayInputStream (baos.toByteArray());
instead.

I'm not sure if this is the best way to do it, but this seems to work, but not exactly as I imagined:
File tempFile = File.createTempFile("config", ".properties");
BufferedWriter bw = new BufferedWriter(new FileWriter(tempFile));
bw.write("Helloworld!!!!");
bw.close();
File tempTarFile = File.createTempFile("config", ".tar");
FileOutputStream dest = new FileOutputStream( tempTarFile );
TarOutputStream out = new TarOutputStream( new BufferedOutputStream( dest ) );
out.putNextEntry(new TarEntry(tempFile, "config.properties"));
BufferedInputStream origin = new BufferedInputStream(new FileInputStream( tempFile ));
int count;
byte data[] = new byte[2048];
while((count = origin.read(data)) != -1) {
out.write(data, 0, count);
}
out.flush();
origin.close();
out.close();
tempFile.delete();
tempTarFile.delete();
InputStream inputStream = new FileInputStream(tempTarFile);
//to check if it worked:
// OutputStream outputStream = null;
//
// try {
//
// // write the inputStream to a FileOutputStream
// outputStream =
// new FileOutputStream(new File("config.tar"));
//
// int read = 0;
// byte[] bytes = new byte[1024];
//
// while ((read = inputStream.read(bytes)) != -1) {
// outputStream.write(bytes, 0, read);
// }
//
// System.out.println("Done!");
//
// } catch (IOException e) {
// e.printStackTrace();
// } finally {
// if (inputStream != null) {
// try {
// inputStream.close();
// } catch (IOException e) {
// e.printStackTrace();
// }
// }
// if (outputStream != null) {
// try {
// // outputStream.flush();
// outputStream.close();
// } catch (IOException e) {
// e.printStackTrace();
// }
//
// }
// }

java.io.IOException: Error: End-of-File, expected line Issue with PDFBox

I am trying to read the PDF text from the PDF which is opened in the browser.
After clicking on a button 'Print' the below URL opens up in the new tab.
https://myappurl.com/employees/2Jb_rpRC710XGvs8xHSOmHE9_LGkL97j/details/listprint.pdf?ids%5B%5D=2Jb_rpRC711lmIvMaBdxnzJj_ZfipcXW
I have executed the same program with other web URLs and found to be working fine. I have used the same code that is used here (Extract PDF text).
And i am using the below versions of PDFBox.
<dependency>
<groupId>org.apache.pdfbox</groupId>
<artifactId>pdfbox</artifactId>
<version>1.8.9</version>
</dependency>
<dependency>
<dependency>
<groupId>org.apache.pdfbox</groupId>
<artifactId>fontbox</artifactId>
<version>1.8.9</version>
</dependency>
Below is the code that is working fine with other URLS :
public boolean verifyPDFContent(String strURL, String reqTextInPDF) {
boolean flag = false;
PDFTextStripper pdfStripper = null;
PDDocument pdDoc = null;
COSDocument cosDoc = null;
String parsedText = null;
try {
URL url = new URL(strURL);
BufferedInputStream file = new BufferedInputStream(url.openStream());
PDFParser parser = new PDFParser(file);
parser.parse();
cosDoc = parser.getDocument();
pdfStripper = new PDFTextStripper();
pdfStripper.setStartPage(1);
pdfStripper.setEndPage(1);
pdDoc = new PDDocument(cosDoc);
parsedText = pdfStripper.getText(pdDoc);
} catch (MalformedURLException e2) {
System.err.println("URL string could not be parsed "+e2.getMessage());
} catch (IOException e) {
System.err.println("Unable to open PDF Parser. " + e.getMessage());
try {
if (cosDoc != null)
cosDoc.close();
if (pdDoc != null)
pdDoc.close();
} catch (Exception e1) {
e.printStackTrace();
}
}
System.out.println("+++++++++++++++++");
System.out.println(parsedText);
System.out.println("+++++++++++++++++");
if(parsedText.contains(reqTextInPDF)) {
flag=true;
}
return flag;
}
And The below is the Stacktrace of the exception that im getting
java.io.IOException: Error: End-of-File, expected line
at org.apache.pdfbox.pdfparser.BaseParser.readLine(BaseParser.java:1517)
at org.apache.pdfbox.pdfparser.PDFParser.parseHeader(PDFParser.java:372)
at org.apache.pdfbox.pdfparser.PDFParser.parse(PDFParser.java:186)
at com.kareo.utils.PDFManager.getPDFContent(PDFManager.java:26)
Updating the image which i took when debugging at URL and File.
Please help me out. Is this something with 'https'???

We all know that file stream is like a pipe. Once the data flows past, it cannot be used again. so you can:
1.Convert input stream to file.
public void useInputStreamTwiceBySaveToDisk(InputStream inputStream) {
String desPath = "test001.bin";
try (BufferedInputStream is = new BufferedInputStream(inputStream);
BufferedOutputStream os = new BufferedOutputStream(new FileOutputStream(desPath))) {
int len;
byte[] buffer = new byte[1024];
while ((len = is.read(buffer)) != -1) {
os.write(buffer, 0, len);
}
} catch (IOException e) {
e.printStackTrace();
}
File file = new File(desPath);
StringBuilder sb = new StringBuilder();
try (BufferedInputStream is = new BufferedInputStream(new FileInputStream(file))) {
int len;
byte[] buffer = new byte[1024];
while ((len = is.read(buffer)) != -1) {
sb.append(new String(buffer, 0, len));
}
System.out.println(sb.toString());
} catch (IOException e) {
e.printStackTrace();
}
}
2.Convert input stream to data.
public void useInputStreamTwiceSaveToByteArrayOutputStream(InputStream inputStream) {
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
try {
byte[] buffer = new byte[1024];
int len;
while ((len = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, len);
}
} catch (IOException e) {
e.printStackTrace();
}
// first read InputStream
InputStream inputStream1 = new ByteArrayInputStream(outputStream.toByteArray());
printInputStreamData(inputStream1);
// second read InputStream
InputStream inputStream2 = new ByteArrayInputStream(outputStream.toByteArray());
printInputStreamData(inputStream2);
}
3.Marking and resetting with input stream.
public void useInputStreamTwiceByUseMarkAndReset(InputStream inputStream) {
StringBuilder sb = new StringBuilder();
try (BufferedInputStream bufferedInputStream = new BufferedInputStream(inputStream, 10)) {
byte[] buffer = new byte[1024];
//Call the mark method to mark
//The number of bytes allowed to be read by the flag set here after reset is the maximum value of an integer
bufferedInputStream.mark(bufferedInputStream.available() + 1);
int len;
while ((len = bufferedInputStream.read(buffer)) != -1) {
sb.append(new String(buffer, 0, len));
}
System.out.println(sb.toString());
// After the first call, explicitly call the reset method to reset the flow
bufferedInputStream.reset();
// Read the second stream
sb = new StringBuilder();
int len1;
while ((len1 = bufferedInputStream.read(buffer)) != -1) {
sb.append(new String(buffer, 0, len1));
}
System.out.println(sb.toString());
} catch (IOException e) {
e.printStackTrace();
}
}
then you can repeat the read operation for the same input stream many times.