We Keep Coding

Java - Prevent PrintWriter to overwrite the entire file everytime

I have a simulation program that needs to write certain results to a csv file very frequently during execution. I have found that there is something wrong with the printwriter which dramatically slows down running my program, as the output file is getting larger in size (near to 1 million rows). I doublt it's overwriting the entire file each time from the beginning, wheras I just need to append a single line at the bottom each time when it's being called. below is the code related to the writing fuctions.
one of the writing fuctions:
public void printHubSummary(Hub hub, String filePath) {
try {
StringBuilder sb = new StringBuilder();
String h = hub.getHub_code();
String date = Integer.toString(hub.getGs().getDate());
String time = hub.getGs().getHHMMFromMinute(hub.getGs().getClock());
String wgt = Double.toString(hub.getIb_wgt());
sb.append(h+","+date+","+time+","+wgt);
// System.out.println("truck print line: " + sb);
FileWriter.writeFile(sb.toString(),filePath);
}
catch (Exception e) {
System.out.println("Something wrong when outputing truck summary file!");
e.printStackTrace();
}
}
the file writer code: (should be where the problem is!)
public static boolean writeFile(String newStr, String filename) throws IOException {
boolean flag = false;
String filein = newStr + "\r\n";
String temp = "";
FileInputStream fis = null;
InputStreamReader isr = null;
BufferedReader br = null;
FileOutputStream fos = null;
PrintWriter pw = null;
try {
File file = new File(filename);
fis = new FileInputStream(file);
isr = new InputStreamReader(fis);
br = new BufferedReader(isr);
StringBuffer buf = new StringBuffer();
for (int j = 1; (temp = br.readLine()) != null; j++) {
buf = buf.append(temp);
buf = buf.append(System.getProperty("line.separator"));
}
if (buf.length() > 0 && buf.charAt(0) == '\uFEFF') {
buf.deleteCharAt(0);
}
buf.append(filein);
fos = new FileOutputStream(file);
byte[] unicode = {(byte)0xEF, (byte)0xBB, (byte)0xBF};
fos.write(unicode);
pw = new PrintWriter(fos);
pw.write(buf.toString().toCharArray());
pw.flush();
flag = true;
} catch (IOException e1) {
throw e1;
} finally {
if (pw != null) {
pw.close();
}
if (fos != null) {
fos.close();
}
if (br != null) {
br.close();
}
if (isr != null) {
isr.close();
}
if (fis != null) {
fis.close();
}
}
return flag;
}
An update on code modification. I have freezed the operations of repeatitively overwrting the entire file. It appears to solve the problem, but writing for sometime it's slowed down as well. Is it the best arrangement for wrting very large file? what other modifications can be done to make it even more efficient?
public static boolean writeFile1(String newStr, String filename) throws IOException {
boolean flag = false;
String filein = newStr + "\r\n";
String temp = "";
FileInputStream fis = null;
InputStreamReader isr = null;
BufferedReader br = null;
FileOutputStream fos = null;
PrintWriter pw = null;
try {
File file = new File(filename);
fis = new FileInputStream(file);
isr = new InputStreamReader(fis);
br = new BufferedReader(isr);
StringBuffer buf = new StringBuffer();
// for (int j = 1; (temp = br.readLine()) != null; j++) {
// buf = buf.append(temp);
// buf = buf.append(System.getProperty("line.separator"));
// }
// if (buf.length() > 0 && buf.charAt(0) == '\uFEFF') {
// buf.deleteCharAt(0);
// }
buf.append(filein);
fos = new FileOutputStream(file,true);
byte[] unicode = {(byte)0xEF, (byte)0xBB, (byte)0xBF};
fos.write(unicode);
pw = new PrintWriter(fos);
pw.write(buf.toString().toCharArray());
pw.flush();
flag = true;
} catch (IOException e1) {
throw e1;
} finally {
if (pw != null) {
pw.close();
}
if (fos != null) {
fos.close();
}
if (br != null) {
br.close();
}
if (isr != null) {
isr.close();
}
if (fis != null) {
fis.close();
}
}
return flag;
}

Provide a second argument to the FileOutputStream constructor to specify whether or not to use append mode, which will add to the end of the file rather than overwriting it.
fos = new FileOutputStream(file, true);
Alternatively, you could create a single static PrintWriter in append mode, which will probably be faster as it reduces garbage collection.

Use the Files / Path / Java NIO2 which is richer: the code below would need Java 7 at least.
Path path = Paths.get(filename);
try (BufferedWriter bw = Files.newBufferedWriter(
path, StandardCharsets.UTF_8, StandardOpenOption.APPEND, StandardOpenOption.CREATE, StandardOpenOption.WRITE)) {
bw.append(filein);
bw.newLine();
}
Your cue here is the StandardOpenOption.
You will probably have to do some additional code before to write the Unicode part (and fix the StandardCharsets.UTF_8):
if (Files.notExists(path)) {
Files.write(path, new byte[] {(byte)0xEF, (byte)0xBB, (byte)0xBF});
}
Also, try to not use StringBuffer in a local method, use StringBuilder: you don't need synchronisation most of the time.

I'm confused about PDF files

If I have my code this way round the PDF say's its invalid and cannot be opened, but if I swap them and have B before A it works fine? why is this and what would I have to do to get it working? TIA
InputStream in = new BufferedInputStream(conn.getInputStream());
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
//A
String line = "";
StringBuilder builder = new StringBuilder();
try {
while ((line = reader.readLine()) != null) {
builder.append(line);
}
} catch (IOException e) {
e.printStackTrace();
}
//B
File directory = Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOWNLOADS);
File outputFile = new File(directory, "goo.pdf");
FileOutputStream fos = null;
try {
fos = new FileOutputStream(outputFile);
} catch (FileNotFoundException e) {
e.printStackTrace();
}
byte[] buffer = new byte[1024];
int len1 = 0;//init length
while (true) {
try {
if (!((len1 = in.read(buffer)) != -1)) break;
} catch (IOException e) {
e.printStackTrace();
}
try {
fos.write(buffer, 0, len1);
} catch (IOException e) {
e.printStackTrace();
}
}

An InputStream can only be read once.
In 'A', the stream is read and the contents are put in a StringBuilder.
In 'B', the stream (now empty) is read and piped to a file.
By having A first, the output file will always be empty.
Simply remove A as it's not doing anything for you here.

Java: How to perform dos2unix on multiple xml files in a zip file?

I have a form to upload a zip file and then convert all the xml files inside the zip to unix format if they are in dos format. Right now I receive the input as an InputStream. How do I process the file in inputstream and perform (dos2unix) on it to convert it to unix format?
I tried to convert the stream to a file and then convert them but didn't work
public void uploadFile(UploadAuditConfig transaction,String fileType, InputStream in, String delimiter) {
ZipInputStream zipInputStream = new ZipInputStream(in);
ZipEntry entry = null;
do{
entry = zipInputStream.getNextEntry();
//need to convert this entry to unix format if it is dos before I pass it to processFile method
if(entry != null && !entry.isDirectory()) {
List<Map<String,String>> list =processFile(zipInputStream, delimiter);
zipInputStream.closeEntry();
}
}while(entry!=null);
}
public List<Map<String, String>> processFile(InputStream in, String
delimiter){
List<Map<String,String>> acesList = new ArrayList<>();
XMLInputFactory xif = XMLInputFactory.newInstance();
XMLStreamReader xsr = xif.createXMLStreamReader(new InputStreamReader(in));
while (xsr.nextTag() == XMLStreamConstants.START_ELEMENT) {
File file = new File("/tmp/" + "out" + i + ".xml");
FileWriter fw = new FileWriter(file);
if (!file.exists())
file.createNewFile();
FileOutputStream fos = new FileOutputStream(file, true);
t.transform(new StAXSource(xsr), new StreamResult(fos));
fos.close();
if (i == 0) {
JSONObject xmlJSONObjHeader = XML.toJSONObject(content);
Object o = JsonPath.parse(xmlJSONObjHeader.toString()).read("$['Header']['BrandAAIAID']");
brandaaiaid = String.valueOf(o);
logger.info("Brand id: " + brandaaiaid);
file.delete();
fw.close();
i++;
}
}
return acesList;
}
Expected: Unix formatted file from inputstream

I was able to convert the inputstream to a file and then convert the file to unix using dos2unix and then again passing the file as inputstream.
OutputStream outputStream = null;
File file1 = new File("/tmp/output.txt");
try
{
outputStream = new FileOutputStream(file1);
int read = 0;
byte[] bytes = new byte[1024];
while ((read = in1.read(bytes)) != -1) {
outputStream.write(bytes, 0, read);
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally
{
if(outputStream != null)
{
try {
outputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
ExecuteShell.executeJavaCommand(file1.getAbsolutePath());
logger.info("action=output.txt converted from dos to unix");
InputStream in = null;
try {
in = new FileInputStream(file1);
} catch (FileNotFoundException e) {
e.printStackTrace();
}

java.io.IOException: Error: End-of-File, expected line Issue with PDFBox

I am trying to read the PDF text from the PDF which is opened in the browser.
After clicking on a button 'Print' the below URL opens up in the new tab.
https://myappurl.com/employees/2Jb_rpRC710XGvs8xHSOmHE9_LGkL97j/details/listprint.pdf?ids%5B%5D=2Jb_rpRC711lmIvMaBdxnzJj_ZfipcXW
I have executed the same program with other web URLs and found to be working fine. I have used the same code that is used here (Extract PDF text).
And i am using the below versions of PDFBox.
<dependency>
<groupId>org.apache.pdfbox</groupId>
<artifactId>pdfbox</artifactId>
<version>1.8.9</version>
</dependency>
<dependency>
<dependency>
<groupId>org.apache.pdfbox</groupId>
<artifactId>fontbox</artifactId>
<version>1.8.9</version>
</dependency>
Below is the code that is working fine with other URLS :
public boolean verifyPDFContent(String strURL, String reqTextInPDF) {
boolean flag = false;
PDFTextStripper pdfStripper = null;
PDDocument pdDoc = null;
COSDocument cosDoc = null;
String parsedText = null;
try {
URL url = new URL(strURL);
BufferedInputStream file = new BufferedInputStream(url.openStream());
PDFParser parser = new PDFParser(file);
parser.parse();
cosDoc = parser.getDocument();
pdfStripper = new PDFTextStripper();
pdfStripper.setStartPage(1);
pdfStripper.setEndPage(1);
pdDoc = new PDDocument(cosDoc);
parsedText = pdfStripper.getText(pdDoc);
} catch (MalformedURLException e2) {
System.err.println("URL string could not be parsed "+e2.getMessage());
} catch (IOException e) {
System.err.println("Unable to open PDF Parser. " + e.getMessage());
try {
if (cosDoc != null)
cosDoc.close();
if (pdDoc != null)
pdDoc.close();
} catch (Exception e1) {
e.printStackTrace();
}
}
System.out.println("+++++++++++++++++");
System.out.println(parsedText);
System.out.println("+++++++++++++++++");
if(parsedText.contains(reqTextInPDF)) {
flag=true;
}
return flag;
}
And The below is the Stacktrace of the exception that im getting
java.io.IOException: Error: End-of-File, expected line
at org.apache.pdfbox.pdfparser.BaseParser.readLine(BaseParser.java:1517)
at org.apache.pdfbox.pdfparser.PDFParser.parseHeader(PDFParser.java:372)
at org.apache.pdfbox.pdfparser.PDFParser.parse(PDFParser.java:186)
at com.kareo.utils.PDFManager.getPDFContent(PDFManager.java:26)
Updating the image which i took when debugging at URL and File.
Please help me out. Is this something with 'https'???

We all know that file stream is like a pipe. Once the data flows past, it cannot be used again. so you can:
1.Convert input stream to file.
public void useInputStreamTwiceBySaveToDisk(InputStream inputStream) {
String desPath = "test001.bin";
try (BufferedInputStream is = new BufferedInputStream(inputStream);
BufferedOutputStream os = new BufferedOutputStream(new FileOutputStream(desPath))) {
int len;
byte[] buffer = new byte[1024];
while ((len = is.read(buffer)) != -1) {
os.write(buffer, 0, len);
}
} catch (IOException e) {
e.printStackTrace();
}
File file = new File(desPath);
StringBuilder sb = new StringBuilder();
try (BufferedInputStream is = new BufferedInputStream(new FileInputStream(file))) {
int len;
byte[] buffer = new byte[1024];
while ((len = is.read(buffer)) != -1) {
sb.append(new String(buffer, 0, len));
}
System.out.println(sb.toString());
} catch (IOException e) {
e.printStackTrace();
}
}
2.Convert input stream to data.
public void useInputStreamTwiceSaveToByteArrayOutputStream(InputStream inputStream) {
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
try {
byte[] buffer = new byte[1024];
int len;
while ((len = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, len);
}
} catch (IOException e) {
e.printStackTrace();
}
// first read InputStream
InputStream inputStream1 = new ByteArrayInputStream(outputStream.toByteArray());
printInputStreamData(inputStream1);
// second read InputStream
InputStream inputStream2 = new ByteArrayInputStream(outputStream.toByteArray());
printInputStreamData(inputStream2);
}
3.Marking and resetting with input stream.
public void useInputStreamTwiceByUseMarkAndReset(InputStream inputStream) {
StringBuilder sb = new StringBuilder();
try (BufferedInputStream bufferedInputStream = new BufferedInputStream(inputStream, 10)) {
byte[] buffer = new byte[1024];
//Call the mark method to mark
//The number of bytes allowed to be read by the flag set here after reset is the maximum value of an integer
bufferedInputStream.mark(bufferedInputStream.available() + 1);
int len;
while ((len = bufferedInputStream.read(buffer)) != -1) {
sb.append(new String(buffer, 0, len));
}
System.out.println(sb.toString());
// After the first call, explicitly call the reset method to reset the flow
bufferedInputStream.reset();
// Read the second stream
sb = new StringBuilder();
int len1;
while ((len1 = bufferedInputStream.read(buffer)) != -1) {
sb.append(new String(buffer, 0, len1));
}
System.out.println(sb.toString());
} catch (IOException e) {
e.printStackTrace();
}
}
then you can repeat the read operation for the same input stream many times.

Java http - read large file

I am reading a big file in a Java program, using http access. I read the stream, and then I apply some criteria. Would it be possible to apply the criteria on the read stream, so I will have a light result (I'm reading big files)?
Here is my code for reading the file:
public String getMyFileContent(URLConnection uc){
String myresult = null;
try {
InputStream is = uc.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
int numCharsRead;
char[] charArray = new char[1024];
StringBuffer sb = new StringBuffer();
while ((numCharsRead = isr.read(charArray)) > 0) {
sb.append(charArray, 0, numCharsRead);
}
myresult = sb.toString();
}
catch (MalformedURLException e) {
e.printStackTrace();
}
catch (IOException e) {
e.printStackTrace();
}
return result;
}
And in another method, I then apply the criteria (to parse the content).
I couldn't achieve to do like this:
public String getMyFileContent(URLConnection uc){
String myresult = null;
try {
InputStream is = uc.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
int numCharsRead;
char[] charArray = new char[1024];
StringBuffer sb = new StringBuffer();
while ((numCharsRead = isr.read(charArray)) > 0) {
sb.append(charArray, 0, numCharsRead);
//Apply my criteria here on the stream ??? Is it possible ???
}
myresult = sb.toString();
}
catch (MalformedURLException e) {
e.printStackTrace();
}
catch (IOException e) {
e.printStackTrace();
}
return myresult;
}

The template I would use is
InputStreamReader isr = new InputStreamReader(uc.getInputStream());
int numCharsRead;
char[] charArray = new char[1024];
while ((numCharsRead = isr.read(charArray)) > 0) {
//Apply my criteria here on the stream
}
however since it is text, this might be more useful
InputStream is = uc.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(uc.getInputStream()));
String line;
while ((line = br.readLine()) != null) {
//Apply my criteria here on each line
}