We Keep Coding

Java code works on my laptop, but not on my desktop?

I use IntelliJ. I have a class that manages student grades. It can edit files, which I have a new .temp file being written, and renamed. Then the old file gets deleted. On my laptop (mac) this works fine, but on my desktop (windows) everything works, but the old file is not deleted, and temp is not renamed.
Below is my method to edit the files:
private static void editStuGrade() throws Exception {
System.out.println("Enter a course grade you want to change (q1,q2,q3,mid,final): ");
String editInput = command.next();
System.out.println("Enter a new score: ");
String newGrade = command.next();
Path p = Paths.get(System.getProperty("user.dir"),"src","Assignment1", student.name + ".txt");
File inputFile = new File(p.toString());
FileInputStream inputStream = new FileInputStream(inputFile);
InputStreamReader inputStreamReader = new InputStreamReader(inputStream);
BufferedReader reader = new BufferedReader(inputStreamReader);
File outputFile = new File(p + ".temp");
FileOutputStream outputStream = new FileOutputStream(outputFile);
OutputStreamWriter outputStreamWriter = new OutputStreamWriter(outputStream);
BufferedWriter writer = new BufferedWriter(outputStreamWriter);
if (editInput.equalsIgnoreCase("q1")) {
writer.write(student.name + "," + student.id + "," + newGrade + "," + student.quiz2
+ "," + student.quiz3 + "," + student.midterm + "," + student.finalTest);
writer.close();
} else if (editInput.equalsIgnoreCase("q2")) {
writer.write(student.name + "," + student.id + "," + student.quiz1 + "," + newGrade
+ "," + student.quiz3 + "," + student.midterm + "," + student.finalTest);
writer.close();
} else if (editInput.equalsIgnoreCase("q3")) {
writer.write(student.name + "," + student.id + "," + student.quiz1 + "," + student.quiz2
+ "," + newGrade + "," + student.midterm + "," + student.finalTest);
writer.close();
} else if (editInput.equalsIgnoreCase("mid")) {
writer.write(student.name + "," + student.id + "," + student.quiz1 + "," + student.quiz2
+ "," + student.quiz3 + "," + newGrade + "," + student.finalTest);
writer.close();
} else if (editInput.equalsIgnoreCase("final")) {
writer.write(student.name + "," + student.id + "," + student.quiz1 + "," + student.quiz2
+ "," + student.quiz3 + "," + student.midterm + "," + newGrade);
writer.close();
}
inputFile.delete();
outputFile.renameTo(new File(p.toString()));
System.out.println("Successful.");
}

Windows is a special snowflake. Unlike other OSes (or rather, Windows File System, unlike other file systems) does not let you delete any open files, and does not let you rename or delete directories of they contain any open files. In contrast to other OSes which don't mind at all; files on disk are merely 'pointers', and any process that opens a file also gets a pointer. The file isn't truly removed from disk until all pointers are gone, so, you can just delete files - if that file is still open, no problem - as long as it is the file doesn't disappear. It's very similar to java garbage collection in that way.
But not so on windows.
Your code has a bug in it - you aren't managing your resources. This is resulting in the files being open, and then you try to delete them - this works on non-windows filesystems but isn't allowed on windows - you can't delete files even if you're the very process that still has them open.
Resources MUST be closed, and the responsibility to do this lies on you. Given that code can exit in many ways (not just 'by running to the end of a method', but also: With return, by throwing an exception, by using break or continue for flow control, etc). Therefore, trying to write code by hand that ensures your resource is closed for all possible exit paths is annoying and error prone, so, don't. Use java's language features:
Do not EVER open a resource unless you do so in a try-with block.
Looks like this:
try (var outputStream = new FileOutputStream(outputFile)) {
// outputStream exists here and can be interacted with as normal
}
No matter how code flow 'exits' that try block, the resource is closed automatically once it does. This is good - not just because this lets you delete those files, but also because every process gets a limited number of files they get to open, so if you fail to close, any non-trivial app will soon hard-crash due to having too many open files.
What are resources? The javadoc will tell you, and use common sense. most InputStream and OutputStreams are - any type that implements AutoClosable tends to be. If you new X() them up you definitely have to close them. If you're invoking a method that sounds like it 'makes' the resource (example: socket.getInputStream or Files.newInputStream), you have to close them.
Use try () {} to do this.
Once you do so, you can delete these files just fine, even on windows.

Is java multi thread can optimize multiple file writing

I have a file of 400+ GB like:
ID Data ...4000+columns
001 dsa
002 Data
… …
17201297 asdfghjkl
I wish to chunk down the file as per ID to get faster data retrieval as like:
mylocation/0/0/1/data.json
mylocation/0/0/2/data.json
.....
mylocation/1/7/2/0/1/2/9/7/data.json
my code is working fine but whatever writer I'm using with loop end closing it takes at least 159,206 milisoconds for 0.001% completion of file creation.
In that case can multithread be an option to reduce Time complexity (as like writing 100 or 1k files at a time)?
My Current code is:
int percent = 0;
File file = new File(fileLocation + fileName);
FileReader fileReader = new FileReader(file); // to read input file
BufferedReader bufReader = new BufferedReader(fileReader);
BufferedWriter fw = null;
LinkedHashMap<String, BufferedWriter> fileMap = new LinkedHashMap<>();
int dataCounter = 0;
while ((theline = bufReader.readLine()) != null) {
String generatedFilename = generatedFile + chrNo + "//" + directory + "gnomeV3.json";
Path generatedJsonFilePath = Paths.get(generatedFilename);
if (!Files.exists(generatedJsonFilePath)) {// create directory
Files.createDirectories(generatedJsonFilePath.getParent());
files.createFile(generatedJsonFilePath);
}
String jsonData = DBFileMaker(chrNo, theline, pos);
if (fileMap.containsKey(generatedFilename)) {
fw = fileMap.get(generatedFilename);
fw.write(jsonData);
} else {
fw = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(generatedFilename)));
fw.write(jsonData);
fileMap.put(generatedFilename, fw);
}
if (dataCounter == 172 * percent) {// As I know my number of rows
long millisec = stopwatch.elapsed(TimeUnit.MILLISECONDS);
System.out.println("Upto: " + pos + " as " + (Double) (0.001 * percent)
+ "% completion successful." + " took: " + millisec + " miliseconds");
percent++;
}
dataCounter++;
}
for (BufferedWriter generatedFiles : fileMap.values()) {
generatedFiles.close();
}

That really depends on your storage. Magnetic disks really like sequential writes, so multithreading would probably have a bad effect on their performance. However, SSDs may benefit from multithreaded writing.
What you should do is Either separate your code to 2 threads, where one thread creates the buffers of data to be written to disk and the second thread only writes the data. This way your disk would always keep busy and not wait for more data to be generated.
Or to have a single thread that generates the buffers to be written, but to use java nio in order to write the data asynchronously, while going on to generate the next buffer.

Problems with read and write an ArrayList of Objects from a file

I am having problems to save an ArrayList of a object to a file and them read that. I am searching in StackOverflow and I cannot find my error...
Either I have and Classnotfound issue OR a read empty values...
Does anybody help me? Looks that writing is working well to me.
That is the code:
// calls using the CLASS
// Lets save distribution IFF asked to
if (shouldSavePoiDistribution){
List<POInode> listOfPOIs = new ArrayList<POInode>();
for(Node n : Runtime.nodes) {
if (n instanceof POInode){
listOfPOIs.add((POInode) n);
}
}
GlobalWriteAndLoadPositions saver = new GlobalWriteAndLoadPositions();
saver.write(Global.distributionFolder,listOfPOIs);
}
// lets load a distribution IFF asked to
if (shouldLoadPoiDistribution){
Global.lastPOIloaded = 0;
GlobalWriteAndLoadPositions loader = new GlobalWriteAndLoadPositions();
Global.listOfLoadedPOIs = loader.load(Global.distributionFile);
}
///// CLASS to read and write object in file.
// The file look be correct its is created and the size varies...
// reading returns the corret size of the arraylist, but only wrong values IFF I try/catch "ois.readObject();" for ClassNotFound, otherwise, it raises an exception
package sinalgo.runtime;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
import projects.nodes.nodeImplementations.POInode;
public class GlobalWriteAndLoadPositions {
public GlobalWriteAndLoadPositions(){
}
public void write(String folder, List<POInode> POIlist) throws IOException {
int nextFileItarator = (int) Files.list(Paths.get(folder)).count();
nextFileItarator++;
FileOutputStream fout= new FileOutputStream (folder + nextFileItarator + ".txt");
System.out.print("\n[Global] Trying SAVE a distribution on " + folder + nextFileItarator + ".txt" + ": ");
ObjectOutputStream oos = new ObjectOutputStream(fout);
oos.writeObject(POIlist);
oos.close();
fout.close();
POIlist.forEach((a)->System.out.print("POI " + a.ID + " # (" + a.getPosition().xCoord +" , " + a.getPosition().yCoord + ") | " ));
System.out.println("\n");
}
public ArrayList<POInode> load(String file) throws IOException{
System.out.print("\n[Global] Trying LOAD a distribution: " + Global.distributionFile + " ||> ");
FileInputStream fin = new FileInputStream(file);
ObjectInputStream ois = new ObjectInputStream(fin);
List<POInode> listOfLoadedPOIs = new ArrayList<POInode>();
listOfLoadedPOIs = (ArrayList<POInode>) ois.readObject(); // THIS LOOK WRONG, I MEAN, Does not find the class POInode
ois.close();
fin.close();
listOfLoadedPOIs.forEach((a)->System.out.print("POI " + a.ID + " # (" + a.getPosition().xCoord +" , " + a.getPosition().yCoord + ") | " ));
System.out.println("\n");
return (ArrayList<POInode>) listOfLoadedPOIs;
}
}
Please, how to solve this?

Firstly, how do you use those methods?
The main trouble goes there.
Also, not bad will be to see POInode class.
BTW A Smal suggestion is with using try with resources instead of closing streams as you did:
// writing looks fine
public void write(String folder, ArrayList<POInode> POIlist) throws IOException {
int nextFileItarator = (int) Files.list(Paths.get(folder)).count();
nextFileItarator++;
try (FileOutputStream fout = new FileOutputStream(folder + nextFileItarator + ".txt");
ObjectOutputStream oos = new ObjectOutputStream(fout)) {
oos.writeObject(POIlist);
}
}
// I read the quantity OK, but empty or wrong values...
public ArrayList<POInode> load(String file) throws IOException {
ArrayList<POInode> listOfLoadedPOIs = new ArrayList<POInode>();
try (FileInputStream fin = new FileInputStream(file);
ObjectInputStream ois = new ObjectInputStream(fin)) {
// readObject() throws ClassnotFound or empty values.
listOfLoadedPOIs = (ArrayList<POInode>) ois.readObject();
}
return (ArrayList<POInode>) listOfLoadedPOIs;
}
Following your code style much better will be called close() from final block.
However, as I said earlier a much better idea is to use try with resources. It is suggested by Joshua Bloch in his Effective Java, Item 9: Prefer try-with-resources to try-finally.

Given your own comment from
listOfLoadedPOIs = (ArrayList<POInode>) ois.readObject();
// readObject() throws ClassnotFound or empty values.
we can deduct what is going on.
There are two cases:
Your prior call to your write method - used an empty list. And then your read code works fine; but gives you, surprise, the empty list you wrote.
Your prior call to your write method - used an NON empty list. But the JVM that is running your "read" code ... doesn't know the class of the objects stored in that list.
So your real problem boils down to: "what does it mean when java tells me about ClassNotFound"; and the answer to that could be found here.
Long story short: your classpath setup when you test your "read" part is incomplete; as java can't find the .class file for the kinds of objects you stored in those files.
The other issue with your code: when you "build" those streams on each other, like
WhateverStream inner = ...
SomeOtherStream outer = new SomeOtherStream(inner);
then you only need to call close()/... on outer, but not on inner. Actually it might even be another bug in your code that you do inner.close() followed by outer.close()!

The fastest ways to check if a file exists in java

Currently i am tasked with making a tool that can check whether a link is correct or not using java. The link is fed from Jericho HTML Parser, and my job is only to check whether the file is exist / the link is correct or not. That part is done, the hard part is to optimize it, since my code run (i have to say) rather sluggishly on 65ms per run
public static String checkRelativeURL(String originalFileLoc, String relativeLoc){
StringBuilder sb = new StringBuilder();
String absolute = Common.relativeToAbsolute(originalFileLoc, relativeLoc); //built in function to replace the link from relative link to absolute path
sb.append(absolute);
sb.append("\t");
try {
Path path = Paths.get(absolute);
sb.append(Files.exists(path));
}catch (InvalidPathException | NullPointerException ex) {
sb.append(false);
}
sb.append("\t");
return sb.toString();
}
and on this line it took 65 ms
Path path = Paths.get(absolute);
sb.append(Files.exists(path));
I have tried using
File file = new File(absolute);
sb.append(file.isFile());
It's still ran around 65~100ms.
So is there any other faster way to check whether a file exists or not other than this?
Since i am processing more than 70k html files and every milliseconds counts, thanks :(
EDIT:
I tried listing all the files into some List, and it doesn't really helps since it take more than 20mins just to list all the file....
The code that i use to list all the file
static public void listFiles2(String filepath){
Path path = Paths.get(filepath);
File file = null;
String pathString = new String();
try {
if(path.toFile().isDirectory()){
DirectoryStream<Path> stream = Files.newDirectoryStream(path);
for(Path entry : stream){
file = entry.toFile();
pathString = entry.toString();
if(file.isDirectory()){
listFiles2(pathString);
}
if (file.isFile()){
filesInProject.add(pathString);
System.out.println(pathString);
}
}
stream.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}

If you know in advance the target OS set (usually it is the case), ultimately the fastest way will be to list so many files through a shell, by invoking a process e.g. using Runtime.exec.
On Windows you can do with
dir /s /b
On Linux
ls -R -1
You can check what is the OS and use appropriate command (error or resort to directory stream if not supported).
If you wish simplicity and don't need to report a progress, you can avoid dealing with the process IO and store the list to a temporary file e.g. ls -R -1 > /tmp/filelist.txt. Alternatively, you can read from the process output directly. Read with a buffered stream, a reader or alike, with large enough buffer.
On SSD it will complete in a blink of an eye and on modern HDD in seconds (half million files is not a problem with this approach).
Once you have the list, you can approach it differently depending on maximum files count and memory requirements. If requirements are loose, e.g. desktop program, you can do with very simple code e.g. pre-loading the complete file list to a HashSet and check existence when needed. Shortening path by removing common root will require much less memory. You can also reduce memory by keeping only filename hash instead of full name (common root removal will probably reduce more).
Or you can optimize it further if you wish, the question just reduces now to a problem of checking existense of a string in a list of strings stored in memory or file, which has many well known optimal solutions.
Bellow is very loose, simplistic sample for Windows. It executes dir on HDD (not SSD) drive root with ~400K files, reads the list and benchmarks (well, kind of) time and memory for string set and md5 set approaches:
public static void main(String args[]) throws Exception {
final Runtime rt = Runtime.getRuntime();
System.out.println("mem " + (rt.totalMemory() - rt.freeMemory())
/ (1024 * 1024) + " Mb");
long time = System.currentTimeMillis();
// windows command: cd to t:\ and run recursive dir
Process p = rt.exec("cmd /c \"t: & dir /s /b > filelist.txt\"");
if (p.waitFor() != 0)
throw new Exception("command has failed");
System.out.println("done executing shell, took "
+ (System.currentTimeMillis() - time) + "ms");
System.out.println();
File f = new File("T:/filelist.txt");
// load into hash set
time = System.currentTimeMillis();
Set<String> fileNames = new HashSet<String>(500000);
try (BufferedReader reader = new BufferedReader(new InputStreamReader(
new FileInputStream(f), StandardCharsets.UTF_8),
50 * 1024 * 1024)) {
for (String line = reader.readLine(); line != null; line = reader
.readLine()) {
fileNames.add(line);
}
}
System.out.println(fileNames.size() + " file names loaded took "
+ (System.currentTimeMillis() - time) + "ms");
System.gc();
System.out.println("mem " + (rt.totalMemory() - rt.freeMemory())
/ (1024 * 1024) + " Mb");
time = System.currentTimeMillis();
// check files
for (int i = 0; i < 70_000; i++) {
StringBuilder fileToCheck = new StringBuilder();
while (fileToCheck.length() < 256)
fileToCheck.append(Double.toString(Math.random()));
if (fileNames.contains(fileToCheck))
System.out.println("to prevent optimization, never executes");
}
System.out.println();
System.out.println("hash set 70K checks took "
+ (System.currentTimeMillis() - time) + "ms");
System.gc();
System.out.println("mem " + (rt.totalMemory() - rt.freeMemory())
/ (1024 * 1024) + " Mb");
// Test memory/performance with MD5 hash set approach instead of full
// names
time = System.currentTimeMillis();
Set<String> nameHashes = new HashSet<String>(50000);
MessageDigest md5 = MessageDigest.getInstance("MD5");
for (String name : fileNames) {
String nameMd5 = new String(md5.digest(name
.getBytes(StandardCharsets.UTF_8)), StandardCharsets.UTF_8);
nameHashes.add(nameMd5);
}
System.out.println();
System.out.println(fileNames.size() + " md5 hashes created, took "
+ (System.currentTimeMillis() - time) + "ms");
fileNames.clear();
fileNames = null;
System.gc();
Thread.sleep(100);
System.gc();
System.out.println("mem " + (rt.totalMemory() - rt.freeMemory())
/ (1024 * 1024) + " Mb");
time = System.currentTimeMillis();
// check files
for (int i = 0; i < 70_000; i++) {
StringBuilder fileToCheck = new StringBuilder();
while (fileToCheck.length() < 256)
fileToCheck.append(Double.toString(Math.random()));
String md5ToCheck = new String(md5.digest(fileToCheck.toString()
.getBytes(StandardCharsets.UTF_8)), StandardCharsets.UTF_8);
if (nameHashes.contains(md5ToCheck))
System.out.println("to prevent optimization, never executes");
}
System.out.println("md5 hash set 70K checks took "
+ (System.currentTimeMillis() - time) + "ms");
System.gc();
System.out.println("mem " + (rt.totalMemory() - rt.freeMemory())
/ (1024 * 1024) + " Mb");
}
Output:
mem 3 Mb
done executing shell, took 5686ms
403108 file names loaded took 382ms
mem 117 Mb
hash set 70K checks took 283ms
mem 117 Mb
403108 md5 hashes created, took 486ms
mem 52 Mb
md5 hash set 70K checks took 366ms
mem 48 Mb

Issues with running Runtime.getRuntime().exec

I'm using process = Runtime.getRuntime().exec(cmd,null,new File(path));
to execute some SQL in file (abz.sql)
Command is:
"sqlplus "+ context.getDatabaseUser() + "/"
+ context.getDatabasePassword() + "#"
+ context.getDatabaseHost() + ":"
+ context.getDatabasePort() + "/"
+ context.getSid() + " #"
+ "\""
+ script + "\"";
String path=context.getReleasePath()+ "/Server/DB Scripts";
It is executing that file but not getting exit. Hence I tried using:
Writer out = new OutputStreamWriter(process.getOutputStream());
out.append("commit;\r\n");
out.append("exit \r\n");
System.out.println("---------"+out);
out.close();
This it complete block that I m using:
if(context.getConnectionField()=="ORACLE")
{
String cmd=
"sqlplus "+ context.getDatabaseUser() + "/"
+ context.getDatabasePassword() + "#"
+ context.getDatabaseHost() + ":"
+ context.getDatabasePort() + "/"
+ context.getSid() + " #"
+ "\""
+ script +"\"";
String path=context.getReleasePath()+ "/Server/DB Scripts";
process = Runtime.getRuntime().exec(cmd,null,new File(path));
out = new OutputStreamWriter(process.getOutputStream());
out.append("commit;\r\n");
out.append("exit \r\n");
System.out.println("---------"+out);
out.close();
Integer result1 = null;
while (result1 == null) {
try {
result1 = process.waitFor();
}
catch (InterruptedException e) {}
}
if(process.exitValue() != 0)
return false;
return true;
}

The code shown fails to read the error stream of the Process. That might be blocking progress. ProcessBuilder was introduced in Java 1.5 and has a handy method to redirectErrorStream() - so that it is only necessary to consume a single stream.
For more general tips, read & implement all the recommendations of When Runtime.exec() won't.

I can see a few issues here. The version of 'exec' that you are using will tokenize the command string using StringTokenizer, so unusual characters in the password (like spaces) or the other parameters being substituted are accidents waiting to happen. I recommend switching to the version
Process exec(String[] cmdarray,
String[] envp,
File dir)
throws IOException
It is a bit more work to use but much more robust.
The second issue that there are all kinds of caveat about whether or not exec will run concurrently with the Java process (see http://download.oracle.com/javase/1.4.2/docs/api/java/lang/Process.html). So you need to say which operating system you're on. If it does not run concurrently then your strategy of writing to the output stream cannot work!
The last bit of the program is written rather obscurely. I suggest ...
for (;;) {
try {
process.waitFor();
return process.exitValue() == 0;
} catch ( InterruptedException _ ) {
System.out.println( "INTERRUPTED!" ); // Debug only.
}
}
This eliminates the superfluous variable result1, eliminates the superfluous boxing and highlights a possible cause of endless looping.
Hope this helps & good luck!