I recently added filelocks to my downloader asynctask:
FileOutputStream file = new FileOutputStream(_outFile);
file.getChannel().lock();
and after download completes, file.close() to release lock.
From a called BroadcastReceiver (different thread), I need to go through the files and see which are downloaded and which are still locked. I started with trylock:
for (int i=0; i<files.length; i++) {
try {
System.out.print((files[i]).getName());
test = new FileOutputStream(files[i]);
FileLock lock = test.getChannel().tryLock();
if (lock != null) {
lock.release();
//Not a partial download. Do stuff.
}
} catch (Exception e) {
e.printStackTrace();
} finally {
test.close();
}
}
Unfortunately I read the file is truncated (0 bytes) when the FileOutputStream is created.
I set it to append, but the lock doesn't seem to take effect, all appear to be un-locked (fully downloaded)
Is there another way to check if a write-lock is applied to the file currently, or am I using the wrong methods here? Also, is there a way to debug file-locks, from the ADB terminal or Eclipse?
None of this is going to work. Check the Javadoc. Locks are held on behalf of the entire process, i.e. the JVM, not by individual threads.
My first thought would be to open it for append per the javadocs
test = new FileOutputStream(files[i], true); // the true specifies for append
Related
I need help with running parallel operations. The goal of the code is to extract a large amount of small files from the same tar in different folders in a very short time
This is the code:
public void decompress(File archive, File destination) throws RuntimeException {
try (InputStream in = new FileInputStream(archive);
BufferedInputStream buff = new BufferedInputStream(in);
TarArchiveInputStream is = (TarArchiveInputStream) new ArchiveStreamFactory().createArchiveInputStream("tar", buff)
) {
TarArchiveEntry entry;
while ((entry = is.getNextTarEntry()) != null) {
File file = new File(destination, entry.getName());
file.getParentFile().mkdirs();
Files.write(file.toPath(), is.readAllBytes());
}
} catch (IOException | ArchiveException e) {
e.printStackTrace();
}
}
When I execute one time this operation, it takes ~900ms
But when I do something like this to execute the same operation, multiple times in parallel it takes 20000ms:
ExecutorService EXECUTOR_SERVICE = Executors.newFixedThreadPool(20);
File archive = ...;
for (int i = 0; i < 5; i++) {
File directory = new File("Dir_" + i);
EXECUTOR_SERVICE.submit(() -> decompress(archive, directory));
}
or
File archive = ...;
for (int i = 0; i < 5; i++) {
File directory = new File("Dir_" + i);
new Thread(() -> decompress(archive, directory)).start();
}
One suspicion is that the directories contain many files, hence File.mkdirs does needlessly much checks.
The constructor of BufferedInputStream may have a custom buffer size. Never helped much, but it might be with your disk. Also with parallelism it could help to prevent much "disk head movements."
You probably already tried Files.copy but still, it might have a better memory behavior that readAllBytes.
So the version becomes (eschewing File in favor of Path):
public void decompress(File archive, File destination) throws RuntimeException {
final int bufferSize = 1024 * 128;
Path archivePath = archive.toPath();
Path destinationPath = destination.toPath();
try (InputStream in = new FileInputStream(archive);
BufferedInputStream buff = new BufferedInputStream(in, bufferSize);
TarArchiveInputStream is = (TarArchiveInputStream)
new ArchiveStreamFactory().createArchiveInputStream("tar", buff)
) {
Path oldFileParent = destinationPath;
oldFileParent.createDirectories();
TarArchiveEntry entry;
while ((entry = is.getNextTarEntry()) != null) {
Path file = Paths.get(destinationPath, entry.getName());
Path fileParent = file.getParent();
if (!fileParent.equals(oldFileParent)) {
oldFileParent = fileParent;
oldFileParent.createDirectories();
}
Files.copy(is, file);
//Files.write(file, is.readAllBytes());
}
} catch (IOException | ArchiveException e) {
e.printStackTrace();
}
}
Throwing a RuntimeException and capturing the IOException/ArchiveException without throwing it back (as new IllegalStateException(e)) is a matter of taste.
Now to adding parallelism: disk output is probably the bottleneck. Writing two files to the same disk in parallel means skipping back and forth on the disk. Small files might just do.
Better seems to parallelize reading a next file and then in another thread write it.
Two threads might theoretically perform better than many threads with enhightened disk traffic. readAllBytes might then be appropriate, to let the writing thread not use is.
As in the tar entry maybe the file size is kept too, that would allow to check whether readAllBytes is efficient enough - for large files.
Logging was mentioned in this question. It is known, that that can consume much time, and with parallelism becomes even more critical. But you seem to be aware of it. You wrote having written your own logger. For a library System.Logger is actually best. It is a façade that uses any logger the application provides. This would have prevented the logger vulnaribility hidden in library dependencies of the past year.
Ignoring the fact that you are not decompressing the file in parallel here (you are running multiple threads decompressing the same file concurrently, essentially overwriting the result), there may be several reasons for this performance hit. I/O is one, so it depends on the underlying implementation. Also, what is the Logger you are using there? While other parts of your code doesn't seem to be shared among multiple threads, the static call to Logger is something that is shared.
Also note: java.nio uses FileChannels which provide synchronous I/O, so depending on how you create the channels, you may get into similar situations (though I don't believe this applies here).
I am working on an Android App that changes the CPU Frequency when a foreground app changes. The frequencies for the foreground app is defined in my application itself. But while changing the frequencies my app has to open multiple system files and replace the frequency with my text. This makes my UI slow and when I change apps continuously, it makes the systemUI crash. What can I do to write these multiple files all together at the same time?
I have tried using ASynctaskLoader but that too crashes the SystemUI later.
public static boolean setFreq(String max_freq, String min_freq) {
ByteArrayInputStream inputStream = new ByteArrayInputStream(max_freq.getBytes(Charset.forName("UTF-8")));
ByteArrayInputStream inputStream1 = new ByteArrayInputStream(min_freq.getBytes(Charset.forName("UTF-8")));
SuFileOutputStream outputStream;
SuFileOutputStream outputStream1;
try {
if (max_freq != null) {
int cpus = 0;
while (true) {
SuFile f = new SuFile(CPUActivity.MAX_FREQ_PATH.replace("cpu0", "cpu" + cpus));
SuFile f1 = new SuFile(CPUActivity.MIN_FREQ_PATH.replace("cpu0", "cpu" + cpus));
outputStream = new SuFileOutputStream(f);
outputStream1 = new SuFileOutputStream(f1);
ShellUtils.pump(inputStream, outputStream);
ShellUtils.pump(inputStream1, outputStream1);
if (!f.exists()) {
break;
}
cpus++;
}
}
} catch (Exception ex) {
}
return true;
}
I assume SuFile and SuFileOutputStream are your custom implementations extending Java File and FileOutputStream classes.
Couple of points need to be fixed first.
f.exists() check should be before initializing OutputStream, otherwise it will create the file before checking exists or not. This makes your while loop to become an infinite loop.
as #Daryll suggested, use the number of CPUs with while/for loop. I suggest using for loop.
close your streams after pump(..) method call.
If you want to keep the main thread free, then you can do something like this,
see this code segment:
public static void setFreq(final String max_freq, final String min_freq) {
new Thread(new Runnable() {
//Put all the stuff here
}).start();
}
This should solve your problem.
Determine the number of CPUs before hand and use that number in your loop rather than using a while (true) having to do SuFile.exists() every cycle.
I don't know what SuFileOutputStream is but you may need to close those file output streams or find a faster way to write the file if that implementation is too slow.
I am writing a small program to help with planning future workouts. I am nearly finished however saving and loading is giving me some trouble. The program works with a list of "ride"(a custom class) objects that hold a number of qualities (like a Dat, and then some ints and doubles)
right now, I have two methods, a "saver" and a "loader":
public void saver() {
try{ // Catch errors in I/O if necessary.
// Open a file to write to, named SavedObj.sav.
FileOutputStream saveFile=new FileOutputStream("SaveObj.sav");
// Create an ObjectOutputStream to put objects into save file.
ObjectOutputStream save = new ObjectOutputStream(saveFile);
// Now we do the save.
for (int x = 0; x < rides.size(); x++) {
save.writeObject(rides.get(x).getDate());
save.writeObject(rides.get(x).getMinutes());
save.writeObject(0);
save.writeObject(rides.get(x).getIF());
save.writeObject(rides.get(x).getTss());
}
// Close the file.
save.close(); // This also closes saveFile.
}
catch(Exception exc){
exc.printStackTrace(); // If there was an error, print the info.
}
}
public void loader() {
try{
// Open file to read from, named SavedObj.sav.
FileInputStream saveFile = new FileInputStream("SaveObj.sav");
// Create an ObjectInputStream to get objects from save file.
ObjectInputStream save = new ObjectInputStream(saveFile);
Ride worker;
while(save.available() > 0) {
worker = new Ride((Date)save.readObject(), (int)save.readObject(), (double)save.readObject(), (double)save.readObject(), (int)save.readObject());
addRide(worker.getDate(), worker.getMinutes(), 0, worker.getIF(), worker.getTss());
}
// Close the file.
save.close(); // This also closes saveFile.
}
catch(Exception exc){
exc.printStackTrace(); // If there was an error, print the info.
}
}
When I run the program, neither "save" nor "load" return any errors. A .sav file is created when one does not exist, and is edited each time the program is executed. Yet, the program never restores data from previous sessions. Please let me know if more information is required.
Thanks in advance for the help!
Don't use available() which returns the number of bytes that can be read without blocking. It doesn't mean what all bytes were read.
If your objects are never null, you could use Object readObject() to check if all data were read from the inputstream.
Date date = null;
while( (date = (Date)save.readObject()) != null) {
worker = new Ride(date, (int)save.readObject(), (double)save.readObject(), (double)save.readObject(), (int)save.readObject());
addRide(worker.getDate(), worker.getMinutes(), 0, worker.getIF(), worker.getTss());
}
Otherwise if read values may be null, you could serialize directly the Ride object or a class containing all fields to serialize rather than unitary fields which could be null With this, the check to know if all data were read with Object readObject() could work.
Do not use available() as a condition. It just tells you whether there is some byte available for immediate reading without any delay, it does not mean the stream has reached its end.
Also you should maybe add a BufferedInputStream and BufferedOutputStream between the Object and File streams, that's almost always a good idea.
To solve your issue you could e. g. first write an integer in the save method that tells you how many objects are in the file and on load read that integer and then make a simple for loop with this amount.
Or you could throw in a PushbackInputStream in the row and then as EOF check use its read() method. It will return -1 on EOF and you can abort reading. If it returns anything else, you unread() the read byte and use the ObjectInputStream that you placed on top.
First, a simple test code:
package javaapplication23;
import java.io.IOException;
import java.util.logging.FileHandler;
public class JavaApplication23 {
public static void main(String[] args) throws IOException {
new FileHandler("./test_%u_%g.log", 10000, 100, true);
}
}
This test code creates with Java 7 only one File "test_0_0.log", no matter, how often I run the program. This is the expected behaviour because the append parameter in the constructor is set to true.
But if I run this sample in Java 8, every run creates a new File (test_0_0.log, test_0_1.log, test_0_2.log,...). I think this is a bug.
Imho, the related change in Java is this one:
## -413,18 +428,18 ##
// object. Try again.
continue;
}
- FileChannel fc;
+
try {
- lockStream = new FileOutputStream(lockFileName);
- fc = lockStream.getChannel();
- } catch (IOException ix) {
- // We got an IOException while trying to open the file.
- // Try the next file.
+ lockFileChannel = FileChannel.open(Paths.get(lockFileName),
+ CREATE_NEW, WRITE);
+ } catch (FileAlreadyExistsException ix) {
+ // try the next lock file name in the sequence
continue;
}
+
boolean available;
try {
- available = fc.tryLock() != null;
+ available = lockFileChannel.tryLock() != null;
// We got the lock OK.
} catch (IOException ix) {
// We got an IOException while trying to get the lock.
## -440,7 +455,7 ##
}
// We failed to get the lock. Try next file.
- fc.close();
+ lockFileChannel.close();
}
}
(In full: OpenJDK changeset 6123:ac22a52a732c)
I know that normally the FileHandler gets closed by the Logmanager, but this is not the case, if the system or the application crashes or the process gets killed. This is why I do not have a "close" statement in the above sample code.
Now I have two questions:
1) What is your opinion? Is this a bug? (Almost answered in the following comments and answers)
2) Do you know a workaround to get the old Java 7 behavior in Java 8? (The more important question...)
Thanks for your answers.
Closing of the FileHandler deletes the 'lck' file. If the lock file exists at all under a JDK8 version that is less than update 40 (java.util.logging), the FileHandler is going to rotate. From the OpenJDK discussion, the decision was made to always rotate if the lck file exists in addtion to if the current process can't lock it. The reason given is that it is always safer to rotate when the lock file exists. So this gets really nasty if you have rotating pattern in use with a mix of JDK versions because the JDK7 version will reuse the lock but the JDK8 version will leave it and rotate. Which is what you are doing with your test case.
Using JDK8 if I purge all log and lck files from the working directory and then run:
public static void main(String[] args) throws IOException {
System.out.println(System.getProperty("java.runtime.version"));
new FileHandler("./test_%u.log", 10000, 100, true).close();
}
I always see a file named 'test_0.log.0'. I get the same result using JDK7.
Bottom line is that is that you have to ensure your FileHandlers are closed. If it is never garbaged collected or removed from the logger tree then LogManager will close your FileHandler. Otherwise you have to close it. After that is fixed, purge all lock files before running your new patched code. Then be aware that if the JVM process crashed or is killed the lock file won't be deleted. If you have an I/O error on close your lock file won't be deleted. When the next process starts, the FileHandler will rotate.
As you point out, it is possible to use up all of the lock files on JDK8 if the above conditions occur over 100 runs. A simple test for this is to run the following code twice without deleting the log and lck files:
public static void main(String[] args) throws Exception {
System.out.println(System.getProperty("java.runtime.version"));
ReferenceQueue<FileHandler> q = new ReferenceQueue<>();
for (int i=0; i<100; i++) {
WeakReference<FileHandler> h = new WeakReference<>(
new FileHandler("./test_%u.log", 10000, 2, true), q);
while (q.poll() != h) {
System.runFinalization();
System.gc();
System.runFinalization();
Thread.yield();
}
}
}
However, the test case above won't work if JDK-6774110 is fixed correctly. The issue for this can be tracked on the OpenJDK site under RFR: 8048020 - Regression on java.util.logging.FileHandler and FileHandler webrev.
my code,
synchronized (countInfo) {
count++;
countInfo = new File(dto.findMyLocation()+"\\Properties\\countInfo"+Start.session.getId()+".txt");
BufferedWriter writer = new BufferedWriter(new FileWriter(countInfo));
writer.write(String.valueOf(count));
writer.close();
}
here countInfo is reference to a file and the method in which this piece of code is written my multiple threads. i am getting a null pointer exception at the "synchronized (countInfo)" line. i know this is because at this line it is not yet known wat countInfo is initialized to, so for this i will have to move the
`
countInfo = new File(dto.findMyLocation()+"\\Properties\\countInfo"+Start.session.getId()+".txt");
line outside of the synchronous block. but if i do that , then all the threads accessing this method of mine will crate a new file. But my goal is only one of the threads (the first thread to come into this method ) has to create a file and all other threads should just read the info in the created file. How can i achieve this?? Please help. I am new to java and Multi Threading! please enrich my knowledge!! thanks in advance!
UPDATE - an image that explains the flow.
the lines marked in YELLOW, cross session file access will never happen as i have used the session ID appended to the file name.
It will help you if you read a bit about threadsafe singlton or doublecheck locking mechanism.
For you scenario you can do something like this (may be for clear code seprate the logic for file creation and data writing):
//make countinfo volatile
public volatile File countInfo = null;
.
.
public void writeIntoFile(){
countInfo = getFile();
synchronized (countInfo) {
count++;
BufferedWriter writer = new BufferedWriter(new FileWriter(countInfo));
writer.write(String.valueOf(count));
writer.close();
}
}
public File getFile(){
if(countInfo==null){
synchronized (this){
if(countInfo==null){
countInfo = new File(dto.findMyLocation()+"\\Properties\\countInfo"+Start.session.getId()+".txt");
}
}
}
return countInfo;
}