In my Ruby on Rails app I'm having a routine that writes to a file (through a java application) and then reads the written file.
write_to_file(file.path, data)
read_file(file.path)
Most of the time this works just fine. But some times it looks like the file write had not happened but there were no errors either. And when I retry the routine with the same data it has worked each time.
I have begun to think if the file write happens asynchronously and the file is actually read before the data is written to the disk. Would this be possible?
write_to_file calls a java application through a socket connection that takes care of the writing. Java application returns a simple json string back to Rails.
This question is really "what does the Java code do?" and is not really a Ruby question. It's not even really a Java question, because the Java language allows (of course) any kind of implementation.
The Java code could certainly be returning before the file is available for reading. We have no idea. It could be posting a request to a queue, and then returning, for example.
The Java code is what you need to look at. If you don't want to bother with that, you could always do something like this:
sleep 0.01 until File.readable?(file.path)
This is a bit crude and there are more elegant ways to do this, but it would work.
You might be experiencing file buffering where small amounts of data aren't written to the file unless it's flushed. I'm not sure what interface you're using here, but the flush method is intended to deal with this exact situation.
Related
Currently I am using file.delete() but it is showing a security risk for this as files deleted like this can be recovered by different means. So please provide me a correct way to delete a file. The security risk depicted here is provided by a testing tool called Quixxi and it checks for any vulnerability in app.
The reason a "deleted" file is recoverable is because a delete operation simply unlinks the file in the filesystem, so the directory no longer considers that file part of it. The contents on disk (or whatever storage) still exist on that device.
If you want to guarantee the contents can never be recovered, you have to overwrite the contents first. There are no built-in functions to do this - you'd have to find a library or write the code yourself. Typically you'd write something like all 0s over the file (make sure to flush to media), write all 1s, write a pattern of 01 repeating, 10 repeating, something like that. After you've written with garbage patterns to media (flush) a few times, then you issue the delete.
Not possible in JRE, unfortunately. The JVM is not designed for that, and you need OS-dependent utilities.
The answer by user1676075 contains a mistake. Let's go by steps.
As pointed out already, Java's File.delete method only unlinks the file leaving its contents on disk. It actually invokes the underlying OS APIs to perform this unlink operation.
The problem occurs when you want to overwrite contents in Java.
Java can open a file for overwrite, but will leverage OS utils to do so. And the OS will likely:
Unlink the allocated space on disk
Link the file to a new free area of disk
The result is that you are now writing tons of zeroes... somewhere else!!!
And even if you managed to write zeroes on the same sectors used by the original file, Gutmann method exists for a reason. Gutmann utilities require root/Administrator (Super User) permissions and direct DMA access to precisely control where the writes have to occur.
And with SSDs, things changes. Actually, it might get easier! At this point, I should provide source for SSDs having a CLEAR instructions to replace a sector with zeroes and that privacy-savy disk controllers do that. But maybe pretend you have read nothing.
This will be a sufficient answer for now, because we have demonstrated that there is no out-of-the-box and straightforward way to securely clear a file in Java.
What Java allows, and is called Java Native Interfaces (please also see Java Native Access), is to call native code from Java. So, you got your Gutmann tool in C++ ready? Are you running root? You can write code to invoke Gutmann-ish erasure from Java, but that's a whole other point.
Never tried, but surely feasible
I need to execute a java simulation program for a very long time, many hours or maybe days, and i wish i could do it on a server.
I've been heard about Cloud Computing, and i'm searching a free platform, or a very cheap one.
For example, i found Oracle Cloud, but i am open to any type of solution.
On the link there are several points to follow to deploy an application, it seems a bit complicated and you also have to install Maven.
Do you think there is a simpler solution, or that this one could be my best try?
I mean, my program consists on a few .class files, i wish i could edit/compile/run the main class very easily, like with a kind of shell or cmd.
Unfortunately, i know very little about web programming, so i don't know even if this would be possible.
However, assuming i can launch my program, and log in after a very long time, will i be able to read the results?
Or is it possible to write a text file to read the results later?
i wish i could do it on a server. I've been heard about Cloud Computing
A possible solution is using AWS Lambda. You only provide your Java code in a ZIP file and it will run in a "server-less" environment. What this means is that you don't have to setup a server yourself instead AWS will manage everything for you.
i'm searching a free platform, or a very cheap one.
It's not for free, but it's pretty cheap though: https://aws.amazon.com/lambda/pricing/
However, assuming i can launch my program, and log in after a very long time, will i be able to read the results? Or is it possible to write a text file to read the results later?
I would not recommend writing it to a text file, instead look at solutions such as S3 Buckets or Elasticsearch with Kibana
I am trying to autograde some work submitted by others. Say an executable is called foo. It is supposed to write to standard out but in some cases, annoyingly, it instead opens a file and writes to that instead.
Is there some way (in linux) of running foo so that all its output, no matter if written to a file or to standard out, is piped to standard out?
I do have the source code for the submissions as well which are in Java.
(Any suggested tags most welcome. I wasn't sure what to choose.)
Added notes
I can't just give 0 to everyone who writes to the wrong place. I can of course give them a bad grade but that doesn't help me.
I don't know where they have written to. Some seem to write to files in subdirectories.
This answer shows how to wrap the write() system call with code of your own. You could do the same for open() (and fopen() and a few more) to trap writes to files. (Do click through to the earlier question it references, which has a good discussion as well as pointers to more information about this technique.) I suppose something similar is possible on Windows, although their terminology is bound to be slightly different. (The U*x libc buzzword is LD_PRELOAD.)
Similarly, if your submissions are in source form, you could hack your compiler to wrap or replace the corresponding Java primitives. If your students are entry-level, there are probably just a few common functions you need to identify and replace. But then, maybe you could just grep for them in the submissions, and be done. (On the other hand, beginners are more likely to stumble over a web page which somehow convinces them that they need to write their own operating system in order to write to standard output...)
So I've been looking around for a way to write from an Arduino directly onto a file on the PC, and basically I've found out there's no native way to do so. I wanted to do this in order to then read the file from a C++/Java program, and use the information in it. I also wanted to do this in real-time at some point, so it would be kind of like sending information from the Arduino over to the Java/C++ program for processing.
However, I've seen multiple people state on other forums that you can link the Serial output to some program running on the PC, and then use that program to write the output to a file. However, each time, they neglect to write out how exactly to do this.
The main purpose I wanted to write from the Arduino directly to a file was to read this file from another (Java/C++) program, so the above would be great for me. So how can I get the Serial output into a Java (much more preferably, as I might want to use Swing later) or C++ program, to then use this information in the program itself, or write it to a file? Real-time sending would be a great help.
If the above isn't possible, MATLAB might do, but to be clear, I would much rather be able to interface with Java/C++. Or both Java and MATLAB.
EDIT: To be more specific about what exactly I'd like to do, it is to sort of 'trigger' the Java program to read from the Serial output when a new line has been written (so it reads each line separately) and store it in a string in the Java program, then process it, all at once, and then sleep until another new line is written to the Serial port.
The following Links show you, how you can implement the serial communication in Java and C++.
Java :
http://playground.arduino.cc/interfacing/java
C++ :
http://playground.arduino.cc/Interfacing/CPPWindows
If you want to write the data stream to a local file, you can do that, for instance, with ofstream (C++) or PrintWriter (Java).
Furthermore, there are a several additional libraries for other programming languages such as c# (cmdMessenger).
Just in case anyone was looking for the easiest way to do this, it is hands down to ignore C++ and Java and use MATLAB.
There's a great short tutorial at AllAboutEE that I used and it solved most of my problems. Instead of plotting the data at the end, just use fprintf in MATLAB to output the data to a file.
So I have a Samba file server on which my Java app needs to write some files. The thing is that there is also another php application (if a php script is even considered an application) that is aggressively pulling the same directory for new files.
Sometimes, the php script is pulling the file before my Java app is done writing it completely to the disk. Here is a little bit of ascii art to help visualize what I currently have (but doesn't work):
Samba share
/foo (my java app drops file here)
/bar (the directory that the php is pulling)
What I'm currently doing is when the file meets some criterias, it's being moved to /bar and then picked up by the php for more processing. I've tried different thing such has setting the file non writable and non readable before calling renameTo.
I've looked a little bit at FileLocks but it doesn't seem to be able to lock future files. So I am wondering what kind of possiblities I have here? What could I use to lock the file from being picked up before it's fully written without touching the php (because, well, it's php and I don't really have the right to modify it right now).
Thanks
Edit 1
I've got some insight on what the php script is really doing if it can help in any way.
It's reading the directory file in loop (using readdir without sleeping).
As soon as it finds a filename other than "." and "..", it calls file_get_contents and that's where it fails because the file is not completely written to disk (or not even there since the Java code might not even had time to write it between the readdir and file_get_contents.
Edit 2
This Java application is replacing an old php script. When they implemented it, they had the same problem I'm having right now. They solved it by writing the new file in /bar/tmp(with file_put_contents) and then use rename to move it to bar (it looks like rename is supposed to be atomic). And it's been working fine so far. I can't and won't believe that Java can't do something better than what php does...
I think this is due to the fact read locks are shared (multiple process can apply read locks to the same file and read it together).
One approach you can do is to create a separate temporary lock file (eg: /bar/file1.lock) while /bar/file1 hasn't finished copying. Delete the lock file as soon as the file copying is finished.
Then alter the php code to ensure the file isn't being locked before it reads.
You mentioned that you tried FileLock, but keep in mind the disclaimer in the javadoc for that method:
Whether or not a lock actually prevents another program from accessing
the content of the locked region is system-dependent and therefore
unspecified. The native file-locking facilities of some systems are
merely advisory, meaning that programs must cooperatively observe a
known locking protocol in order to guarantee data integrity.
You also mentioned you are using File.renameTo, which also has some caveats (mentioned in the javadoc):
Many aspects of the behavior of this method are inherently
platform-dependent: The rename operation might not be able to move a
file from one filesystem to another, it might not be atomic, and it
might not succeed
Instead of File.renameTo, Try Files.move with the ATOMIC_MOVE option. You'll have to catch AtomicMoveNotSupportedException and possibly fall back to some alternative workaround in case an atomic move is not possible.
You could create a hardlink with Files.createLink(Paths.get('/foo/myFile'), 'Paths.get('/bar/myFile')) then delete the original directory entry (in this example, /foo/myFile.
Failing that, a simple workaround that doesn't require modification to the PHP is to use a shell command or system call to move the file from /foo to /bar. You could, for example, use ProcessBuilder to call mv, or perhaps call ln to create a symlink or hardlink in /bar. You might still have the same problem with mv if /foo and /bar are on different filesystems.
If you have root privileges on the server, you could also try implementing mandatory file locking. I found an example in C, but you could call the C program from Java or adapt the example to Java using JNA (or JNI if you want to punish yourself).