I am trying to autograde some work submitted by others. Say an executable is called foo. It is supposed to write to standard out but in some cases, annoyingly, it instead opens a file and writes to that instead.
Is there some way (in linux) of running foo so that all its output, no matter if written to a file or to standard out, is piped to standard out?
I do have the source code for the submissions as well which are in Java.
(Any suggested tags most welcome. I wasn't sure what to choose.)
Added notes
I can't just give 0 to everyone who writes to the wrong place. I can of course give them a bad grade but that doesn't help me.
I don't know where they have written to. Some seem to write to files in subdirectories.
This answer shows how to wrap the write() system call with code of your own. You could do the same for open() (and fopen() and a few more) to trap writes to files. (Do click through to the earlier question it references, which has a good discussion as well as pointers to more information about this technique.) I suppose something similar is possible on Windows, although their terminology is bound to be slightly different. (The U*x libc buzzword is LD_PRELOAD.)
Similarly, if your submissions are in source form, you could hack your compiler to wrap or replace the corresponding Java primitives. If your students are entry-level, there are probably just a few common functions you need to identify and replace. But then, maybe you could just grep for them in the submissions, and be done. (On the other hand, beginners are more likely to stumble over a web page which somehow convinces them that they need to write their own operating system in order to write to standard output...)
Related
Foreword
I've seen some questions related to this in other languages and done quite some research on this. First, before we begin going down the rabbit hole, I must describe why I'm doing this. I'm pretty printing a binary tree and I found a way to do it that I don't think anyone else has found (can't find it anywhere on the web I've searched forever). It solves a lot of the problems related to spacing and conflicts and I'll probably provide it as an answer to prettyprint Btree questions later. Anyways, an essential part of doing the above is to be able to move the line feed (the cursor) one line up. I will also settle for moving the carriage return one line up.
I have made three EDITs to this question named "Edit 1-3" near the bottom.
My specific Windows version (might be relevant): Windows 10 v1607
OS build: 14393.2035
The problem??
The problem is that this is easily done with ANSI escape characters
ANSI escape codes: https://en.wikipedia.org/wiki/ANSI_escape_code
With "\033[A" being the specific code I want to implement (got it from a question about moving one line up in python).
The problem
So problem solved right? Oh no, not even close. The problem is, apparently windows had this """wonderful""" update that may have broke everything ANSI/console related. The below links to a question about this but with different ANSI requirements however the root problem is the same.
How to print color in console using System.out.println?
tl:dr It doesn't work for windows for some reason, you need a big library to do this or something else.
Sooo then I tried doing some in-depth research on the update itself and why this happens to be windows specific (of course) and I was led to this.
the root cause of windows problem: https://github.com/Microsoft/WSL/issues/1173
tl;dr:
Basically, the update changed a default setting for cmd.exe which is the default console that java uses. What changed was how the console setting for ANSI codes are processed for 'child applications' AKA java (stuff that uses the console but isn't console itself). The default Before the update was that child applications inherited the default of console which was to have ANSI enabled. NOW (because apparently, the previous default broke things) they changed it so that child applications DO NOT inherit the default settings from the console. This means java has to effectively set this console mode itself if java wants to have ANSI enabled. There are obviously a plethora of libraries for the java command to do that but of the good available ones, there are some issues. Firstly they require maven and I don't want maven, also I don't like NEEDING a million extra libraries to do something very simple and it adds to the number of things my code relies on (which is bad). So I'm trying to avoid simple solutions like "just use JANSI" to setConsoleMode for ANSI.
What I've tried after this
after I was done reading/understanding that I tried going into my Registry Editor and trying to change the default setting of the console (cmd.exe) to always enable ANSI even for child applications (AKA java). It had been suggested in the GitHub discussion but to my surprise the specific setting wasn't even there for me. (it was supposed to be in Computer -> HKEY_CUURENT_USER -> Console between "TrimLeadingZeros" and "WindowAlpha").
Try 2
Since I couldn't edit the console setting without big libraries full of things I don't want, I tried thinking outside the box and I messed around in java settings. I discovered where exactly java sets it's console and discovered that I could change that setting to use a different console in java. I had recently installed git and known that git bash was available so I tried that using the below question (and google in general) as a starting point.
https://superuser.com/questions/1196463/start-sh-exe-bash-with-given-path
Third time is the charm?
I couldn't get the darn thing to work. It didn't return any errors or do anything new, like at all. It was even set 'for the current project' so it should've done something different when I ran my project no? I believe part of the problem is that my root git folder is screwy (not in program files). This is probably because the computer I'm using is a work computer and maybe some setting there affects where git was installed, I don't know but what I do know is that when I changed consoles I tried this path:
C:\Users\abbotts1\AppData\Local\Programs\Git\bin\bash.exe
I also tried:
C:\Users\abbotts1\AppData\Local\Programs\Git\bash.exe
and
C:\Users\abbotts1\AppData\Local\Programs\Git\bin\sh.exe
and
"C:\Users\abbotts1\AppData\Local\Programs\Git\bin\sh.exe" --login -i
And after each change of console I tried this code:
public class ExpressionEvaluator {
public static void main(String[] args) {
System.out.println("1");
System.out.println("\033[A");
System.out.println("2");
}
}
And I always got:
1
Extra Line here
2
As the output. If I removed the 'move cursor up' ANSI println statement attempt I got:
1
2
What I wanted was:
2
1
(the whole point being, I'm able to move up a line freely)
I've also tried making them print statements and that didn't work either.
So I'm now at wits end
So here is what I want, an answer something like:
A nonintrusive way to change the setting for cmd.exe to allow child applications like java to use ANSI BY DEFAULT when ran
OR a nonintrusive library that doesn't require maven and a million other things to enable ANSI so I can run the ANSI in java in the console and get my desired output
OR a programmatic workaround that allows me to effectively print one line up without this whole ANSI thing
OR help to configure IntelliJ console so that I can actually use a different console and use ANSI to print one line up
To address the obvious concerns
Since 1 and 4 are superuser questions and 2 is 'I need a library' and offtopic I'm more talking about point 3 here. If there is no workaround for this and the answer is one of the offtopic ones just tell me there is no workaround. I don't know how else to ask this question since it is a 'programming specific problem' it just has many solutions some of which are not 'on the topic' because they aren't programming solutions. If the only answer is 'the solution isn't on topic here' then I'll go ask in the appropriate place. Let me know if I should delete this question when I go to ask it somewhere else.
What I think might work for on topic part of this
Since 3 is on topic here I'll discuss what I'm thinking:
Maybe I could make some sort of system to print only to certain arrays instead of moving up and down lines (i.e. have an array to represent the lines, traverse the tree and instead of moving up and down lines, just switching the array to print to). I don't see this as very efficient and Its kind of a waste of arrays and processing power but if its a solution I'm willing to hear it. That's all I can 'think of' but most of what I've tried is trying to get ANSI to work.
For future reference:
what is the appropriate way to ask these questions with multiple off-topic solutions/ solution questions? Is it better just not to ask them at all? Do solutions that involve questions constitute a chameleon question? I don't feel like bringing the meta effect upon myself.
Edit 1: where I'm at so far:
I've tried the whole git bash thing again and boy was it a process. The actual git bash path in the terminal needs to be in quotes with the --login -I arguments coming afterward. Ex:
C:\Git\bin\sh.exe --login -i
This setting is for the java terminal. THEN you have to set a windows environment variable named PATH to your java JDK. Go to your path environment variables (there are a million and one youtube videos for that) and set a new variable named PATH to your JDK. Ex:
C:\Program Files\Java\jdk-10.0.1\bin
This question:
How can i make gitbash find the javac command?
Goes through that process extensively
Tl;dr You have to set a windows path variable this in order for git bash to recognize the javac and java commands.
Once that's done you need to actually run your java and javac commands like you would in any terminal. BTW be careful because paths in git bash require a two '\' notation instead of the normal '\' so your source directory path might look like this:
C:\IdeaProjects\Calculator\src\
Then you just run:
javac ClassName.java
java ClassName
BUT THEN it doesn't actually print the ANSI output, it prints the raw escape characters. Also, I found out I was using the WRONG escape sequence (I had the wrong number to represent the 'esc' button since the 'esc' key is represented as some number, but I had the wrong hex number I was using like x330 or something). I also learned that the notation is like this:
'esc key hex number' + '[' + 'parameter hex values separated by commas'
so this might look like:
\x1B [ A
where the actual letters and numbers are hex value stuff (without the obvious 0x...) and the first escape hex value has an x in it (why?). Anyways, when running them in Java as strings you need to escape the escape character (duh right?) with an extra '\' so, for example, the code might look like:
System.out.println("\\\x1B [A");
I just noticed that stack overflow escapes these too so I
actually have three '\' but for you guys, it only displays two '\',
weird right? Anyways back to what I was saying
BUT STILL, the output doesn't actually work! This is where I'm at. I've done the above and I 99.9% know git bash is installed right and runs fine but when I run this:
public class ExpressionEvaluator {
public static void main(String[] args) {
System.out.println("1");
System.out.println("\\x1B [A");
System.out.println("2");
}
}
I get this in java console (not git bash??):
1
\x1B [A
2
and this in git bash:
1
\x1B [A
2
What I actually want is:
2
1
Because the ANSI escape character is supposed to move my cursor/linefeed/whatever, one line upwards. The same thing happens if I run the above code but instead use the ANSI code: "\x1B [F". Only raw ANSI is output. I'm pretty sure git bash was supposed to be 'natively ANSI aware' and I've seen people say that on websites so I don't know why it isn't working.
And I still don't know for sure if those are two separate console outputs or the same console output. I really can't tell so if anyone wants to leave a comment saying 'yeah its the same dummy', I'd appreciate it because I can't find a definitive source out on the web that it is. I think it is but nothing other than the console setting in IntelliJ indicates that as true.
I've heard rumors of a TERM variable that needs to be set or otherwise manipulated on windows. I've checked myself what it is using:
echo $TERM
in git bash and I got back:
cygwin
So I don't exactly know if that's good or bad because I've literally gone through all the search terms you can think of and they all lead to the same basic page of results for 'git bash colors not working' and most of them involve windows 7 (don't have it) installing maven/jansi (don't want or shouldn't need it) or some other language that isn't Java and using some other IDE which isn't IntelliJ. Some pages that do have my specific requirements have said something about TERM supposed to be xterm or some other thing like xterm-256 or something for 'color' output, something like this. I am so unfamiliar with this stuff so I don't even know where to begin.
Too long give us a tl;dr
I need to know why git bash is printing raw ANSI instead of actually using the ANSI.
what I know
I'm using git bash with IntelliJ, 99.9% sure I have my path set correctly, I am able to run my java class from git bash, I have it set as the IntelliJ terminal and I currently have the windows TERM variable set to cygwin.
What I don't know
I don't know what TERM needs to be and can't find it on the web, I can't tell if the IntelliJ console that appears when I click the green arrow 'run' is the same as the git bash console, and I can't figure out if some other thing is preventing me from actually interpreting the ANSI.
What I need
I need a simple explanation, something straight out of r/ELI5 of what is wrong with git bash if anything and how to fix it. If it can't be explained simply or nothing is wrong then maybe I'll try another supposedly 'natively ANSI aware' terminal. I think Powershell was another option that was listed. My best bet is that the TERM variable needs to be something else, or git bash was never really natively ANSI aware and capable, to begin with. I've seen other questions with the same problem for colors but their fixes are for older versions/different languages and things or they don't actually work. I have yet to find a good page for 'git bash outputs raw ANSI in IntelliJ' and I've used variations of those exact words for hours now. All I can get is long GitHub discussions on the 'bugs' related to this and they confuse me, don't lead to solutions, and may be active or just don't contain any resolution.
Edit 2
After doing some more research I've learned that my previous escape code was correct:
\033[A
\x1B[A
should be similar.
I also learned that it isn't the console I'm using that's the problem, its Windows itself. I now know this because I've tried compiling and running on cmd.exe , git bash and powershell. To change the default setting talked about earlier (consoles not supporting ansi for child applications) you have to EXPLICITLY enable it via the program itself rather than rely on a console or something.
link to a question that explains this in Python here:
How to use the new support for ANSI escape sequences in the Windows 10 console?
tl;dr
The method that they use is something called getConsoleMode and setConsoleMode and VIRTUALTERMINALPROCESSING flags. Apparently you need to use these to actually set the console mode to support ANSI. I don't currently know if those are things that are in java hidden somewhere or (likely) something that needs to be added to the base java libraries. I'm going to try and figure out how they actually get the ctypes thing from that question (seems to be what they're importing to sue these methods) and get the methods I need. Once I do I'll post that as an answer unless one of you figures it out before I do and can explain it better.
Apparently, the escape sequences work fine if you only use them from the console but If you use a "child application" then they don't work. So at the very least we now for sure know the root cause of the problem.
Edit 3
Found this which is highly relevant especially the console virtual terminal sequences section (lefthand navbar):
https://learn.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences
This which, near the bottom has a whole C-implementation of how to enable the console to read ANSI. Apparently, this doesn't need libraries at all but the process to actually change the console defaults to use code like this requires sysadmin privileges, intimate knowledge of the program files and a whole host of other things (at least if you're before the windows 10 update when color support was changed). Now it's still disabled by default but can be enabled. I don't yet know how to try ANSI from the console directly. I've tried multiple
echo \x1B[(insert ANSI code here)
but none of the commands seem to work in ANY terminal (cmd, git bash, powershell). They just return the raw code
\x1B[(whatever the ANSI code was)
I obviously am new to the console so I might be using the wrong command, if so feel free to enlighten me but the examples I've seen use echo. ANYWAYS, I thought that calling the ANSI directly from terminal was supposed to work since its supposedly enabled by default just not child applications (post windows 10 update) BUT maybe its not, maybe its disabled by default and even when enabled its still disabled for child applications (Java) unless explicitly changed in said child application (java). I'm going to try to see if it's possible to enable ANSI directly from the console, or if the linked C-code needs to be directly ported to Java or ran in the console just to work. The problem is I don't know how to get the imports/includes that the C-code uses and use it for code in java. I'd rather not just accept a coded 'solution' in C and try and use that alongside java code, I'd rather translate, understand it better and have my own code that does the same thing.
Another option I've been told is something called ANSICON which is like some sort of plugin that you install in the console with the -i flag and that's supposed to enable ANSI at least in-console. I found this
https://community.liferay.com/blogs/-/blogs/enable-ansi-colors-in-windows-command-prompt
the above explains that process in a little more detail.
My specific windows version and the version of the update
Another thing I learned was that the specific version of windows 10 that the 'update' that changed ANSI console behavior was like windows 10 v1151 or something, I'll try and find the webpage to source this directly but I currently have windows 10 v1607 so I think I should be good. Included in that was the actual OS build which I have 14393.2035 and I think that was identified as a particularly intermediate update to this process in one of my previous links (I believe its the one with the whole GitHub discussion about the update, you can find it here: https://github.com/Microsoft/WSL/issues/1173). I have a work computer so I can't really make system updates because I'm not an administrator and I doubt IT would let an intern go around updating work computers.
Anyways, I'll continue on trying to see if I can get the C-code to java, I'll test it and then try and post an answer. If ya'll are ahead of me let me know.
The python way you reference in "Edit 2" (https://stackoverflow.com/a/36760881/309816) simply invokes Windows-specific native code (kernel32, which is non-portable) to "fix" this.
I suppose you are OK with that and want to do the same in Java (i.e. invoke kernel32 when you detect Windows)...
A very lightweight library for achieving the same in Java is JNA which has out-of-the-box wrapper for kernel32 (see: https://java-native-access.github.io/jna/4.2.1/com/sun/jna/platform/win32/Kernel32.html)
You seem to be after this method: https://java-native-access.github.io/jna/4.2.1/com/sun/jna/platform/win32/Wincon.html#SetConsoleMode-com.sun.jna.platform.win32.WinNT.HANDLE-int-
Hope this works for you.
EDIT: technically, you only need jna.jar (see getting started here: https://github.com/java-native-access/jna/blob/master/www/GettingStarted.md), but I would suggest you also use jna-platform.jar so that you don't need write the code that generates the mappings for kernel32 at runtime yourself.
Home for JNA: https://github.com/java-native-access/jna
I think adding 1 (or 2 if you add jna-platform) jars that have a very specific scope (doing native calls without all the JNI preparation overhead) is lightweight enough. You don't need to generate any headers, or change anything in your compilation process. It will just work by adding those jars to your classpath.
You should also clarify in your question that this is about Windows. Maybe edit the title to: "Printing up a line in java console (reverse of '\n') on Windows" as this is really about a platform-specific concern that you want to address with Java.
Currently I am using file.delete() but it is showing a security risk for this as files deleted like this can be recovered by different means. So please provide me a correct way to delete a file. The security risk depicted here is provided by a testing tool called Quixxi and it checks for any vulnerability in app.
The reason a "deleted" file is recoverable is because a delete operation simply unlinks the file in the filesystem, so the directory no longer considers that file part of it. The contents on disk (or whatever storage) still exist on that device.
If you want to guarantee the contents can never be recovered, you have to overwrite the contents first. There are no built-in functions to do this - you'd have to find a library or write the code yourself. Typically you'd write something like all 0s over the file (make sure to flush to media), write all 1s, write a pattern of 01 repeating, 10 repeating, something like that. After you've written with garbage patterns to media (flush) a few times, then you issue the delete.
Not possible in JRE, unfortunately. The JVM is not designed for that, and you need OS-dependent utilities.
The answer by user1676075 contains a mistake. Let's go by steps.
As pointed out already, Java's File.delete method only unlinks the file leaving its contents on disk. It actually invokes the underlying OS APIs to perform this unlink operation.
The problem occurs when you want to overwrite contents in Java.
Java can open a file for overwrite, but will leverage OS utils to do so. And the OS will likely:
Unlink the allocated space on disk
Link the file to a new free area of disk
The result is that you are now writing tons of zeroes... somewhere else!!!
And even if you managed to write zeroes on the same sectors used by the original file, Gutmann method exists for a reason. Gutmann utilities require root/Administrator (Super User) permissions and direct DMA access to precisely control where the writes have to occur.
And with SSDs, things changes. Actually, it might get easier! At this point, I should provide source for SSDs having a CLEAR instructions to replace a sector with zeroes and that privacy-savy disk controllers do that. But maybe pretend you have read nothing.
This will be a sufficient answer for now, because we have demonstrated that there is no out-of-the-box and straightforward way to securely clear a file in Java.
What Java allows, and is called Java Native Interfaces (please also see Java Native Access), is to call native code from Java. So, you got your Gutmann tool in C++ ready? Are you running root? You can write code to invoke Gutmann-ish erasure from Java, but that's a whole other point.
Never tried, but surely feasible
In my Ruby on Rails app I'm having a routine that writes to a file (through a java application) and then reads the written file.
write_to_file(file.path, data)
read_file(file.path)
Most of the time this works just fine. But some times it looks like the file write had not happened but there were no errors either. And when I retry the routine with the same data it has worked each time.
I have begun to think if the file write happens asynchronously and the file is actually read before the data is written to the disk. Would this be possible?
write_to_file calls a java application through a socket connection that takes care of the writing. Java application returns a simple json string back to Rails.
This question is really "what does the Java code do?" and is not really a Ruby question. It's not even really a Java question, because the Java language allows (of course) any kind of implementation.
The Java code could certainly be returning before the file is available for reading. We have no idea. It could be posting a request to a queue, and then returning, for example.
The Java code is what you need to look at. If you don't want to bother with that, you could always do something like this:
sleep 0.01 until File.readable?(file.path)
This is a bit crude and there are more elegant ways to do this, but it would work.
You might be experiencing file buffering where small amounts of data aren't written to the file unless it's flushed. I'm not sure what interface you're using here, but the flush method is intended to deal with this exact situation.
I am currently trying to have Logstash work on Solaris with the File Input method. But I am encountering some bugs (see LOGSTASH-665). After digging a lot, it appears that native support for File.stat on my system (SunOS 5.10, JDK 1.6.0_21, 32 bit) is totally deficient, so I am looking for a way to properly handle it.
Specifically I want to access the inode information. Based on the metadata I can gather about the file (like its path and whatever is available on solaris), I want to calculate a number which is unique for that file, and which changes when the file is replaced by another file which has the same name. At first I thought about simply using a hash of the file path and used this function as a replacement, but indeed, when the file is rolled over the number does not change, so I need to also access the ctime information...
..Or make a system call to get the ls -li result to get the real inode number by another way.
Problem is that I never used ruby before, I am more used to java, so I am struggling to find a solution. Every suggestion will be appreciated.
The best solution I know of is to wrap the native call using JNI or JNA.
There do appear to be a couple of projects that have done this, although I haven't used either of them. See this question: Is there a Java library of Unix functions?
So I have a Samba file server on which my Java app needs to write some files. The thing is that there is also another php application (if a php script is even considered an application) that is aggressively pulling the same directory for new files.
Sometimes, the php script is pulling the file before my Java app is done writing it completely to the disk. Here is a little bit of ascii art to help visualize what I currently have (but doesn't work):
Samba share
/foo (my java app drops file here)
/bar (the directory that the php is pulling)
What I'm currently doing is when the file meets some criterias, it's being moved to /bar and then picked up by the php for more processing. I've tried different thing such has setting the file non writable and non readable before calling renameTo.
I've looked a little bit at FileLocks but it doesn't seem to be able to lock future files. So I am wondering what kind of possiblities I have here? What could I use to lock the file from being picked up before it's fully written without touching the php (because, well, it's php and I don't really have the right to modify it right now).
Thanks
Edit 1
I've got some insight on what the php script is really doing if it can help in any way.
It's reading the directory file in loop (using readdir without sleeping).
As soon as it finds a filename other than "." and "..", it calls file_get_contents and that's where it fails because the file is not completely written to disk (or not even there since the Java code might not even had time to write it between the readdir and file_get_contents.
Edit 2
This Java application is replacing an old php script. When they implemented it, they had the same problem I'm having right now. They solved it by writing the new file in /bar/tmp(with file_put_contents) and then use rename to move it to bar (it looks like rename is supposed to be atomic). And it's been working fine so far. I can't and won't believe that Java can't do something better than what php does...
I think this is due to the fact read locks are shared (multiple process can apply read locks to the same file and read it together).
One approach you can do is to create a separate temporary lock file (eg: /bar/file1.lock) while /bar/file1 hasn't finished copying. Delete the lock file as soon as the file copying is finished.
Then alter the php code to ensure the file isn't being locked before it reads.
You mentioned that you tried FileLock, but keep in mind the disclaimer in the javadoc for that method:
Whether or not a lock actually prevents another program from accessing
the content of the locked region is system-dependent and therefore
unspecified. The native file-locking facilities of some systems are
merely advisory, meaning that programs must cooperatively observe a
known locking protocol in order to guarantee data integrity.
You also mentioned you are using File.renameTo, which also has some caveats (mentioned in the javadoc):
Many aspects of the behavior of this method are inherently
platform-dependent: The rename operation might not be able to move a
file from one filesystem to another, it might not be atomic, and it
might not succeed
Instead of File.renameTo, Try Files.move with the ATOMIC_MOVE option. You'll have to catch AtomicMoveNotSupportedException and possibly fall back to some alternative workaround in case an atomic move is not possible.
You could create a hardlink with Files.createLink(Paths.get('/foo/myFile'), 'Paths.get('/bar/myFile')) then delete the original directory entry (in this example, /foo/myFile.
Failing that, a simple workaround that doesn't require modification to the PHP is to use a shell command or system call to move the file from /foo to /bar. You could, for example, use ProcessBuilder to call mv, or perhaps call ln to create a symlink or hardlink in /bar. You might still have the same problem with mv if /foo and /bar are on different filesystems.
If you have root privileges on the server, you could also try implementing mandatory file locking. I found an example in C, but you could call the C program from Java or adapt the example to Java using JNA (or JNI if you want to punish yourself).