We Keep Coding

Java UTF-8/UTF-16LE Input Conversion

I am trying to get my java console application to be able to take commands both from Windows PS and Linux Terminal while being run. It isn't working properly though because of a special character (like äüö etc) in a command.
When the program is being executed in Linux everything works fine because of UTF-8. My simple program just takes an input via a scanner and checks whether or not it matches a specific String. My expected command is read from a .txt file and is in UTF-8 format.
I tried to
Arrays.equals(userInputAsString.getBytes(StandardCharsets.UTF_8), expectedInputToExecuteCommandAsString.getBytes(StandardCharsets.UTF_8))
But unfortunately it doesn't work while being run in PS.
I hope someone may be willing to help me!

Is there a way to verify if a file is a special block file in Java?

I have a special use case where I need to verify if a file is a special block file (only if my java program is run on Linux). My code can already detect if it's ran on Windows or Linux, so if it's windows I simply invalidate the verification.
For Linux, I was reading about the different FileAttributeView interfaces; and the first thing that popped up was to use the PosixFileAttributeView (since block and char device files are Linux-specific), but after looking further into its Javadoc, it only gives access to the different RWX (read, write, execute permissions relative to the owner, group and others) but nothing about special block files or other file attributes. On the other hand, the BasicFileAttributeView is supported by all filesystems and has an instance method called isOther(), this could work but there has to be some other way that's more specific than just knowing that a file is not a regular file, directory or symlink. Knowing that it's something other is just too vague.. Any help is deeply appreciated.

Wow, this is actually harder than I imagined!
To ground this in Linux, what you're looking for is the st_mode field from stat() system call (in C-world). Its different values indicate whether the file is a character device, block device, directory, etc... The first character on each line of the return from ls -la represent that field: b for block device, d for directory, - for regular file, and on. For example, if you run ls -la /dev/sda you'll probably get something like brw-rw---- 1..., because sda is a block device. The first line of ls -la /opt will be dr-xr-xrx... since opt is a directory.
(https://linux.die.net/man/2/stat, for reference)
Poking around the Javadocs, I didn't find a file class that allows direct access to this st_mode field. Like you mentioned, there are methods to determine if the file is a directory, file, or other, but this is not sufficient to distinguish between all the options and figure out if it's a block device.
I did find this project which looks like it lets you get at that stat() struct in Java. It is a fair bit of overhead. Perhaps you could extract the bits you need for this application, but it might not be trivial.
The specific method you'd need to call is isBlockDev()
Hope that helps a little bit, curious to see if you figure something out, or if there's a better answer out there!

Printing up a line in java console (reverse of '\n')

Foreword
I've seen some questions related to this in other languages and done quite some research on this. First, before we begin going down the rabbit hole, I must describe why I'm doing this. I'm pretty printing a binary tree and I found a way to do it that I don't think anyone else has found (can't find it anywhere on the web I've searched forever). It solves a lot of the problems related to spacing and conflicts and I'll probably provide it as an answer to prettyprint Btree questions later. Anyways, an essential part of doing the above is to be able to move the line feed (the cursor) one line up. I will also settle for moving the carriage return one line up.
I have made three EDITs to this question named "Edit 1-3" near the bottom.
My specific Windows version (might be relevant): Windows 10 v1607
OS build: 14393.2035
The problem??
The problem is that this is easily done with ANSI escape characters
ANSI escape codes: https://en.wikipedia.org/wiki/ANSI_escape_code
With "\033[A" being the specific code I want to implement (got it from a question about moving one line up in python).
The problem
So problem solved right? Oh no, not even close. The problem is, apparently windows had this """wonderful""" update that may have broke everything ANSI/console related. The below links to a question about this but with different ANSI requirements however the root problem is the same.
How to print color in console using System.out.println?
tl:dr It doesn't work for windows for some reason, you need a big library to do this or something else.
Sooo then I tried doing some in-depth research on the update itself and why this happens to be windows specific (of course) and I was led to this.
the root cause of windows problem: https://github.com/Microsoft/WSL/issues/1173
tl;dr:
Basically, the update changed a default setting for cmd.exe which is the default console that java uses. What changed was how the console setting for ANSI codes are processed for 'child applications' AKA java (stuff that uses the console but isn't console itself). The default Before the update was that child applications inherited the default of console which was to have ANSI enabled. NOW (because apparently, the previous default broke things) they changed it so that child applications DO NOT inherit the default settings from the console. This means java has to effectively set this console mode itself if java wants to have ANSI enabled. There are obviously a plethora of libraries for the java command to do that but of the good available ones, there are some issues. Firstly they require maven and I don't want maven, also I don't like NEEDING a million extra libraries to do something very simple and it adds to the number of things my code relies on (which is bad). So I'm trying to avoid simple solutions like "just use JANSI" to setConsoleMode for ANSI.
What I've tried after this
after I was done reading/understanding that I tried going into my Registry Editor and trying to change the default setting of the console (cmd.exe) to always enable ANSI even for child applications (AKA java). It had been suggested in the GitHub discussion but to my surprise the specific setting wasn't even there for me. (it was supposed to be in Computer -> HKEY_CUURENT_USER -> Console between "TrimLeadingZeros" and "WindowAlpha").
Try 2
Since I couldn't edit the console setting without big libraries full of things I don't want, I tried thinking outside the box and I messed around in java settings. I discovered where exactly java sets it's console and discovered that I could change that setting to use a different console in java. I had recently installed git and known that git bash was available so I tried that using the below question (and google in general) as a starting point.
https://superuser.com/questions/1196463/start-sh-exe-bash-with-given-path
Third time is the charm?
I couldn't get the darn thing to work. It didn't return any errors or do anything new, like at all. It was even set 'for the current project' so it should've done something different when I ran my project no? I believe part of the problem is that my root git folder is screwy (not in program files). This is probably because the computer I'm using is a work computer and maybe some setting there affects where git was installed, I don't know but what I do know is that when I changed consoles I tried this path:
C:\Users\abbotts1\AppData\Local\Programs\Git\bin\bash.exe
I also tried:
C:\Users\abbotts1\AppData\Local\Programs\Git\bash.exe
and
C:\Users\abbotts1\AppData\Local\Programs\Git\bin\sh.exe
and
"C:\Users\abbotts1\AppData\Local\Programs\Git\bin\sh.exe" --login -i
And after each change of console I tried this code:
public class ExpressionEvaluator {
public static void main(String[] args) {
System.out.println("1");
System.out.println("\033[A");
System.out.println("2");
}
}
And I always got:
1
Extra Line here
2
As the output. If I removed the 'move cursor up' ANSI println statement attempt I got:
1
2
What I wanted was:
2
1
(the whole point being, I'm able to move up a line freely)
I've also tried making them print statements and that didn't work either.
So I'm now at wits end
So here is what I want, an answer something like:
A nonintrusive way to change the setting for cmd.exe to allow child applications like java to use ANSI BY DEFAULT when ran
OR a nonintrusive library that doesn't require maven and a million other things to enable ANSI so I can run the ANSI in java in the console and get my desired output
OR a programmatic workaround that allows me to effectively print one line up without this whole ANSI thing
OR help to configure IntelliJ console so that I can actually use a different console and use ANSI to print one line up
To address the obvious concerns
Since 1 and 4 are superuser questions and 2 is 'I need a library' and offtopic I'm more talking about point 3 here. If there is no workaround for this and the answer is one of the offtopic ones just tell me there is no workaround. I don't know how else to ask this question since it is a 'programming specific problem' it just has many solutions some of which are not 'on the topic' because they aren't programming solutions. If the only answer is 'the solution isn't on topic here' then I'll go ask in the appropriate place. Let me know if I should delete this question when I go to ask it somewhere else.
What I think might work for on topic part of this
Since 3 is on topic here I'll discuss what I'm thinking:
Maybe I could make some sort of system to print only to certain arrays instead of moving up and down lines (i.e. have an array to represent the lines, traverse the tree and instead of moving up and down lines, just switching the array to print to). I don't see this as very efficient and Its kind of a waste of arrays and processing power but if its a solution I'm willing to hear it. That's all I can 'think of' but most of what I've tried is trying to get ANSI to work.
For future reference:
what is the appropriate way to ask these questions with multiple off-topic solutions/ solution questions? Is it better just not to ask them at all? Do solutions that involve questions constitute a chameleon question? I don't feel like bringing the meta effect upon myself.
Edit 1: where I'm at so far:
I've tried the whole git bash thing again and boy was it a process. The actual git bash path in the terminal needs to be in quotes with the --login -I arguments coming afterward. Ex:
C:\Git\bin\sh.exe --login -i
This setting is for the java terminal. THEN you have to set a windows environment variable named PATH to your java JDK. Go to your path environment variables (there are a million and one youtube videos for that) and set a new variable named PATH to your JDK. Ex:
C:\Program Files\Java\jdk-10.0.1\bin
This question:
How can i make gitbash find the javac command?
Goes through that process extensively
Tl;dr You have to set a windows path variable this in order for git bash to recognize the javac and java commands.
Once that's done you need to actually run your java and javac commands like you would in any terminal. BTW be careful because paths in git bash require a two '\' notation instead of the normal '\' so your source directory path might look like this:
C:\IdeaProjects\Calculator\src\
Then you just run:
javac ClassName.java
java ClassName
BUT THEN it doesn't actually print the ANSI output, it prints the raw escape characters. Also, I found out I was using the WRONG escape sequence (I had the wrong number to represent the 'esc' button since the 'esc' key is represented as some number, but I had the wrong hex number I was using like x330 or something). I also learned that the notation is like this:
'esc key hex number' + '[' + 'parameter hex values separated by commas'
so this might look like:
\x1B [ A
where the actual letters and numbers are hex value stuff (without the obvious 0x...) and the first escape hex value has an x in it (why?). Anyways, when running them in Java as strings you need to escape the escape character (duh right?) with an extra '\' so, for example, the code might look like:
System.out.println("\\\x1B [A");
I just noticed that stack overflow escapes these too so I
actually have three '\' but for you guys, it only displays two '\',
weird right? Anyways back to what I was saying
BUT STILL, the output doesn't actually work! This is where I'm at. I've done the above and I 99.9% know git bash is installed right and runs fine but when I run this:
public class ExpressionEvaluator {
public static void main(String[] args) {
System.out.println("1");
System.out.println("\\x1B [A");
System.out.println("2");
}
}
I get this in java console (not git bash??):
1
\x1B [A
2
and this in git bash:
1
\x1B [A
2
What I actually want is:
2
1
Because the ANSI escape character is supposed to move my cursor/linefeed/whatever, one line upwards. The same thing happens if I run the above code but instead use the ANSI code: "\x1B [F". Only raw ANSI is output. I'm pretty sure git bash was supposed to be 'natively ANSI aware' and I've seen people say that on websites so I don't know why it isn't working.
And I still don't know for sure if those are two separate console outputs or the same console output. I really can't tell so if anyone wants to leave a comment saying 'yeah its the same dummy', I'd appreciate it because I can't find a definitive source out on the web that it is. I think it is but nothing other than the console setting in IntelliJ indicates that as true.
I've heard rumors of a TERM variable that needs to be set or otherwise manipulated on windows. I've checked myself what it is using:
echo $TERM
in git bash and I got back:
cygwin
So I don't exactly know if that's good or bad because I've literally gone through all the search terms you can think of and they all lead to the same basic page of results for 'git bash colors not working' and most of them involve windows 7 (don't have it) installing maven/jansi (don't want or shouldn't need it) or some other language that isn't Java and using some other IDE which isn't IntelliJ. Some pages that do have my specific requirements have said something about TERM supposed to be xterm or some other thing like xterm-256 or something for 'color' output, something like this. I am so unfamiliar with this stuff so I don't even know where to begin.
Too long give us a tl;dr
I need to know why git bash is printing raw ANSI instead of actually using the ANSI.
what I know
I'm using git bash with IntelliJ, 99.9% sure I have my path set correctly, I am able to run my java class from git bash, I have it set as the IntelliJ terminal and I currently have the windows TERM variable set to cygwin.
What I don't know
I don't know what TERM needs to be and can't find it on the web, I can't tell if the IntelliJ console that appears when I click the green arrow 'run' is the same as the git bash console, and I can't figure out if some other thing is preventing me from actually interpreting the ANSI.
What I need
I need a simple explanation, something straight out of r/ELI5 of what is wrong with git bash if anything and how to fix it. If it can't be explained simply or nothing is wrong then maybe I'll try another supposedly 'natively ANSI aware' terminal. I think Powershell was another option that was listed. My best bet is that the TERM variable needs to be something else, or git bash was never really natively ANSI aware and capable, to begin with. I've seen other questions with the same problem for colors but their fixes are for older versions/different languages and things or they don't actually work. I have yet to find a good page for 'git bash outputs raw ANSI in IntelliJ' and I've used variations of those exact words for hours now. All I can get is long GitHub discussions on the 'bugs' related to this and they confuse me, don't lead to solutions, and may be active or just don't contain any resolution.
Edit 2
After doing some more research I've learned that my previous escape code was correct:
\033[A
\x1B[A
should be similar.
I also learned that it isn't the console I'm using that's the problem, its Windows itself. I now know this because I've tried compiling and running on cmd.exe , git bash and powershell. To change the default setting talked about earlier (consoles not supporting ansi for child applications) you have to EXPLICITLY enable it via the program itself rather than rely on a console or something.
link to a question that explains this in Python here:
How to use the new support for ANSI escape sequences in the Windows 10 console?
tl;dr
The method that they use is something called getConsoleMode and setConsoleMode and VIRTUALTERMINALPROCESSING flags. Apparently you need to use these to actually set the console mode to support ANSI. I don't currently know if those are things that are in java hidden somewhere or (likely) something that needs to be added to the base java libraries. I'm going to try and figure out how they actually get the ctypes thing from that question (seems to be what they're importing to sue these methods) and get the methods I need. Once I do I'll post that as an answer unless one of you figures it out before I do and can explain it better.
Apparently, the escape sequences work fine if you only use them from the console but If you use a "child application" then they don't work. So at the very least we now for sure know the root cause of the problem.
Edit 3
Found this which is highly relevant especially the console virtual terminal sequences section (lefthand navbar):
https://learn.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences
This which, near the bottom has a whole C-implementation of how to enable the console to read ANSI. Apparently, this doesn't need libraries at all but the process to actually change the console defaults to use code like this requires sysadmin privileges, intimate knowledge of the program files and a whole host of other things (at least if you're before the windows 10 update when color support was changed). Now it's still disabled by default but can be enabled. I don't yet know how to try ANSI from the console directly. I've tried multiple
echo \x1B[(insert ANSI code here)
but none of the commands seem to work in ANY terminal (cmd, git bash, powershell). They just return the raw code
\x1B[(whatever the ANSI code was)
I obviously am new to the console so I might be using the wrong command, if so feel free to enlighten me but the examples I've seen use echo. ANYWAYS, I thought that calling the ANSI directly from terminal was supposed to work since its supposedly enabled by default just not child applications (post windows 10 update) BUT maybe its not, maybe its disabled by default and even when enabled its still disabled for child applications (Java) unless explicitly changed in said child application (java). I'm going to try to see if it's possible to enable ANSI directly from the console, or if the linked C-code needs to be directly ported to Java or ran in the console just to work. The problem is I don't know how to get the imports/includes that the C-code uses and use it for code in java. I'd rather not just accept a coded 'solution' in C and try and use that alongside java code, I'd rather translate, understand it better and have my own code that does the same thing.
Another option I've been told is something called ANSICON which is like some sort of plugin that you install in the console with the -i flag and that's supposed to enable ANSI at least in-console. I found this
https://community.liferay.com/blogs/-/blogs/enable-ansi-colors-in-windows-command-prompt
the above explains that process in a little more detail.
My specific windows version and the version of the update
Another thing I learned was that the specific version of windows 10 that the 'update' that changed ANSI console behavior was like windows 10 v1151 or something, I'll try and find the webpage to source this directly but I currently have windows 10 v1607 so I think I should be good. Included in that was the actual OS build which I have 14393.2035 and I think that was identified as a particularly intermediate update to this process in one of my previous links (I believe its the one with the whole GitHub discussion about the update, you can find it here: https://github.com/Microsoft/WSL/issues/1173). I have a work computer so I can't really make system updates because I'm not an administrator and I doubt IT would let an intern go around updating work computers.
Anyways, I'll continue on trying to see if I can get the C-code to java, I'll test it and then try and post an answer. If ya'll are ahead of me let me know.

The python way you reference in "Edit 2" (https://stackoverflow.com/a/36760881/309816) simply invokes Windows-specific native code (kernel32, which is non-portable) to "fix" this.
I suppose you are OK with that and want to do the same in Java (i.e. invoke kernel32 when you detect Windows)...
A very lightweight library for achieving the same in Java is JNA which has out-of-the-box wrapper for kernel32 (see: https://java-native-access.github.io/jna/4.2.1/com/sun/jna/platform/win32/Kernel32.html)
You seem to be after this method: https://java-native-access.github.io/jna/4.2.1/com/sun/jna/platform/win32/Wincon.html#SetConsoleMode-com.sun.jna.platform.win32.WinNT.HANDLE-int-
Hope this works for you.
EDIT: technically, you only need jna.jar (see getting started here: https://github.com/java-native-access/jna/blob/master/www/GettingStarted.md), but I would suggest you also use jna-platform.jar so that you don't need write the code that generates the mappings for kernel32 at runtime yourself.
Home for JNA: https://github.com/java-native-access/jna
I think adding 1 (or 2 if you add jna-platform) jars that have a very specific scope (doing native calls without all the JNI preparation overhead) is lightweight enough. You don't need to generate any headers, or change anything in your compilation process. It will just work by adding those jars to your classpath.
You should also clarify in your question that this is about Windows. Maybe edit the title to: "Printing up a line in java console (reverse of '\n') on Windows" as this is really about a platform-specific concern that you want to address with Java.

Odd behaviour of Runtime.getRuntime().exec()

I have some Java code which executes a certain command on a Windows machine. However, it's behaviour is bizarre in that the same command works on one machine, but fails on three others. As far as I can tell each of the four machines are identical.
The command is this:
cmd /c %APP_HOME%\..\..\some\more\path\executable -arg1=value1 -arg2=value2 -errorlogname=%APP_HOME%\logs\errors.log -arg3 some more parameters
Where %APP_HOME% is a system variable set to the value "D:/path/to/program".
The error I get is this:
Application specific error message about not being able to open one of the logs or "a related file":
d:/path/to/program\logs. errno = 13, Permission denied
The Java I'm using looks like this:
Runtime.getRuntime().exec(cmdStr);
Where cmdStr is a String containing the command as described above. Now, things that I know are;
I should really be using the overloaded Runtime.exec(String[]) method
The command uses a mix of forward and back slashes
These are valid comments which I'll probably end up fixing the problem with, but what I'm trying to work out is why the exact same code fails on only 3 out of 4 machines.
And yes, all the paths on mentioned in the command exist on the box.
And it gets more strange. If I execute the following (very similar) command it works fine on all four boxes.
cmd /c %APP_HOME%\..\..\some\more\path\executable -arg1=value1 -arg2=value2 -errorlogname=D:\path\to\program\logs\errors.log -arg3 some more parameters
In the above working version, the only difference is the %APP_HOME% reference has been replaced with the absolute path for the -errorlogname argument. But the variable has been left to point at location of the executable.
Now the fact that the identical code runs fine on one box leads me suspect that the problem is not the Java code. Similarly, the mix of slash directions I don't see being a problem, because this concoction works on one machine.
In my view, it should be failing on every machine. Not just one of them.
Obviously, if everything really was identical it would break (or work) on every machine. But all the things I've thought to look at are identical.
Can anyone else suggest anything else to consider?
Many thanks.

I suspect %APP_HOME% is actually not set on the failing machine, but you happen to be launching the script in the correct directory for the relative path to the executable to work. To debug this further you're going to have to actually verify that %APP_HOME% has the right value by printing it out.

Proper method to find user's My Documents folder on Windows with Java?

For whatever reason, I sometimes need to find the current user's My Documents folder on Windows in a Java program to read some files. But as far as I can tell, there is no way to do it that isn't severely flawed.
The first wrong way: System.getProperty("user.home");
Why it won't work:
It only returns the \username\ folder; I'd need to add "\Documents\" on to the end to get the Documents folder... and that only works in English.
Sun bugs 6519127 and 4787931. Java finds the user home folder on Windows by reading a deprecated registry key* to find the Desktop then taking the parent; this method has multiple known problems that will easily cause a completely wrong folder to be returned. The bugs are 3.75 years and 8 years old with no fix.
The second wrong way: Using a registry-reading program to get the Personal folder of the user, which is My Documents (but i18n'd).
Why it won't work:
While it fixes the English-only problem, it's still using the same deprecated registry area, so the bugs still apply to it.
The deprecated registry key says to use a native call (SHGetKnownFolderPath) which I obviously can't do from Java.
The third wrong way:
JFileChooser fr = new JFileChooser();
FileSystemView fw = fr.getFileSystemView();
File documents = fw.getDefaultDirectory();
Why it won't work: It works great!
Except when it doesn't. While I had a program that used this open and running in the background, I opened a DirectX game (Fallout: New Vegas). The Java program immediately terminated with no stack trace. Always reproducible (for me on that game, and who knows what else). Couldn't find a Sun bug#.
So is there any method to find a user's Documents folder, on Windows, from Java, that doesn't have known problems?
(This is a nice big question.)
*(The key is "HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders")

There's no pure java way to do it, but you could use the JNA wrapper over JNI to do it without having to write any native code yourself. There's a good example of how to get the Documents folder on Windows halfway down the responses at:
What is the best way to find the users home directory in Java?

A time consuming, but reliable way of finding the 'Documents' folder of a windows user: Make your java app execute a bat script that uses Reg.exe (a windows system file) to find the value of the reg key which has the path in it. Then use a pipeline in the same bat file to send that data to the 'findstr' function which windows command prompt has. Use another pipeline to output the returned value to a text file. Then, simply make your java app read that text file, and delete it once its done :) Worked well enough for me.
Code for the bat file:
# echo off
Title Find Documents Folder
Reg Query "HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders" |findstr "Personal">>DocPath.dat
exit

There is a custom Java API that someone built (their website no longer works), but there code remains on Google Code:
http://winfoldersjava.googlecode.com/files/WinFoldersJava_1.1.zip
There are two DLL's that need to be referenced, one for each architecture(x86 and x64).

user.home is not "my documents", but users home folder, like on Unix ~/.
To get to "My documents" you can use System.getProperty("user.home")+"\Documents"; irrespective of the language system. Try it.