We Keep Coding

Java: Runtime.exec() and Unicode symbols on Windows: how to make it work with non-English letters?

Intro
I am using Runtime.exec() to execute some external command and I am using parameters that contain non-English characters. I simply want to run something like this:
python test.py шалом
It works correctly in cmd directly, but is incorrectly handled via Runtime.exec.getRuntime()("python test.py шалом")
On Windows my external program fails due to unknown symbols passed to it.
I remember similar issue from early 2010s (!) - JDK-4947220, but I thought it is already fixed since Java core 1.6.
Environments:
OS: Name Microsoft Windows 10 Pro (Version 10.0.18362 Build 18362)
Java: jdk1.8.0_221
Code
To understand the question - the best way is to use code snippet listed below:
import java.io.BufferedReader;
import java.io.InputStreamReader;
public class MainClass {
private static void foo(String filename) {
try {
BufferedReader input = new BufferedReader(
new InputStreamReader(
Runtime.getRuntime().exec(filename).getInputStream()));
String line;
while ((line = input.readLine()) != null) {
System.out.println(line);
}
input.close();
} catch (Exception e) { /* ... */ }
}
public static void main(String[] args) {
foo("你好.bat 你好"); // ??
foo("привет.bat привет"); // ??????
foo("hi.bat hi"); // hi
}
}
Where .bat file contains only simple #echo %1
The output will be:
??
??????
hi
PS
System.out.println("привет") - works fine and prints everything correctly
Questions are the following:
1) Is this issue related to Utf-8 utf-16 formats?
2) How to fix this issue? I do not like this answer as it looks like a very dangerous and ugly workaround.
3) Does anyone know why file names of batch file is not broken and this file can be found, but the argument gets broken? May be it is problem of #echo?

Yes, issue is related with UTF. Theoretically a setting 65001 codepage for cmd that executes the bat files should solve the issue (along with setting UTF-8 charset as default from the Java side)
Unfortunately there a bug in Windows mentioning here Java, Unicode, UTF-8, and Windows Command Prompt
So there's no simple and complete solution. What it's possible to do is to set the same default language-specific encoding, like cp1251 Cyrillic, for both java and cmd. Not all languages are well reflected in the windows encodings, for example Chinese is one of them.
If there's some non-technical restriction on the windows system to change default encoding to the language-specific one for all cmd processes, the java code will be more complicated. At beginning new cmd process have to be created and to its stdin/stdout streams should be attached reader with UTF-16LE (for `cmd /U' process) and writer with CP1251 from different threads. First command sending to stdin from java should be 'chcp 1251' and second is the name of bat-file with its parameters.
Complete solution still may use UTF-16LE for reading of cmd output but to pass a text in, other universal encoding should be used, for example base64, which again leads to increasing complexity

How to use input from a java code and transfer it into an external application (as well as retrieving output)?

I'm actually not that great and am relatively new at Java. I wish to receive input from the user, and want to input this data into an external application.
This application processes the data and provides an output. I wish to retrieve this output using the Java code.
I have attempted in doing this but, I haven't got the slightest idea on how to start this script.
Nothin' on the internet seems to answer this question. If you have any idea or any new functions that can be useful, please help me in doing so.
Since I'm starting from ground zero, any help is appreciated.
Thanks so much.

To communicate with an external application you need to first define the communication way. For example:
Will this application read the output from a file?
If that statement it's true, then you need to learn serialization:
Will this application read the input from the standard output (like a command-line application)
If that statement it's true then you need to send with System.out.print().
Will this application get the data over HTTP.
Then you need to learn about REST and or RPC architectures.
Assuming that it will be a command-line application, then you could use something like this:
public class App
{
public static void main(String... args)
{
// You need to implement your business logic here. Not just print whatever the user passes as arguments of the command-line.
for(String arg : args)
{
System.out.print(arg);
}
}
}

There's a lot going on here but I'll suggest an example for each part of this question and assume this is just going to be written in Java, and suggesting an iterative design/development approach.
receive input from the user::getting arguments from the command line can work, but I think most users want to use familiar user interfaces like excel to input large amounts of data. Have them export files to .csv or look into reading excel files directly with apache poi. The latter is not for beginners, but not terrible to figure out or find examples. The former should be easy to figure out if you look into reading files and splitting them line by line on the delimiter. Here's an example of that:
try (BufferedReader reader = new BufferedReader(new FileReader(new File("user_input.csv"))) {
String currentLine = reader.readLine();
while (currentLine != null) {
String splitLine[] = currentLine.split(","); //choose delimiter here
//process cells as needed
//write output somewhere so other program can read it later
currentLine = reader.readLine();
}
}
catch (IOException ex) {
System.out.println(ex.getMessage()); //maybe write to an error log
System.exit(1);
}
"input" data to other app::you can use pipes if you're at the command line. but I'd recommend you write to a file and have the other app read it. here's an expansion of the previous code snippet showing how to write to a file as that might be more practical and easier to log/archive/debug.
try (BufferedReader reader = new BufferedReader(new FileReader(new File("user_input.csv")));
BufferedWriter writer = new BufferedWriter(new FileWriter(new File("process_me.csv")))) {
String currentLine = reader.readLine();
while (currentLine != null) {
String splitLine[] = currentLine.split(","); //choose delimiter here
//process cells as needed
writer.write(processed_stuff);
currentLine = reader.readLine();
}
}
catch (IOException ex) {
System.err.println(ex.getMessage());
System.exit(1);
}
Then retrieving output::can just be reading another file with another Java program. This way you're communicating between programs using the file system. You must agree upon file formats and directories though. And you'll be limited to having both programs on the same server.
To make this at scale, you could use web services assuming the other program you're making requests to is a web service or has one wrapped around it. You can send your file and receive some response using URLConnection. This is where things will get much more complex, but now everything in your new program is just one Java program and the other code can live on another server.
Building the app first with those "intermediate" files between the user input code, the external code, and the final code will help you focus on perfecting the business logic, then you can worry about just communication over the network.

Using files as arguments for a Java program in the terminal

java someJavaProgram fsa.fsa <test.txt
That, apparently, is a legitimate command to take with two files as arguments for a Java program in the terminal - one to read in, and then the other (and I think the idea is that it prints the output to the terminal directly). someJavaProgram, fsa.fsa and test.txt are all files in the same directory (being someProject/src, and someJavaProgram in the default package).
However, the response I am given in the terminal just says:
FSA file not found - please scan in the appropriate file.
Testing file not found, please scan in the new relevant file.
My question is two-fold:
What is this command and what is it for?
Does it need refining or modifying or is it the program that needs improvement?
I should note that I wrote the code in Eclipse, where I simply hardcoded filepaths into the program. I'm not sure if that affects anything but it's related.
EDIT: The filepaths and related code are as follows:
private static final String FILE_PATH = "src/test.txt";
private static final String FSA_PATH = "src/fsa1.fsa";
...
public static void main(String[] args) throws FileNotFoundException {
interpretAutomaton();
testAutomaton();
}
...
interpretAutomaton() {
...
Scanner fsaScanner = new Scanner(new BufferedReader(new FileReader(FSA_PATH)));
...
testAutomaton() {
...
Scanner fileScanner = new Scanner(new BufferedReader(new FileReader(FILE_PATH)));
*Both are surrounded by try/catch blocks - which clearly work!
Thanks to anyone who can help clarify on the matter!

Based on the comments so far, to answer your actual questions:
1) The command has four elements:
java - execute the java program
someJavaProgram - the name of the Java class to execute
fsa.fsa - the first argument to the java program, accessible via argv[]
<test.txt - standard input redirection, the contents of the file will be available on the program's standard input, ie. System.in
The net effect is to run your Java program with one argument and one file's contents on the standard input.
2) Both the command line and the program look like they need to change:
change the command line to:
java someJavaProgram fsa.fsa test.txt
That is, remove the <. You will also need to check the paths to the files are correct. This command line assume you are in the same directory as the files when you execute it.
Change your code to use the filenames on the command line rather than the hard-coded names.

Replace string in text file using Java

I have a text file (file.txt) which contains multiple lines:
/location/test/file.csv
/location/test1/file2.csv
/location/test2/file.exe
Using ECMA, I would like to replace all instance of "/" with "\". However, the code below only replaces the first line and eliminate lines 2 and 3.
This is the result of the file.txt file after I run the code (as I said, lines 2 and 3 are missing):
\location\test\file.csv
Can anyone please help?
function ReadFile ()
{
var file = new java.io.BufferedReader(new java.io.FileReader("C:\\Test\\file.txt"));
var fileWriter = new java.io.FileWriter("C:\\Test\\file.txt",false);
while ((line = file.readLine()) != null)
{
println(line);
if (line.contains ("/"))
line = line.replace("/","\\");
fileWriter.write(line);
fileWriter.close();
}
}
ReadFile ();

So I managed to run this code using Rhino. It does in fact run. I made some changes to filenames in order to get it running on my mac, but it is essentially the same code:
function ReadFile () {
var file = new java.io.BufferedReader(new java.io.FileReader("file"));
var fileWriter = new java.io.FileWriter("file2",false);
while ((line = file.readLine()) != null) {
if (line.contains ("/"))
line = line.replace("/","\\");
fileWriter.write(line + "\n");
}
fileWriter.close();
file.close();
}
ReadFile ();
So the bugs you had were:
Reading and writing to the same file. This is awkward using streams. Basically, don't do it. I changed it to write to file2.
Closing the writer inside your reader loop. Close the writer at the end, once closed, you can no longer write to it.
Not closing the file you are reading from.
For those interested in how I got this running, on OS X, using rhino:
brew install rhino
rhino example.js
I had to remove the println because that wasn't recognised but that wasn't a critical part. JS on the JVM. Fun! Except nothing is async.....
There are other JS engines too, but rhino worked.

JavaScript String object has a replace method that takes a regular expression that can replace characters in a String. However, the Java in your JavaScript won't work because you are mixing up two languages.
http://www.w3schools.com/jsref/jsref_replace.asp

The problem is that you're writing over the file as you're reading it, so as soon as your first write(line) completes, the file no longer has the next two lines. Either use a second, temporary file to write to until you've finished processing the file, or keep a list/array of all the new lines in memory, and then write out the file at the end.
I'm really curious to know, though, how you managed to get your current program to even run. In my experience, Java and ECMA/Javascript are two completely separate languages, but it looks like you're using javascript code against Java libraries. What's up with that?

Why does this java function not indicate that filepath is invalid

I'm trying to check a user entered file path is valid for the OS, and I thought this could be achieved with Paths.get(filePath) so I wrote this simple function
public static boolean isValidFilename(String s)
{
try
{
Paths.get(s);
return true;
}
catch (InvalidPathException e)
{
return false;
}
}
but whilst on Windows 8 it will successfully return false for
c:\File::
it will not for
c:\File..
Is this a bug, or misunderstanding on my part

It's a misunderstanding on your part. A colon is not a legal filename character in Windows, but a period '.' is.
The second file is valid.... it's the same file as C:/File... (i.e. C://File.. and C:/File.. are the same location)
Edit: See this question here: How to create a filename with a trailing period in Windows?
I have tried it in my system and unfortunately my results are mixed.... still the above answers are good reading....
Edit2: I have run the following code, and it works for me now:
File file = new File("\\\\?\\C:/Temp/File..");
FileOutputStream fos = new FileOutputStream(file);
fos.write("Foo".getBytes());
fos.close();
It appears that the \\?\ construct works in java.io.File, but not in java.nio.file.Path
If you print out the exception you also get more of an idea of what the problem is.....
java.nio.file.InvalidPathException: Illegal char <:> at index 7: C:/File::
at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:194)
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:165)
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:89)
at sun.nio.fs.WindowsPath.parse(WindowsPath.java:106)
at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:267)
at java.nio.file.Paths.get(Paths.java:95)

Your results seem to be ok.
c://File.. is a valid filename in windows, and although I can't find any documentation about it, the two .. seems to be handled in a special way - it is just stripped from the name, so file.. is the same as file or even file...... It's probably some interaction with how windows treats file extensions.
c:/File:: is not, since you can't have a : in a filename
(see e.g. http://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx)