We Keep Coding

Transforming JSON with XSLT using SaxonEE and Python

I am attempting to write a Python script that transforms JSON to a text file (CSV) with XSLT.
With saxon-ee-10.5.jar, I can successfully perform the desired transformation by running the following command (Windows 10):
java -cp saxon-ee-10.5.jar com.saxonica.Transform -it -xsl:styling.xslt -o:result.csv
How can I achieve the same result by using Python? I have been trying with Saxon-EE/C, but I am not sure if what I want to happen is possible.
Here is an example of what I have tried so far. My XSLT already defines an $in parameter for the initial.json file, but the PyXslt30Processor.apply_templates_returning_file() seems to require a call to PyXslt30Processor.set_initial_match_selection(), of which I am not sure if non-XML files can be passed.
from saxonc import PySaxonProcessor
with PySaxonProcessor(license=True) as proc:
xslt30proc = proc.new_xslt30_processor()
xslt30proc.set_initial_match_selection(file_name='initial.json')
content = xslt30proc.apply_templates_returning_file(
stylesheet_file='styling.xslt',
output_file='result.csv'
)
print(content)
Is what I want to accomplish possible with Saxon-EE/C, or should I try techniques of calling Java from Python?

I think you want to use call_template... instead of apply-templates, e.g. https://www.saxonica.com/saxon-c/doc/html/saxonc.html#PyXslt30Processor-call_template_returning_file with
xslt30proc.call_template_returning_file(None, stylesheet_file='styling.xslt',
output_file='result.csv'
)
Using None as the template name should be identical to using -it on the command line, i.e. start by calling the template named xsl:initial-template.
Don't use xslt30proc.set_initial_match_selection in that case.
It might, however, help, to set xslt30proc.set_cwd('.') before the call_template_returning_file call.

IP as Linux array element throws UnknownHostException but as constant works

I have the following script in the directory /home/test/javacall that parses csv of IP pair , invokes a sh file that calls an executable jar to get output from these IPs.
In the below code ip1=${IPArray[0]} throws UnknownHostException from java.
But If I use the ip directly ip1="10.10.10.10" java code works fine. I did System.out.println from java and I got the same IP displayed in both cases. But in the case of ip1=${IPArray[0]} only, I get the exception.
#!/bin/bash
INPUT="IPPairs.csv"
array=()
while IFS="," read var1 var2 ; do
echo $var1 $var2
pairString="$var1***$var2"
array+=("$pairString")
done < $INPUT
for i in "${array[#]}" ; do
echo $i
IPString=$(echo $i | tr '***' ' ')
read -ra IPArray <<< "$IPString"
ip1=${IPArray[0]}
#ip1="10.10.10.10"
ip2=${IPArray[1]}
source /home/test/javacall/javacmd.sh "$ip1" "/home/test/javacall/out.txt" "show running-config all-properties"
done
Exception:
com.jcraft.jsch.JSchException: java.net.UnknownHostException: 10.10.10.10
at com.jcraft.jsch.Util.createSocket(Util.java:349)
at com.jcraft.jsch.Session.connect(Session.java:215)
at com.jcraft.jsch.Session.connect(Session.java:183)

That string (357\273\277) indicates that your csv file is encoded with a Byte-Order Mark (BOM) at the front of the file. The read command is not interpreting the BOM as having special meaning, just passing on the raw characters, so you see them as part of your output.
Since you didn't indicate how your source file is generated, you may be able to adjust the settings on that end to prevent writing the BOM, which is optional in many cases. Alternatively, you can work around it various ways on the script side. These questions both offer some examples:
How can I remove the BOM from a UTF-8 file?
Cygwin command not found bad characters found in .bashrc 357\273\277
But honestly, if you just follow Charles Duffy's advice and run your file through dos2unix before parsing it, it should clean this up for you automatically. i.e.:
...
array=()
dos2unix $INPUT
while IFS="," read var1 var2 ; do
...
Or, building on Charles' version:
#!/usr/bin/env bash
case $BASH_VERSION in ''|[123].*) echo "ERROR: Bash 4.0+ needed" >&2; exit 1;; esac
INPUT="IPPairs.csv"
declare -A pairs=( )
dos2unix $INPUT
while IFS=$',\r' read -r var1 var2 _ ; do
pairs[$var1]=$var2
done <"$INPUT"
for ip1 in "${!pairs[#]}"; do
ip2=${pairs[$ip1]}
# Using printf %q causes nonprintable characters to be visibly shown
printf 'Processing pair: %q and %q\n' "$ip1" "$ip2" >&2
done
Do note that running dos2unix in your script is not necessarily the best approach, as the file only needs to be converted once. Generally speaking, it shouldn't hurt anything, especially with such a small file. Nonetheless, a better approach would be to run dos2unix as part of whatever process pushes your csv to the server, and keep it out of this script.

System.out.println() only shows visible characters.
If your input file contains DOS newlines, System.out.println() won't show them, but they'll still be present in your command line, and parsed as part of the IP address to connect to, causing an UnknownHostException. Converting it to a UNIX text file, as with dos2unix, or using :set fileformat=unix in vim, is typically the quickest way to fix this.
BTW, if you don't need ordering retained, an associative array is typically a more appropriate data structure to use to store pairs:
#!/usr/bin/env bash
case $BASH_VERSION in ''|[123].*) echo "ERROR: Bash 4.0+ needed" >&2; exit 1;; esac
declare -A pairs=( )
while IFS=$',\r' read -r var1 var2 _ ; do
pairs[$var1]=$var2
done <"$input"
for ip1 in "${!pairs[#]}"; do
ip2=${pairs[$ip1]}
# Using printf %q causes nonprintable characters to be visibly shown
printf 'Processing pair: %q and %q\n' "$ip1" "$ip2" >&2
done
In the above, using IFS=$',\r' prevents LF characters (from the "CRLF" sequence that makes up a DOS newline) from becoming either part of var1 or var2. (Adding an _ placeholder variable to consume any additional content in a given line of the file adds extra insurance towards this point).

How to escape all special characters for ffmpeg drawtext filter in java

Does anyone have a good recipe for escaping all of the special characters (',%,\,:,{,}) from a String in java, that will be used in an ffmpeg drawtext filter chain? Trying to use replaceAll with different combinations of escaping has been an exercise in frustration!
String myTextString = "Bob's special\cool mix:stuff # 40% off";
Runtime.getRuntime().exec(new String[] { "ffmpeg",...., "filter_complex", "drawtext=enable='between(t,0,10)':x=10:y=10:fontfile=Roboto-Black.ttf:text='" + myTextString + "':fontcolor=#a43ddb:fontsize=14", ... });
ffmpeg drawtext filter: https://ffmpeg.org/ffmpeg-filters.html#drawtext-1

Alright...after banging my head against a wall for getting the right escape patterns to satisfy both java and ffmpeg I came up with this:
MyDrawTextString.replaceAll("\\\\", "\\\\\\\\\\\\\\\\").replaceAll("'", "'\\\\\\\\\\\\\''").replaceAll("%", "\\\\\\\\\\\\%").replaceAll(":", "\\\\\\\\\\\\:");
Looks insane, but it works! Note: I had to double my backslashes in my answer here to get this to display correctly too :-P Dang those backslashes.
The key is ffmpeg drawtext needs 3 backslashes to escape (',%,:) and single quotes need to also be wrapped in a second pair of single quotes. Java String needs 2 backslashes to make one and java replaceAll regex needs to have 2 backslashes to make a single one in a string. Therefore you need (2+2)*3 backslashes to escape things in drawtext filter string!

Just put your text into a text file (e.g. myText.txt) and use the textfile option:
-> myText.txt:
This is my text with special characters: ,(,),'
Then instead of using:
ffmpeg -i test.mpg -vf drawtext="This is my text with special characters :,(,),'"
Use the following command:
ffmpeg -i test.mpg -vf drawtext=textfile=textFile.txt

for Python (in Colab)
Hi, I ran into the same issue using Google Colab and Python. For those looking for a solution, this might help.
I execute the ffmpeg commandline as follows:
!ffmpeg ... -filter_complex "$texts" ...
... where texts refers to a string variable containing the mentioned filteres with drawtext option.
For me worked:
texts = ... # init
def normalize_text(t):
return t\
.replace("\\", "\\\\")\
.replace('"', '""')\
.replace("'", "''")\
.replace("%", "\\%")\
.replace(":", "\\:")
texts = normalize_text(texts) #normalize
!ffmpeg ... #execute
As you can see, escaping it once has worked for me. Note: this function might be extended to include certain other characters which will result in an error message being displayed upon execution, something along the lines of "filter could not be parsed" or "no option XXX" and more.
Thanks guys

space in path using java

Hi, I have a big problem. I'm making a java program and I have to call an exe file in a folder that have whitespace. This program also has 2 arguments that always have whitspace in the path.
Example:
C:\Users\Program File\convert image\convert.exe C:\users\image exe\image.jpeg C:\Users\out put\out.bmp
I have to do this in Windows but i want generalize it for every OS.
My code is:
Runtime run = Runtime.getRuntime();<br/>
String path_current = System.getProperty("user.dir");<br/>
String [] uno = new String[]{"cmd","/c",path_current+"\\\convert\\\convert.exe",path_current+"\\\f.jpeg", path_current+"\\\fr.bmp"};<br/>
Process proc2 = run.exec(uno);<br/>
proc2.waitFor();<br/>
This does not work. I tried removing the String array and inserting a simple String with "\"" before and after the path but that didn't work. How do I resolve this?

you may want to use :
http://commons.apache.org/io/api-1.4/org/apache/commons/io/FilenameUtils.html#separatorsToSystem(java.lang.String)
see also this answer :
Is there a Java utility which will convert a String path to use the correct File separator char?

Remove "cmd" and "/c", and use a single forward slash instead of your triple backslaches.

Spaces in java execute path for OS X

On OS X, I am trying to .exec something, but when a path contains a space, it doesn't work. I've tried surrounding the path with quotes, escaping the space, and even using \u0020.
For example, this works:
Runtime.getRuntime().exec("open /foldername/toast.sh");
But if there's a space, none of these work:
Runtime.getRuntime().exec("open /folder name/toast.sh");
Runtime.getRuntime().exec("open \"/folder name/toast.sh\"");
Runtime.getRuntime().exec("open /folder\\ name/toast.sh");
Runtime.getRuntime().exec("open /folder\u0020name/toast.sh");
Ideas?
Edit: Escaped backslash... still no worky.

There's a summary of this problem on Sun's forums... seems to be a pretty common issue not restricted to OS X.
The last post in the thread summarizes the proposed solution. In essence, use the form of Runtime.exec that takes a String[] array:
String[] args = new String[] { "open", "\"/folder name/toast.sh\"" };
or (the forum suggests this will work too)
String[] args = new String[] { "open", "folder name/toast.sh" };

Try this:
Runtime.getRuntime().exec("open /folder\\ name/toast.sh");
"\ " will just put a space in the string, but "\ " will put a "\ " in the string, which will be passed to the shell, and the shell will escape the space.
If that doesn't work, pass in the arguments as an array, one element for each argument. That way the shell doesn't get involved and you don't need bizarre escapes.
Runtime.getRuntime().exec(new String[]{"open", "/folder name/toast.sh"});

Paul's option works, but you still must escape the spaces like so:
Runtime.getRuntime().exec(new String[]{"open", "/folder\\ name/toast.sh"});
The thing that sucks about using a String array is that each param and its option must be in their own element. For instance you cannot do this:
Runtime.getRuntime().exec(new String[]{"executable", "-r -x 1", "/folder\\ name/somefile"});
But instead must specify it like so:
Runtime.getRuntime().exec(new String[]{"executable", "-r", "-x", "1", "/folder\\ name/somefile"});

In Kotlin, I was able to escape white spaces using templated strings.
private const val chromeExec = "/Applications/Google
Chrome.app/Contents/MacOS/Google Chrome"
Runtime.getRuntime().exec(arrayOf("$browserExec", url))