We Keep Coding

Transforming JSON with XSLT using SaxonEE and Python

I am attempting to write a Python script that transforms JSON to a text file (CSV) with XSLT.
With saxon-ee-10.5.jar, I can successfully perform the desired transformation by running the following command (Windows 10):
java -cp saxon-ee-10.5.jar com.saxonica.Transform -it -xsl:styling.xslt -o:result.csv
How can I achieve the same result by using Python? I have been trying with Saxon-EE/C, but I am not sure if what I want to happen is possible.
Here is an example of what I have tried so far. My XSLT already defines an $in parameter for the initial.json file, but the PyXslt30Processor.apply_templates_returning_file() seems to require a call to PyXslt30Processor.set_initial_match_selection(), of which I am not sure if non-XML files can be passed.
from saxonc import PySaxonProcessor
with PySaxonProcessor(license=True) as proc:
xslt30proc = proc.new_xslt30_processor()
xslt30proc.set_initial_match_selection(file_name='initial.json')
content = xslt30proc.apply_templates_returning_file(
stylesheet_file='styling.xslt',
output_file='result.csv'
)
print(content)
Is what I want to accomplish possible with Saxon-EE/C, or should I try techniques of calling Java from Python?

I think you want to use call_template... instead of apply-templates, e.g. https://www.saxonica.com/saxon-c/doc/html/saxonc.html#PyXslt30Processor-call_template_returning_file with
xslt30proc.call_template_returning_file(None, stylesheet_file='styling.xslt',
output_file='result.csv'
)
Using None as the template name should be identical to using -it on the command line, i.e. start by calling the template named xsl:initial-template.
Don't use xslt30proc.set_initial_match_selection in that case.
It might, however, help, to set xslt30proc.set_cwd('.') before the call_template_returning_file call.

Capture generated output file path and name using CSSDK

We are in the process of converting over to using the XSLT compiler for page generation. I have a Xalan Java extention to exploit the CSSDK and capture some meta data we have stored in the Extended Attributes for output to the page. No problems in getting the EA's rendered to the output file.
The problem is that I don't know how to dynamically capture the file path and name of the output file.
So just as POC, I have the CSVPath hard coded to the output file in my Java extension. Here's a code sample:
CSSimpleFile sourceFile = (CSSimpleFile)client.getFile(new CSVPath("/some-path-to-the-output.jsp"));
Can someone point me in the CSSDK to where I could capture the output file?

I found the answer.
First, get or create your CSClient. You can use the examples provided in the cssdk/samples. I tweaked one so that I captured the CSClient in the method getClientForCurrentUser(). Watch out for SOAP vs Java connections. In development, I was using a SOAP connection and for the make_toolkit build, the Java connection was required for our purposes.
Check the following snippet. The request CSClient is captured in the static variable client.
CSSimpleFile sourceFile = (CSSimpleFile)client.getFile(new CSVPath(XSLTExtensionContext.getContext().getOutputDirectory().toString() + "/" + XSLTExtensionContext.getContext().getOutputFileName()));

Scraping Data. Save File?

I am trying to scrape data from a website which uses javascript to load much of their content. Right now I am using jSoup to parse html pages, however since much of the content is loaded using javascript I haven't been able to parse the data I want.
How should I go about getting this javascript content? Should I first save the page then load and parse it using jSoup? If so, what should I use to load javascript content before I save? Is there an API which you would recommend that could output html?
Currently using java.

You might be interested in checking out pjscrape (disclaimer: this is my project). It's a command-line tool using PhantomJS to allow scraping using JavaScript and jQuery in a full browser context - among other things, you can define a "ready" function for the page and wait to scrape until the function (which might check for the existence of certain DOM elements, etc) returns true.
The other option, depending on the page, is to use a console like Firebug to figure out what data is being loaded (i.e. what URLs are being retrieved by the AJAX calls on the page), and scrape the data directly from those URLs.

If the data are generated with javascript then the data are in the downloaded page.
Better is to directly parse them on the fly as you do with plain HTML or Text parsing.
If you cannot isolate tokens with jSoup API just parse them using direct String options, as a plain text.

I tried using htmlUnit however I found it very slow.
I ended up using the curl command line function within java which worked for my purposes.
String command = "curl "+url;
Process p = Runtime.getRuntime().exec(command);
BufferedReader stdInput = new BufferedReader(new InputStreamReader(p.getInputStream()));
while ((s = stdInput.readLine()) != null) {
html = html+s+"\n";
}
return html;

PDFJET Does Not Work For En Empty Constructor

i want to use pdfjet for a Google app engine project.
i downloaded the Java jar from the pPdfjet home page.
i followed an example given in a stack-overflow example and the examples given in the home page.
all the examples uses an empty constructor: PDF pdf=new PDF();. However when i try to use it,
it says that the constructor PDF() is undefined, further more all the method shown do not work:
pdf.wrap(): is undefined
pdf.save("Example_03.pdf"): is undefined

It looks like the examples on their web page are out of date. Look at the examples in the zip download instead. This simple example works for me:
OutputStream out = new FileOutputStream("test.pdf");
PDF pdf = new PDF(out);
Page page = new Page(pdf, Letter.PORTRAIT);
pdf.flush();
out.close();

Ok this is easy. Actually instead of taking from req.getOutputStream() directly create and instance of BytArrayOutputStream and use that.
For sending it just use out.toArray() as add it to the attacement part.

Are there any java libraries for validating user supplied HTML, on the server side?

I have a service which takes the user supplied rich text (can have HTML tags) and saves it into the database. That data gets used by some other application. But sometimes the user supplied data has missing HTML tags and wrong closing tags. I want to validate if the user supplied data is valid HTML or not and depending on that I want to warn the user.
Are there any java libraries to do HTML validation?

You can try JTidy, but it's too slow for simple HTML cleaning.
If you want just process HTML you can try NekoHTML, it's lightweight and fast

You can try JTidy.
JTidy is a Java port of HTML Tidy, a
HTML syntax checker and pretty
printer.

You can use Jsoup, from the project README
Here is an example:
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
...
String markup = "<body><head>...";
Jsoup.isValid(markup, null);
Instead of null, you can pass a Whitelist ? object as second parameter to the isValid method.
Plus, you can easily install this library using Gradle

Validator.nu, which implements the HTML5 spec, IMO.

There's a great thing called NekoHTML which is just a thin wrapper over the Apache Xerces parser that turns on error-recovery/correction. It doesn't validate so much as error-correct, so you can process the result as XML, i.e. run it through XPaths or XSLTs. It has worked flawlessly for me for several months on completely arbitrary HTML from 3rd-party sites.