We Keep Coding

SDMX-ML: SAS libname XML

Eurostat data can be downloaded via a REST API. The response format of the API is a XML file formatted according to the SDMX-ML standard. With SAS, very conveniently, one can access XML files with the libname statement and the XML or XMLv2 engine.
Currently, I am using the xmlv2 engine together with the automap= option to generate an xmlmap to access the data. It works. But the resulting SAS data sets are very unstructured, and for another data set to be downloaded the data structure might change. Also the request might depend on the DSD-file that Eurostat provides for each database item within a different XML file.
Here comes the code:
%let path = /your/working/directory/;
filename map "&path.map.txt";
filename resp "&path.resp.txt";
proc http
URL="http://ec.europa.eu/eurostat/SDMX/diss-web/rest/data/cdh_e_fos/..PC.FOS1.BE/?startperiod=2005&endPeriod=2011"
METHOD="GET"
OUT=resp;
run;quit;
libname resp XMLv2 automap=REPLACE xmlmap=map;
proc datasets;
copy out=WORK in=resp;
run;quit;
With the code above, you can view all downloaded data in your WORK library. Its a mess.
To download another time series change parameters of the URL according to Eurostat's description.
So here is my question
Is there a way to easily generate a xmlmap from a call to the DSD file so that the data are stored in a well structured way?
As the SDMX-ML standard is widely used in public institutions such as the ECB, Eurostat, OECD... I am wondering if somebody has implemented requests to the databases, already. I know about the tool from Banca Italia which uses a javaObject. However, I was wondering if there might be a solution without the javaObject.

Get EC2 Instance XML Description using AWS Java SDK?

We have a scenario in which we need to retrieve the description info for EC2 instances running on AWS. To accomplish this, we are using the AWS Java SDK. In 90% of our use case, the com.amazonaws.services.ec2.model.Instance class is exactly what we need. However, there is also a small use-case where it would be beneficial to get the raw XML describing the instance. That is, the XML data before it is converted into the Instance object. Is there any way to obtain both the Instance object and the XML string using the AWS Java SDK? Is there a way to manually convert from one to the other? Or, would we be forced to make a separate call using HttpClient or something similar to get the XML data?

Make an EC2Client by adding request handler and override the beforeUnmarshalling() method like below
AmazonEC2ClientBuilder.standard().withRegion("us-east-1")
.withRequestHandlers(
new RequestHandler2() {
#Override
public HttpResponse beforeUnmarshalling(Request<?> request, HttpResponse httpResponse) {
// httpResponse.getContent() is the raw xml response from AWS
// you either save it to a file or to a XML document
return new HTTPResponse(...);
// if you consumed httpResponse.getContent(), you need to provide new HTTPResponse
}
}
).build():

If you have xml (e.g. from using AWS rest API directly), then you can use com.amazonaws.services.ec2.model.transform.* classes to convert xml to java objects. Unfortunately, it only provides classes required for SDK itself. So you, for example, can convert raw XML to an Instance using InstanceStaxUnmarshaller, but can't convert Instance to XML unless you write such converter.
Here is an example how to parse an Instance XML:
XMLEventReader eventReader = XMLInputFactory.newInstance().createXMLEventReader(new StringReader(instanceXml));
StaxUnmarshallerContext suc = new StaxUnmarshallerContext(eventReader, new TreeMap<>());
InstanceStaxUnmarshaller isu = new InstanceStaxUnmarshaller();
Instance i = isu.unmarshall(suc);
System.out.println(i.toString());
You probably can try to intercept raw AWS response, so that you can keep raw XML while still using SDK most of the time. But I wouldn't call that easy as it will require quite a bit of coding.

You could use JAXB.marshal like following. JAXB (Java Architecture for XML Binding) could convert Java object to / from XML file.
StringWriter sw = new StringWriter();
JAXB.marshal(instance, sw);
String xmlString = sw.toString();

You can use AWS rest API to replace Java SDK. A bonus will be slight performance gain because you'll not send statistic data to Amazon as the SDK does.

Using PHP to supply an in memory pdf from a Java generator

I am creating a web front end for clients to download their reports. The program that the company uses is written in Java and consists of 31 mysql tables with the 4 reports needed having over 140 points of data each. I am making the website in PHP and report generator in Java but I am having an issue finding the information on how to get the in memory PDF to load directly in the clients browser. I figured on using a TCP client/server going from Java to PHP but how do I code it so that it doesn't have to be written to the server drive and be supplied as a link. I also have no wish to rewrite 17,000 lines of Java to PHP as I am new to PHP. Is there anyway to get this done?
Thank you in advance.

Depending on how long the java app would take to run, you could consider using proc_open to create a pipe to the java programme:
<?php
$desc = array(array('pipe','r'),array('pipe','w'));
$resource= proc_open('bash',$desc,$pipes);
if (!is_resource($resource))
{
throw new Exception('PDF stream creation failed');
}
usleep(5);//for safety
fwrite($pipes[0],'java mkPDF paras'."\n");//double quoted \n is crucial
fclose($pipes[0]);
usleep(100);//while java app is running;
$pdf = stream_get_contents($pipes[1]);
fclose($pipes[1]);
proc_close($resource);
?>
This is just a basic example, that gets the streamed pdf in one big lump, so it's far from perfect. What you -IMO- should look into is getting the stream in chunks using a while loop. The php man page contains a couple of examples of this, basically, repleace
usleep(100);//while java app is running;
$pdf = stream_get_contents($pipes[1]);
with:
usleep(10);//while java app is running;
$pdf = '';
while (($pdfBuff = fgets($pipes[1],1024)) !== null)
{
$pdf.=$pdfBuff;//or echo, or whatever
}
fclose($pipes[1]);
The latter is untested, so I'm not sure if that will work in your case... but you get the idea

As said by #Elias directly send web request to java application and stream the pdf to php.
Using web services,
I. Develop some web services on java side that will send only data in any of the format like XML, jSon etc.
II. Write a code to consume these web services and develop your code to generate the pdf and stream these pdf's.
There is one pdf generating lib. Please find the link here

Scraping Data. Save File?

I am trying to scrape data from a website which uses javascript to load much of their content. Right now I am using jSoup to parse html pages, however since much of the content is loaded using javascript I haven't been able to parse the data I want.
How should I go about getting this javascript content? Should I first save the page then load and parse it using jSoup? If so, what should I use to load javascript content before I save? Is there an API which you would recommend that could output html?
Currently using java.

You might be interested in checking out pjscrape (disclaimer: this is my project). It's a command-line tool using PhantomJS to allow scraping using JavaScript and jQuery in a full browser context - among other things, you can define a "ready" function for the page and wait to scrape until the function (which might check for the existence of certain DOM elements, etc) returns true.
The other option, depending on the page, is to use a console like Firebug to figure out what data is being loaded (i.e. what URLs are being retrieved by the AJAX calls on the page), and scrape the data directly from those URLs.

If the data are generated with javascript then the data are in the downloaded page.
Better is to directly parse them on the fly as you do with plain HTML or Text parsing.
If you cannot isolate tokens with jSoup API just parse them using direct String options, as a plain text.

I tried using htmlUnit however I found it very slow.
I ended up using the curl command line function within java which worked for my purposes.
String command = "curl "+url;
Process p = Runtime.getRuntime().exec(command);
BufferedReader stdInput = new BufferedReader(new InputStreamReader(p.getInputStream()));
while ((s = stdInput.readLine()) != null) {
html = html+s+"\n";
}
return html;

Implementating Showdown.js markdown parser on the server side for ColdFusion

This is a "fact finding" question to see how difficult it would be to create a ColdFusion UDF to parse markdown on the server using the showdown.js parser. There is already a java implementation that utilizes showdown.js (see code at the end of this post) and I want to see how to go about implementing it for ColdFusion. I have no experience in Java and I would not particularly call myself "a programmer," but I don't want this to stop me from trying.
Summary
I would like to run Shadown.js server-side in order to convert markdown to HTML.
Why?
Saving two versions of a user entry, one in markdown format and another in HTML, allows us to display the raw markdown version to the end user in case they wanted to edit their entry.
Why not use a server-side parser?
For two reasons:
As of now there are no ColdFusion markdown parsers for this specific purpose
Using Showdown.js on the client-side, and then a different parser on the server-side will result in inconsistent markup between the preview displayed to the client and the version stored in the database. Given that markdown is loosely defined, most parser implementations will have subtle differences.
There is a very good blog entry that discusses the issue.
Why not do all the parsing on the client-side and post both versions?
This does not strike me as a secure solution. I also think users would potentially be able to post markdown with HTML that does not match.
Are there any existing implementations?
There is one implementation called CFShowdown, but it's not for this specific purpose. Rather, it's for handling output on a page. The comments section of the aforementioned blog features a pure Java implementation written by a user called David:
ScriptEngineManager manager = new ScriptEngineManager();
ScriptEngine jsEngine = manager.getEngineByName("js");
try
{
jsEngine.eval(new InputStreamReader(getClass().getResourceAsStream("showdown.js")));
showdownConverter = jsEngine.eval("new Showdown.converter()");
}
catch (Exception e)
{
log.error("could not create showdown converter", e);
}
try
{
return ((Invocable) jsEngine).invokeMethod(
showdownConverter,
"makeHtml",
markdownString
) + "";
}
catch (Exception e)
{
log.error("error while converting markdown to html", e);
return "[could not convert input]";
}
Objective
Create a java class that would allow us to use this implementation with a ColdFusion UDF or a custom tag inside a component, something along the lines of <cfset html = getMarkdown(string)>
Since I have no experience with Java, I want to get some advice and input from users on where and how to start going about this task. I created a

Have files showdown.js and a file markdown.txt (example below) in the same directory.
showdown.cfm
<cfscript>
manager = createObject("java", "javax.script.ScriptEngineManager").init();
jsEngine = manager.getEngineByName("js");
showdownJS = fileRead('#getDirectoryFromPath(getCurrentTemplatePath())#/showdown.js');
jsEngine.eval(showdownJS);
showdownConverter = jsEngine.eval("new Showdown.converter()");
markdownString = fileRead("#getDirectoryFromPath(getCurrentTemplatePath())#/markdown.txt");
args = [markdownString];
result = jsEngine.invokeMethod(
showdownConverter,
"makeHtml",
args
) & "";
</cfscript>
markdown.txt
Showdown Demo
-------------
You can try out Showdown on this page:
- Type some [Markdown] text on the left side.
- See the corresponding HTML on the right.
For a Markdown cheat-sheet, switch the right-hand window from *Preview* to *Syntax Guide*.
Showdown is a JavaScript port of the original Perl version of Markdown. You can get the full [source code] by clicking on the version number at the bottom of the page.
Also check out [WMD, the Wysiwym Markdown Editor][wmd]. It'll be open source soon; email me at the address below if you'd like to help me test the standalone version.
**Start with a [blank page] or edit this document in the left window.**
[Markdown]: http://daringfireball.net/projects/markdown/
[source code]: http://attacklab.net/showdown/showdown-v0.9.zip
[wmd]: http://wmd-editor.com/
[blank page]: ?blank=1 "Clear all text"
Update
Here's a version that takes Adam Presley's work in Java and does it all in a CFC. Note I took that little bit of magic he added at the end of showdown.js and put it into a CFC function whose return value is appended (i.e. showdownAdapterJS()).
CFC
<cfcomponent output="false" accessors="true">
<cffunction name="init" output="false" access="public" returntype="Showdown" hint="Constructor">
<cfset variables.manager = createObject("java", "javax.script.ScriptEngineManager").init()>
<cfset variables.engine = manager.getEngineByName("javascript")>
<cfreturn this/>
</cffunction>
<cffunction name="toHTML" output="false" access="public" returntype="any" hint="">
<cfargument name="markdownText" type="string" required="true"/>
<cfset var local = structNew()/>
<cfset var bindings = variables.engine.createBindings()>
<cfset var result = "">
<cftry>
<cfset bindings.put("markdownText", arguments.markdownText)>
<cfset variables.engine.setBindings(bindings, createObject("java", "javax.script.ScriptContext").ENGINE_SCOPE)>
<cfset var showdownJS = fileRead('#getDirectoryFromPath(getCurrentTemplatePath())#/showdown.js')>
<cfset showdownJS &= showdownAdapterJS()>
<cfset result = engine.eval(showdownJS)>
<cfcatch type="javax.script.ScriptException">
<cfset result = "The script had an error: " & cfcatch.Message>
</cfcatch>
</cftry>
<cfreturn result>
</cffunction>
<cffunction name="showdownAdapterJS" output="false" access="private" returntype="string" hint="">
<cfset var local = structNew()/>
<cfsavecontent variable="local.javascript">
<cfoutput>#chr(13)##chr(10)#var __converter = new Showdown.converter();
__converter.makeHtml(markdownText);</cfoutput>
</cfsavecontent>
<cfreturn local.javascript>
</cffunction>
</cfcomponent>
Usage
<cfset showdown = createObject("component", "Showdown").init()>
<cfset markdownString = fileRead("#getDirectoryFromPath(getCurrentTemplatePath())#/markdown.txt")>
<cfoutput>#showdown.toHTML(markdownString)#</cfoutput>

You can run server-side javascript in CF by using CFGroovy - which basically allows you to run any JSR-223 scripting language inline with CFML.
Ben Nadel has an example of running server-side javascript using CFGroovy and Rhino
The example has everything you need - assuming you have the javascript code already put together.

Actually, I've already wrapped up Showdown in a Java library that can be used in ColdFusion. The example I provide, in what I admit is poor documentation, uses a custom tag, but you can use the Java component just as easily like so.
<cfset obj = createObject('java', 'com.adampresley.cfshowdown.Showdown').init() />
<cfset parsedText = obj.toHTML(trim(someMarkdownContent)) />
Perhaps that helps? Either way, long live Markdown! :)

Given that Markdown is not a regular language, and a majority of implementations are a series of regular expressions, there are bound to be differences between them, as you noted. There is almost no avoiding it.
If your objective is just to:
Provide a client-side markdown editor with live-preview (like the Stack Overflow question/answer editor), and
Store an identically-processed copy of the generated html for end-user display
Then I see only two real options:
Do all markdown processing server-side, and accomplish your preview using AJAX to submit the markdown and get the updated preview html (using the same library that you'll ultimately use to generate the stored html), or
Do all markdown processing client-side, and submit both the raw markdown and the generated HTML as parts of your content composition form and store both; so that you can display the original markdown for editing purposes and the generated HTML for display purposes.
Personally I would go with option 2.