We Keep Coding

How to read dynamic website content in java

As per the html the source code is:
{result.data}
While requesting the URL result.data is set with 100 and am able to see the value as 100 in the browser. Where as while I am trying to execute the java program with the same url request I am unable to see the value as I have seen in browser.
URL url = new URL(site)
url.openConnection() etc..
I wanted to get the same content as I have seen in the browser through java program.

Your question is not very descriptive, but I guess you are trying to scrape data from the site.
You can use the following libraries for this task:
Jaunt (http://jaunt-api.com)
Jsoup (http://jsoup.org/cookbook/extracting-data/dom-navigation)
HTMLUnit

To what i understand, you want to do one of the below things :
Instead of reading the result line by line, you want to parse it as an XML to as to traverse to div(s) and other html tags.
For this purpose i would suggest you to use jsoup library.
When you hit the URL: www.abcd.com/number=500 in browser, it loads an empty div and on load it fetches data from somewhere, this data which it fetches on load, you want to fetch this using java ?
For this, there must be some js in the resulting page, which is fetching data by hitting some service on page load, you will need to look up in the page to know the service details and instead of hitting this URL (www.abcd.com/number=500) you will need to hit that service to get data.

SDMX-ML: SAS libname XML

Eurostat data can be downloaded via a REST API. The response format of the API is a XML file formatted according to the SDMX-ML standard. With SAS, very conveniently, one can access XML files with the libname statement and the XML or XMLv2 engine.
Currently, I am using the xmlv2 engine together with the automap= option to generate an xmlmap to access the data. It works. But the resulting SAS data sets are very unstructured, and for another data set to be downloaded the data structure might change. Also the request might depend on the DSD-file that Eurostat provides for each database item within a different XML file.
Here comes the code:
%let path = /your/working/directory/;
filename map "&path.map.txt";
filename resp "&path.resp.txt";
proc http
URL="http://ec.europa.eu/eurostat/SDMX/diss-web/rest/data/cdh_e_fos/..PC.FOS1.BE/?startperiod=2005&endPeriod=2011"
METHOD="GET"
OUT=resp;
run;quit;
libname resp XMLv2 automap=REPLACE xmlmap=map;
proc datasets;
copy out=WORK in=resp;
run;quit;
With the code above, you can view all downloaded data in your WORK library. Its a mess.
To download another time series change parameters of the URL according to Eurostat's description.
So here is my question
Is there a way to easily generate a xmlmap from a call to the DSD file so that the data are stored in a well structured way?
As the SDMX-ML standard is widely used in public institutions such as the ECB, Eurostat, OECD... I am wondering if somebody has implemented requests to the databases, already. I know about the tool from Banca Italia which uses a javaObject. However, I was wondering if there might be a solution without the javaObject.

How can I get json documents from couchbase in java?

I have a couchbase database that is shared between multiple applications, storing documents as json. I cannot seem to get data into my java app, since it appears to be dependent on native java binary serialization.
This code:
CouchbaseClient client = new CouchbaseClient(hosts,"bucket","");
System.out.println((String)client.get("someKey"));
results in
net.spy.memcached.transcoders.SerializingTranscoder: Failed to decompress data
java.util.zip.ZipException: Not in GZIP format
since it is trying to deserialize by default. I notice that I can provide my own transcoder, but I really only want the raw string data so I can json parse it myself using gson or whatever. None of the available transcoders seem to give me this.
The couchbase docs have an example for setting json, but none for reading it. How are people reading json into java?

First off, this problem will go away soon in that the Couchbase "2.0 SDKs" implement common flags between each other so this kind of problem doesn't come up. Michael's blogs are a good read if you want to see what's happening here. The reason for the problem in the first place is that in the 1.x series, Couchbase was trying to stay compatible with existing application code and memcached. In the memcached world, the clients were all written by different people at different times.
Based on the exception, I believe you're probably trying to read an item stored by .NET. I have a sample transcoder you can use for this from a few weeks ago.

Make sure you are using latest CB java client:
<dependencies>
<dependency>
<groupId>com.couchbase.client</groupId>
<artifactId>couchbase-client</artifactId>
<version>1.4.4</version>
</dependency>
</dependencies>
see: Couchbase Java Client Library 1.4
I have my service that uses CB client running just fine. Here is how I create client:
CouchbaseConnectionFactoryBuilder cfb = new CouchbaseConnectionFactoryBuilder();
cfb.setOpTimeout(10000);
cfb.setOpQueueMaxBlockTime(5000);
CouchbaseClient client = new CouchbaseClient(cfb.buildCouchbaseConnection(baseURIs, bucketName, ""));
And here is an example how I get a raw string and convert it to POJOs:
MyPOJO get(CouchbaseClient client, String key)
{
com.google.gson.Gson gson = new com.google.gson.Gson();
String jsonValue = (String) client.get(key);
return gson.fromJson(jsonValue, MyPOJO.class);
}
Also, update your question with the sample JSON doc that causing this issue. Perhaps it has something to do with the format of the document itself.

Leveraging the wordpress API in an android reader app

I'm writing an android magazine reader for a campus publication I work for. We use wordpress to publish our website, and I want to leverage the wordpress REST API to pull stories (posts) directly from the website, without publishers having to take any additional steps to publish posts on the app after publishing them on the site. I'll do this by getting JSON objects representing posts and deserializing them into POJOs of the Story class (defined in the android application), around which views will then be built dynamically.
I've just discovered the Wordpress REST API and am really excited because I think that the implementation as described above is going to be pretty simple. Are there any obvious roadblocks that I'm missing that might complicate things?
I know that the API responds with a "content" parameter that is a string containing the HTML code for the post, with references to included images/media in the appropriate places. How can I get Android to load that html and display it properly in a WebViewer?

If you don't want to parse the html and separately load images and other resources, simply use
loadDataWithBaseURL like so:
WebView storyView = (WebView) findViewById( .... );
String htmlToDisplay = ....;
storyView.loadDataWithBaseURL( "http://storysite.com/', htmlToDisplay, mimeType, encoding, "" );
The baseURL will be prepended to all relative partial URIs found in the document, so that the WebView can take care of loading all other assets for you.

Porting a Python JSON API to Java (GWT)

I am trying to port this bit of Python code to Java http://www.bemasher.net/archives/1002 (scroll down to the bottom for the code)
But because everything is dynamically typed, I'm having difficulty porting this. I need to be able to use a Java implementation of Pythons json. Right now I'm using gson ( http://code.google.com/p/google-gson/ ), but I'm open to anything.
Neither of gson's fromJson's signatures
public <T> T fromJson(String json, Class<T> classOfT) throws JsonParseException
public <T> T fromJson(String json, Type typeOfT) throws JsonSyntaxException
match Pythons
data = json.loads(response)
This is the JSON that I am trying to read: http://www.ows.newegg.com/Stores.egg/Categories/1
How would I go about getting the class or type of the JSON above and read through it? All the other examples I've seen you have to know what it is.
Thanks!

It basically fetches http://www.ows.newegg.com/Stores.egg/Menus and dumps its contents to a string (response), then it parses the JSON using json.loads() and puts that into data.
Then it will iterate through the contents of data and build a HTML list using lxml. The list itself should look like:
<ul>
<li>{StoreTitle}</li>
...
</ul>
Where {StoreID} and {StoreTitle} are the IDs and titles fetched and parsed from that URL.
References: urllib2, json, lxml

I forgot to mention that I was using Google Web Toolkit - sorry about that. I went ahead and accepted #NullUserException 's answer though I'm writing this for anyone else who may come across this.
When converting this from Python to the Java (GWT) equivalant, I found that there was already support in GWT for JSON, and even better it works on the client side saving trips to the server. (source: JSON GWT API)
To keep thoe code clean I am using a wrapper for it that can be found on Google Code here.