I am trying to build my new App and I need some data for it.
so I was wondering if I can get JSON data (or query link or any source file) of any site for now and for future.
I need to get data from this site:
https://study.ekb.eg/ - https://www.ekb.eg/
Note: https://www.ekb.eg/ is the main source data for https://study.ekb.eg/
Thanks
The site needs to be specifically programmed to return JSON data for you. Regular, normal sites are not like this. They generally just return HTML
So the site owner would need to set up an API for you to consume
Eurostat data can be downloaded via a REST API. The response format of the API is a XML file formatted according to the SDMX-ML standard. With SAS, very conveniently, one can access XML files with the libname statement and the XML or XMLv2 engine.
Currently, I am using the xmlv2 engine together with the automap= option to generate an xmlmap to access the data. It works. But the resulting SAS data sets are very unstructured, and for another data set to be downloaded the data structure might change. Also the request might depend on the DSD-file that Eurostat provides for each database item within a different XML file.
Here comes the code:
%let path = /your/working/directory/;
filename map "&path.map.txt";
filename resp "&path.resp.txt";
proc http
URL="http://ec.europa.eu/eurostat/SDMX/diss-web/rest/data/cdh_e_fos/..PC.FOS1.BE/?startperiod=2005&endPeriod=2011"
METHOD="GET"
OUT=resp;
run;quit;
libname resp XMLv2 automap=REPLACE xmlmap=map;
proc datasets;
copy out=WORK in=resp;
run;quit;
With the code above, you can view all downloaded data in your WORK library. Its a mess.
To download another time series change parameters of the URL according to Eurostat's description.
So here is my question
Is there a way to easily generate a xmlmap from a call to the DSD file so that the data are stored in a well structured way?
As the SDMX-ML standard is widely used in public institutions such as the ECB, Eurostat, OECD... I am wondering if somebody has implemented requests to the databases, already. I know about the tool from Banca Italia which uses a javaObject. However, I was wondering if there might be a solution without the javaObject.
I am unable to retrive inline images/screen shot from Java in Lotus Notes from
document.getItemValueString('Body')
By above function am i able retrive text available in mailbody not inline images.
Please provide your suggestions in order to retrive inlines images from the mail body
Thanks in advance.
LSP Jyothi
First of all: Body is a NotesRichtextItem. You would have to use the NotesRichtextItem- methods and properties to get the inline- image... if there where any for that purpose.
Inline- images are not handled by any means in LotusScript. To get them, you need to:
Export the document as XML
Find the part in the XML that represents the inline image
take the Base64- encoded value there and convert it into a binary format, use Mime- Classes for that (Trick).
Write the data to a file
There is a lot of code involved in doing this. I just post the "crucial" parts of the code here (untested, no syntax check, just as a starting point):
EDIT: Sorry, I am not an expert in Java and only saw the tag "lotusscript", therefor my example is LotusScript- Code (should be similar with java, and I think Base64- operations are alreays built in in java, no need to use the Mime- Trick)
Dim strDxl as String
Dim strFoundBase64 as String
Dim exporter as NotesDXLExporter
Dim stream as NotesStream
Dim docConvert as NotesDocument
Dim mimeEntity as NotesMimeEntity
Set exporter = session.CreateDXLExporter
exporter.ConvertNotesBitmapsToGIF = True
strDxl = exporter.Export(document)
'- Search through strDxl and find everything that is in the following tags:
'- <gif></gif>, <gif originalformat='notesbitmap'></gif>, <jpeg></jpeg>, <png></png>
strFoundBase64 = ...'assign text between tags
'- use Mime class to convert to binary
Set docConvert = New NotesDocument( document.ParentDatabase )
Set mimeEntity = docConvert.CreateMIMEEntity
Call mimeEntity.SetContentFromBytes(strFoundBase64, "image/gif", ENC_BASE64)
'- Write result to file
Set stream = ses.CreateStream
Call stream.Open( "C:\Temp\image.gif", "binary")
Call mimeEntity.GetContentAsBytes(stream)
Call stream.Close()
I have a canvas painted by the user.
In the JavaScript I do:
var data = canvas.toDataURL().substr(22);
// snipped code that sets up xhr POST to "d/"
var params = "img=" + encodeURIComponent(data);
xhr.send(params);
I substr(22) to get rid of "data:image/png;base64,"
Then in app engine I do:
doodle.setProperty("img", new Text(req.getParameter("img")));
So I am setting the img property of the doodle Entity to the canvas.toDataURL().substr(22)
Now, when I want to retrieve the image, I do:
if (debugimg) {
resp.setContentType("text/plain");
resp.getWriter().print(((Text)groove.getProperty("img")).getValue());
}
else {
resp.setContentType("image/png;base64");
resp.getWriter().print(((Text)groove.getProperty("img")).getValue());
}
But for the life of me, the image never comes up.
Here is an example. I drew this, and can save it and render it in JavaScript.
https://yougotadoodle.appspot.com/d.jsp?id=1483002
If I use debugimg, this is what is being saved:
http://yougotadoodle.appspot.com/d?id=1483002&img=true&debugimg=true
But when I try to serve it with setContentType("image/png;base64") or even just "image/png" you get a broken picture:
http://yougotadoodle.appspot.com/d?id=1483002&img=true
I have tried a few different things, including not substr(22)ing it. Any ideas?
I tried using a Blob(), so storing it like this:
doodle.setProperty("img", new Blob(req.getParameter("img").getBytes()));
and reading it like this:
resp.getWriter().print(((Blob)groove.getProperty("img")).getBytes());
But that seemed to spit out somethign like this:
[B#1f11e0f
You have to decode this string before serving it as image/png because it is the Base64 encoded version.
I tested it locally in Python and your Hello SO! worked perfectly after decoding the given string. I'm not sure how to do it in Java but it should be fairly easy.
Here are three code snippets that have worked for me on the JS side (jpg) through the put to a blob property. May not be optimal, but it does work. HTH. -stevep
Create canvas render:
imgFinalData = canvas.toDataURL('image/jpg', 1.0);
Setup variable for POST to GAE:
f64 = imgFinalData.substr(imgFinalData.indexOf(',')+1).toString();
Post to GAE (fd is an array used to store mutiple POST vars):
fd.push('bytes=' + escape(f64));
//Here is the call with fd that handles the post:
postXmlHttpRequest(url, fd.join('&'), handlePostFinal);
One the GAE side (Python):
Property that stores POST data (line from entity class):
bytes = db.BlobProperty(required=True, indexed=False)
How the post data is processed b/4 put:
data = urllib.unquote(self.request.get('bytes'))
data = data.replace(' ','+')
bytes = base64.b64decode(data + '=' * (4 - len(data) % 4))
Property line inside the entity statement for put:
bytes = db.Blob(bytes),
I have the following feeds from my vendor,
http://scores.cricandcric.com/cricket/getFeed?key=4333433434343&format=xml&tagsformat=long&type=schedule
I wanted to get the data from that xml files as java objects, so that I can insert into my database regularly.
The above data is nothing but regular updates from the vendor, so that I can update in my website.
can you please suggest me what are my options available to get this working
Should I use any webservices or just Xstream
to get my final output.. please suggest me as am a new comer to this concept
Vendor has suggested me that he can give me the data in following 3 formats rss, xml or json, I am not sure what is easy and less consumable to get it working
I would suggest just write a program that parses the XML and inserts the data directly into your database.
Example
This groovy script inserts data into a H2 database.
//
// Dependencies
// ============
import groovy.sql.Sql
#Grapes([
#Grab(group='com.h2database', module='h2', version='1.3.163'),
#GrabConfig(systemClassLoader=true)
])
//
// Main program
// ============
def sql = Sql.newInstance("jdbc:h2:db/cricket", "user", "pass", "org.h2.Driver")
def dataUrl = new URL("http://scores.cricandcric.com/cricket/getFeed?key=4333433434343&format=xml&tagsformat=long&type=schedule")
dataUrl.withReader { reader ->
def feeds = new XmlSlurper().parse(reader)
feeds.matches.match.each {
def data = [
it.id,
it.name,
it.type,
it.tournamentId,
it.location,
it.date,
it.GMTTime,
it.localTime,
it.description,
it.team1,
it.team2,
it.teamId1,
it.teamId2,
it.tournamentName,
it.logo
].collect {
it.text()
}
sql.execute("INSERT INTO matches (id,name,type,tournamentId,location,date,GMTTime,localTime,description,team1,team2,teamId1,teamId2,tournamentName,logo) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)", data)
}
}
Well... you could use an XML Parser (stream or DOM), or a JSON parser (again stream of 'DOM'), and build the objects on the fly. But with this data - which seems to consist of records of cricket matches, why not go with a csv format?
This seems to be your basic 'datum':
<id>1263</id>
<name>Australia v India 3rd Test at Perth - Jan 13-17, 2012</name>
<type>TestMatch</type>
<tournamentId>137</tournamentId>
<location>Perth</location>
<date>2012-01-14</date>
<GMTTime>02:30:00</GMTTime>
<localTime>10:30:00</localTime>
<description>3rd Test day 2</description>
<team1>Australia</team1>
<team2>India</team2>
<teamId1>7</teamId1>
<teamId2>1</teamId2>
<tournamentName>India tour of Australia 2011-12</tournamentName>
<logo>/cricket/137/tournament.png</logo>
Of course you would still have to parse a csv, and deal with character delimiting (such as when you have a ' or a " in a string), but it will reduce your network traffic quite substantially, and likely parse much faster on the client. Of course, this depends on what your client is.
Actually you have RESTful store that can return data in several formats and you only need to read from this source and no further interaction is needed.
So, you can use any XML Parser to parse XML data and put the extracted data in whatever data structure that you want or you have.
I did not hear about XTREME, but you can find more information about selecting the best parser for your situation at this StackOverflow question.