How to parse json to a database? - java

I have a complex json written in a string. I know java and a little of mysql. I need to make a database out of the json.
I'm using some twitter data so the tweets contain the description of the user who tweeted it and in case it's been retweeted, it contains the description of the user who tweeted it before this user.
My objective is to create a user table ( or array or any other data structure ) which contains all the tweets this user tweeted, and all his tweets which have been retweeted.
The tweet object contains around 50-80 objects so giving an example here will make this post really long.
Example
StatusJSONImpl{createdAt=Wed Sep 28 12:04:55 IST 2011, id=118936707830775808, text='RT #nytimesbits: Google's Biggest Threat Is Google http://t.co/kTNqJFJC', source='web', isTruncated=false, inReplyToStatusId=-1, inReplyToUserId=-1, isFavorited=false, inReplyToScreenName='null', geoLocation=null, place=null, retweetCount=6, wasRetweetedByMe=false, contributors=null, annotations=null, retweetedStatus=StatusJSONImpl{createdAt=Wed Sep 28 05:35:26 IST 2011, id=118838689248985088, text='Google's Biggest Threat Is Google http://t.co/kTNqJFJC', source='The New York Times', isTruncated=false, inReplyToStatusId=-1, inReplyToUserId=-1, isFavorited=false, inReplyToScreenName='null', geoLocation=null, place=null, retweetCount=6, wasRetweetedByMe=false, contributors=null, annotations=null, retweetedStatus=null, userMentionEntities=[], urlEntities=[URLEntityJSONImpl{start=34, end=54, url=http://t.co/kTNqJFJC, expandedURL=http://nyti.ms/pR9DfX, displayURL=nyti.ms/pR9DfX}], hashtagEntities=[], user=UserJSONImpl{id=14434070, name='NYTimes Bits Blog', screenName='nytimesbits', location='The Cloud', description='News and analysis on tech and business. Also here: select retweets from NYT tech writers and friends. Account maintained by David F. Gallagher (#davidfg).', isContributorsEnabled=true, profileImageUrl='http://a1.twimg.com/profile_images/108833947/bits75_normal.jpg', profileImageUrlHttps='https://si0.twimg.com/profile_images/108833947/bits75_normal.jpg', url='http://nytimes.com/bits', isProtected=false, followersCount=53180, status=null, profileBackgroundColor='9ae4e8', profileTextColor='000000', profileLinkColor='0000ff', profileSidebarFillColor='e0ff92', profileSidebarBorderColor='87bc44', profileUseBackgroundImage=true, showAllInlineMedia=false, friendsCount=139, createdAt=Fri Apr 18 20:49:26 IST 2008, favouritesCount=5, utcOffset=-18000, timeZone='Eastern Time (US & Canada)', profileBackgroundImageUrl='http://a3.twimg.com/profile_background_images/4780380/twitter_post.png', profileBackgroundImageUrlHttps='https://si0.twimg.com/profile_background_images/4780380/twitter_post.png', profileBackgroundTiled=true, lang='en', statusesCount=6360, isGeoEnabled=false, isVerified=true, translator=false, listedCount=4671, isFollowRequestSent=false}}, userMentionEntities=[UserMentionEntityJSONImpl{start=3, end=15, name='NYTimes Bits Blog', screenName='nytimesbits', id=14434070}], urlEntities=[URLEntityJSONImpl{start=51, end=71, url=http://t.co/kTNqJFJC, expandedURL=http://nyti.ms/pR9DfX, displayURL=nyti.ms/pR9DfX}], hashtagEntities=[], user=UserJSONImpl{id=17989546, name='Wolfgang Fasching-K.', screenName='wwwof', location='Vienna', description='Digital ist besser. Fokus: IT & Internet, World News & US Politik, Medien & Pop/Kultur. http://www.riverone.at', isContributorsEnabled=false, profileImageUrl='http://a0.twimg.com/profile_images/67758989/SF050069-w_normal.JPG', profileImageUrlHttps='https://si0.twimg.com/profile_images/67758989/SF050069-w_normal.JPG', url='null', isProtected=false, followersCount=59, status=null, profileBackgroundColor='C0DEED', profileTextColor='333333', profileLinkColor='0084B4', profileSidebarFillColor='DDEEF6', profileSidebarBorderColor='C0DEED', profileUseBackgroundImage=true, showAllInlineMedia=false, friendsCount=64, createdAt=Tue Dec 09 17:09:35 IST 2008, favouritesCount=0, utcOffset=3600, timeZone='Vienna', profileBackgroundImageUrl='http://a3.twimg.com/profile_background_images/234523169/Naschmarkt-Wien-Juni10-2010-s.jpg', profileBackgroundImageUrlHttps='https://si0.twimg.com/profile_background_images/234523169/Naschmarkt-Wien-Juni10-2010-s.jpg', profileBackgroundTiled=true, lang='en', statusesCount=269, isGeoEnabled=false, isVerified=false, translator=false, listedCount=4, isFollowRequestSent=false}}

For JSON parsing, I recommend Jackson. Also, in order to validate your input, you should have a look at JSON Schema (for which I have an implementation if you want).
Here is how to parse a JSON in a string using Jackson:
final ObjectMapper mapper = new ObjectMapper();
final JsonNode node = mapper.readTree(yourInput);
// Access members:
node.get(0); // access node 0 of an array
for (final JsonNode entry: node) {
... // cycle through array nodes
}
node.get("foo"); // access property "foo" of an object
node.get("foo").getTextValue(); // access as a text
// etc etc
It also has a s*load of options to serialize to POJOs if that's what you want.

Your first step will be to parse the JSON to get an object graph, using a library like gson or any of several others.
Then (and this seems really general, but it's a pretty open question) it's a matter of determining what the schema should be, creating the tables, and looping through the object graph populating them.
You might look at "document databases" (so-called NoSQL) rather than SQL ones if you're allowed to, as they usually allow the schema to be more fluid.

If your problem is just with Twitter, you can look for dedicated APIs like Twiiter4J or Spring Social, that should provides ready java beans for tweets.
If you're realizing a small project Gson is the best solution for parsing. But if your making something more sophisticated, I suggest you to use Jackson for parsing and Hibernate as a middleware between application and sql database.

Related

how to read recurring events from outlook pst file using libpst

I'm using the java libpst 0.9.3 (https://mvnrepository.com/artifact/com.pff/java-libpst/0.9.3) to read calendar events from a local outlook pst file.
I struggle to get the proper information for recurring appopintments incl. exceptions.
When scanning the appointments, I use
myAppointment.getRecurrenceType()!= 0
to check whether this is an recurring event and then try to use the class
PSTAppointmentRecurrence(byte[] recurrencePattern,
PSTAppointment appt,
PSTTimeZone tz)
to create a recurrend appointment object but I fail to create an instance of this class.
I tried this:
myRecurrendAppointment= new PSTAppointmentRecurrence(myAppointment.getRecurrenceStructure(), myAppointment, ???)
??? tz= where do I get the system timezone as an instance of PSTTimeZone?
thanks in advance
Chris

How to get liferay form name through code in java

I am writing the code to fetch liferay forms through java but I am unable to do so. I am using recordset to do it but it is only fetching DDL recordset not forms.
public List<DDLRecordSet> getRecordSets() {
long groupId = themeDisplay.getScopeGroupId();
System.out.println("groupId:::"+groupId);
long liveGroupId = themeDisplay.getScopeGroup().getLiveGroupId();
System.out.println("LiveGroupID::"+ liveGroupId);
List<DDLRecordSet> results = DDLRecordSetLocalServiceUtil.getRecordSets(themeDisplay.getScopeGroupId());
if (liveGroupId != 0) {
results.addAll(DDLRecordSetLocalServiceUtil.getRecordSets(groupId));
}
System.out.println("Result::"+ results);
return result;
}
The output is shown only for DDL recordset not of form recordset.
Output=
Result::[{mvccVersion=0, uuid=7ca2308a-29f0-b344-8eeb-13eda08fe745, recordSetId=46003, groupId=20124, companyId=20101, userId=35402, userName=Test1 Test1, versionUserId=0, versionUserName=, createDate=Fri Jun 11 05:06:35 GMT 2021, modifiedDate=Fri Jun 11 05:06:35 GMT 2021, DDMStructureId=34991, recordSetKey=46002, version=1.0, name=, description=, minDisplayRows=10, scope=0, settings=, lastPublishDate=null}]
Please let me know where am I wrong.
To fetch all the created forms of a group/site, you can fetch the DDMFormInstances via DDMFormInstanceLocalServiceUtil.getFormInstances(groupId) (or better to use osgi references instead). For me, it is not clear what is required in your scenario (form, form-fields, form-data).
If you want the submitted data (depending on your handler this might differ), you can usually look for the corresponding DDMFormInstanceRecord and check the storageId which might be contentId of DDMContent which holds your data (usually in json).

RestFB get events in Java

Using FQL, by means of which finds events that contain a given word. FQL works only in API version <2.1. By which I use the Graph API Explorer to display events. Eg.
search?q=york&type=event
Example of FQL
SELECT Eid, name, location, start_time, description, pic_small, creator, event venue FROM WHERE start_time> "Sun Jun 21 0:00:35 GMT 2015" AND (CONTAINS ("york")
I would like to make a search events by using RestFB not using FQL, but do not know how. The documentation is scarce.
I answered this already on github, but perhaps someone else find this useful.
Your special case is not in the documentation, but you can transfer the knowledge you find in the documentation and solve your problem: http://restfb.com/#searching
Connection<Event> eventList =
facebookClient.fetchConnection("search", Event.class,
Parameter.with("q", "york"), Parameter.with("type", "event"));
Now, you can iterate over the eventList.
Here you can find how this can be done: http://restfb.com/#fetching-connections

regex extraction of data

I have a hundred Whois files of different top level domains(.com, .se, .uk, .cz etc.). Each has a different format. My main task is to extract information such as registrar, registrant, expiry date, updated date etc. The below code works for com. net. org & info. I am using Java SE 6.
Admin contact: "\\bAdmin\\sEmail:\\s*\\w+\\-*\\w*\\.*\\w*#\\w+(\\.\\w+)+"
Technical contact: "\\bTech\\sEmail:\\s*\\w+\\-*\\w*\\.*\\w*#\\w+(\\.\\w+)+"
Whois Registrant: "\\bRegistrant\\sName:\\s*\\w+\\-*\\.*\\w+\\s*\\w*"
Registrar: "\\bRegistrar:\\w+\\.*\\w*"
Registered on Date: "\\bCreation\\sDate:\\s*\\d+-\\d+-\\d+T\\d+:\\d+:\\d+Z"
Expiry Date: "\\bExpiry\\sDate:\\s*\\d+-\\d+-\\d+T\\d+:\\d+:\\d+Z"
Updated Date: "\\bUpdated\\sDate:\\s*\\d+-\\d+-\\d+T\\d+:\\d+:\\d+Z"
Name Servers: "\\bName\\sServer:\\s*\\w+\\d*\\.*\\w*\\-*\\w*(\\.\\w+)+"
Registrant Status: "\\bDomain\\sStatus:\\s*\\w+"
How do I add alternatives for each of the above points for other TLDs. For example :
I would like to have Name Servers:
"\\bName\\sServer:\\s*\\w+\\d*\\.*\\w*\\-*\\w*(\\.\\w+)+"
OR
alternative pattern
OR
alternative Pattern
Is it doable? If not is there an alternative way?
Alternative patterns can be concatenated with the | operator:
"\\bName\\sServer:\\s*\\w+\\d*\\.*\\w*\\-*\\w*(\\.\\w+)+|alternative pattern|alternative Pattern"
(If this isn't what you need, then your question should be reformulated.)

How to show formatted contact numbers?

I have a field called phone number which has values such as 0833888999 that I want to format as following 0833 888 999.
The answer that Rachana offered blew is true for few countries but not all countries.
Therefore,
I am using this Google library to format contact numbers from different countries; however, the problem is that after persisting the contact numbers in database I can not search for them on database, for example the library would format contact numbers of different countries in different format, for example add space or "-" between them that makes hibernate unable to find them.
+18182223333 >>> +1 818-222-3333
+441135558888 >>> +44 113 555 8888
Hibernate
.add(Restrictions.ilike("user.phone","+18182223333");
Try this
<s:property value="getText('{0,number,0### ### ###}',{phone})"/>
Where,phone=0833888999
Hope this will help you also see Using Struts2 Tags to Formatting Numbers
you will get clear idea about number formatting
I think you should keep the raw phone number (e.g. 0833888999) in the database and it's the View responsibility to format it accordingly.
You can have a separate table "country_phone_format" holding a country_id and a phone_format and you could fetch the phone_format as a #ManyToOne entity so that you have both the raw data and the format to properly display it into the View.
The PhoneFormat could be easily cached with the second level cache, as they should be rarely modified.
Just as Vlad mentions, you should keep the raw phone number (e.g. 0833888999) in the database (or save it with the country and area code if you prefer) and leave th responsibility to format it accordingly to the View.
You can use Type Conversion to convert to/from the format you desire - and take advantage of the google library you mention. Something like the following can get you started (abbreviated just to get the gist of it):
public class MyConverter extends StrutsTypeConverter {
public Object convertFromString(Map context, String[] values, Class toClass) {
String phoneNr = values[0]
MyPhoneObject phone = googleLibrary.doYourMagic(phoneNr);
return phone;
}
public String convertToString(Map context, Object o) {
googleLibrary.parseString( ((MyPhoneObject)o).rawPhoneNr() );
}
}
Don't forget to register the converter in xwork-conversion.properties
my.PhoneObject=path.to.MyConverter

Categories