I have a problem: I need to parse a JSON file in Java where each line represents a tweet and follows the standard JSON of Twitter. I do not need all the information, I attach two photos to show you which fields I need. I would do it without using any support library. Thank you!
This is what I did for now. I do not think it's the best way to do it, especially going ahead I'll be in trouble because the names of many fields repeat
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
public class TweetCorpus implements Iterable<Tweet>
{
private List<Tweet> tweets;
public static TweetCorpus parseFile(File file)
{
List<Tweet> tweets = new ArrayList<>();
try(BufferedReader br = Files.newBufferedReader(file.toPath()))
{
while(br.ready())
{
String tweet = br.readLine();
//System.out.println(tweet);
if(!tweet.isEmpty())
{
long l = Long.parseLong(tweet.substring(tweet.indexOf("\"id\":") + 5, tweet.indexOf(",\"id_str\":")));
String t = tweet.substring(tweet.indexOf(",\"text\":\"") + 9, tweet.indexOf(",\"source\":"));
tweets.add(new Tweet(l, t));
}
}
}
catch(IOException e)
{
e.printStackTrace();
}
return new TweetCorpus(tweets);
}
public int getTweetCount() { return tweets.size(); }
public TweetCorpus(List<Tweet> tweets)
{
this.tweets = tweets;
}
#Override
public Iterator<Tweet> iterator()
{
return tweets.iterator();
}
public static void main(String[] args)
{
TweetCorpus t = parseFile(new File("C:\\Users\\acer\\Desktop\\Moroder\\Uni\\1 Anno - 2 Semestre\\Metodologie Di Programmazione\\Progetto\\HM4Test\\tweetsCorpus.js"));
t.getTweetCount();
}
}
json media/retweet tweet
json "normal" tweet
You can use Gson or Jackson java library to parse json to Tweet object. Their are tools online which generates pojo from json, which you can use with jackson to parse your json string to object.
Once you have json values in an object, you can use getters/setters to extract/modify the values you are interested in from input json.
Well writing your own parser would be a reinventing the wheel kind of task. But if your need is to write your own parser, refer to jackson project on github for inspiration on design and maintenance.
This would help you in making a generic application.
Quick reference for jackson parser ,
https://dzone.com/articles/processing-json-with-jackson
Re-inventing a JSON parser using only readLine() is a really bad idea. If you don't have experience writing parsers by hand, you will end up with a lot of bad code that is really hard to understand. Just use a library. There are tons of good JSON libraries for Java.
Jackson
GSON
Boon
Example code:
static class User {
String id, name;
}
static class MyTweet {
String id, text;
User user;
}
// if the entire file is a JSON array:
void parse(Reader r) {
List<MyTweet> tweets = objectMapper.readValue(
r, new TypeReference<List<MyTweet>>(){});
}
// if each line is a single JSON object:
void parse(BufferedReader r) {
while (r.ready()) {
String line = r.readLine();
MyTweet tweet = objectMapper.readValue(line, MyTweet.class);
}
}
Related
This is my first post in StackOverflow!
I have been trying to upload a CSV file, parse it, and create a json file from it.
I found this tutorial using Jackson parser: https://kalliphant.com/jackson-convert-csv-json-example/
(I picked that because of the speed of processing), but I found errors when I tried it and I did not find why am I getting those, I am using intellij IDEA actually, and I tried using reload project and download sources from maven, but it did not solve the issue.
I have been looking for this error on the internet but I did not find any relevant.
I put the exact same code from the tutorial and I am getting those errors:
Errors I got:
Thank you!
The errors you're getting appear to be related to bad imports at your class file. Now even if that was OK, what you're trying to do you still not work.
Firstly, your CSV file is missing a header (above country), secondly as with normal JSON serialization/deserialization you need to perform this action against an object (a simple pojo). In your case your attempting to do this using object which is wrong -- both syntactically as well as conceptually.
With the above on hand try the following. Modify your CSV file to look like so:
country population mortality
spain 13000000 10000
france 30000000 15000
united kingdom 40000000 22000
belgium 20000000 50000
us 25000000 30000
The try the following code:
public class CsvParser {
public static void main(String... args) {
CsvSchema schema = CsvSchema
.emptySchema()
.withHeader();
ObjectReader reader = new CsvMapper()
.readerFor(MortalityEntry.class)
.with(schema);
List<MortalityEntry> results = null;
try {
MappingIterator<MortalityEntry> iterator = reader.readValues(Paths.get("/tmp/input.csv").toFile());
results = iterator.readAll();
} catch (IOException e) {
e.printStackTrace();
}
ObjectMapper objectMapper = new ObjectMapper();
try {
objectMapper.writeValue(Paths.get("/tmp/output.json").toFile(), results);
} catch (IOException e) {
e.printStackTrace();
}
}
private static class MortalityEntry {
private String country;
public String getCountry() { return country; }
public void setCountry(String country) { this.country = country; }
private Integer population;
public Integer getPopulation() { return population; }
public void setPopulation(Integer population) { this.population = population; }
private Integer mortality;
public Integer getMortality() { return mortality; }
public void setMortality(Integer mortality) { this.mortality = mortality; }
}
}
As you can see I'm using a simple pojo MortalityEntry to deserialize (from CSV) and serialize (to JSON), letting Jackson do its magic.
This simple example should be enough to get you going.
I'm using Jackson 2.4 in Java to do some JSON legwork. I make a call to a remote server with Apache HttpGet, deserialize the results with Jackson into a POJO, manipulate those results, and then serialize them with Jackson to push back to a remote server with HttpPost.
The issue I'm finding is that Jackson is translating unicode literals into unicode characters, which I need it not to do thanks to encoding issues on each end. For example, I might have this in the JSON:
"field1": "\u00a2"
But Jackson is converting the "\u00a2" to "ยข" when it's deserialized, which causes problems with the remote server. It has to be maintained as escaped unicode. If I use something like Apache EntityUtils (specifying UTF-8) or even make the call from my web browser to get the data, the escaped unicode is preserved, so I know that it's coming in properly from the server. If I have Jackson consume the input stream from the entity on the response, it does the conversion automatically.
I've tried writing with a JsonGenerator that is explicitly set to UTF-8 to write to the HttpPost. It didn't work, remote server still rejected it. I've dug through the configuration options for ObjectMapper and JsonParser, but I don't see anything that would override this behavior. Escaping non-ASCII, sure, but that's not what I need to do here. Maybe I'm missing something obvious, but I can't get Jackson to deserialize this string without replacing the escaped unicode.
EDIT: Well, my bad, the only literals having problems have 3 or 5 leading slashes, not just one. That's some screwiness, but Java seems to be what's unpacking it by default during the deserialization, even if the raw text that came back from the server preserves it. Still not sure how to get Java to preserve this without checking an insane amount of text.
What you are expecting is outside scope of Jackosn. It's java that converts the String while reading it. For same reason, if you have a properties file with value \u00a2 and read it using jdk API, you will get converted value. Depending on the file size, either you can double escape char \ before passing the string to Json or "escape" the string back using your Deserializer (only for string) and something like below:
Thank you
package com.test.json;
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.DeserializationContext;
import com.fasterxml.jackson.databind.JsonDeserializer;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.module.SimpleModule;
import java.io.IOException;
import java.util.Map;
public class Jackson {
static ObjectMapper _MAPPER = new ObjectMapper();
public static void main(String[] args) throws Exception {
String json = "{\"field1\": \"\\u00a2\",\"field2\": \"\\u00a2 this\",\"numberField\": 121212}";
SimpleModule testModule
= new SimpleModule("StOvFl", _MAPPER.version()).addDeserializer(String.class,
new UnEscapedSerializaer());
_MAPPER.registerModule(testModule);
Map m = _MAPPER.readValue(json, new TypeReference<Map<String, Object>>() {
});
System.out.println("m" + m);
}
}
class UnEscapedSerializaer extends JsonDeserializer<String> {
#Override
public String deserialize(JsonParser jp, DeserializationContext ctxt)
throws IOException, JsonProcessingException {
String s = jp.getValueAsString();
return org.apache.commons.lang.StringEscapeUtils.StringEscapeUtils.escapeJava(s);
}
}
Another way to custom Jackson's behavior is customized JsonParser. See jackson's source code of JsonFactory, ReaderBasedJsonParser;
The key methond is _finishString2() which is used to do 'decodeEscaped', so we can write a JsonParser extends ReaderBasedJsonParser and override the _finishString2 method:
public class MyJsonParser extends ReaderBasedJsonParser {
#Override
protected void _finishString2() throws IOException {
char[] outBuf = _textBuffer.getCurrentSegment();
int outPtr = _textBuffer.getCurrentSegmentSize();
final int[] codes = _icLatin1;
final int maxCode = codes.length;
while (true) {
if (_inputPtr >= _inputEnd) {
if (!loadMore()) {
_reportInvalidEOF(": was expecting closing quote for a string value");
}
}
char c = _inputBuffer[_inputPtr++];
int i = (int) c;
if (i < maxCode && codes[i] != 0) {
if (i == INT_QUOTE) {
break;
} else {
//c = _decodeEscaped();
//do nth
}
}
// Need more room?
if (outPtr >= outBuf.length) {
outBuf = _textBuffer.finishCurrentSegment();
outPtr = 0;
}
// Ok, let's add char to output:
outBuf[outPtr++] = c;
}
_textBuffer.setCurrentLength(outPtr);
}
public static void main(String[] args) throws IOException {
String json = "{\"field1\": \"\\u00a2\",\"field2\": \"\\u00a2 this\",\"numberField\": 121212}";
ObjectMapper objectMapper = new ObjectMapper(new MyJsonParserFactory());
Object o = objectMapper.readValue(json, Object.class);
System.out.println(o);
}
}
Full demo code here
I'm developing desktop software with JavaFX and Java Spark which is basically a barebones framework for developing web apps, but I'm trying to use it strictly to put/get sensitive methods and variables on a server so that users can't access them. REST seems to be the correct approach but I'm struggling to understand 2 interrelated REST concepts. The Spark docs are light but I've integrated the few good tutorials into a working demo below but I've hit a wall. I'll briefly explain:
With the help of Postman I've been able to put a few records onto the server by using a path of http://localhost:4567/secrets and Body of:
{
"title" : "demofield1",
"content" : "12345"
}
Each record contains a title as an identifier (demofield1) and content as the sensitive data that should remain hidden from users at all times (12345). It's pretty trivial to put these strings onto a server and then get them by using title as a parameter, shown below. The demo code simply has a Model class for creating and returning a record (secret), a JSON conversion method, and a get and put Spark method. Secrets are stored locally in a HashMap for now, I'm assuming a real app would simply swap in a server DB.
The get method works as expected, returning the correct JSON record and storing the content as a String with this line: String secretString = model.getCertainSecret(title).getContent();
With that said...
Questions (partial answers fully appreciated too):
secretString above now holds a confidential value (12345) which is obtained using a supposedly secure REST method. But couldn't a user simply reverse-engineer my source code and write System.out.println(secretString) and have that 12345 revealed? I don't understand how a simple string is protected after retrieving it from the server, despite not being explicitly shown. The code seems correct yet the value is easily obtainable. What am I missing?
How do you put entire java methods on a server? A lot of code I need to protect isn't just strings but methods containing Tasks, Platform.runLater()->, and needs to interact with other desktop software. For example, one of my methods uses JACOB to identify when a certain area of a third-party software is clicked. I can't even fathom what a get/put would look like in that context.
My assumption was that a server-side DB would store all content from my put requests, but I don't understand how it stores and returns a method? Should I be reading about servlets or SaaS or something? I'm focused on desktop users.
Code:
import com.fasterxml.jackson.core.JsonParseException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;
import lombok.Data;
import org.apache.log4j.BasicConfigurator;
import java.io.IOException;
import java.io.StringWriter;
import java.util.HashMap;
import java.util.Map;
import static spark.Spark.get;
import static spark.Spark.put;
public class ServerDemo
{
private static final int HTTP_BAD_REQUEST = 400;
#Data
static class NewSecretPayload {
private String title;
private String content;
public boolean isValid() {
return title != null && !title.isEmpty();
}
}
public static class Model {
private int nextId = 1;
private Map<String, Secret> secrets = new HashMap<>();
#Data
class Secret {
private int id;
private String title;
private String content;
}
public int createSecret(String title, String content){
int id = nextId++;
Secret secret = new Secret();
secret.setId(id);
secret.setTitle(title);
secret.setContent(content);
secrets.put(title, secret);
return id;
}
public Secret getCertainSecret(String titleToUse){
if(null != secrets.get(titleToUse)){
return secrets.get(titleToUse);
}else{
return null;
}
}
}
public static String dataToJson(Object data) {
try {
ObjectMapper mapper = new ObjectMapper();
mapper.enable(SerializationFeature.INDENT_OUTPUT);
StringWriter sw = new StringWriter();
mapper.writeValue(sw, data);
return sw.toString();
} catch (IOException e) {
throw new RuntimeException("IOException from a StringWriter?");
}
}
public static void main(String[] args) {
Model model = new Model();
BasicConfigurator.configure();
put("/secrets", (request, response) -> {
try {
ObjectMapper mapper = new ObjectMapper();
NewSecretPayload creation = mapper.readValue(request.body(), NewSecretPayload.class);
if (!creation.isValid()) {
response.status(HTTP_BAD_REQUEST);
return "";
}
int id = model.createSecret(creation.getTitle(), creation.getContent());
response.status(200);
response.type("application/json");
return id;
} catch (JsonParseException jpe) {
response.status(HTTP_BAD_REQUEST);
return "";
}
});
get("/secrets/:title", (req, res) -> {
String title = req.params(":title");
if (model.getCertainSecret(title) != null) {
res.status(200);
res.type("application/json");
String secretString = model.getCertainSecret(title).getContent();
return dataToJson(model.getCertainSecret(title));
}
res.status(400);
return new ResponseError("No user with title "+title+" was found", title);
});
}
}
Lets dig down to your first problem "Keeping string secret" :--
Restrict : The simplest way is not to provide the data to malicious user.
Masking : Mask the data you are providing to end user. You will have the original data mapped to masked data. You will provide masked data to end user. Here the end user can never get the original data as it is a one way process. When end user sends masked-data you can always retrieve the original data from it.
Encrypting : If the end user needs to see the data you can encrypt it and send it. You can make a sanity check of your code before decrypting the data. The sanity check can give you idea if the code is ever modified. If code fails the sanity check you can always exit the process.
I have a JSON array that I'd like to map that looks like this:
{
"library": [
{
"key":"val"
},
{
"key":"val"
}
]
}
Is there a way to parse this using the object mapper starting at the array rather than at the root? I know you can do a manual node parse, but I would prefer not to do that if possible. any help with this would be greatly appreciated.
Jackson offers three principal ways to parse json: to a map, to an object, to a jackson node tree. None of these methods offer a way to start from anywhere other than the root. To start from somewhere other than the root, you need to parse your way to there from the root, which means you need to start parsing from the root! :)
That being said, if for example you use mapping to an object, it is very easy to get the array you need out of the object:
package test;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Map;
import org.codehaus.jackson.JsonParseException;
import org.codehaus.jackson.map.JsonMappingException;
import org.codehaus.jackson.map.ObjectMapper;
public class Test {
static String json = "{\"library\": [{\"key\":\"val\"},{\"key\":\"val\"}]}";
static class JsonClass {
private ArrayList<Map<?,?>> library;
public ArrayList<Map<?, ?>> getLibrary() {
return library;
}
public void setLibrary(ArrayList<Map<?, ?>> library) {
this.library = library;
}
}
public static void main(String[] args)
throws JsonParseException, JsonMappingException, IOException {
JsonClass parsed = new ObjectMapper().readValue(json, Test.JsonClass.class);
System.out.println(parsed.getLibrary());
}
}
Running this prints:
[{key=val}, {key=val}]
An alternative would be to use a streaming parser... it can pick any node, without bothering about understanding the whole structure. I believe Gson has that. But in your case it would probably be an overkill to use a streaming parser: it makes sense when the overall structure is complex, you need to process a big stream fast, and are interested in relatively small part of the data. These do not seem to apply to your scenario.
Google places API returns a JSON when it requested for a place under food category which includes the details of several places.
I want to create an object array where each object contains details of a specific place.
I have used GSON library for my implementation and it works fine for a dummy JSON object but not with the JSON result given from Google place API and 'JsonSyntaxException' is thrown.
I look for a solution for following matters..
1 How can I proceed with GSON and given JSON object to create my object array or
2 Is there any other way to accomplish my task (still using JSON result)
Thanks.
update
Class PlaceObject
import java.util.List;
public class PlaceObject {
private List<String> results;
}
Class JSONconverter
import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import com.google.gson.Gson;
public class JSONconverter {
public static void main(String[] args) {
Gson gson = new Gson();
try {
BufferedReader br = new BufferedReader(
new FileReader("c:\\placeAPI.json"));
//convert the json string back to object
PlaceObject obj = gson.fromJson(br, PlaceObject.class);
//obj.results = null ,when debugged thats the problem
System.out.println("Result: " + obj);
} catch (IOException e) {
e.printStackTrace();
}
}
}
The link of JSON
http://www.mediafire.com/?8mmnuxuopimhdnz
I like working with gson
btw. there is another relevant thread
Jersey client's documentation proposes to use Jackson library (see wiki)
You can also take a look at Genson library http://code.google.com/p/genson/.
It provides an out of box integration with jersey. You only need to have the jar in your classpath.