We Keep Coding

Hadoop read JSON from HDFS

I'm trying to read an JSON file into my hadoop mapreduce algorithm.
How can i do this? I've put a file 'testinput.json' into /input in my HDFS memory.
When calling the mapreduce i execute hadoop jar popularityMR2.jar populariy input output, with input stating the input directory in the dhfs memory.
public static class PopularityMapper extends Mapper<Object, Text, Text, Text>{
protected void map(Object key, Text value,
Context context)
throws IOException, InterruptedException {
JSONParser jsonParser = new JSONParser();
try {
JSONObject jsonobject = (JSONObject) jsonParser.parse(new FileReader("hdfs://input/testinput.json"));
JSONArray jsonArray = (JSONArray) jsonobject.get("votes");
Iterator<JSONObject> iterator = jsonArray.iterator();
while(iterator.hasNext()) {
JSONObject obj = iterator.next();
String song_id_rave_id = (String) obj.get("song_ID") + "," + (String) obj.get("rave_ID")+ ",";
String preference = (String) obj.get("preference");
System.out.println(song_id_rave_id + "||" + preference);
context.write(new Text(song_id_rave_id), new Text(preference));
}
}catch(ParseException e) {
e.printStackTrace();
}
}
}
My mapper function now looks like this. I read the file from the dhfs memory. But it always returns an error, file not found.
Does someone know how i can read this json into a jsonobject?
Thanks

FileReader cannot read from HDFS, only local Filesystem.
The filepath comes from the Job parameters - FileInputFormat.addInputPath(job, new Path(args[0]));
You wouldn't read the file in the Mapper class, anyway.
MapReduce defaults to read line-delimited files, so your JSON objects would have to be one per-line such as
{"votes":[]}
{"votes":[]}
From the mapper, you would parse the Text objects into JSONObject like so
protected void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
JSONParser jsonParser = new JSONParser();
try {
JSONObject jsonobject = (JSONObject) jsonParser.parse(value.toString());
JSONArray jsonArray = (JSONArray) jsonobject.get("votes");
If you only have one JSON object in the file, then you probably shouldn't be using MapReduce.
Otherwise, you would have to implement a WholeFileInputFormat and set that in the Job
job.setInputFormatClass(WholeFileInputFormat.class);

Tried reading the JSON from HDFS path using the following function using pydoop library and it is working as expected.Hope it helps.
import pydoop.hdfs as hdfs
def lreadline(inputJsonIterator):
with hdfs.open(inputJsonIterator,mode='rt') as f:
lines = f.read().split('\n')
return lines

How to get the differences between two jsonArrays?

I`m trying to get the diferences between two jsonArrays, in different files, and print it in a new file.
What is the best practice for this?
I hope you can help me.
Thanks!
I`m using eclipse. I´ve tried with Maps.difference reading the files with a fileReader.
enter code here ://Reading the file
File jsonInputFileMod = new File("../MENU.json");
InputStream isMod;
is = new FileInputStream(jsonInputFileMod);
// Create JsonReader from Json.
JsonReader readerMod = Json.createReader(is);
// Get the JsonObject structure from JsonReader.
JsonArray empObjMod = readerMod.readArray();
readerMod.close();
//Creating maps
Map [] mapArray = new Map [empObj.size()];
for(int i=0; i<empObj.size(); i++){
mapArray[i] = (Map) empObj.get(i);
}
Map [] mapArrayMod = new Map [empObjMod.size()];
for(int i=0; i<empObjMod.size(); i++){
mapArrayMod[i] = (Map) empObjMod.get(i);
//Comparation
if(mapArray.length==mapArrayMod.length){
String [] dif = new String [mapArray.length];
FileWriter salida = new FileWriter("../diferences.json");
for(int i=0; i<mapArray.length; i++){
dif[i] = Maps.difference(mapArray[i], mapArrayMod[i]).toString();
salida.write("\n\n JSON : " + i + "\n\n");
//salida.write(Maps.difference(mapArray[i], mapArrayMod[i]).toString().replace("[", "\n\t["));
salida.write(dif[i]);
}
salida.close();

Character encoding of parsed strings is wrong only after building jar

I am writing a program that generates PDF files of printable exams. I have all the exam questions stored in a JSON file. The catch is that the exam is in Czech, so there are many special characters (specifically ěščřžýáíé). When I run the program in Idea, it works perfectly - the output is exactly as it is supposed to be.
But when I build the jar executable, the generated files have chunks of wrong encoded text. Specifically anything that went through the JSON parser. Everything hard coded like headers etc. is encoded properly, so the mistake must be in the parser.
The JSON input file is encoded in UTF-8.
I use these two methods to parse the JSON file.
private static Category[] parseJSON(){
JSONParser jsonParser = new JSONParser();
Category[] categories = new Category[0];
try (FileReader reader = new FileReader("otazky.json")){
// Read JSON file
Object obj = jsonParser.parse(reader);
JSONArray categoryJSONList = (JSONArray) obj;
java.util.List<JSONObject> categoryList = new ArrayList<>(categoryJSONList);
categories = new Category[categoryJSONList.size()];
int i = 0;
for (JSONObject category : categoryList) {
categories[i] = parseCategoryObject(category);
i++;
}
} catch (ParseException | IOException e) {
e.printStackTrace();
}
return categories;
}
private static Category parseCategoryObject(JSONObject category) {
String categoryName = (String) category.get("name");
int generateCount = (int) (long) category.get("generateCount");
JSONArray questionsJSONArray = (JSONArray) category.get("questions");
java.util.List<JSONObject> questionJSONList = new ArrayList<>(questionsJSONArray);
Question[] questions = new Question[questionJSONList.size()];
int j = 0;
for (JSONObject question : questionJSONList) {
JSONArray answers = (JSONArray) question.get("answers");
String s = (String) question.get("question");
String[] a = new String[answers.size()];
for (int i = 0; i < answers.size(); i++) {
a[i] = answers.get(i).toString();
}
int c = (int) (long) question.get("correct");
Question q = new Question(s, a, c);
questions[j] = q;
j++;
}
return new Category(categoryName, questions, generateCount);
}
The output looks like this:
...
Právnà norma:
a) je obecnÄ› závaznĂ© pravidlo chovánĂ, kterĂ© nemusĂ mĂt urÄŤitou formu,
b) nemĹŻĹľe bĂ˝t součástĂ právnĂho pĹ™edpisu,
...
While I would need it to look like this:
...
Právní norma:
a) je obecně závazné pravidlo chování, které nemusí mít určitou formu,
b) nemůže být součástí právního předpisu,
...

Benjamin Urquhart suggested that I try using InputStringReader and FileInputStream instead of FileReader to read the file, because with FileReader you cannot specify the encoding (system default is used). I find those two methods hard to use, but I found an alternative - Files.readAllLines, which is fairly easy to use, and it worked.

Loop through json file appending key value to List

I am trying to loop through a JSON file and append the value each time to patientList. So far I believe I have done the hard part, however the simplest part seems to be taking a lot of time, that is appending the values to patientList. My getJsonFile method gets the path of the JSON file. The format of the JSON file is below. I am able to print jsonArray so I know I am good up to that point, but lost after that.
Json file.
[{"patient":1},{"patient":2},{"patient":3},{"patient":4},{"patient":5},{"patient":6},{"patient":7},{"patient":8},{"patient":9}]
getJsonFile method.
private List<Integer> getJsonFile(String path)
{
List<Integer> patientList = new ArrayList<>();
try (FileReader reader = new FileReader(path))
{
JSONParser jsonParser = new JSONParser();
JSONArray jsonArray = (JSONArray)jsonParser.parse(reader);
System.out.println(jsonArray);
// Update patientList
for (int i = 0; i < jsonArray.size(); i++ )
{
patientList.add(jsonArray(i));
}
}
catch(IOException | ParseException | NullPointerException e)
{
e.printStackTrace();
}
return patientList;
}

Your JSONArray contains objects: {"patient":1}
So you could not add patientList.add(jsonArray(i));
You have to access the int value inside that object:
JSONObject patient = jsonArray.getJsonObject(i);
patientList.add(patient.getInt("patient");
Edit
Well. You are using the simple-json library with quite limit feature and outdated. In this case you have to cast the data yourself:
JSONObject patient = (JSONObject)jsonArray.get(i);
patientList.add((Integer)patient.get("patient");
I recommend you remove this lib and use existing JSON feature of Java. If you want more advance feature, Jackson/GSon is the library to use.

Parsing list of json

I need to parse list of json stored in a single file !
What I have done so far is,
test.json file contains:
{"location":"lille","lat":28.4,"long":51.7,"country":"FR"}
with this file I have the code below
public class JsonReader {
public static void main(String[] args) throws ParseException {
JSONParser parser = new JSONParser();
try {
Object obj = parser.parse(new FileReader("c:\\test.json"));
JSONObject locationjson= (JSONObject) obj;
String location= (String) locationjson.get("location");
System.out.printf("%s",location);
long lat = (Long) locationjson.get("lat");
System.out.printf("\t%d",lat);
//similarly for other objects
This is a working code and I am able to print only one json in the file test.json
Now if I have to print a list of json in file: test1.json as shown below: each line is a single valid json and there are list of json in a single file. What I need is to parse each json and print it in each line. Will using a bean class work?
{"Atlas":{"location":"lille","lat":28.4,"long":51.7,"country":"FR"}}
{"Atlas":{"location":"luxum","lat":24.1,"long":54.7,"country":"LU"}}
{"Atlas":{"location":"ghent","lat":28.1,"long":50.1,"country":"BE"}}
{"Atlas":{"location":"alborg","lat":23.4,"long":53.7,"country":"DN"}}
Your help is appreciated !

The JSON should have a root node.
If you don't have that, you can read from the file line-by-line, and pass each line into the JSONParser wrapped in a StringReader (since the JSONParser.parse() method takes a Reader).
e.g.
BufferedReader in
= new BufferedReader(new FileReader("test.json"));
while(!done) {
String s = in.readLine();
if (s == null) {
done = true;
}
else {
StringReader sr = new StringReader(s);
// etc...
}
}
Edit: I've assumed you're using JSONParser. If you're using a different parser (which one?) then it may take a String argument.

JSONParser.parse() also takes the String as an aurgument.
Read the file with FileReader and for each line that you read use JSONParser.parse(String) method.

First of all:make sure you have created a class just for the son item for each row that has the string properties in the header file and synthesized in the implementation file.Then create a property array in the implementation file that will be doing the parsing.In the parse file..use NSJSONSerialization in your retrieve data method:
-(void)retrieveData{
NSURL * url =[NSURL URLWithString:getDataURL];
NSData *data =[NSData dataWithContentsOfURL:url];
json = [NSJSONSerialization JSONObjectWithData:data options:kNilOptions error:nil];
newsArray = [[NSMutableArray alloc]init];
for (int i = 0; i < json.count; i++){
{
NSString * cID =[[json objectAtIndex:i]objectForKey:#"Name1"];
NSString * cName = [[json objectAtIndex:i]objectForKey:#"name2"];
NSString * cState =[[json objectAtIndex:i]objectForKey:#"name3"];
NSString * cPopulation =[[json objectAtIndex:i]objectForKey:#"Edition"];
NSString * cCountry =[[json objectAtIndex:i]objectForKey:#"name4"];
City *myCity = [[City alloc]initWithCityID:cID andCityName:cName andCityState:cState andCityPopulation:cPopulation andCityCountry:cCountry];
[newsArray addObject:myCity];
}
[self.myTableView reloadData];
}
}
Then retrieve the son objects or arrays and enter them in the name 1 name 2 section to parse them to your table.Also make sure you have already setup your Table and assigned it a cell identifier and indexed to row.
-(UITableViewCell *)tableView:(UITableView *)tableView cellForRowAtIndexPath:(NSIndexPath *)indexPath{
//static NSString *strIdentifier=#"identity";
static NSString *cellIdentifier = #"Cell";
UITableViewCell *cell = [tableView dequeueReusableCellWithIdentifier:cellIdentifier];
if(cell == nil){
cell = [[UITableViewCell alloc]initWithStyle:UITableViewCellStyleSubtitle reuseIdentifier:cellIdentifier];
}
// cell.textLabel.text = [NSString stringWithFormat:#"Cell %d",indexPath.row];
City *currentCity =[newsArray objectAtIndex:indexPath.row];
cell.textLabel.text = currentCity.Name1;
cell.detailTextLabel.text = currentCity.Name2;
return cell;
}