I'm using Java to convert JSON to Avro and store these to GCS using Google DataFlow.
The Avro schema is created on runtime using SchemaBuilder.
One of the fields I define in the schema is an optional LONG field, it is defined like this:
SchemaBuilder.FieldAssembler<Schema> fields = SchemaBuilder.record(mainName).fields();
Schema concreteType = SchemaBuilder.nullable().longType();
fields.name("key1").type(concreteType).noDefault();
Now when I create a GenericRecord using the schema above, and "key1" is not set, when putting the resulted GenericRecord to the context of my DoFn: context.output(res); I get the following error:
Exception in thread "main" org.apache.beam.sdk.Pipeline$PipelineExecutionException: org.apache.avro.UnresolvedUnionException: Not in union ["long","null"]: 256
I also tried doing the same thing with withDefault(0L) and got the same result.
What do I miss?
Thanks
It works fine for me when trying as below and you can try to print the schema that will help to compare also you can remove the nullable() for long type to try.
fields.name("key1").type().nullable().longType().longDefault(0);
Provided the complete code that I used to test:
import org.apache.avro.AvroRuntimeException;
import org.apache.avro.Schema;
import org.apache.avro.SchemaBuilder;
import org.apache.avro.SchemaBuilder.FieldAssembler;
import org.apache.avro.SchemaBuilder.RecordBuilder;
import org.apache.avro.file.DataFileReader;
import org.apache.avro.file.DataFileWriter;
import org.apache.avro.generic.GenericData.Record;
import org.apache.avro.generic.GenericDatumReader;
import org.apache.avro.generic.GenericDatumWriter;
import org.apache.avro.generic.GenericRecord;
import org.apache.avro.generic.GenericRecordBuilder;
import org.apache.avro.io.DatumReader;
import org.apache.avro.io.DatumWriter;
import java.io.File;
import java.io.IOException;
public class GenericRecordExample {
public static void main(String[] args) {
FieldAssembler<Schema> fields;
RecordBuilder<Schema> record = SchemaBuilder.record("Customer");
fields = record.namespace("com.example").fields();
fields = fields.name("first_name").type().nullable().stringType().noDefault();
fields = fields.name("last_name").type().nullable().stringType().noDefault();
fields = fields.name("account_number").type().nullable().longType().longDefault(0);
Schema schema = fields.endRecord();
System.out.println(schema.toString());
// we build our first customer
GenericRecordBuilder customerBuilder = new GenericRecordBuilder(schema);
customerBuilder.set("first_name", "John");
customerBuilder.set("last_name", "Doe");
customerBuilder.set("account_number", 999333444111L);
Record myCustomer = customerBuilder.build();
System.out.println(myCustomer);
// writing to a file
final DatumWriter<GenericRecord> datumWriter = new GenericDatumWriter<>(schema);
try (DataFileWriter<GenericRecord> dataFileWriter = new DataFileWriter<>(datumWriter)) {
dataFileWriter.create(myCustomer.getSchema(), new File("customer-generic.avro"));
dataFileWriter.append(myCustomer);
System.out.println("Written customer-generic.avro");
} catch (IOException e) {
System.out.println("Couldn't write file");
e.printStackTrace();
}
// reading from a file
final File file = new File("customer-generic.avro");
final DatumReader<GenericRecord> datumReader = new GenericDatumReader<>();
GenericRecord customerRead;
try (DataFileReader<GenericRecord> dataFileReader = new DataFileReader<>(file, datumReader)){
customerRead = dataFileReader.next();
System.out.println("Successfully read avro file");
System.out.println(customerRead.toString());
// get the data from the generic record
System.out.println("First name: " + customerRead.get("first_name"));
// read a non existent field
System.out.println("Non existent field: " + customerRead.get("not_here"));
}
catch(IOException e) {
e.printStackTrace();
}
}
}
If I understand your question correctly, you're trying to accept JSON strings and save them in a Cloud Storage bucket, using Avro as your coder for the data as it moves through Dataflow. There's nothing immediately obvious from your code that looks wrong to me. I have done this, including saving the data to Cloud Storage and to BigQuery.
You might consider using a simpler, and probably less error prone approach: Define a Java class for your data and use Avro annotations on it to enable the coder to work properly. Here's an example:
import org.apache.avro.reflect.Nullable;
import org.apache.beam.sdk.coders.AvroCoder;
import org.apache.beam.sdk.coders.DefaultCoder;
#DefaultCoder(AvroCoder.class)
public class Data {
public long nonNullableValue;
#Nullable public long nullableValue;
}
Then, use this type in your DnFn implementations like you likely already are. Beam should be able to move the data between workers properly using Avro, even when the fields marked #Nullable are null.
I'm trying to read a JSON file in my java application using the org/json/json/20171018 repository (http://central.maven.org/maven2/org/json/json/20171018/ -> json-20171018.jar). My JSON file looks like this:
{
"manifest_version": 2,
"name": "Chrome Extension",
"version": "0.1",
"permissions": [
"tabs"
],
"content_scripts": [
{
"matches": [
"<all_urls>"
],
"js": ["content.js"]
}
],
"background": {
"matches": [
"google.de",
"youtube.com",
"wikipedia.de"
],
"scripts": ["background.js"],
"persistent": true
}
}
I'm interested in the background section more specific in the links the background matches to. So I've created first a JSONObject of the whole file, then a JSONObject of the background section and then a JSONArray of the type matches. But unfortunately I'm getting this error showing up when I run the program:
Exception in thread "main" org.json.JSONException: JSONObject["matches"] not found.
at org.json.JSONObject.get(JSONObject.java:520)
at org.json.JSONObject.getJSONArray(JSONObject.java:714)
at Json.main(Json.java:19)
My java code looks like this:
import org.json.JSONArray;
import org.json.JSONObject;
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;
public class Json {
public static void main(String[] args){
String loc = new String("chromeAdon/manifest.json");
File file = new File(loc);
try {
String content = new String(Files.readAllBytes(Paths.get(file.toURI())));
JSONObject json = new JSONObject(content);
JSONObject json2 = new JSONObject(json.getJSONObject("background"));
JSONArray jarray = json2.getJSONArray("matches");
for (int i=0;i<jarray.length();i++){
System.out.println(jarray.getString(0));
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
Does anyone know where my mistake is?
You are wrapping the JSON object returned by getJSONObject("background"), which is not needed.
Try just using the returned object:
JSONObject jsonContent = new JSONObject(content);
JSONObject jsonBackground = jsonContent.getJSONObject("background");
JSONArray jsonArrayMatches = jsonBackground.getJSONArray("matches");
below is my java code to read from a json file
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.util.Iterator;
import org.json.simple.JSONArray;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
import org.json.simple.parser.ParseException;
public class FileIO {
#SuppressWarnings("unchecked")
//public static Showtime [] movie = MoviesListing.movie();
public void importTxt() {
int j = 0;
JSONParser parser = new JSONParser();
try {
JSONArray a = (JSONArray) parser.parse(new FileReader("C:/Users/Glambert/Dropbox/java/New folder/perfection/UPdate/json.txt"));
for (Object o : a)
{
JSONObject person = (JSONObject) o;
MoviesListing.movie[j].assign((String) person.get("moviename"));
MoviesListing.movie[j].assignShowingStatus((String) person.get("showingstatus"));
MoviesListing.movie[j].assignSynopsis((String) person.get("synopsis"));
MoviesListing.movie[j].assignDirector((String) person.get("director"));
MoviesListing.movie[j].assignCast((String) person.get("cast"));
MoviesListing.movie[j].assignReviewerRate((double) person.get("reviewerRate"));
MoviesListing.movie[j].assignPastReviews((String) person.get("pastreviews"));
MoviesListing.movie[j].assignMovieRating((String) person.get("movierating"));
MoviesListing.movie[j].assignShowtimes((JSONArray) person.get("showtimes"));
j++;
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} catch (ParseException e) {
e.printStackTrace();
}
}
It works from reading a json file like this:
[
{
"moviename":"Goosebumps",
"showingstatus":"Now Showing",
"synopsis":"Upset...",
"director":"Rob Letterman",
"cast":"Jack Black, Dylan Minnette, Odeya Rush, Amy Ryan",
"reviewerRate": 3.2 ,
"pastreviews":"Goosebumps...",
"movierating":"PG",
"showtimes":[
"09.30",
"12.30"
],
"job":"Teacher"
},
{
"moviename":"Bridge of Spies",
"showingstatus":"Now Showing",
"synopsis":"Tom Hanks stars..",
"director":"Steven Spielberg",
"cast":"Tom Hanks, Mark Rylance",
"reviewerRate": 3.9,
"pastreviews":"JBridge...",
"movierating":"PG-13",
"showtimes":[
"09.40",
"10.45"
],
}
]
However, I want to write/edit into the txt file in the same, or at least close to this format. Anyone knows how to do that?
In the JSON format you have provided, remove the comma after last array:
....
"showtimes":[
"09.40",
"10.45"
]
}
]
Solution:
You can use (Download the Jar file from Maven)
http://mvnrepository.com/artifact/com.google.code.gson/gson
And Create a Bean for Your Data as below:
MovieBean.java
private String moviename;
private String showingstatus;
...
private ArrayList<Double> showtimes;
private String job;
And After that you can use,
Gson methods fromJson and toJson for creating JSON to Bean and Bean to JSON respectively.
You can edit the data in Bean and save it again on the same content file.
I'm using the following code to parse info from a site and it works expect that the older in the last for loop goes out of whack. names() comes out as
["569","570","565","566","567","568","562","563","564"]
those number should be in numeric order but they aren't. Is there a good way to fix this?
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.StringWriter;
import java.io.UnsupportedEncodingException;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Iterator;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import java.util.SortedMap;
import java.util.TreeMap;
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;
import org.xmlpull.v1.XmlPullParser;
import org.xmlpull.v1.XmlPullParserFactory;
import android.annotation.SuppressLint;
import android.util.Log;
public class JSON {
private String html;
private String version = "version";
private String pageString = null;
private String urlString = "http://frc-manual.usfirst.org/a/GetAllItems/ManualID=3";
public volatile boolean parsingComplete = true;
public JSON(String page){
this.pageString = page;
}
public String getHTML(){
return html;
}
public String getVersion(){
return version;
}
#SuppressLint("NewApi")
public void readAndParseJSON(String in) {
try {
JSONObject reader = new JSONObject(in);
JSONObject head = reader.getJSONObject("data").getJSONObject("SubChapter").getJSONObject("3").getJSONObject("children").getJSONObject(pageString);
html = head.getString("item_content_text");
if(head.has("children")){
JSONObject children = head.getJSONObject("children");
JSONArray sub1 = new JSONArray(children.names().toString());
for(int i=sub1.length()-1;i>=0;i--){
JSONObject children2 = children.getJSONObject(Integer.toString(sub1.getInt(i)));
html = html + "<h2>" + children2.getString("secdisp")+ " " + children2.getString("item_name") + "</h2>";
html = html + children2.getString("item_content_text");
if(children2.has("children")){
JSONObject children3 = children2.getJSONObject("children");
JSONArray sub2 = new JSONArray(children3.names().toString());
html = html + sub2;
for(int j=sub2.length()-1;j>=0;j--){
JSONObject children4 = children3.getJSONObject((String) sub2.get(j));
html = html + "<h3>" + children4.getString("secdisp")+ " " + children4.getString("item_name") + "</h3>";
html = html + children4.getString("item_content_text");
}
}
}
}
JSONObject main = reader.getJSONObject("data");
version = main.getString("LatestManualUpdate");
parsingComplete = false;
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public void fetchJSON(){
Thread thread = new Thread(new Runnable(){
#Override
public void run() {
try {
URL url = new URL(urlString);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setReadTimeout(10000 /* milliseconds */);
conn.setConnectTimeout(15000 /* milliseconds */);
conn.setRequestMethod("GET");
conn.setDoInput(true);
// Starts the query
conn.connect();
InputStream stream = conn.getInputStream();
String data = convertStreamToString(stream);
readAndParseJSON(data);
stream.close();
} catch (Exception e) {
e.printStackTrace();
}
}
});
thread.start();
}
static String convertStreamToString(java.io.InputStream is) {
java.util.Scanner s = new java.util.Scanner(is).useDelimiter("\\A");
return s.hasNext() ? s.next() : "";
}
}
The order of the keys in a JSONObject is undefined (see the documentation for keys()). In practice, this means they will likely be ordered according to their hash codes, which is clearly not what you want. If you need to keep these items in a specific order, you could either:
Use a different JSON parser. There are plenty available with a google search; I'd suggest that perhaps a "push-parser" may be the most appropriate type, as this will be guaranteed to give you the items in the object one after another in the order they are defined. json-simple is one such parser. Look at their examples 5 & 6 in the "decoding examples" page. Alternatively, see example 4 for how to change the type of Map it uses for storing JSON objects in its object-model mode, and note that a LinkedHashMap preserves the order the values are added to it.
Sort the list of names before you use them
Change the JSON to use an array of objects which contain the number rather than an object with the numbers as keys, as arrays are (obviously) kept in the order they appear in the original JSON.
You are expecting something that is outside the representational model for JSON.
According to the JSON specification:
"An object is an unordered set of name/value pairs."
The fact that you can imply an order is not relevant. The JSON information model does not recognize that such an ordering exists. That is why most JSON parsers ignore your implied ordering, and many in-memory JSON object representations have no way of recording it.
So what should you do?
My recommendation would be to change your JSON schema. For example, instead of this:
{ "name1" : "value1", "name2" : "value2", "name3" : "value3" }
write:
[ [ "name1", "value1" ], [ "name2", "value2" ], [ "name3", "value3" ] ]
or
[ { "name": "name1", "val" : "value1" },
{ "name": "name2", "val" : "value2" },
{ "name": "name3", "val" : "value3" } ]
... both of which are guaranteed to preserve the order of the entries.
It is also possible to "hack" a solution; i.e. there are ways to get some JSON parsers to preserve the order of name/value pairs that is implied by the serial form. (I'm not going to explain how, because I think it is a really bad idea.)
The problem with doing this is that it ignores interoperability. One of the most important reasons for using JSON (or XML, or any other standard format) is that people can use any spec-conformant software to generate and parse the "stuff" used for data interchange. But what you are doing is taking away that advantage by adding extra requirements that directly contradict the JSON spec.
I am trying to learn more about MySQL and using Java (on Android) to access and retrieve information from a database on my WAMS server. The way my app is setup is that it has an initial login screen which also grabs the "uid" of the username that's logging in (from a different table) and stores it.
Upon login (which is functional - I setup a toast notification that displays the retrieved username and uid of the user logging in), it goes to a new screen (dashboard.xml) which has a TextView field setup to display the retrieved data (from table posted below) associated with the stored "uid". Here is the table I am trying to pull data from:
http://db.tt/4izVQuGB
Now, I have setup a PHP file that queries my db for rows that are associated with a specific "uid". I have tested this file using an HTML form.
$connect = mysql_connect($dbhost, $dbuser, $dbpass) or die("connection error");
mysql_select_db($dbdb)or die("database selection error");
//Retrieve the User ID
$uid = $_POST['uid'];
//Query
$query = mysql_query("SELECT * FROM node WHERE uid='$uid' AND type='goal'");
//store # of rows returned
$num_rows = mysql_num_rows($query);
if ($num_rows >= 1) {
while($results=mysql_fetch_assoc($query)) {
//Store the returned data into a variable
$output = $results;
//encode the returned data in JSON format
echo json_encode($output);
}
mysql_close();
}
The result I get by testing the PHP file using uid value of 1 is:
{"nid":"1","vid":"1","type":"goal","language":"","title":"test","uid":"1","status":"1","created":"1342894493","changed":"1342894493","comment":"2","promote":"1","moderate":"0","sticky":"1","tnid":"0","translate":"0"}
{"nid":"2","vid":"2","type":"goal","language":"","title":"test2","uid":"1","status":"1","created":"1342894529","changed":"1342894529","comment":"2","promote":"1","moderate":"0","sticky":"1","tnid":"0","translate":"0"}
{"nid":"5","vid":"5","type":"goal","language":"","title":"run","uid":"1","status":"1","created":"1343506987","changed":"1343506987","comment":"2","promote":"1","moderate":"0","sticky":"1","tnid":"0","translate":"0"}
{"nid":"9","vid":"9","type":"goal","language":"","title":"run to the
hills","uid":"1","status":"1","created":"1343604338","changed":"1343605100","comment":"2","promote":"0","moderate":"0","sticky":"0","tnid":"0","translate":"0"}
Now, I have written some android code which sets up httppost and is supposed to retrieve the "titles" in my database table. I know it is wrong (obviously since it doesn't work) but I am confused as to what to do next.
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.ArrayList;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.NameValuePair;
import org.apache.http.client.HttpClient;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.message.BasicNameValuePair;
import org.json.JSONObject;
import android.app.Activity;
import android.content.Intent;
import android.content.SharedPreferences;
import android.os.Bundle;
import android.view.View;
import android.view.View.OnClickListener;
import android.widget.Button;
import android.widget.EditText;
import android.widget.TextView;
import android.widget.Toast;
public class Dashboard extends Activity implements OnClickListener {
// variable declarations
String uid = "1";
// create textview to display retrieved data
TextView display;
HttpClient httpclient;
HttpPost httppost;
HttpResponse httpresponse;
HttpEntity httpentity;
ArrayList<NameValuePair> resultArray;
#Override
public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.dashboard);
display = (TextView) findViewById(R.id.test);
// initialize HttpClient
httpclient = new DefaultHttpClient();
// initialize HttpPost
httppost = new HttpPost("http://192.168.1.112/android/fetch.php");
try {
// Create new List
List<NameValuePair> resultList = new ArrayList<NameValuePair>();
resultList.add(new BasicNameValuePair("uid", uid));
httppost.setEntity(new UrlEncodedFormEntity(resultList));
httpresponse = httpclient.execute(httppost);
httpentity = httpresponse.getEntity();
InputStream instream = entity.getContent();
try {
// store incoming stream in an array
JSONArray jArray = new JSONArray(streamToString(instream));
JSONObject jData = null;
for (int i = 0; i < jArray.length(); i++) {
jData = jArray.getJSONObject(i);
String goals = jData.getString("title");
display.setText(goals);
}
//} catch (JSONException e) {
//Toast.makeText(this, "No entries found", Toast.LENGTH_LONG).show();
} catch (Exception e) {
Toast.makeText(this, e.toString(), Toast.LENGTH_LONG)
.show();
}
} catch (Exception e) {
e.printStackTrace();
Notifications error = new Notifications();
error.userPassErrorDialog();
}
}
private static String streamToString(InputStream is) {
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
StringBuilder sb = new StringBuilder();
String line = null;
try {
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
is.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return sb.toString();
}
public void onClick(View v) {
// TODO Auto-generated method stub
}
}
I get the following error when testing it in the Android emulator:
http://db.tt/2vg9MqYh
Any help or suggestions will be greatly appreciated.
In your Android app, you expect a JSONArray:
// store incoming stream in an array
JSONArray jArray = new JSONArray(streamToString(instream));
However, in your PHP file you only output multiple separate JSON objects instead of a real array. I think, you should collect all items from the database in an PHP array first and then encode and output it only once.
My PHP skills are a bit rusted, but I hope this one will work:
//store # of rows returned
$num_rows = mysql_num_rows($query);
if ($num_rows >= 1) {
$output = array();
while($results = mysql_fetch_assoc($query)) {
// append row to output
$output[] = results
}
mysql_close(); // shouldn't that be outside the if block?
//encode the returned data in JSON format
echo json_encode($output);
}
I would expect the output then to be like this (maybe without indentation):
[
{"nid":"1","vid":"1","type":"goal","language":"","title":"test","uid":"1","status":"1","created":"1342894493","changed":"1342894493","comment":"2","promote":"1","moderate":"0","sticky":"1","tnid":"0","translate":"0"},
{"nid":"2","vid":"2","type":"goal","language":"","title":"test2","uid":"1","status":"1","created":"1342894529","changed":"1342894529","comment":"2","promote":"1","moderate":"0","sticky":"1","tnid":"0","translate":"0"},
{"nid":"5","vid":"5","type":"goal","language":"","title":"run","uid":"1","status":"1","created":"1343506987","changed":"1343506987","comment":"2","promote":"1","moderate":"0","sticky":"1","tnid":"0","translate":"0"},
{"nid":"9","vid":"9","type":"goal","language":"","title":"run to the hills","uid":"1","status":"1","created":"1343604338","changed":"1343605100","comment":"2","promote":"0","moderate":"0","sticky":"0","tnid":"0","translate":"0"}
]
The problem lies in encoding and decoding of JSON. from your JSON response it looks like you are receiving JSON object from server also please try to validate you JSON response here. run your php file in browser, copy the entire response on the JSON validator and check the brackets that you are receiving the response in.
1. If your response starts with '[' it is and array and if it starts with '{' it is a JSON Object. while parsing JSON you have defined JSON array first but the server response is JSON object. While using JSON you have to be careful on server side for the format of response it will send and you have to be careful on the client side for the format of response you receive. I am posting example script for you.
-> Server side
if (mysql_num_rows($result)>0){
$response["data"] = array(); //this is an array
while($row= mysql_fetch_array($result))
{
$data = array(); //here I have created another temp array
$data["name"] = $row["name"];
$data["surname"] = $row["surname"];
array_push($response["data"], $data); //this makes an array of objects in the response
}}
}//endif
else{
echo "no input";
}}
mysql_close();
echo json_encode($response); //and finally I echo it as an JSON object
As this php script will return me one object of array of objects ( bit complex isn't it!!) below is the format of response
-> validated JSON response
{
"data": [
{
"name": "Setu",
"surname": "Desai",
}
]
}
and to decode this my client site script need to be the following
-> parsing JSON object
JSONObject snObject = new JSONObject(jsonString);
JSONArray snArray = snObject.getJSONArray("data");
for (int i = 0; i < snArray.length(); i++) {
JSONObject snObject2 = snArray.getJSONObject(i);
String surname = snObject2.getString("surname");
surnamearray.add(surname);
}
the simple way to understand is to validate you JSON response and identify the position of JSON array and objects and then start decoding.