We Keep Coding

Cloud Dataflow:Reading entire json array file from Cloud Storage and create a PCollection of json object

I have a json array file with content as below
[ {
"MemberId" : "1234",
"Date" : "2017-07-03",
"Interactions" : [ {
"Number" : "1327",
"DwellTime" : "00:03:05"
} ]
}, {
"MemberId" : "5678",
"Date" : "2017-07-03",
"Interactions" : [ {
"Number" : "1172",
"DwellTime" : "00:01:26"
} ]
} ]
I wanted to create a PCollection of Java Object mapped to each Json present in Json array

JSON formatted like this (records spread over multiple lines instead of one per line) is hard for a data processing tool like beam/dataflow to process in parallel - from a random point in the file, you cannot be sure where the next record begins. You can do it by reading from the beginning of the file, but then you're not really reading in parallel.
If it's possible, reformatting it so that it's one record per line would let you use something like TextIO to read in the file.
If not, you'll need to read the file in one go.
I would suggest a couple possible approaches:
Write a ParDo that reads from the file using the gcs API
This is pretty straight forward. You'll do all the reading in one ParDo and you'll need to implement the connection code inside of that pardo. Inside the pardo you would write the same code you would as if you're reading the file in a normal java program. The pardo will emit each java object as a record.
Implement a filebasedsource
File based sources will work - when the fileOrPatternSpec is "gs://..." it knows how to read from GCS. You'll need to make sure to set fileMetadata.isReadSeekEfficient to false so that it won't try to split the file. I haven't tried it, but I believe the correct way to do that is to set it inside of the single file constructor of FBS (ie, your class's override of FileBaseSource(MetaData, long, long)
TextSource/XmlSource (and their accompanying wrappers TextIO/XmlIO) are examples of this, except that they try to implement splitting - yours will be much simpler since it won't.

How do I serve csv files using java spark?

I am using spark to run the server side for a web application I am writing. I searched the documentation a bit, but I came up empty.. is there a way to serve data to the frontend such that it automatically downloads for the user as a csv file? My data that I am attempting to serve as csv looks something like this.
// example data... returned by "getData()"
JSONArray test = new JSONArray()
.put(
new JSONArray().put("foo").put("bar")
)
.put(
new JSONArray().put(1).put(2)
);
// route
get("/csv/:path", "application/json", (req, res) -> {
res.type("text/csv");
JSONArray reply = getData(req);
return data;
});
I was taking a look at the ResponseTransformers section of the documentation, but I couldn't figure out how to serve my data as a downloadable csv file instead of a json object. I'm assuming the ResponseTransformer would somehow need to be subclassed, but I couldn't find an example to do what I want. Could anyone provide me with an example, or point me in the direction of some docs that explain how to do this?
EDIT : I was able to, on the javascript side, call my route like this.
window(route);
which allowed me to select a program on my computer to download the response. However, the data looks like this in notepad
[["foo","bar"],[1,2]]
So, close.. but not quite a csv file. I was hoping the output would look more like this.
foo,bar
1,2

I think you could use a StringBuilder to render your csv file, as this answer does. I also think that the second parameter of your request "application/json" could also be removed. It would look like this:
// route
get("/csv/:path", (req, res) -> {
res.type("text/csv");
StringBuilder sb = new StringBuilder();
sb.append("id");
sb.append(',');
sb.append("Name");
sb.append('\n');
sb.append("1");
sb.append(',');
sb.append("Zack");
sb.append('\n');
return sb.toString();
});

Parsing a java.io.File into JSON using GSON

I'm sending a file over the network which is in JSON format and wanting to retrieve information from it. The file is created using the File.createTempFile method. Here's the code:
File patchFile = File.createTempFile("indexer", ".pf", null);
try(FileOutputStream fos = new FileOutputStream(patchFile)) {
byte[] patchFileBytes = new byte[payloadLength];
buffer.readBytes(patchFileBytes);
fos.write(patchFileBytes);
}
I'm trying to find the best way to read the patchFile as JSON and get values from it. The generated files structure looks like this(on the client):
{
"0": [
{
"fileId": "Cache.dat",
"fileChecksum": "d41d8cd98f00b204e9800998ecf8427e"
},
{
"fileId": "Character.fbx",
"fileChecksum": "d41d8cd98f00b204e9800998ecf8427e"
},
{
"fileId": "Skybox.png",
"fileChecksum": "d41d8cd98f00b204e9800998ecf8427e"
}
],
"1": [
{
"fileId": "indexer.pf",
"fileChecksum": "f8130e38ce5c58ccbf200d24c2629632"
}
]
}
Currently I've got the file being sent over and read into memory as you can see above, but I'm unsure as to where to go from here on actually reading the data, could anyone help me out?
I'm using GSON.

What I would suggest is to avoid writing to a temporary file, instead wrap the buffer with a reader and use in one of the API's
(I'm assuming that the buffer is an InputStream, then use java.io.InputStreamReader to convert it into a Reader)
GSON has a lot of API's to perform your required task:
Examples:
To follow the iterative approach, then look at com.google.gson.JsonParser#parse it accepts reader or string to get elements.
If you know what is the type of the element being deserialized, then you can look at using com.google.gson.Gson#fromJson methods.
See the following API docs, it might be helpful:
JsonStreamParser
JsonParser
Gson

You did not mention how big your file can be. If it is not too big then you could just use Apache Commons IO and IOUtils.toString and read the entire file content into String. And since you already know GSON it should be pretty simple from there.

How to append object to a json file with JSR 353 (Java API for JSON Processing)

Using JSR-353 (https://jsonp.java.net/index.html)
I would like to open a json file and append some object in the root array, eg :
[{"foo":"bar"}]
I would like with a code about like this :
try(JsonGenerator writer = Json.createGenerator(new FileOutputStream(this.file))){
writer.writeStartObject().write("hello", "world").writeEnd();
} catch (IOException e) {
e.printStackTrace();
}
And obtain in the end :
[
{"foo":"bar"},
{"hello":"world"}
]
Note : I don't want to have to load the full json in-memory to append my data.

Note : I don't want to have to load the full json in-memory to append
my data.
Basically, you can't. You would have to parse the full data structure, so that your write(..) would know where to write. Otherwise, it's just appending somewhere and that might break the JSON format.
So read the JSON from the file, generate a JsonArray from it. Create a new JsonObject from your values. Add it to the array. Then write the full array.

You can't simply "append". In the general case you must read in the JSON, modify the tree-structured memory image, then "serialize" it back to linear JSON.
In very simple cases such as the above you could in theory seek to the end, backspace over the closing ], then write out ,, the second object, and a new closing ], but it's not a general solution to updating JSON.

Adding elements to an JSON object in a external JSON file?

(After months of surfing the internet, talking to the school's computing department and try code out, I still don't get how to do it, but I do know more specific about what I trying to do)
Previously I said I want to "Add lines" to a existing JSON file.
What I want to do is simply add an element to an JSON object from a file, then save the file.
However I am still confused about how to do it.
The process I am guessing is to use ajax to load the content of the file (the JSON code in the file) into a variable then add the new element into the object then save the file.
I have seen a lot of code but are all just too confusing and looks like its for webpages. I am trying to edit a file on the computer as a program which I think webpage related code such as xmlhttp requests are irrelevant as the file is in a folder in appdata.
I have been confused and thought Java and Javascript were the same thing, I know now they're not.
What code or functions would I look for and how would it be used in the code?
(Please don't post pseudocode because I have no idea how to write the code for them since I have literally no idea how to code anything other than a html webpage and some php. Other coding language like Java, Javascript and Python I have little knowledge with but not enough to write a program alone.)

I think it would be best to use code that somebody else has already written to manipulate the JSON. There are plenty of libraries for that, and the best would be the officially specified one, JSON-P. What you would do is this:
Go to http://jsonp.java.net/ and download JSON-P. (You will have to examine the page carefully to find the link to "JSON Processing RI jar".) You will need to include this JAR in your class path while you write your program.
Add imports to your program for javax.json.*.
Write this code to do the job (you will have to catch JsonExceptions and IOExceptions):
JsonReader reader = Json.createReader(new FileReader("launcher_profiles.json"));
JsonObject file = reader.readObject();
reader.close();
JsonObject profiles = file.getJsonObject("profiles");
JsonObject newProfile = Json.createObjectBuilder()
.add("name", "New Lines")
.add("gameDir", "New Lines")
.add("lastVersionId", "New Lines")
.add("playerUUID", "")
.build();
JsonObjectBuilder objectBuilder = Json.createObjectBuilder()
.add("New Profile Name", newProfile);
for (java.util.Map.Entry<String, JsonValue> entry : profiles.entrySet())
objectBuilder.add(entry.getKey(), entry.getValue());
JsonObject newProfiles = objectBuilder.build();
// Now, figure out what I have done so far and write the rest of the code yourself! At the end, use this code to write out the new file:
JsonWriter writer = Json.createWriter(new FileWriter("launcher_profiles.json"));
writer.writeObject(newFile);
writer.close();