I am looking for a utility which converts one json format to another by respecting at the conversion definitions from a preferably xml file. Is there any library doing something like this in java ?
For example source json is:
{"name":"aa","surname":"bb","accounts":[{"accountid":10,"balance":100}]}
target json is :
{"owner":"aa-bb","accounts":[{"accountid":10,"balance":100}]}
sample config xml :
t.owner = s.name.concat("-").concat(surname)
t.accounts = t.accounts
Ps:Please dont post solutions for this example, it is just for giving an idea, there will be quite different scenarios in mapping.
Is this what u need?
Open input file.
Read / parse JSON from file using a JSON library.
Convert in-memory data structure to new structure.
Open output file
Unparse in-memory data structure to file using JSON library.
I can use the following code to read a single json file but I need to read multiple json files and merge them into one Dataframe. How can I do this?
DataFrame jsondf = sqlContext.read().json("/home/spark/articles/article.json");
Or is there a way to read multiple json files into JavaRDD then convert to Dataframe?
To read multiple inputs in Spark, use wildcards. That's going to be true whether you're constructing a dataframe or an rdd.
context.read().json("/home/spark/articles/*.json")
// or getting json out of s3
context.read().json("s3n://bucket/articles/201510*/*.json")
You can use exactly the same code to read multiple JSON files. Just pass a path-to-a-directory / path-with-wildcards instead of path to a single file.
DataFrameReader also provides json method with a following signature:
json(jsonRDD: JavaRDD[String])
which can be used to parse JSON already loaded into JavaRDD.
function spark.read.json accepts list of file as a parameter.
spark.read.json(List_all_json file)
This will read all the files in the list and return a single data frame for all the information in the files.
Using pyspark, if you have all the json files in the same folder, you can use df = spark.read.json('folder_path'). This instruction will load all the json files inside the folder.
For reading performance, I recommend you for providing dataframe the schema:
import pyspark.sql.types as T
billing_schema = billing_schema = T.StructType([
T.StructField('accountId', T.LongType(),True),
T.StructField('accountName',T.StringType(),True),
T.StructField('accountOwnerEmail',T.StringType(),True),
T.StructField('additionalInfo',T.StringType(),True),
T.StructField('chargesBilledSeparately',T.BooleanType(),True),
T.StructField('consumedQuantity',T.DoubleType(),True),
T.StructField('consumedService',T.StringType(),True),
T.StructField('consumedServiceId',T.LongType(),True),
T.StructField('cost',T.DoubleType(),True),
T.StructField('costCenter',T.StringType(),True),
T.StructField('date',T.StringType(),True),
T.StructField('departmentId',T.LongType(),True),
T.StructField('departmentName',T.StringType(),True),
T.StructField('instanceId',T.StringType(),True),
T.StructField('location',T.StringType(),True),
T.StructField('meterCategory',T.StringType(),True),
T.StructField('meterId',T.StringType(),True),
T.StructField('meterName',T.StringType(),True),
T.StructField('meterRegion',T.StringType(),True),
T.StructField('meterSubCategory',T.StringType(),True),
T.StructField('offerId',T.StringType(),True),
T.StructField('partNumber',T.StringType(),True),
T.StructField('product',T.StringType(),True),
T.StructField('productId',T.LongType(),True),
T.StructField('resourceGroup',T.StringType(),True),
T.StructField('resourceGuid',T.StringType(),True),
T.StructField('resourceLocation',T.StringType(),True),
T.StructField('resourceLocationId',T.LongType(),True),
T.StructField('resourceRate',T.DoubleType(),True),
T.StructField('serviceAdministratorId',T.StringType(),True),
T.StructField('serviceInfo1',T.StringType(),True),
T.StructField('serviceInfo2',T.StringType(),True),
T.StructField('serviceName',T.StringType(),True),
T.StructField('serviceTier',T.StringType(),True),
T.StructField('storeServiceIdentifier',T.StringType(),True),
T.StructField('subscriptionGuid',T.StringType(),True),
T.StructField('subscriptionId',T.LongType(),True),
T.StructField('subscriptionName',T.StringType(),True),
T.StructField('tags',T.StringType(),True),
T.StructField('unitOfMeasure',T.StringType(),True)
])
billing_df = spark.read.json('/mnt/billingsources/raw-files/202106/', schema=billing_schema)
Function json(String... paths) takes variable arguments. (documentation)
So you can change your code like this:
sqlContext.read().json(file1, file2, ...)
Is there a way (any jmeter plugin) by which we can have the JMeter script read all the contents(String) from external text file ?
I have a utility in java which uses Jackson ObjectMapper to convert a arraylist to string and puts it to a text file in the desktop. The file has the JSON info that i need to send in the jmeter Post Body.
I tried using ${__FileToString()} but it was unable to deserialize the instance of java.util.ArrayList. It was also not reading all the values properly.
I am looking for something like csv reader where i just give the file location. I need all the json info present in the file. Need to extract it and assign to the post body.
Thanks for your help !!!
If your question is about how to deserialize ArrayList in JMeter and dynamically build request body, you can use i.e. Beanshell PreProcessor for it.
Add a Beanshell PreProcessor as a child of your request
Put the following code into the PreProcessor's "Script" area:
FileInputStream in = new FileInputStream("/path/to/your/serialized/file.ser");
ObjectInput oin = new ObjectInputStream(in);
ArrayList list = (ArrayList) oin.readObject();
oin.close();
in.close();
for (int i = 0; i < list.size(); i++) {
sampler.addArgument("param" + i, list.get(i).toString());
}
The code will read file as ArrayList, iterate through it and add request parameter like:
param1=foo
param2=bar
etc.
This is the closest answer I'm able to provide, if you need more exact advice - please elaborate your question. In the meantime I recommend you to get familiarized with How to use BeanShell: JMeter's favorite built-in component guide to learn about scripting in JMeter and what do pre-defined variables like "sampler" in above code snippet mean.
I'm sending a file over the network which is in JSON format and wanting to retrieve information from it. The file is created using the File.createTempFile method. Here's the code:
File patchFile = File.createTempFile("indexer", ".pf", null);
try(FileOutputStream fos = new FileOutputStream(patchFile)) {
byte[] patchFileBytes = new byte[payloadLength];
buffer.readBytes(patchFileBytes);
fos.write(patchFileBytes);
}
I'm trying to find the best way to read the patchFile as JSON and get values from it. The generated files structure looks like this(on the client):
{
"0": [
{
"fileId": "Cache.dat",
"fileChecksum": "d41d8cd98f00b204e9800998ecf8427e"
},
{
"fileId": "Character.fbx",
"fileChecksum": "d41d8cd98f00b204e9800998ecf8427e"
},
{
"fileId": "Skybox.png",
"fileChecksum": "d41d8cd98f00b204e9800998ecf8427e"
}
],
"1": [
{
"fileId": "indexer.pf",
"fileChecksum": "f8130e38ce5c58ccbf200d24c2629632"
}
]
}
Currently I've got the file being sent over and read into memory as you can see above, but I'm unsure as to where to go from here on actually reading the data, could anyone help me out?
I'm using GSON.
What I would suggest is to avoid writing to a temporary file, instead wrap the buffer with a reader and use in one of the API's
(I'm assuming that the buffer is an InputStream, then use java.io.InputStreamReader to convert it into a Reader)
GSON has a lot of API's to perform your required task:
Examples:
To follow the iterative approach, then look at com.google.gson.JsonParser#parse it accepts reader or string to get elements.
If you know what is the type of the element being deserialized, then you can look at using com.google.gson.Gson#fromJson methods.
See the following API docs, it might be helpful:
JsonStreamParser
JsonParser
Gson
You did not mention how big your file can be. If it is not too big then you could just use Apache Commons IO and IOUtils.toString and read the entire file content into String. And since you already know GSON it should be pretty simple from there.
(After months of surfing the internet, talking to the school's computing department and try code out, I still don't get how to do it, but I do know more specific about what I trying to do)
Previously I said I want to "Add lines" to a existing JSON file.
What I want to do is simply add an element to an JSON object from a file, then save the file.
However I am still confused about how to do it.
The process I am guessing is to use ajax to load the content of the file (the JSON code in the file) into a variable then add the new element into the object then save the file.
I have seen a lot of code but are all just too confusing and looks like its for webpages. I am trying to edit a file on the computer as a program which I think webpage related code such as xmlhttp requests are irrelevant as the file is in a folder in appdata.
I have been confused and thought Java and Javascript were the same thing, I know now they're not.
What code or functions would I look for and how would it be used in the code?
(Please don't post pseudocode because I have no idea how to write the code for them since I have literally no idea how to code anything other than a html webpage and some php. Other coding language like Java, Javascript and Python I have little knowledge with but not enough to write a program alone.)
I think it would be best to use code that somebody else has already written to manipulate the JSON. There are plenty of libraries for that, and the best would be the officially specified one, JSON-P. What you would do is this:
Go to http://jsonp.java.net/ and download JSON-P. (You will have to examine the page carefully to find the link to "JSON Processing RI jar".) You will need to include this JAR in your class path while you write your program.
Add imports to your program for javax.json.*.
Write this code to do the job (you will have to catch JsonExceptions and IOExceptions):
JsonReader reader = Json.createReader(new FileReader("launcher_profiles.json"));
JsonObject file = reader.readObject();
reader.close();
JsonObject profiles = file.getJsonObject("profiles");
JsonObject newProfile = Json.createObjectBuilder()
.add("name", "New Lines")
.add("gameDir", "New Lines")
.add("lastVersionId", "New Lines")
.add("playerUUID", "")
.build();
JsonObjectBuilder objectBuilder = Json.createObjectBuilder()
.add("New Profile Name", newProfile);
for (java.util.Map.Entry<String, JsonValue> entry : profiles.entrySet())
objectBuilder.add(entry.getKey(), entry.getValue());
JsonObject newProfiles = objectBuilder.build();
// Now, figure out what I have done so far and write the rest of the code yourself! At the end, use this code to write out the new file:
JsonWriter writer = Json.createWriter(new FileWriter("launcher_profiles.json"));
writer.writeObject(newFile);
writer.close();