I am currently trying to create an ingestion job workflow using kafka in Talend Studio. The job will read the json data in topic "work" and store into the hive table.
Snippet of json:
{"Header": {"Vers":"1.0","Message": "318","Owner": {"ID": 102,"FID": 101},"Mode":"8"},"Request": {"Type":"4","ObjType": "S","OrderParam":[{"Code": "OpType","Value": "30"},{"Code": "Time","Value": "202"},{"Code": "AddProperty","ObjParam": [{"Param": [{"Code": "Sync","Value": "Y"}]}]}]}}
{"Header": {"Vers":"2.0","Message": "318","Owner": {"ID": 103,"FID": 102},"Mode":"8"},"Request": {"Type":"5","ObjType": "S","OrderParam":[{"Code": "OpType","Value": "90"},{"Code": "Time","Value": "203"},{"Code": "AddProperty","ObjParam": [{"Param": [{"Code": "Sync","Value": "Y"}]}]}]}}
Talend workflow:
My focus in this question is not the talend component. But the java code in tJava component that uses the java to fetch and read the json.
Java code:
String output=((String)globalMap.get("tLogRow_1_OUTPUT"));
JSONObject jsonObject = new JSONObject(output);
System.out.println(jsonObject);
String sourceDBName=(jsonObject.getString("Vers"));
The code above able to get the data from tLogRow in "output" variable. However, it gives an error where it read null value for json object. What should I do to correctly get the data from json accordingly?
You can use a tExtractJsonFields instead of a tJava. This component extracts data from your input String following a json schema that you can define in metadata. With this you could extract all fields from your input .
Related
I use Java (with Quarkus) for the backend.
Vue3 for the frontend.
API and Websockets to transition data between the two.
I am building a chat message application.
So far I was able to send a message inside the backend under a JSON format, do stuff in Java with this message (which holds the message inside a MessageJson class that I have created, hence the data type is now type of MessageJson), transform it into a String, then and send it back to the frontend. I wish to transform the String in the frontend, into a Json.
Here is my function which send the message back to the front (using Websockets):
public MessageResponseForm processBackend(MessageJson messageJson) {
MessageResponseForm messageResponseForm = new MessageResponseForm();
messages(messageJson);
webSocketConfig.sendMessage("U001", messageJson.toString());
return messageResponseForm;
}
The frontend retrieve it without issue, in the String format.
But I need it back into Json format.
I use the function JSON.parse() in Vue.js but it gives me this:
Which is normal in my opinion, because I should receive something like this from the backend:
"{"toRecipient":"U002","fromSender":"U00555","messageContent":"ewww555"}" .....
But instead I got this:
"MessageJson(toRecipient=U002, fromSender=U00555, messageContent=ewww555, dateCreated=2022-9-23 14:15:57, companyName=test)"
My Class MessageJson does not seems to keep the correct Json format.
How to prevent this?
Do I need to use the #Produces annotations in Quarkus? If yes, where exactly?
I found no other method related to ToString()...
EDIT: Do I need to use the dependency GSON to do the job, or can I do this natively?
I have a JSON data in the properties file and trying to retrieve it in java. When I am trying to retrieve the JSON data with the property name it's giving only first string/word from the JSON.
Inside the property file, I have the below content.
profile: {"fname": "ABC","lname": "XYZ","meetings":{"morning":10,"evening":60}}
I am trying to read the content using property name 'profile' as a string and I am getting below error message.
Expected ',' instead of ''
can someone help me with the issue, I tried to escape and unescape but still have the same issue
It may depend on what you are using to deserialize the JSON, but well formed JSON is a single element, so what you have needs to be inside of a container. That is, your file content should be:
{ profile: {"fname": "ABC","lname": "XYZ","meetings":{"morning":10,"evening":60}}}
You can do it like this:
profile={"fname": "ABC","lname": "XYZ","meetings":{"morning":10,"evening":60}}
Or if you want to do it in multiple lines
profile={\
"fname": "ABC",\
"lname": "XYZ",\
"meetings":{\
"morning":10,\
"evening":60\
}\
}
I have a web server which returns JSON data that I would like to load into an Apache Spark DataFrame. Right now I have a shell script that uses wget to write the JSON data to file and then runs a Java program that looks something like this:
DataFrame df = sqlContext.read().json("example.json");
I have looked at the Apache Spark documentation and there doesn't seem a way to automatically join these two steps together. There must be a way of requesting JSON data in Java, storing it as an object and then converting it to a DataFrame, but I haven't been able to figure it out. Can anyone help?
You could store JSON data into a list of Strings like:
final String JSON_STR0 = "{\"name\":\"0\",\"address\":{\"city\":\"0\",\"region\":\"0\"}}";
final String JSON_STR1 = "{\"name\":\"1\",\"address\":{\"city\":\"1\",\"region\":\"1\"}}";
List<String> jsons = Arrays.asList(JSON_STR0, JSON_STR1);
where each String represents a JSON object.
Then you could transform the list to an RDD:
RDD<String> jsonRDD = sc.parallelize(jsons);
Once you've got RDD, it's easy to have DataFrame:
DataFrame data = sqlContext.read().json(jsonRDD);
I'm testing a standalone java application using JMeter and I'm using a OS Process Sampler to execute the jar file. I've given JSON string in java program and I've used publish method to post JSON string to Queue. Now I want to capture the data in JMeter response data and I want to do parameterisation for those fields and I want to use CSV Data Set Config to pass the values to fields and each user should take unique data.
The JSON string looks like below and I want to parameterise these valuse from JMeter response data and pass values from CSV file.
[{
"id": 100001,
"status": "Pending",
"priorityCode": "1",
"miniTeamId": "234256",
"requestName": "General"
}]
Given your CSV file looks like:
100001,Pending,1,234256,General
100002,Done,2,234257,General
etc.
Configure your CSV Data Set config as follows:
Filename: path to your .csv file (better full path)
Variable Names: id,status,priorityCode,miniTeamId,requestName
other fields can be left as they are. By default JMeter will read next line from CSV file by each thread on each loop, when file end will be reached - JMeter will start over.
JMeter Variables populated by the CSV Data Set Config can be referred as ${variableName} or ${__V(variableName)} so parametrised request body should look like:
[
{
"id": ${id},
"status": "${status}",
"priorityCode": "${priorityCode}",
"miniTeamId": "${miniTeamId}",
"requestName": "${requestName}"
}
]
See Using CSV DATA SET CONFIG for more detailed information on parametrisation of JMeter tests using CSV files.
Also be aware of the following test elements:
__csvRead() function - if you need to get data from multiple CSV files
JSON Path Extractor - if you need to get values from response JSON
UPD: passing parametrized JSON string via OS Process Sampler
I am using Apache PIG to reduce data originally stored in CSV format and want to output in Avro. Part of my PIG script calls a java UDF that appends a few fields to the input Tuple and passes the modified Tuple back. I am modifying the output, PIG, schema when doing this using:
Schema outSchema = new Schema(input).getField(1).schema;
Schema recSchema = outSchema.getField(0).schema;
recSchema.add(new FieldSchema("aircrafttype", DataType.CHARARRAY));
Inside the public Schema outputSchema(Schema input) method of my UDF.
Within the exec method, I append java.lang.String values to the input Tuple and return the edited Tuple to the PIG script. This, and all subsequent operations work fine. If I output to CSV format using PigStorage(',') there are no problems. When I attempt to output using
STORE records INTO '$out_dir' USING org.apache.pig.piggybank.storage.avro.AvroStorage('
{
"schema":{
"type":"record", "name":"my new data",
"fields": [
{"name":"fld1", "type":"long"},
{"name":"fld2", "type":"string"}
]}
}');
I get the following error:
java.io.IOException: java.lang.ClassCastException: java.lang.String cannot be cast to org.apache.avro.util.Utf8
I have attempted appending the character fields to the Tuple (within my UDF) as char[] and Utf8 types, but that makes PIG angry before I even get to trying to write out data. I have also attempted modifying my Avro schema to allow for null types in every field.
I'm using PIG v0.11.1 and Avro v1.7.5, any help is much appreciated.
This was a PIG version issue. My UDF was built into a jar-with-dependencies including PIG v0.8.1. The mix of PIG versions 0.8.1 and 0.11.1 was causing the problems, AVRO had nothing to do with it.