We Keep Coding

Java Spark: How to get value from a column which is JSON formatted string for entire dataset?

Needs some help here. I am trying to read data from Hive/CSV. There is a column whose type is string and the value is json formatted string. It is something like this:
| Column Name A |
|----------------------------------------------------------|
|"{"key":{"data":{"key_1":{"key_A":[123]},"key_2":[456]}}}"|
How can I get the value of key_2 and insert it to a new column?
I tried to create a new function to the get value via Gson
private BigDecimal getValue(final String columnValue){
JsonObject jsonObject = JsonParser.parseString(columnValue).getAsJsonOBject();
return jsonObject.get("key").getAsJsonObject().get("key_1").getAsJsonObject().get("key_2").getAsJsonArray().get(0).getAsBigDecimal();
}
But how i can apply this method to the whole dataset?
I was trying to achieve something like this:
Dataset<Row> ds = souceDataSet.withColumn("New_column", getValue(sourceDataSet.col("Column Name A")));
But it cannot be done as the data types are different...
Could you please give any suggestions?
Thx!
hx!
------------------Update---------------------
As #Mck suggested, I used get_json_object.
As my value contains "
"{"key":{"data":{"key_1":{"key_A":[123]},"key_2":[456]}}}"
I used substring to removed " and make the new string like this
{"key":{"data":{"key_1":{"key_A":[123]},"key_2":[456]}}}
Code for substring
DataSet<Row> dsA = sourceDataSet.withColumn("Column Name A",expr("substring(Column Name A, 2, length(Column Name A))"))
I used dsA.show() and confirmed the dataset looks correct.
Then I used following code try to do it
Dataset<Row> ds = dsA.withColumn("New_column",get_json_object(dsA.col("Column Name A"), "$.key.data.key_2[0]"));
which returns null.
However, if the data is this:
{"key":{"data":{"key_2":[456]}}}
I can get value 456.
Any suggestions why I get null?
Thx for the help!

Use get_json_object:
ds.withColumn(
"New_column",
get_json_object(
col("Column Name A").substr(lit(2), length(col("Column Name A")) - 2),
"$.key.data.key_2[0]")
).show(false)
+----------------------------------------------------------+----------+
|Column Name A |New_column|
+----------------------------------------------------------+----------+
|"{"key":{"data":{"key_1":{"key_A":[123]},"key_2":[456]}}}"|456 |
+----------------------------------------------------------+----------+

Parsing Properties file

I am trying to parse a Properties file that has the following format:
CarModel=Prius
CarMake=Toyota
Option1=Transmission
OptionValue1a=Manual
OptionValue1b=Automatic
Option2=Brakes
OptionValue2a=Regular
OptionValue2b=ABS
My question is, what if there are various forms of the Properties file? For instance, what if a Properties file has 3 options for Option 1, and another Properties file has 2 options for Option 1? Right now my code looks like this:
Properties props = new Properties();
FileInputStream x = new FileInputStream(filename);
props.load(x);
String carModel = props.getProperty("CarModel");
if(!carModel.equals(null)){
String carMake = props.getProperty("CarMake");
String option1 = props.getProperty("Option1");
String option1a = props.getProperty("OptionValue1a");
String option1b = props.getProperty("OptionValue1b");
etc. I'm thinking I need a lot of 'if' statements, but I'm unsure how to implement them. Any ideas?

Are you sure you want to use a properties file? I suggest using YAML.
I am trying to parse a Properties file that has the following format:
CarModel: Prius
CarMake: Toyota
Transmission:
- Manual
- Automatic
Brakes:
- Regular
- ABS
Using SnakeYAML you can do
Map<String, Object> car = (Map) new Yaml().load(new FileReader(filename));
Note the lines starting with - are turned into a list.

If you must stick with Properties, I suggest putting the list in a property.
CarModel=Prius
CarMake=Toyota
Options=Transmission Manual|Automatic,\
Brakes Regular|ABS
This way you can read the options like
String options = prop.getProperty("Options");
for(String option : options.split("\\s*,\\s*")) {
String[] parts = option.split("\\s+");
String optionType = parts[0];
String[] optionChoices = parts[1].split("[|]");
}
This way you can have any number of options with any number of choices.

How do I extract variable data from a line of string with tags?

I'm trying to write Java code to go to a website, read the HTML code line-by-line, extract certain pieces of data, including an embedded URL to go to another website, and repeat the process 100 times.
I've been able to isolate most of the pieces of data I need using expressions like:
s.ranking = line.substring(line.indexOf(">")+1, line.length() -7);
But I'm having problems with the following line:
<strong>Writer:</strong> Dylan <br/><strong>Producer:</strong> Tom Wilson&nbsp <br/><strong>Released:</strong> July '65, Columbia<br/>12 weeks; No. 2</p>
I need to extract and save the Writer data (Dylan). The producer data (Tom Wilson) and the Release date data (July '65). Some of the pages will have multiple writers and will be labeled "Writers:", and some will have multiple producers, labeled "Producers:"
How do I capture "Dylan" ,"Tom Wilson" and "July '65" from the above line in Java?
Thank you very much!
DM

The best approach is to use HTML parser. But as i read your comment " I'm doing this for a class and am learning about finding, isolating and extracting data."
What you can do something like :
String producer = "Producer:";
String writer = "Writer:";
String released = "Released:";
String s = "<strong>Writer:</strong> Dylan <br/><strong>Producer:</strong> Tom Wilson&nbsp <br/><strong>Released:</strong> July '65, Columbia<br/>12 weeks; No. 2</p> ";
int writerIndex = s.lastIndexOf(writer);
int producerIndex = s.lastIndexOf(producer);
int realesedIndex = s.lastIndexOf(released);
String writerExtracted = s.substring(writerIndex + writer.length(),
producerIndex).replaceAll("\\<.*?>", "");
System.out.println(writerExtracted);
String producerExtracted = s.substring(
producerIndex + producer.length(), realesedIndex).replaceAll(
"\\<.*?>", "");
System.out.println(producerExtracted);
String releasedExtracted = s.substring(
realesedIndex + released.length(), s.length()).replaceAll(
"\\<.*?>", "");
System.out.println(releasedExtracted);
Output:
Dylan
Tom Wilson&nbsp
July '65, Columbia12 weeks; No. 2
NOTE: you can get rid of signs such as &#039 or &nbsp using another regex ...

string interpolation java 1.3 and below(if possible)?

I have a simple string, where I need to insert a few numbers and strings.
Say String a = "My name is %s. I am %d years old".
I also need to insert same number or string at several of these holes.
I need a solution which works for ancient versions of java atleast upto 1.3
I know about String.format (JDK 5+). I read about formatter, my head hurts!!
please help.

Your only option is to use MessageFormat here.
You'd type:
String s = "My name is {0}. I am {1} years old";
and use the appropriate method to render this to a string. For instance:
String ret = MessageFormat.format(s, "John", 32);
I'd like to put a link to the javadoc, but... I don't know how much has changed since 1.3! (well, link added, it can't hurt)
(it should be noted that even in 2013, Java's ResourceBundles still use MessageFormat and read property files in ISO-8859-1, not UTF-8)

What about this one
String text = "The user {0} has email address {1}."
String msg = MessageFormat.format(text, params);
And this other
String text = "The user {name} has email address {email}.";
Object[] params = { "nameRobert", "rhume55#gmail.com" };
Map map = new HashMap();
map.put("name", "Robert");
map.put("email", "rhume55#gmail.com");
System.out.println("1st : " + MapFormat.format(text, map));

java code corresponding to Newtonsoft.Json.JsonConvert.SerializeObject(Object source,Newtonsoft.Json.JsonSerializerSettings()) in .net?

I have a code in .net that serializes a request to the json format ... The code is something like this.
var ops = new Newtonsoft.Json.JsonSerializerSettings();
ops.NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore;
ops.MissingMemberHandling = Newtonsoft.Json.MissingMemberHandling.Ignore;
ops.DefaultValueHandling = Newtonsoft.Json.DefaultValueHandling.Ignore;
ops.Converters.Add(new Newtonsoft.Json.Converters.JavaScriptDateTimeConverter());
String strSO = Newtonsoft.Json.JsonConvert.SerializeObject(source,
bIndent ? Newtonsoft.Json.Formatting.Indented : Newtonsoft.Json.Formatting.None,
ops);
I tried the java code corresponding to this portion but it doesn't work.

From my understanding, the Newtonsoft serializer takes an object with member variables and outputs a json string that represents that object.
So you can do something like:
Product product = new Product();
product.Name = "Apple";
product.Expiry = new DateTime(2008, 12, 28);
product.Price = 3.99M;
product.Sizes = new string[] { "Small", "Medium", "Large" };
string output = JsonConvert.SerializeObject(product);
And you'll get an output string like:
{"Name": "Apple",
"Expiry": "\/Date(1230375600000+1300)\/",
"Price": 3.99,
"Sizes": ["Small", "Medium", "Large"]
}
Now the bad news is that the BlackBerry library that you're using doesn't use reflection to examine the structure of objects it serialises. It is a formatter rather than a serializer.
The good news is that it is pretty easy to use. The documentation is here:
http://www.blackberry.com/developers/docs/6.0.0api/org/json/me/package-summary.html
In short, to write an object such as the one above, you would do something like:
myString = new JSONStringer()
.object()
.key("Name")
.value("Apple")
.key("Expiry")
.value("Date("+myDate.getTime()+")")
.endObject()
.toString();
..and so on. Note that you are constructing the JSON structure element by element, rather than having the JSON library assume that your object is the exact structure of the data you wish to output.
Hopefully this will give you some idea of how to proceed.

If your question is "Does anyone know of a Java equivalent to Newtonsoft.Json for .NET for serializing in JSON format?"
Check the bottom of http://json.org