Get value of inside of another array after exploding an array - java

I have a json which include multiple array.
I have json as:
"eList": [
{
"eList": [
{
"eOne": "86"
}
],
"idList": [
"67890",
"24328"
]
}
]
I have exploded the id list as:
df=df.withColumn("idList",explode(col("eList.idList")))
It works fine till here.
I want the value of eOne from elist. I want the output as follows:
| idList | eList |
| -------- | ----- |
| 67890 | 86 |
| 24328 | null |
explode function is not going to work. I dont have anything unique rows to partition by window? How am i supposed to proceed?

Related

JDBC pagination in one to many relation and mapping to pojo?

we have a messaging logs table and we are using this table to provide a search UI which lets to search messages by id or status or auditor or date. Table audit looks like below
+-----------+----------+---------+---------------------+
| messageId | auditor | status | timestamp |
+-----------+----------+---------+---------------------+
| 10 | program1 | Failed | 2020-08-01 10:00:00 |
| 11 | program2 | success | 2020-08-01 10:01:10 |
| 12 | program3 | Failed | 2020-08-01 10:01:15 |
+-----------+----------+---------+---------------------+
Since in a given date range we could have many messages matching the criteria so we added pagination for the query. Now as a new feature we are adding another table with one to many relation which contain tags as the possible reasons for the failure. The table failure_tags will look like below
+-----------+----------+-------+--------+
| messageId | auditor | type | cause |
+-----------+----------+-------+--------+
| 10 | program1 | type1 | cause1 |
| 10 | program1 | type1 | cause2 |
| 10 | program1 | type2 | cause3 |
+-----------+----------+-------+--------+
Now for a general search query for a status = 'Failed' and using left join with the other table will retrieve 4 rows as below
+-----------+----------+-------+--------+---------------------+
| messageId | auditor | type | cause | timestamp |
+-----------+----------+-------+--------+---------------------+
| 10 | program1 | type1 | cause1 | 2020-08-01 10:00:00 |
| 10 | program1 | type1 | cause2 | 2020-08-01 10:00:00 |
| 10 | program1 | type2 | cause3 | 2020-08-01 10:00:00 |
| 12 | program3 | | | 2020-08-01 10:01:15 |
+-----------+----------+-------+--------+---------------------+
The requirement is to since the 3 rows of messageId 10 belongs to same message the requirement is to merge the rows into 1 in json response, so the response will have only 2 elements
[
{
"messageId": "10",
"auditor": "program1",
"failures": [
{
"type": "type1",
"cause": [
"cause1",
"cause2"
]
},
{
"type": "type2",
"cause": [
"cause3"
]
}
],
"date": "2020-08-01 10:00:00"
},
{
"messageId": "12",
"auditor": "program3",
"failures": [],
"date": "2020-08-01 10:01:15"
}
]
Because of this merge for a pagination request of 10 elements after fetching from the database and merging would result in less than 10 results.
The 1 solution, I could think of is after merging, if its less than page size, initiate a search again do the combining process and take the top 10 elements. Is there any better solution to get all the results in 1 query instead of going twice or more to DB ?
We use generic spring - JDBC not the JPA.

SQL query using Spark and java language

I have two dataframe on spark .
The first dataframe1 is :
+--------------+--------------+--------------+
|id_z |longitude |latitude |
+--------------+--------------+--------------+
|[12,20,30 ] |-7.0737816 | 33.82666 |
|13 |-7.5952683 | 33.5441916 |
+--------------+--------------+--------------+
The second dataframe2 is :
+--------------+--------------+---------------+
|id_z2 |longitude2 |latitude2 |
+--------------+--------------+---------------+
| 14 |-8.5952683 | 38.5441916 |
| 12 |-7.0737816 | 33.82666 |
+--------------+--------------+---------------+
I want to apply the logic of the following request.
String sql = "SELECT * FROM dataframe2 WHERE dataframe2 .id_z2 IN ("'"
+ id_z +"'") and longitude2 = "'"+longitude+"'" and latitude = "'"+latitude+"'"";
I prefer not to have a join, is it possible to do this?
I really need your help , or just a starting point that will make things easier for me.
Thnak you

Only compare YYYY-MM part of date using Springboot and MongoDB

I am facing issue with mongoDb date fields.
I have one column in MongoDB collect which is date type. Which store whole date in ISO format.
As like below.
2020-05-13T14:00:00.000
Now from my api i am passing 2020-05 means YYYY-MM and i want to fetch all records with 2020-05 value in db.
I had tried with multiple option as below.
Date mydate=new Date();
Criteria criterid = Criteria
.where("date").is(mydate)
I have below data in DB
|---------------------|------------------|------------------|
| Name | ID | Date |
|---------------------|------------------|------------------|
| A | 1 |2020-05-13T14:00:00|
|---------------------|------------------|------------------|
| B | 2 |2020-12-31T14:00:00|
|---------------------|------------------|------------------|
| C | 3 |2020-07-17T14:00:00|
|---------------------|------------------|------------------|
| D | 4 |2020-05-29T14:00:00|
|---------------------|------------------|------------------|
enter code here
And if i will pass "2020-05" then it should returns
|---------------------|------------------|--------------------|
| Name | ID | Date |
|---------------------|------------------|--------------------|
| A | 1 |2020-05-13T14:00:00 |
|---------------------|------------------|--------------------|
| D | 4 |2020-05-29T14:00:00 |
|---------------------|------------------|--------------------|
Is there anything in mongoDb like STR_TO_DATE() in MySQL and TO_DATE() in oracle`**.
I am trying to implement this in Spring boot application.
Try above query.
It will help you.

One solution would be this one:
db.collection.find({
$expr: {
$and: [
{ $eq: [{ $year: "$Date" }, 2020] },
{ $eq: [{ $month: "$Date" }, 5] }
]
}
})

Get abstract syntax tree of JSON string

Problem Description
I am writing a simple JSON analyzer in order to achieve syntax analysis of JSON strings. I am receiving random structure JSON strings and I want to export the syntax structure. As result, I want to get a tree structure which describes the format of the JSON string (key, value, arrays etc) and types of every element. I've already found the syntax definition of the JSON (described below)
object
{}
{ members }
members
pair
pair , members
pair
string : value
array
[]
[ elements ]
elements
value
value , elements
value
string
number
object
array
true
false
null
Example
JSON String:
{"widget": {
"null": null,
"window": {
153: "This is string",
"boolean": true,
"int": 500,
"float": 5.555
}
}}
And I want to get something like:
{ KEY_STR : {
KEY_STR : null
KEY_ARRAY : {
KEY_INT: VALUE_STR,
KEY_STR: VALUE_BOOL,
KEY_STR: VALUE_INT,
KEY_STR: VALUE_FLOAT
}
}}
I am using JAVA with GSON library.
How I want to use that
I am interested to export the abstract tree in order to create my own messages automatically
My Question
I have started to implement that by using JsonParser. I am parsing the JSON object and then I am defining for every key and value the type. But I am wondering if I am on a good way or I am discovering the wheel. Is there anything already exist to export the abstract syntax tree or I should implement that by myself?

I have no idea what JsonParser will export. But in general, parsing something, then exporting an AST form from an AST data structure, reading the AST, and then extracting values from the read AST seems like just a lot of overhead goo to build and maintain.
What you should do is build the JSON parser into your application, parse the JSON to an AST data structure, and simply process that AST structure directly.
Frankly, JSON is simple enough so you could write your own recursive descent parser to parse JSON and build the AST, leading you back to the first solution. See https://stackoverflow.com/a/2336769/120163
If you absolutely insist on exporting it, you can find tools that will do that off the shelf.
Our DMS Software Reengineering Toolkit will do that, although it might be a bit heavyweight for this kind of application.
One of the nice things about JSON is the simple grammar. Here's the grammar that DMS uses:
-- JSON.atg: JSON domain grammar for DMS
-- Copyright (C) 2011-2018 Semantic Designs, Inc.; All Rights Reserved
--
-- Update history:
-- Date Initials Changes Made
-- 2011/09/02 CW Created
--
-- Note that dangling commas in lists are off-spec
-- but I (CW) hate dealing with them
--
-- I'm not sure if JSON is supposed to have more than one entity per file
-- but I'm allowing it for robustness
--
-- TODO: name/value pair lists should be associative-commutative
JSON_text = ;
JSON_text = JSON_text object ;
JSON_text = JSON_text array ;
-- unordered set of name/value pairs
-- should be able to use an associative-commutative property directive
object = '{' name_value_pair_list '}' ;
-- empty production is for empty list, but will also allow multiple commas
name_value_pair_list = ;
name_value_pair_list = name_value_pair_list ',' ;
name_value_pair_list = name_value_pair_list name_value_pair ;
name_value_pair = STRING ':' value ;
-- ordered collection of values
array = '[' value_list ']' ;
value_list = value ;
value_list = value_list ',' value ;
value_list = value_list ',' value ',' ;
value = STRING ;
value = NUMBER_INT ;
value = NUMBER_FLOAT ;
value = object ;
value = array ;
value = 'true' ;
value = 'false' ;
value = 'null' ;
Yes, it almost exactly matches the abstract grammar provided by the OP.
Now, with that, you can ask DMS to parse a file and export its AST with this command:
run ..\DomainParser +AST ..\..\..\Examples\One.js
For the JSON file One.js, containing this text:
{
"from": "http://json.org/example.html"
}
{
"glossary": {
"title": "example glossary",
"GlossDiv": {
"title": "S",
"GlossList": {
"GlossEntry": {
"ID": "SGML",
"SortAs": "SGML",
"GlossTerm": "Standard Generalized Markup Language",
"Acronym": "SGML",
"Abbrev": "ISO 8879:1986",
"GlossDef": {
"para": "A meta-markup language, used to create markup languages such as DocBook.",
"GlossSeeAlso": ["GML", "XML"]
},
"GlossSee": "markup"
}
}
}
}
}
<rest of file snipped>
The parser produces an S-expression:
(JSON_text#JSON=2#59406e0^0 Line 1 Column 1 File C:/DMS/Domains/JSON/Examples/One.js
(JSON_text#JSON=2#2199f60^1#59406e0:1 Line 1 Column 1 File C:/DMS/Domains/JSON/Examples/One.js
(JSON_text#JSON=2#21912a0^1#2199f60:1 Line 1 Column 1 File C:/DMS/Domains/JSON/Examples/One.js
(JSON_text#JSON=2#593df00^1#21912a0:1 Line 1 Column 1 File C:/DMS/Domains/JSON/Examples/One.js
|(JSON_text#JSON=2#593d420^1#593df00:1 Line 1 Column 1 File C:/DMS/Domains/JSON/Examples/One.js
| (JSON_text#JSON=2#593c580^1#593d420:1 Line 1 Column 1 File C:/DMS/Domains/JSON/Examples/One.js
| (JSON_text#JSON=1#593bec0^1#593c580:1 Line 1 Column 1 File C:/DMS/Domains/JSON/Examples/One.js)JSON_text
| (object#JSON=4#593c560^1#593c580:2 Line 1 Column 1 File C:/DMS/Domains/JSON/Examples/One.js
| (name_value_pair_list#JSON=7#593c520^1#593c560:1 Line 2 Column 5 File C:/DMS/Domains/JSON/Examples/One.js
| |(name_value_pair_list#JSON=5#593c420^1#593c520:1 Line 2 Column 5 File C:/DMS/Domains/JSON/Examples/One.js)name_value_pair_list
| |(name_value_pair#JSON=8#593c4e0^1#593c520:2 Line 2 Column 5 File C:/DMS/Domains/JSON/Examples/One.js
| | (STRING#JSON=24#593c400^1#593c4e0:1[`from'] Line 2 Column 5 File C:/DMS/Domains/JSON/Examples/One.js)STRING
| | (STRING#JSON=24#593c480^1#593c4e0:2[`http://json.org/example.html'] Line 2 Column 13 File C:/DMS/Domains/JSON/Examples/One.js)STRING
| |)name_value_pair#593c4e0
| )name_value_pair_list#593c520
| )object#593c560
| )JSON_text#593c580
| (object#JSON=4#593d400^1#593d420:2 Line 5 Column 1 File C:/DMS/Domains/JSON/Examples/One.js
| (name_value_pair_list#JSON=7#593d3c0^1#593d400:1 Line 6 Column 5 File C:/DMS/Domains/JSON/Examples/One.js
| (name_value_pair_list#JSON=5#593c5c0^1#593d3c0:1 Line 6 Column 5 File C:/DMS/Domains/JSON/Examples/One.js)name_value_pair_list
| (name_value_pair#JSON=8#593d380^1#593d3c0:2 Line 6 Column 5 File C:/DMS/Domains/JSON/Examples/One.js
| |(STRING#JSON=24#593c5a0^1#593d380:1[`glossary'] Line 6 Column 5 File C:/DMS/Domains/JSON/Examples/One.js)STRING
| |(object#JSON=4#593d360^1#593d380:2 Line 6 Column 17 File C:/DMS/Domains/JSON/Examples/One.js
| | (name_value_pair_list#JSON=7#593d340^1#593d360:1 Line 7 Column 9 File C:/DMS/Domains/JSON/Examples/One.js
| | (name_value_pair_list#JSON=6#593c720^1#593d340:1 Line 7 Column 9 File C:/DMS/Domains/JSON/Examples/One.js
| | (name_value_pair_list#JSON=7#593c6c0^1#593c720:1 Line 7 Column 9 File C:/DMS/Domains/JSON/Examples/One.js
| | |(name_value_pair_list#JSON=5#593c600^1#593c6c0:1 Line 7 Column 9 File C:/DMS/Domains/JSON/Examples/One.js)name_value_pair_list
| | |(name_value_pair#JSON=8#593c640^1#593c6c0:2 Line 7 Column 9 File C:/DMS/Domains/JSON/Examples/One.js
| | | (STRING#JSON=24#593c5e0^1#593c640:1[`title'] Line 7 Column 9 File C:/DMS/Domains/JSON/Examples/One.js)STRING
| | | (STRING#JSON=24#593c620^1#593c640:2[`example glossary'] Line 7 Column 18 File C:/DMS/Domains/JSON/Examples/One.js)STRING
| | |)name_value_pair#593c640
| | )name_value_pair_list#593c6c0
| | )name_value_pair_list#593c720
| | (name_value_pair#JSON=8#593d320^1#593d340:2 Line 8 Column 17 File C:/DMS/Domains/JSON/Examples/One.js
| | (STRING#JSON=24#593c700^1#593d320:1[`GlossDiv'] Line 8 Column 17 File C:/DMS/Domains/JSON/Examples/One.js)STRING
| | (object#JSON=4#593d300^1#593d320:2 Line 8 Column 29 File C:/DMS/Domains/JSON/Examples/One.js
| | |(name_value_pair_list#JSON=7#593d2e0^1#593d300:1 Line 9 Column 13 File C:/DMS/Domains/JSON/Examples/One.js
| | | (name_value_pair_list#JSON=6#593c880^1#593d2e0:1 Line 9 Column 13 File C:/DMS/Domains/JSON/Examples/One.js
| | | (name_value_pair_list#JSON=7#593c820^1#593c880:1 Line 9 Column 13 File C:/DMS/Domains/JSON/Examples/One.js
| | | (name_value_pair_list#JSON=5#593c760^1#593c820:1 Line 9 Column 13 File C:/DMS/Domains/JSON/Examples/One.js)name_value_pair_list
| | | (name_value_pair#JSON=8#593c7e0^1#593c820:2 Line 9 Column 13 File C:/DMS/Domains/JSON/Examples/One.js
| | | |(STRING#JSON=24#593c740^1#593c7e0:1[`title'] Line 9 Column 13 File C:/DMS/Domains/JSON/Examples/One.js)STRING
| | | |(STRING#JSON=24#593c780^1#593c7e0:2[`S'] Line 9 Column 22 File C:/DMS/Domains/JSON/Examples/One.js)STRING
| | | )name_value_pair#593c7e0
| | | )name_value_pair_list#593c820
| | | )name_value_pair_list#593c880
| | | (name_value_pair#JSON=8#593d2c0^1#593d2e0:2 Line 10 Column 25 File C:/DMS/Domains/JSON/Examples/One.js
| | | (STRING#JSON=24#593c860^1#593d2c0:1[`GlossList'] Line 10 Column 25 File C:/DMS/Domains/JSON/Examples/One.js)STRING
| | | (object#JSON=4#593d2a0^1#593d2c0:2 Line 10 Column 38 File C:/DMS/Domains/JSON/Examples/One.js
| | | (name_value_pair_list#JSON=7#593d280^1#593d2a0:1 Line 11 Column 17 File C:/DMS/Domains/JSON/Examples/One.js
| | | |(name_value_pair_list#JSON=5#593c8c0^1#593d280:1 Line 11 Column 17 File C:/DMS/Domains/JSON/Examples/One.js)name_value_pair_list
| | | |(name_value_pair#JSON=8#593d260^1#593d280:2 Line 11 Column 17 File C:/DMS/Domains/JSON/Examples/One.js
| | | | (STRING#JSON=24#593c8a0^1#593d260:1[`GlossEntry'] Line 11 Column 17 File C:/DMS/Domains/JSON/Examples/One.js)STRING
| | | | (object#JSON=4#593d240^1#593d260:2 Line 11 Column 31 File C:/DMS/Domains/JSON/Examples/One.js
| | | | (name_value_pair_list#JSON=7#593d200^1#593d240:1 Line 12 Column 21 File C:/DMS/Domains/JSON/Examples/One.js
| | | | (name_value_pair_list#JSON=6#593d160^1#593d200:1 Line 12 Column 21 File C:/DMS/Domains/JSON/Examples/One.js
| | | | |(name_value_pair_list#JSON=7#593d120^1#593d160:1 Line 12 Column 21 File C:/DMS/Domains/JSON/Examples/One.js
| | | | | (name_value_pair_list#JSON=6#593cde0^1#593d120:1 Line 12 Column 21 File C:/DMS/Domains/JSON/Examples/One.js
| | | | | (name_value_pair_list#JSON=7#593cd60^1#593cde0:1 Line 12 Column 21 File C:/DMS/Domains/JSON/Examples/One.js
| | | | | (name_value_pair_list#JSON=6#593cca0^1#593cd60:1 Line 12 Column 21 File C:/DMS/Domains/JSON/Examples/One.js
| | | | | |(name_value_pair_list#JSON=7#593cc60^1#593cca0:1 Line 12 Column 21 File C:/DMS/Domains/JSON/Examples/One.js
| | | | | | (name_value_pair_list#JSON=6#593cc00^1#593cc60:1 Line 12 Column 21 File C:/DMS/Domains/JSON/Examples/One.js
| | | | | | (name_value_pair_list#JSON=7#593cb80^1#593cc00:1 Line 12 Column 21 File C:/DMS/Domains/JSON/Examples/One.js
| | | | | | (name_value_pair_list#JSON=6#593cb00^1#593cb80:1 Line 12 Column 21 File C:/DMS/Domains/JSON/Examples/One.js
| | | | | | |(name_value_pair_list#JSON=7#593cac0^1#593cb00:1 Line 12 Column 21 File C:/DMS/Domains/JSON/Examples/One.js
| | | | | | | (name_value_pair_list#JSON=6#593ca60^1#593cac0:1 Line 12 Column 21 File C:/DMS/Domains/JSON/Examples/One.js
| | | | | | | (name_value_pair_list#JSON=7#593ca00^1#593ca60:1 Line 12 Column 21 File C:/DMS/Domains/JSON/Examples/One.js
| | | | | | | (name_value_pair_list#JSON=5#593c900^1#593ca00:1 Line 12 Column 21 File C:/DMS/Domains/JSON/Examples/One.js)name_value_pair_list
| | | | | | | (name_value_pair#JSON=8#593c9c0^1#593ca00:2 Line 12 Column 21 File C:/DMS/Domains/JSON/Examples/One.js
| | | | | | | |(STRING#JSON=24#593c8e0^1#593c9c0:1[`ID'] Line 12 Column 21 File C:/DMS/Domains/JSON/Examples/One.js)STRING
| | | | | | | |(STRING#JSON=24#593c920^1#593c9c0:2[`SGML'] Line 12 Column 27 File C:/DMS/Domains/JSON/Examples/One.js)STRING
| | | | | | | )name_value_pair#593c9c0
| | | | | | | )name_value_pair_list#593ca00
| | | | | | | )name_value_pair_list#593ca60
| | | | | | | (name_value_pair#JSON=8#593caa0^1#593cac0:2 Line 13 Column 41 File C:/DMS/Domains/JSON/Examples/One.js
| | | | | | | (STRING#JSON=24#593ca40^1#593caa0:1[`SortAs'] Line 13 Column 41 File C:/DMS/Domains/JSON/Examples/One.js)STRING
| | | | | | | (STRING#JSON=24#593ca80^1#593caa0:2[`SGML'] Line 13 Column 51 File C:/DMS/Domains/JSON/Examples/One.js)STRING
| | | | | | | )name_value_pair#593caa0
| | | | | | |)name_value_pair_list#593cac0
I've truncated the output because nobody really wants to see the tree. Now, there's a lot of "extra" stuff in tree such as node location, source line numbers, which can all be easily eliminated or ignored.

Read array of Json-Ld Objects into a Model In Apache Jena.How to retrieve individual objects from Model?

This is Json Array of Object(Student Data) . I am loaded that Json-Ld Data in Jena Model
[
{
"#context" : {
"myvocab" : "http://mywebsite.com/vocab/",
"name" : "myvocab:name",
"firstname" : "myvocab:firstname",
"lastname" : "myvocab:lastname",
"rollNumber" : "myvocab:rollNumber"
},
"name" : {
"firstname" : "Dhannan",
"lastname" : "Chaudhary"
},
"rollNumber" : "26"
},
{
"#context" : {
"myvocab" : "http://mywebsite.com/vocab/",
"name" : "myvocab:name",
"firstname" : "myvocab:firstname",
"lastname" : "myvocab:lastname",
"rollNumber" : "myvocab:rollNumber"
},
"name" : {
"firstname" : "Maakin",
"lastname" : "Dhayaal"
},
"rollNumber" : "69"
}
]
This is my model output for above example ( by using SPARQL )
-------------------------------------------------------------------
| Subject | Predicate | Object |
===================================================================
| _:b0 | <http://mywebsite.com/vocab/lastname> | "Chaudhary" |
| _:b0 | <http://mywebsite.com/vocab/firstname> | "Dhannan" |
| _:b1 | <http://mywebsite.com/vocab/lastname> | "Dhayaal" |
| _:b1 | <http://mywebsite.com/vocab/firstname> | "Maakin" |
| _:b2 | <http://mywebsite.com/vocab/rollNumber> | "62" |
| _:b2 | <http://mywebsite.com/vocab/name> | _:b1 |
| _:b3 | <http://mywebsite.com/vocab/rollNumber> | "61" |
| _:b3 | <http://mywebsite.com/vocab/name> | _:b0 |
-------------------------------------------------------------------
From this model I want only Subjects(Resources in term of Jena) of every Student for my case it should ( _:b2 , _:b3) .
But by using model.listSubjects() it gives iterator to all subjects ( _:b0 , _:b1 , _:b2 , _:b3)
My main goal is to be able to get individual models for student 1 and student 2.
How can I achieve this?
Every Suggestions are welcome.

First, you can use RDF Type literal to define Student class as well as the StudentName class (not sure why you'd need to break them up).
You can then check if the subject has the property that you are looking for. You can see how we do this in Eclipse Lyo Jena provider.
Finally, you can model your domain with Lyo modelling tools and generate the POJOs for your domain that can be converted from/to Jena models in a single method call.

Categories