SQLite compare dates between two consecutive rows - java

I have a table of media files:
id title date
The date is in epoch format, and for example, I have 3 photos:
1 xxxxx 1644777107349(6:31:47.349)
2 wwwww 1644777110138(6:31:50.138)
3 zzzzz 1644777117453(6:31:57.453)
I want to get the count of photos that were taken with a gap of 5 to 30 seconds, for example:
2 and 3 were taken with a gap of 7 seconds so I want to get 2. I want to compare every row with the consecutive row.
This is the SQL query:
SELECT Count(*)
FROM (SELECT A._id
FROM media_files A
INNER JOIN media_files B
ON( Abs(A.added_time - B.added_time) >= 5000 )
AND A._id != B._id
AND A.type = 'image'
AND B.type = 'image'
WHERE Abs(A.added_time - B.added_time) <= 30000
GROUP BY A._id)
And for the above example, I always get 3 instead of 2, any idea what is the problem?

Here is a test setup and a query which returns only the images taken between 5 and 30 seconds after the previous one.NB I'm working on mySQL so where I have put 5 and 30 seconds it may be that in SQLlite you need to put 5000 and 30000.
CREATE TABLE media_files (
id INT,
title VARCHAR(25),
added_date DATETIME,
type VARCHAR(25)
);
INSERT INTO media_files VALUES ( 1, 'xxxxx', '2022-02-13 6:30:47.349','image');
INSERT INTO media_files VALUES ( 1, 'xxxxx', '2022-02-13 6:31:27.349','image');
INSERT INTO media_files VALUES ( 1, 'xxxxx', '2022-02-13 6:31:47.349','image');
INSERT INTO media_files VALUES ( 2, 'wwwww', '2022-02-13 6:31:50.138','image');
INSERT INTO media_files VALUES ( 3, 'zzzzz', '2022-02-13 6:31:57.453','image');
SELECT id,
added_date ad,
added_date -
( SELECT MAX(added_date)
FROM media_files m
WHERE m.added_date < mf.added_date ) DIF
FROM media_files mf
WHERE ( added_date -
( SELECT MAX(added_date)
FROM media_files m
WHERE m.added_date < mf.added_date )) > 5
AND ( added_date -
( SELECT MAX(added_date)
FROM media_files m
WHERE m.added_date < mf.added_date )) < 30;
id | ad | DIF
-: | :------------------ | --:
1 | 2022-02-13 06:31:47 | 20
3 | 2022-02-13 06:31:57 | 7

Related

What is the best way to group timestamps into intervals of given length in Java?

I have a table for metrics in Oracle DB where one of the columns is the timestamp. I need to read the metrics from the DB and group them into given intervals of any length( 2 Month or 3 Hour or 1 Days or 2 years etc) between a starting timestamp and ending timestamp. The timestamp will be of format
2020-05-24T18:51:10.018-07:00
I know I can read all the entries from the table and sort them and group them into intervals by converting them all into seconds, but is there a better way to do it ?
You may use match_recognize for this.
with t(ts) as (
select
current_timestamp
+ interval '3' minute
* dbms_random.value(0, level)
from dual
connect by level < 20
)
select *
from t
match_recognize(
order by ts asc
measures
match_number() as grp
all rows per match
pattern(a w5min*)
define
/*Rows within 5 minutes after the first row in bucket*/
w5min as ts - first(ts) < interval '5' minute
)
TS | GRP
:------------------ | --:
2021-11-03 06:20:40 | 1
2021-11-03 06:20:56 | 1
2021-11-03 06:23:27 | 1
2021-11-03 06:23:49 | 1
2021-11-03 06:25:23 | 1
2021-11-03 06:25:36 | 1
2021-11-03 06:32:14 | 2
2021-11-03 06:34:38 | 2
2021-11-03 06:36:29 | 2
2021-11-03 06:36:59 | 2
2021-11-03 06:39:29 | 3
2021-11-03 06:40:17 | 3
2021-11-03 06:41:07 | 3
2021-11-03 06:47:14 | 4
2021-11-03 06:48:31 | 4
2021-11-03 06:52:29 | 5
2021-11-03 06:59:22 | 6
2021-11-03 07:02:05 | 6
2021-11-03 07:04:54 | 7
db<>fiddle here
Whether it’s the best way, who can tell? If you want a Java solution, I suggest this one.
OffsetDateTime[] timestamps = {
OffsetDateTime.parse("2020-05-24T18:51:10.018-07:00"),
OffsetDateTime.parse("2020-03-07T23:45:02.399-08:00"),
OffsetDateTime.parse("2020-05-24T20:01:11.442-07:00"),
OffsetDateTime.parse("2020-03-08T01:03:05.079-08:00"),
OffsetDateTime.parse("2020-05-24T19:32:34.461-07:00"),
};
TemporalAmount intervalLength = Duration.ofHours(2);
List<OffsetDateTime> list = new ArrayList<>(Arrays.asList(timestamps));
list.sort(Comparator.naturalOrder());
List<List<OffsetDateTime>> groups = new ArrayList<>();
if (! list.isEmpty()) {
List<OffsetDateTime> currentGroup = new ArrayList<>();
Iterator<OffsetDateTime> itr = list.iterator();
OffsetDateTime first = itr.next();
currentGroup.add(first);
OffsetDateTime groupEnd = first.plus(intervalLength);
while (itr.hasNext()) {
OffsetDateTime current = itr.next();
if (current.isBefore(groupEnd)) {
currentGroup.add(current);
} else { // new group
groups.add(currentGroup);
currentGroup = new ArrayList<>();
groupEnd = current.plus(intervalLength);
currentGroup.add(current);
}
}
// remember to add last group
groups.add(currentGroup);
}
groups.forEach(System.out::println);
Output:
[2020-03-07T23:45:02.399-08:00, 2020-03-08T01:03:05.079-08:00]
[2020-05-24T18:51:10.018-07:00, 2020-05-24T19:32:34.461-07:00, 2020-05-24T20:01:11.442-07:00]
The advantage of declaring intervalLength a TemporalAmount is you are free to assign either a time-based Duration to it (as above) or a date-based Period.
TemporalAmount intervalLength = Period.ofMonths(3);
In this case the result is just one group:
[2020-03-07T23:45:02.399-08:00, 2020-03-08T01:03:05.079-08:00,
2020-05-24T18:51:10.018-07:00, 2020-05-24T19:32:34.461-07:00,
2020-05-24T20:01:11.442-07:00]
Link
Oracle tutorial: Date Time explaining how to use java.time.

Genrating more rows based on a colum, if criteria not matched SQL

I have the following table, These table has information about the number of files and some other fields . This is a lookup table or reference table.
The oracle version i am using Oracle Database 18c Enterprise E
id nm expected_num_files file_name_expr
1 CVS 3 cvs_d.*.zip
2 CVS 2 cvs_w.*.gz
3 Rite-aid 4 ra_d.*.gz
5 Walgreen 2 wal_d*.txt
I have a audit table which has the information received files. Both of those tables can be joined on id
audit_id id file_nm
123 1 cvs_d1.zip
124 1 cvs_d2.zip
125 2 cvs_w1.gz
126 1 cvs_d3.zip
The ideal case is where all the files received.
Ideal Result
select id , count(*) from auditlog group by id
id count_files
1 3
2 2
3 4
5 2
CurrentResult of audit table
But in current case i recieved only some files
id count_files
1 3
2 1
In order to reach ideal case, i need to populate the dummy records in the final table from the lookup table with empty auditid
I need a final output table should like this.
If i perform the query select id , count(*) from auditlog group by id
on final table, i will get the ideal result that is highlighted above
audit_id id file_nm
123 1 cvs_d1.zip
124 1 cvs_d2.zip
126 1 cvs_d3.zip
-1 2 cvs_w.*.gz
125 2 cvs_w1.gz
-1 3 ra_d*.gz
-1 3 ra_d.*.gz
-1 3 ra_d.*.gz
-1 3 ra_d.*.gz
-1 5 wal_d*.txt
-1 5 wal_d*.txt
We can generate the initail rows easily, but the rows with -1 are generated based on number of files not received form the column(number of files they didnt sent)
Explain final table: As we have 3 records for id 1 in the audit table so we populated them in the final table, But for id 2 we have one record in the audit table we populated that and for other record we populated -1.
As for the data you have given, you can stay with the tables as the are and create a view that provides the data as needed:
WITH /* The following is the lookup data you provided: */
lookup(id, nm, expected_num_files, file_name_expr) AS
(SELECT 1, 'CVS', 3,'cvs_d.*.zip' from dual union all
SELECT 2, 'CVS', 2,'cvs_w.*.gz' from dual union all
SELECT 3, 'Rite-aid',4,'ra_d.*.gz' from dual union all
SELECT 5, 'Walgreen',2,'wal_d*.txt' from dual)
, /* This is the current auditlog as you described: */
auditlog(audit_id, id, file_nm) AS
(select 123, 1, 'cvs_d1.zip' from dual union all
select 124, 1, 'cvs_d2.zip' from dual union all
select 125, 2, 'cvs_w1.gz' from dual union all
select 126, 1, 'cvs_d3.zip' from dual)
, rn AS (SELECT LEVEL rn FROM dual
CONNECT BY LEVEL < (SELECT MAX(expected_num_files) FROM lookup))
/* This is the select you can put into a view: */
SELECT NVL(a.audit_id, -1) AS audit_id
, NVL(a.id,l.id) AS id
, NVL(a.file_nm, l.file_name_expr) AS file_nm
FROM lookup l
/* Create a Row for every expected file: */
JOIN rn r
ON r.rn <= l.expected_num_files
FULL JOIN (SELECT a.*
, row_number() over(PARTITION BY id ORDER BY audit_id) AS rn
FROM auditlog a) a
ON a.id = l.id
AND a.rn = r.rn
ORDER BY 2,1
Result:
AUDIT_ID | ID | FILE_NM
---------+----+-----------
123 | 1 | cvs_d1.zip
124 | 1 | cvs_d2.zip
126 | 1 | cvs_d3.zip
-1 | 2 | cvs_w.*.gz
125 | 2 | cvs_w1.gz
-1 | 3 | ra_d.*.gz
-1 | 3 | ra_d.*.gz
-1 | 3 | ra_d.*.gz
-1 | 3 | ra_d.*.gz
-1 | 5 | wal_d*.txt
-1 | 5 | wal_d*.txt
An other way to write the query is the following:
WITH lookup(id, nm, expected_num_files, file_name_expr) AS
(SELECT 1, 'CVS', 3,'cvs_d.*.zip' from dual union all
SELECT 2, 'CVS', 2,'cvs_w.*.gz' from dual union all
SELECT 3, 'Rite-aid',4,'ra_d.*.gz' from dual union all
SELECT 5, 'Walgreen',2,'wal_d*.txt' from dual)
, auditlog(audit_id, id, file_nm) AS
(select 123, 1, 'cvs_d1.zip' from dual union all
select 124, 1, 'cvs_d2.zip' from dual union all
select 125, 2, 'cvs_w1.gz' from dual union all
select 126, 1, 'cvs_d3.zip' from dual)
, rn AS (SELECT LEVEL rn FROM dual
CONNECT BY LEVEL < (SELECT MAX(expected_num_files) FROM lookup))
SELECT * FROM auditlog
UNION ALL
SELECT -1, l.id, l.file_name_expr
FROM lookup l
JOIN rn r
ON r.rn <= l.expected_num_files - NVL((SELECT COUNT(*) FROM auditlog WHERE id = l.id),0)
ORDER BY 2,1
Is there any other option than cross join. There is performance impact because i have millions of records
That's always hard to judge. A CROSS APPLY may be faster. At the least, it saves the work of generating a lot of rows that end up getting discarded. It might be worth trying.
SELECT coalesce(al.audit_id,-1) audit_id,
s.id,
coalesce(al.file_nm, s.file_name_expr) file_nm
FROM audit_summary s
CROSS APPLY ( SELECT rownum rn FROM dual CONNECT BY rownum <= s.expected_num_files ) ef
LEFT JOIN LATERAL ( SELECT row_number() over ( partition by al.id ORDER BY al.audit_id ) rn,
al.file_nm,
al.audit_id,
al.id
FROM audit_log al
WHERE al.id = s.id) al ON al.rn = ef.rn
ORDER BY 2,3,1;
Here is a full example with data:
WITH audit_summary (id, nm, expected_num_files, file_name_expr ) AS
( SELECT 1, 'CVS', 3, 'cvs_d.*.zip' FROM DUAL UNION ALL
SELECT 2, 'CVS', 2, 'cvs_w.*.gz' FROM DUAL UNION ALL
SELECT 3, 'Rite-aid', 4, 'ra_d.*.gz' FROM DUAL UNION ALL
SELECT 4, 'Walgreen', 2, 'wal_d*.txt' FROM DUAL),
audit_log (audit_id, id, file_nm) AS
( SELECT 123, 1, 'cvs_d1.zip' FROM DUAL UNION ALL
SELECT 124, 1, 'cvs_d2.zip' FROM DUAL UNION ALL
SELECT 125, 2, 'cvs_w1.gz' FROM DUAL UNION ALL
SELECT 126, 1, 'cvs_d3.zip' FROM DUAL )
SELECT coalesce(al.audit_id,-1) audit_id,
s.id,
coalesce(al.file_nm, s.file_name_expr) file_nm
FROM audit_summary s
CROSS APPLY ( SELECT rownum rn FROM dual CONNECT BY rownum <= s.expected_num_files ) ef
LEFT JOIN LATERAL ( SELECT row_number() over ( partition by al.id ORDER BY al.audit_id ) rn,
al.file_nm,
al.audit_id,
al.id
FROM audit_log al
WHERE al.id = s.id) al ON al.rn = ef.rn
ORDER BY 2,3,1;
+----------+----+------------+
| AUDIT_ID | ID | FILE_NM |
+----------+----+------------+
| 123 | 1 | cvs_d1.zip |
| 124 | 1 | cvs_d2.zip |
| 126 | 1 | cvs_d3.zip |
| -1 | 2 | cvs_w.*.gz |
| 125 | 2 | cvs_w1.gz |
| -1 | 3 | ra_d.*.gz |
| -1 | 3 | ra_d.*.gz |
| -1 | 3 | ra_d.*.gz |
| -1 | 3 | ra_d.*.gz |
| -1 | 4 | wal_d*.txt |
| -1 | 4 | wal_d*.txt |
+----------+----+------------+
I was split on whether the LEFT JOIN LATERAL was helpful or not. It's the same as a simple LEFT JOIN with the al.id = s.id condition moved from the lateral view to the join condition. I have a vague idea of the LEFT JOIN LATERAL making it so you can run the query piecemeal (by audit_summary.id) if there are specific files you suspect are missing.

How to get aggregate and column data from Derby database joining three tables

I need to present data from a Derby database in a JTable, but two of the columns are aggregate sums from two one-to-many related tables. Here are example schema:
SHIFTDATA:
ID
DATE
SHIFT
FOOD_COST
OFFICE_SUPPLIES
REP_MAINT
NET_SALES
SALES_TAX
OTHERPAIDOUTS:
ID
SHIFTDATA_ID
LABEL
AMOUNT
DISCOUNTS
ID
SHIFTDATA_ID
DISCOUNT_NAME
AMOUNT
There are 0 or more OTHERPAIDOUTS for a given SHIFTDATA
There are 0 or more DISCOUNTS for a given SHIFTDATA
I need the equivalent of this statement, though I know I can’t combine aggregate expressions with "non-aggregate expressions" in a SELECT statement:
SELECT (S.FOOD_COST + S.OFFICE_SUPPLIES + S.REP_MAINT + SUM(O.AMOUNT)) AS TOTAL_PAIDOUTS,
SUM(D.AMOUNT) AS TOTAL_DISCOUNT,
S.NET_SALES,
S.SALES_TAX
FROM SHIFTDATA S, OTHERPAIDOUTS O, DISCOUNTS D WHERE O.SHIFTDATA_ID=S.ID AND D.SHIFTDATA_ID=S.ID
I see in other threads where adding the GROUP BY clause fixes these situations, but I guess adding in the second aggregate from a third table is throwing me off. I tried GROUP BY S.NET_SALES, S.SALES_TAX, and adding AND S.ID = 278 to the WHERE clause to get a known result, and the TOTAL_PAIDOUTS is correct (there are 3 related records in OTHERPAIDOUTS), but the returned TOTAL_DISCOUNTS is 3 times what it should be.
Needless to say, I’m not a SQL programmer! Hopefully you get the gist of what I’m after. I tried nested SELECT statements but just made a mess of it. This application is still in development, including the database structure, so if a different DB structure would simplify things, that may be an option. Or, if there's another way to programmatically build the table model, I'm open to that as well. Thanks in advance!!
======== Edit =============
In order to check the values from a known record, I'm querying with a specific SHIFTDATA.ID. Following is the sample table records:
SHIFTDATA:
ID |FOOD_COST |OFFICE_SU&|REP_MAINT |NET_SALES |SALES_TAX
------------------------------------------------------
278 |0.00 |5.00 |10.00 |3898.78 |319.79
OTHERPAIDOUTS:
ID |SHIFTDATA_&|LABEL |AMOUNT
---------------------------------------------------------------------------
37 |278 |FOOD COST FUEL |52.00
38 |278 |MAINT FUEL |5.00
39 |278 |EMPLOYEE SHOES |21.48
DISCOUNTS:
ID |ITEM_NAME |SHIFTDATA_&|AMOUNT
---------------------------------------------------------------------------
219 |Misc Discounts |278 |15.91
What I expect to see for this SHIFTDATA row in the JTable:
TOTAL_PAIDOUTS | TOTAL_DISCOUNT |NET_SALES |SALES_TAX
------------------------------------------------------
93.48 |15.91 |3898.78 |319.79
The best I can get is by adding the GROUP BY clause, but grouping by the fields from SHIFTDATA I get:
TOTAL_PAIDOUTS | TOTAL_DISCOUNT |NET_SALES |SALES_TAX
------------------------------------------------------
93.48 |47.73 |3898.78 |319.79
Here is the SQL query with required result.
Here are the table definitions, data, sql and the results:
CREATE TABLE shiftdata (
id int,
foodcost int,
officesuppl int,
repmaint int,
netsales int,
salestax int);
CREATE TABLE otherpaidouts (
id int,
shiftid int,
label varchar(20),
amount int);
CREATE TABLE discounts (
id int,
shiftid int,
itemname varchar(20),
amount int);
Create data for two shifts: 278 and 333. Both shifts have discounts, but only 278 shift has the otherpaidouts.
insert into shiftdata values (278, 0, 5, 10, 3898, 319);
insert into shiftdata values (333, 22, 15, 100, 2111, 88);
insert into otherpaidouts values (37, 278, 'Food Cost FUEL', 52);
insert into otherpaidouts values (38, 278, 'Maint FUEL', 5);
insert into otherpaidouts values (39, 278, 'Empl SHOES', 21);
insert into discounts values (219, 278, 'Misc DISCOUNTS', 15);
insert into discounts values (312, 333, 'Misc DISCOUNTS', 25);
The Query:
SELECT sd.id, sd.netsales, sd.salestax,
IFNULL(
(SELECT SUM(d.amount) FROM discounts d WHERE d.shiftid=sd.id), 0) AS total_discount,
(SELECT sd.foodcost + sd.officesuppl + sd.repmaint + IFNULL(SUM(op.amount), 0) FROM otherpaidouts op WHERE op.shiftid=sd.id) AS total_paidouts
FROM shiftdata sd;
The Result:
+------+----------+----------+----------------+----------------+
| id | netsales | salestax | total_discount | total_paidouts |
+------+----------+----------+----------------+----------------+
| 278 | 3898 | 319 | 15 | 93 |
| 333 | 2111 | 88 | 25 | 137 |
+------+----------+----------+----------------+----------------+
Try a LEFT OUTER JOIN, something like this:
SELECT S.FOOD_COST + S.OFFICE_SUPPLIES + S.REP_MAINT + SUM(O.AMOUNT) AS TOTAL_PAIDOUTS,
SUM(D.AMOUNT) AS TOTAL_DISCOUNT,
S.NET_SALES,
S.SALES_TAX
FROM SHIFTDATA S
LEFT JOIN OTHERPAIDOUTS AS O ON O.SHIFTDATA_ID = S.ID
LEFT JOIN DISCOUNTS AS D ON D.SHIFTDATA_ID = S.ID
Edit
SELECT S.FOOD_COST + S.OFFICE_SUPPLIES + S.REP_MAINT +
( SELECT COALESCE(SUM(AMOUNT), 0) FROM OTHERPAIDOUTS WHERE SHIFTDATA_ID = S.ID ) AS TOTAL_PAIDOUTS,
( SELECT COALESCE(SUM(AMOUNT), 0) FROM DISCOUNTS WHERE SHIFTDATA_ID = S.ID ) AS TOTAL_DISCOUNT,
S.NET_SALES,
S.SALES_TAX
FROM SHIFTDATA S

Get rows based on maximum value for subsets of a table

I want to filter a table based on the values of one column, then get the maximum value for each of these values.
e.g.
id | value
-----------
0 | 10
0 | 22
0 | 50
1 | 33
1 | 4
2 | 5
2 | 23
2 | 33
3 | 22
3 | 50
Filter by rows with IDs 2 and 3, then get the maximum of each id
id | value
-----------
2 | 33
3 | 50
How do I use that using hibernate?
This is my attempt:
List<int> ids = ... // Retreived from elsewhere
Disjunction disjunction = Restrictions.disjunction();
for(int id: ids){
disjunction.add(Restrictions.eq("id", id)); // Specify which IDs
}
#SuppressWarnings("unchecked")
List<Item> items= (List<Item>) sessionFactory.getCurrentSession()
.createCriteria(Item.class)
.add(disjunction)
.setProjection(
Projections.projectionList()
.add(Projections.max("value"))
.add(Projections.groupProperty("id")
)
)
.setResultTransformer(Criteria.DISTINCT_ROOT_ENTITY)
.list();
This is just giving me the 'id' with the highest value (e.g. 3, not the entire row)
I am trying to do this in a spring mvc app.
Thanks in advance
select MAX(id),max(value) from ABCD where id in (110,56001) group by id
Try this Query to execute in the form to get the expected output.
You can specify a WHERE clause in your CriteriaQuery, and then do a multiselect to do the GROUP BY:
CriteriaBuilder cb = em.getCriteriaBuilder();
CriteriaQuery<Object[]> query = cb.createQuery(Object[].class);
Root<Item> item = query.from(Item.class);
query.where(cb.equal(item.get("id"), 2));
query.where(cb.equal(item.get("id"), 3));
query.multiselect(item.get("id"), item.max("value")).groupBy(item.get("id"));
List<Object[]> results = em.createQuery(query).getResultList();
System.out.println("id | value\n-----------");
for(Object[] object : results){
System.out.println(object[0] + " | " + object[1]);
}

How can I convert JSON to a database table dynamically?

I need to save JSON data to an Oracle database. The JSON looks like this(see below). But it doesn't stay in the same format. I might add some additional nodes or modify existing ones. So is it possible to create or modify oracle tables dynamically to add more columns? I was going to do that with Java. I will create a Java class matching the JSON, convert JSON to Java object and persist it to the table. But how can I modify Java class dynamically? Or would it be better idea to do that with PL/SQL? The JSON comes from a mobile device to a REST web service.
{"menu": {
"id": "file",
"value": "File",
"popup": {
"menuitem": [
{"value": "New", "onclick": "CreateNewDoc()"},
{"value": "Open", "onclick": "OpenDoc()"},
{"value": "Close", "onclick": "CloseDoc()"}
]
}
}}
I would suggest that you avoid creating new columns, and instead create a new table that will contain one entry for each of what would have been the new columns. I'm assuming here that the new columns would be menu items. So you would have a "menu" table with these columns:
id file
and you would have a "menuitem" table which would contain one entry for each of your menu items:
id value onclick
So instead of adding columns dynamically, you would be adding records.
I suggested in the comments to change your approach to a NoSQL database like MongoDB. However, if you still feel you need to use a relational database, maybe the EAV model can point you in the right direction.
In summay, you would have a "helper" table that stores which columns an Entity has and their types.
You cannot modify a Java class but you can define the class like a Map and implement the logic to add the desired columns.
Magento, a PHP product, uses EAV in its database.
Mongodb may be your best choice, or you could have a large TEXT field and only extract the columns you are likely to search one.
However, you can CREATE TABLE for additional normalised data and ALTER TABLE to add a column. The later can be particularity expensive.
Use https://github.com/zolekode/json-to-tables/.
Here you go:
import json
from core.extent_table import ExtentTable
from core.table_maker import TableMaker
menu = {
"id": "file",
"value": "File",
"popup": {
"menuitem": [
{"value": "New", "onclick": "CreateNewDoc()"},
{"value": "Open", "onclick": "OpenDoc()"},
{"value": "Close", "onclick": "CloseDoc()"}
]
}
}
menu = json.dumps(menu)
menu = json.loads(menu)
extent_table = ExtentTable()
table_maker = TableMaker(extent_table)
table_maker.convert_json_object_to_table(menu, "menu")
table_maker.show_tables(8)
table_maker.save_tables("menu", export_as="sql", sql_connection="your_connection")
Output:
SHOWING TABLES :D
menu
ID id value popup
0 0 file File 0
1 1 None None None
____________________________________________________
popup
ID
0 0
1 1
____________________________________________________
popup_?_menuitem
ID PARENT_ID is_scalar scalar
0 0 0 False None
1 1 0 False None
2 2 0 False None
____________________________________________________
popup_?_menuitem_$_onclick
ID value onclick PARENT_ID
0 0 New CreateNewDoc() 0
1 1 Open OpenDoc() 1
2 2 Close CloseDoc() 2
3 3 None None None
____________________________________________________
This can be done in MYSQL database:
This code takes a JSON input string and automatically generates
SQL Server CREATE TABLE statements to make it easier
to convert serialized data into a database schema.
It is not perfect, but should provide a decent starting point when starting
to work with new JSON files.
SET NOCOUNT ON;
DECLARE
#JsonData nvarchar(max) = '
{
"Id" : 1,
"IsActive":true,
"Ratio": 1.25,
"ActivityArray":[true,false,true],
"People" : ["Jim","Joan","John","Jeff"],
"Places" : [{"State":"Connecticut", "Capitol":"Hartford", "IsExpensive":true},{"State":"Ohio","Capitol":"Columbus","MajorCities":["Cleveland","Cincinnati"]}],
"Thing" : { "Type":"Foo", "Value" : "Bar" },
"Created_At":"2018-04-18T21:25:48Z"
}',
#RootTableName nvarchar(4000) = N'AppInstance',
#Schema nvarchar(128) = N'dbo',
#DefaultStringPadding smallint = 20;
DROP TABLE IF EXISTS ##parsedJson;
WITH jsonRoot AS (
SELECT
0 as parentLevel,
CONVERT(nvarchar(4000),NULL) COLLATE Latin1_General_BIN2 as parentTableName,
0 AS [level],
[type] ,
#RootTableName COLLATE Latin1_General_BIN2 AS TableName,
[key] COLLATE Latin1_General_BIN2 as ColumnName,
[value],
ROW_NUMBER() OVER (ORDER BY (SELECT 1)) AS ColumnSequence
FROM
OPENJSON(#JsonData, '$')
UNION ALL
SELECT
jsonRoot.[level] as parentLevel,
CONVERT(nvarchar(4000),jsonRoot.TableName) COLLATE Latin1_General_BIN2,
jsonRoot.[level]+1,
d.[type],
CASE WHEN jsonRoot.[type] IN (4,5) THEN CONVERT(nvarchar(4000),jsonRoot.ColumnName) ELSE jsonRoot.TableName END COLLATE Latin1_General_BIN2,
CASE WHEN jsonRoot.[type] IN (4) THEN jsonRoot.ColumnName ELSE d.[key] END,
d.[value],
ROW_NUMBER() OVER (ORDER BY (SELECT 1)) AS ColumnSequence
FROM
jsonRoot
CROSS APPLY OPENJSON(jsonRoot.[value], '$') d
WHERE
jsonRoot.[type] IN (4,5)
), IdRows AS (
SELECT
-2 as parentLevel,
null as parentTableName,
-1 as [level],
null as [type],
TableName as Tablename,
TableName+'Id' as columnName,
null as [value],
0 as columnsequence
FROM
(SELECT DISTINCT tablename FROM jsonRoot) j
), FKRows AS (
SELECT
DISTINCT -1 as parentLevel,
null as parentTableName,
-1 as [level],
null as [type],
TableName as Tablename,
parentTableName+'Id' as columnName,
null as [value],
0 as columnsequence
FROM
(SELECT DISTINCT tableName,parentTableName FROM jsonRoot) j
WHERE
parentTableName is not null
)
SELECT
*,
CASE [type]
WHEN 1 THEN
CASE WHEN TRY_CONVERT(datetime2, [value], 127) IS NULL THEN 'nvarchar' ELSE 'datetime2' END
WHEN 2 THEN
CASE WHEN TRY_CONVERT(int, [value]) IS NULL THEN 'float' ELSE 'int' END
WHEN 3 THEN
'bit'
END COLLATE Latin1_General_BIN2 AS DataType,
CASE [type]
WHEN 1 THEN
CASE WHEN TRY_CONVERT(datetime2, [value], 127) IS NULL THEN MAX(LEN([value])) OVER (PARTITION BY TableName, ColumnName) + #DefaultStringPadding ELSE NULL END
WHEN 2 THEN
NULL
WHEN 3 THEN
NULL
END AS DataTypePrecision
INTO ##parsedJson
FROM jsonRoot
WHERE
[type] in (1,2,3)
UNION ALL SELECT IdRows.parentLevel, IdRows.parentTableName, IdRows.[level], IdRows.[type], IdRows.TableName, IdRows.ColumnName, IdRows.[value], -10 AS ColumnSequence, 'int IDENTITY(1,1) PRIMARY KEY' as datatype, null as datatypeprecision FROM IdRows
UNION ALL SELECT FKRows.parentLevel, FKRows.parentTableName, FKRows.[level], FKRows.[type], FKRows.TableName, FKRows.ColumnName, FKRows.[value], -9 AS ColumnSequence, 'int' as datatype, null as datatypeprecision FROM FKRows
-- For debugging:
-- SELECT * FROM ##parsedJson ORDER BY ParentLevel, level, tablename, columnsequence
DECLARE #CreateStatements nvarchar(max);
SELECT
#CreateStatements = COALESCE(#CreateStatements + CHAR(13) + CHAR(13), '') +
'CREATE TABLE ' + #Schema + '.' + TableName + CHAR(13) + '(' + CHAR(13) +
STRING_AGG( ColumnName + ' ' + DataType + ISNULL('('+CAST(DataTypePrecision AS nvarchar(20))+')','') + CASE WHEN DataType like '%PRIMARY KEY%' THEN '' ELSE ' NULL' END, ','+CHAR(13)) WITHIN GROUP (ORDER BY ColumnSequence)
+ CHAR(13)+')'
FROM
(SELECT DISTINCT
j.TableName,
j.ColumnName,
MAX(j.ColumnSequence) AS ColumnSequence,
j.DataType,
j.DataTypePrecision,
j.[level]
FROM
##parsedJson j
CROSS APPLY (SELECT TOP 1 ParentTableName + 'Id' AS ColumnName FROM ##parsedJson p WHERE j.TableName = p.TableName ) p
GROUP BY
j.TableName, j.ColumnName,p.ColumnName, j.DataType, j.DataTypePrecision, j.[level]
) j
GROUP BY
TableName
PRINT #CreateStatements;
You can find the solution on https://bertwagner.com/posts/converting-json-to-sql-server-create-table-statements/
ALso JSON can be converted to a POJO class in JAVA :
package com.cooltrickshome;
2
import java.io.File;
3
import java.io.IOException;
4
import java.net.MalformedURLException;
5
import java.net.URL;
6
import org.jsonschema2pojo.DefaultGenerationConfig;
7
import org.jsonschema2pojo.GenerationConfig;
8
import org.jsonschema2pojo.Jackson2Annotator;
9
import org.jsonschema2pojo.SchemaGenerator;
10
import org.jsonschema2pojo.SchemaMapper;
11
import org.jsonschema2pojo.SchemaStore;
12
import org.jsonschema2pojo.SourceType;
13
import org.jsonschema2pojo.rules.RuleFactory;
14
import com.sun.codemodel.JCodeModel;
15
public class JsonToPojo {
16
/**
17
* #param args
18
*/
19
public static void main(String[] args) {
20
String packageName="com.cooltrickshome";
21
File inputJson= new File("."+File.separator+"input.json");
22
File outputPojoDirectory=new File("."+File.separator+"convertedPojo");
23
outputPojoDirectory.mkdirs();
24
try {
25
new JsonToPojo().convert2JSON(inputJson.toURI().toURL(), outputPojoDirectory, packageName, inputJson.getName().replace(".json", ""));
26
} catch (IOException e) {
27
// TODO Auto-generated catch block
28
System.out.println("Encountered issue while converting to pojo: "+e.getMessage());
29
e.printStackTrace();
30
}
31
}
32
public void convert2JSON(URL inputJson, File outputPojoDirectory, String packageName, String className) throws IOException{
33
JCodeModel codeModel = new JCodeModel();
34
URL source = inputJson;
35
GenerationConfig config = new DefaultGenerationConfig() {
36
#Override
37
public boolean isGenerateBuilders() { // set config option by overriding method
38
return true;
39
}
40
public SourceType getSourceType(){
41
return SourceType.JSON;
42
}
43
};
44
SchemaMapper mapper = new SchemaMapper(new RuleFactory(config, new Jackson2Annotator(config), new SchemaStore()), new SchemaGenerator());
45
mapper.generate(codeModel, className, packageName, source);
46
codeModel.build(outputPojoDirectory);
47
}
48
}

Categories