We Keep Coding

Remove duplicate records from child table and update parent with newly inserted child records reference id for billions of records

Employee has multiple versions of addresses and given addresses are being duplicate needs to be removed and replaced with single unique address row and update parent table with newly inserted address id reference in optimal way .
CREATE TABLE address (adr_id, ver_id, address) AS
SELECT 1, 1, 'newYork' FROM DUAL UNION ALL
SELECT 1, 2, 'newYork' FROM DUAL UNION ALL
SELECT 1, 3, 'newYork' FROM DUAL UNION ALL
SELECT 4, 1, 'Washington' FROM DUAL UNION ALL
SELECT 4, 2, 'Washington' FROM DUAL;
CREATE TABLE employee (emp_id,adr_id,ver_id,) AS
SELECT 100,1, 1 FROM DUAL UNION ALL
SELECT 200,1, 2 FROM DUAL UNION ALL
SELECT 300,1, 3 FROM DUAL UNION ALL
SELECT 400,4, 1 FROM DUAL UNION ALL
SELECT 500,4, 2 FROM DUAL;
Before
employee
id | adr_id | ver_id
100 1 1
200 1 2
300 1 3
400 4 1
500 4 2
Address
adr_id | ver_id | address
1 1 newYork
1 2 newYork
1 3 newYork
4 1 Washington
4 2 Washington
Expected After
Address
adr_id | ver_id | address
11 0 newYork
12 0 Washington
employee
id | adr_id | ver_id
100 11 1
200 11 2
300 11 3
400 12 1
500 12 2
Note: There is foreign key constraint between Employee and Address table with composite key comprising of adr_id & ver_id
This is being done using stored procedure but its too slow is there any parallel processing can be done here by some way?

Oracle order by via JPA with numeric first

I have column data like this in my db
data
-----
1
2
A
3
4
B
I have below nls settings in place
SELECT * From NLS_SESSION_PARAMETERS;
NLS_LANGUAGE AMERICAN
NLS_TERRITORY AMERICA
NLS_CURRENCY $
NLS_ISO_CURRENCY AMERICA
NLS_ISO_CURRENCY AMERICA
NLS_NUMERIC_CHARACTERS .,
NLS_CALENDAR GREGORIAN
NLS_DATE_FORMAT DD-MON-RR
NLS_DATE_LANGUAGE AMERICAN
NLS_SORT BINARY
NLS_COMP BINARY
When I query the db for **select data from <mytable> order by data ** I am getting the result as below
data
-----
1
2
3
4
A
B
The same query via hibernate is giving the results with alphabets first and numeric later
data
-----
A
B
1
2
3
4
But, I want column to be displayed with numbers first and alphabets later via hibernate
data
------
1
2
3
4
A
B
Can someone help me with this.

You can use regularexpressions to sort in such a way,
select data from <mytable> order by REGEXP_REPLACE(data ,'[^0-9]'), REGEXP_REPLACE(data ,'[0-9]')

That depends on NLS_SORT parameter's value. For example, in my database it is set to CROATIAN and the result is what you wanted:
SQL> select * From nls_session_parameters where parameter = 'NLS_SORT';
PARAMETER VALUE
-------------------- --------------------
NLS_SORT CROATIAN
SQL> with test (data) as
2 (select '1' from dual union all
3 select '2' from dual union all
4 select 'A' from dual union all
5 select '3' from dual union all
6 select '4' from dual union all
7 select 'B' from dual
8 )
9 select data
10 from test
11 order by data;
D
-
A
B
1
2
3
4
6 rows selected.
However, in your database, NLS_SORT = BINARY. Let's try it:
SQL> alter session set nls_sort = 'BINARY';
Session altered.
SQL> with test (data) as
2 (select '1' from dual union all
3 select '2' from dual union all
4 select 'A' from dual union all
5 select '3' from dual union all
6 select '4' from dual union all
7 select 'B' from dual
8 )
9 select data
10 from test
11 order by data;
D
-
1
2
3
4
A
B
6 rows selected.
SQL>
Right; the wrong result.
Therefore, modify NLS_SORT, if that's an option. See valid values by
SQL> select * From v$nls_valid_values where parameter = 'SORT';
PARAMETER VALUE ISDEP
-------------------- -------------------- -----
SORT BINARY FALSE
SORT WEST_EUROPEAN FALSE
SORT XWEST_EUROPEAN FALSE
SORT GERMAN FALSE
<snip>

I believe one of the two options below should work,
Use NLSSORT function in the order by clause explicitly.
select * from my_table order by NLSSORT(data,'NLS_SORT=BINARY');
Create a function based index on the column,
create index mytable_nlssort_index on my_table(nlssort(data, 'nls_sort=''BINARY'''))
Use the query as below when the column is indexed.
select data from my_table order by data;

Java - Hibernate : Generate sequence column based on another field

Need to add a field to the database which will record a sequence number related to that (foreign) id.
Example table data (current):
ID ACCOUNT some_other_stuff
1 1 ...
2 1 ...
3 1 ...
4 2 ...
5 2 ...
6 1 ...
I need to add a sequenceid column which increments separately for each account, achieving:
ID ACCOUNT SEQ some_other_stuff
1 1 1 ...
2 1 2 ...
3 1 3 ...
4 2 1 ...
5 2 2 ...
6 1 4 ...
Note that the sequence is related to account.

Unfortunately this cannot be done with JPA and hibernate. The only solution would be to do it manually in the service. You can use #Generated value on a column but that relies on the database to provide the value. And you cannot create a custom sequence implementation and use #GeneratedValue because that works only for the ID column.

FOREACH in cypher - neo4j

I am very new to CYPHER QUERY LANGUAGE AND i am working on relationships between nodes.
I have a CSV file of table containing multiple columns and 1000 rows.
Template of my table is :
cdrType ANUMBER BNUMBER DUARTION
2 123 456 10
2 890 456 5
2 123 666 2
2 123 709 7
2 345 789 20
I have used these commands to create nodes and property keys.
LOAD CSV WITH HEADERS FROM "file:///2.csv" AS ROW
CREATE (:ANUMBER {aNumber:ROW.aNumber} ),
CREATE (:BNUMBER {bNumber:ROW.bNumber} )
Now I need to create relation between all rows in the table and I think FOREACH loop is best in my case. I created this query but it gives me an error. Query is :
MATCH (a:ANUMBER),(b:BNUMBER)
FOREACH(i in RANGE(0, length(ANUMBER)) |
CREATE UNIQUE (ANUMBER[i])-[s:CALLED]->(BNUMBER[i]))
and the error is :
Invalid input '[': expected an identifier character, whitespace,
NodeLabel, a property map, ')' or a relationship pattern (line 3,
column 29 (offset: 100)) " CREATE UNIQUE
(a:ANUMBER[i])-[s:CALLED]->(b:BNUMBER[i]))"
I need relation for every row. like in my case. 123 - called -> 456 , 890 - called -> 456. So I need visual representation of this calling data that which number called which one. For this I need to create relation between all rows.
any one have idea how to solve this ?

What about :
LOAD CSV WITH HEADERS FROM "file:///2.csv" AS ROW
CREATE (a:ANUMBER {aNumber:ROW.aNumber} )
CREATE (b:BNUMBER {bNumber:ROW.bNumber} )
MERGE (a)-[:CALLED]->(b);
It's not more complex than that i.m.o.
Hope this helps !
Regards,
Tom

Filter and Group multiple DataSets in spark java

I am very new to spark.The below is the requirement am getting to
1st RDD
empno first-name last-name
0 fname lname
1 fname1 lname1
2nd rdd
empno dept-no dept-code
0 1 a
0 1 b
1 1 a
1 2 a
3rd rdd
empno history-no address
0 1 xyz
0 2 abc
1 1 123
1 2 456
1 3 a12
I have to generate a file combining all the RDDs for each employee, and the average emp-count is 200k
Desired output:
seg-start emp-0
seg-emp 0-fname-lname
seg-dept 0-1-a
seg-dept 0-1-b
seg-his 0-1-xyz
seg-his 0-2-abc
seg-end emp-0
seg-start emp-1
......
seg-end emp-1
How can I achieve this by combining RDDs? Please note that the data is not written straight forward as it was shown here, we are converting data to business valid format(ex:- e0xx5fname5lname is 0-fname-lname), so need help from the experts here, as the current batch program runs for hours to write data, thinking of using spark to process this efficiently.