We Keep Coding

multiple columns in parent to use same child table to save data in JPA

I have a requirement where I have to save employee detail to a child table, employee data remains the same for multiple parent records (columns as well), I want to use the existing child table to insert if not present and update if already present, how I can do that using Spring JPA / hibernate?
Parent Table
|id|project_id|owner_emp_id|developer_emp_id|lead_emp_id|tester_emp_id|
|:---- |:------:| -----:|:---- |:------:| -----:|
| 1 | 100 | emp_10 | emp_20 | emp_20 | emp_30 |
| 2 | 200 | emp_11 | emp_21 | emp_22 | emp_30 |
employee child table
|emp_id|first_name|olast_name|email|phone|
|:---- |:------:| -----:|:---- |:------:|
| emp_10 |..| .. | .. | .. |
| emp_20 | .. | .. | .. | .. |
| emp_30 | .. | .. | .. | .. |
| emp_11 | .. | .. | .. | .. |
| emp_21 | .. | .. | .. | .. |
| emp_22 | .. | .. | .. | .. |
from the above example when saving the child table during 1st record emp_20 data need to save only once. similarly, the 2nd record insert emp_30 already presents an update or ignores saving it, how to do this in JPA?

MySQL - Selecting only the default value from column

I am wondering is it possible to select only the default value of empty column?
I have absolutely empty table and I want just to select one of the columns default value - it is important for my JAVA app which is filling the table.
Thanks.

You can get the default from the INFORMATION_SCHEMA.COLUMNS
select COLUMN_DEFAULT
from INFORMATION_SCHEMA.COLUMNS
where TABLE_SCHEMA='your_db' and TABLE_NAME='your_table' and COLUMN_NAME='your_column'

You can define a default value for a column when you create a table, if you just want MySQL to insert it automatically:
create table my_table (i INT DEFAULT 1);
But if you mean you want the default value which is stored in the DB dictionary, you can get it by this query:
SELECT Column_Default
FROM Information_Schema.Columns
WHERE Table_Schema = 'yourSchema'
AND Table_Name = 'yourTableName'
AND Column_Name = 'yourColumnName'

I can only think of two ways:
Inserting a row
Insert a row without specifying a value for that column
Select the column from that row; it will have the default value of the column
Delete the row
...probably all in a transaction so nothing else sees it.
Using describe (explain)
The describe command (aka explain) describes objects in the system, including tables. So if you do explain YourTable, you'll get back information about the table, including its default values.
Here's an example from that linked documentation:
mysql> DESCRIBE City;
+−−−−−−−−−−−−+−−−−−−−−−−+−−−−−−+−−−−−+−−−−−−−−−+−−−−−−−−−−−−−−−−+
| Field | Type | Null | Key | Default | Extra |
+−−−−−−−−−−−−+−−−−−−−−−−+−−−−−−+−−−−−+−−−−−−−−−+−−−−−−−−−−−−−−−−+
| Id | int(11) | NO | PRI | NULL | auto_increment |
| Name | char(35) | NO | | | |
| Country | char(3) | NO | UNI | | |
| District | char(20) | YES | MUL | | |
| Population | int(11) | NO | | 0 | |
+−−−−−−−−−−−−+−−−−−−−−−−+−−−−−−+−−−−−+−−−−−−−−−+−−−−−−−−−−−−−−−−+
So you can extract the default from the Default column in the returned rows.
Ah, of course, there's a third way, see slaakso's answer for it.

SQOOP Export failure

I am trying to export a table from HDFS to SQOOP but I am getting java exceptions.
The query I'm using is as follows:
sqoop export --connect jdbc:mysql://172.31.54.174/Database --driver com.mysql.jdbc.Driver --username user --password userpassword --table accounts --export-dir /user/pri/accounts
While execution this query gives me below error:
17/03/29 07:54:26 INFO mapreduce.Job: map 0% reduce 0%
17/03/29 07:54:30 INFO mapreduce.Job: Task Id : attempt_1489328678238_4886_m_000002_0, Status : FAILED
Error: java.io.IOException: Can't export data, please check failed map task logs
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.RuntimeException: Can't parse input data: '\N'
at accounts.__loadFromFields(accounts.java:691)
at accounts.parse(accounts.java:584)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83)
... 10 more
Caused by: java.lang.IllegalArgumentException: Timestamp format must be yyyy-mm-dd hh:mm:ss[.fffffffff]
at java.sql.Timestamp.valueOf(Timestamp.java:204)
at accounts.__loadFromFields(accounts.java:643)
... 12 more
The file that I am exporting contains data as below :
1,2008-10-23 16:05:05.0,\N,Donald,Becton,2275 Washburn Street,Oakland,CA,94660,5100032418,2014-03-18 13:29:47.0,2014-03-18 13:29:47.0
2,2008-11-12 03:00:01.0,\N,Donna,Jones,3885 Elliott Street,San Francisco,CA,94171,4150835799,2014-03-18 13:29:47.0,2014-03-18 13:29:47.0
I have also created the table accounts and its structure is as follows:
+----------------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------------+-------------+------+-----+---------+-------+
| acct_num | varchar(20) | NO | PRI | | |
| acct_create_dt | datetime | NO | | NULL | |
| acc_close_dt | datetime | YES | | NULL | |
| first_name | varchar(20) | NO | | NULL | |
| last_name | varchar(20) | NO | | NULL | |
| address | varchar(30) | NO | | NULL | |
| city | varchar(20) | NO | | NULL | |
| state | varchar(20) | NO | | NULL | |
| zipcode | varchar(20) | NO | | NULL | |
| phone_number | varchar(20) | YES | | NULL | |
| created | datetime | NO | | NULL | |
| modified | datetime | NO | | NULL | |
+----------------+-------------+------+-----+---------+-------+
I am also attaching a screenshot of the error.

As you can see from your logs '\N' is like escape character so it isnot fitting in varchar.I dont understand why you are adding the same. Also the timestamp format issues is indicated.And also check in your existing data if any column you are using for primary key is repeated itself.

Add --input-null-string '\\N' --input-null-non-string '\\N' in your sqoop export command.

How to Get DB Field name from ConstraintViolationException - Hibernamte

How to get the Name of the DB field causing ConstraintViolationException while inserting in to Database in hibernate.
I have the Table Like
mysql> desc Mytable;
+-------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+-------------+------+-----+---------+----------------+
| id | bigint(11) | NO | PRI | NULL | auto_increment |
| name | varchar(20) | YES | UNI | NULL | |
| city | varchar(20) | YES | UNI | NULL | |
+-------+-------------+------+-----+---------+----------------+
And records inthe table are
mysql> select * from Mytable;
+----+--------+-------+
| id | name | city |
+----+--------+-------+
| 1 | SATISH | BLORE |
+----+--------+-------+
1 row in set (0.00 sec)
Now, im trying to insert
"RAMESH","BLORE" through hibernate.
It is throwing ConstraintViolationException due to "BLORE" (CITY) already Exist.
if im trying to insert.
"SATISH","MLORE" through hibernate
It is throwing ConstraintViolationException due to "SATISH" (NAME) already Exist.
My question is
how to get fieldName who is causing the exception ConstraintViolationException through Hibernate.

Since there might be other constraints that could be violated (e.g. combined keys), you only have the name of the constraint that is violated, which in your case might just be the column name (however, I'm not entirely sure about that). You can get the name of the violated constraint by calling getConstraintName() on the ConstraintViolationException.

Time series database for java?

I want to store millions of time series entries (long time, double value) with Java. (Our monitoring system is currently storing every entry in a large mySQL table but performance is very bad.)
Are there time series databases implemented in java out there?

checkout http://opentsdb.net/ as used by StumbleUpon?
checkout http://square.github.com/cube/ as used by square
I hope to see additional suggestions in this thread.

The performance was bad because of wrong database design. I am using mysql and the table had this layout:
+-------------+--------------------------------------+------+-----+-------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+--------------------------------------+------+-----+-------------------+-----------------------------+
| fk_category | smallint(6) | NO | PRI | NULL | |
| method | enum('min','max','avg','sum','none') | NO | PRI | none | |
| time | timestamp | NO | PRI | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
| value | float | NO | | NULL | |
| accuracy | tinyint(1) | NO | | 0 | |
+-------------+--------------------------------------+------+-----+-------------------+-----------------------------+
My fault was an inapproriate index. After adding a multi column primary key all my queries are lightning fast:
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| job | 0 | PRIMARY | 1 | fk_category | A | 18 | NULL | NULL | | BTREE | | |
| job | 0 | PRIMARY | 2 | method | A | 18 | NULL | NULL | | BTREE | | |
| job | 0 | PRIMARY | 3 | time | A | 452509710 | NULL | NULL | | BTREE | | |
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
Thanks for all you answers!

You can take a look at KDB. It's primarily used by financial companies to fetch market time series data.

What do you need to do with the data and when?
If you are just saving the values for later, a plain text file might do nicely, and then later upload to a database.