i have a scenario where i have a file of the form,
id,class,type
1,234,gg
2,235,kk
3,236,hth
4,237,rgg
5,238,rgr
I also have a table in my database of the form PROPS,
id,class,property
1,7735,abc
2,3454,efg
3,235,hij
4,238,klm
5,24343,xyx
Now the first file and the db table are joined based on class so that final output will be of the form:
id,class,type,property
1,235,kk,hij
2,238,rgr,klm
Now, i can search the db table for each class record of the first file and so forth.
But this will take too much time.
Is there any way to do this same thing through a MySQL STORED PROCEDURE?
My question is whether there is a way to read the first file content line by line(WITHOUT MAKING USE OF A TEMPORARY TABLE), check the class with the class in the db table and insert the result into an output file and return the output file using MYSQL STORED PROCEDURE?
Related
I have two huge tsv files(10mil records) where tsv one file has the attributes id, name, age and other had attributes id, email and phno.
I tried to read the first file and insert the records into the Person table and then read the second file and update the Person table. This approach takes time as the table is first inserted with 10 mil records and then they are updated. IS there any other way to speed this process?
P.S some Id are not there in the 2nd tsv file so I was not able to merge both of them .
Why you don't try LOAD DATA INFILE which is a highly optimized, MySQL-specific statement that directly inserts data into a table from a CSV / TSV file.
There are two ways to use LOAD DATA INFILE. You can copy the data file to the server's data directory (typically /var/lib/mysql-files/) and run:
LOAD DATA INFILE '/path/to/products.csv' INTO TABLE products;
Or you can also store the data file on the client side, and use the LOCAL keyword:
LOAD DATA INFILE '/path/to/products.csv' INTO TABLE products;
High-speed inserts with MySQL
You should also check MySql Documentation - LOAD DATA Statement
And you could use a statement like this one:
LOAD DATA INFILE 'data.txt'
INTO TABLE tbl_name
FIELDS TERMINATED BY ',' ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES;
I am not an advanced Java developer.
I am working on a project to insert records into tables in a SQL database by reading the data in a CSV file. The Employee CSV file has several columns containing data. There is an XML file which contains the mapping information, that is, the XML file says which column in the CSV file contains what information.
I have been successful in reading the CSV file with the mapping in the XML file. I have also been successful in inserting the data in the CSV file into database tables. But there is a catch. The CSV file is a file containing the all the historical records of employees in chronological order (oldest record first). In the scenario where there are multiple records for an employee, his/her last record in the file contains his/her current information and needs to be inserted into the Employee table. All of his/her older records need to be inserted into the Employee_History table in the same order they appear in the CSV file. Column 0 of the CSV file contains the Employee ID. The following is to give an idea of what the CSV file looks like
Emp_ID|First Name|Last Name|Email|Update Date
123|John|Smith|john.smith01#email.com|01/01/2020
234|Bruce|Waye|bruce.wayne#wayneenterprises.com|02/02/2020
123|John|Smith|john.smith02#email.com|02/15/2020
345|Clark|Kent|clark.kent#dailyplanet.com|02/16/2020
123|John|Smith|john.smith03#email.com|02/20/2020 -- **Last record in the CSV file for Emp ID = 123**
Can anyone please tell me the best way to approach this? I am struggling to come up with a way to identify a given custodian's last record in the CSV file.
I need to bind a group of csv file in the format "YYYY-MM-DD hh:mm:ss.csv" that are present in the same folder with a unique table that contains all the data present in all the files.
I need to read the data from a Java EE application thus I would like to create a connection pool inside the application server. I found the CsvJdbc driver that allows the reading of multiple files as a single entity. A good starting point was this page in the section with this paragraph:
To read several files (for example, daily log files) as a single table, set the database connection property indexedFiles. The following example demonstrates how to do this.
The example could be fine for me but the problem is that I do not have a header word in the filename string. So the corresponding table becames an empty string that makes obviously impossible to query the table.
How can I tell the driver to map the pattern to a table that hasn't a header part?
P.S. I already tried to use hsqldb as a frontend to the csv files but it does not support multiple files.
Setup CsvJdbc to read several files as described in http://csvjdbc.sourceforge.net/doc.html and then use an empty table name in the SQL query because your CSV filenames do not have any header before the fileTailPattern regular expression. For example:
props.put("fileTailPattern", "(\\d+)-(\\d+)-(\\d+) (\\d+):(\\d+):(\\d+)");
props.put("fileTailParts", "Year,Month,Day,Hour,Minutes,Seconds");
...
ResultSet results = stmt.executeQuery("SELECT * FROM \"\" AS T1");
Hi I am creating table using schema file and loading table from data file through jdbc. I am doing batch upload using PreparedStatement and executeBatch. Data file contents look like the following structure:
key time rowid stream
X 11:40 1 A
Y 3:30 2 B
Now I am able to load successfully table in database. But I would like to test/verify that same table loaded into database against this same data file. how do I do it? How do compare table in database with data file? I am new to JDBC. Please guide. Thanks in advance.
Like Loki said, you can use a tool like DBUnit. Another option is to make a rudimentary integration test whereby your test generates a dump file of your table and compares this dump with the original "good" file.
You need DBunit . Check more details here : http://dbunit.sourceforge.net/howto.html
DB unit helps you to write test cases against data from database.
I'm trying to create a function in my java application, where the user could select a prior made backup but only import table-rows that aren't in the current database instance. With a MySql database I could dump my tables, rename them inside the .sql to create temporary tables when imported again, and then simply cross query all rows not in the DB. Any idea how I could acomplish something similar in hsqldb from within my java application?
You can do this:
open the backup database
create a text table that is a copy of the main table, e.g. CREATE TEXT TABLE yourtable_copy AS (SELECT * FROM yourtable)
set a file for the table SET TABLE yourtable_copy SOURCE 'filepath'
copy the data to the new table
set the source off with SET TABLE yourtable_copy SOURCE OFF
shutdown the backup database
open the main database
now do the same text table creation and source setting with the main database but do not copy the data, as the backup data is already there and will be opend
do your updates then turn the text source off in the main database
reference http://www.hsqldb.org/doc/2.0/guide/texttables-chapt.html