I had one csv file of 1 cr lines of data. From this file I have to read data from csv and using first field I need do check conditions with in my db and from that Db took one key and append all prevoius data and write to another csv. Which ever I read from CSV I written code for this but it takes days of time to read and write for one 2 lakhs line of data. Here I am using single thread to do this all.
Sample code is I followed below steps:
1).reading data from CSF.
2).Read first field from csv and checking condition(in this i am
checking 5 conditions).
3).And written into CSV.
In my opinion, reading the CSV into the database and then writing a statement to filter the data would be more efficient than reading the file line by line using the code.
You may refer the links below to know more about csv to database import if you use MySQL:
http://dev.mysql.com/doc/refman/5.0/en/mysqlimport.html
http://support.modwest.com/content/6/253/en/how-do-i-import-delimited-data-into-mysql.html
Related
I have wrote a java code to parse xml file using jaxb and then insert the data from xml file to my sql. The code does insertion to different 20 tables based on the xml tags.
My problem is that my code is running for last 3 days and it could insert 5 millions record to my sql data base!
In this way I need to wait a month to finish the job..
Can you give me some hint how can I make my code efficient?
MySQL supports 'LOAD DATA' command and CSV format. So converting your XMLs into prepared CSV and then load by mysql abilities will be fastest solution:
How to import CSV file to MySQL table
I have a word2vec model stored in text file as
also -0.036738 -0.062687 -0.104392 -0.178325 0.010501 0.049380....
one -0.089568 -0.191083 0.038558 0.156755 -0.037399 -0.013798....
The size of the text file is more than 8GB.
I want to read this file into mysql database using the first word as key (in a column) and the rest of the line as another column. Is it possible to do so without reading each line and splitting it?
I went through some related questions but it didn't match what I want.
How to read a file and add its content to database?
read text file content and insert it into a mysql database
you can do it by:
making a simple for loop that iterates over the records on the model
aggregating about 100 records on an array
using mysql bulk insert feature to insert 100s of records at once
use a fast language like go if you can.
This thing you are trying to do, it's very possible, let me know if you need code for this.
I am trying to read data from a csv file has 332,462 KB with 136 columns and 297,388 rows. Then I want to insert into an Oracle database table which has the exactly same column number mapping, except I add one more column at the end of this table to record today's date.
So, everything looks fine, no exceptions, the only thing is I can only read a small part like 7619 row, and the program stops. The finish part in the database is what I want, that is correct, but I don't know why it stops, I tried use readNext(), readAll(), and pass an inputStreamRead to CSVReader, all of these way have the same result.
What is the cause of this? One thing I am think is this csv file has some empty row that the CSVReader read it as the end of the file?
Is there a way to perform SQL queries on CSV text held in memory and read in from a Java Reader e.g. StringReader.
org.h2.tools.Csv.read(Reader reader, String[] colNames) would allow me to retrieve a result set containing all the rows and columns. However, I actually want to perform a query on the CSV text read from the Reader.
The background - I receive a file containing multiple CSV sections for each entity, see Can H2 Database query a CSV file containing multiple sections of different record groups?, and whilst parsing the file I store each of the CSV sections I need in Strings (a String for each one). This shouldn't bog down memory as I only keep the data in memory for a short time and each CSV section is relatively small). I need to perform queries on these CSV sections to build a document in a custom format.
I could write each CSV section to a file (as a set of files) and use CSVREAD, but I don't want to do that as I need my application to be as fast as possible and splitting and writing the sections to disk will thrash the hard drive to death.
You could write a user defined function that returns a result set, and use that to generate the required rows. Within your user defined function, you can use the Csv tool from H2 (actually any Csv tool).
This is not possible directly, since DBMS can usually only query their own optimized data storage. You have to import the text with the mentioned org.h2.tools.Csv.read into a table and perform the queries on that table. The table may be a temporary one, to prevent any writes on the disk, assuming the memory is sufficient.
How do I write and read data in txt file in the form of rows and columns? Also, how do I store many files like this in a folder for a particular name? The idea is just like a Table for a particular person who has many rows and columns.
You'll want to look into comma-separated value files.
In addition to Jeremy's post
Use opencsv to create the csv file
Use CsvJdbc to use JDBC to query against those files. This makes the future transition to DB in future and you will be working with a known API for searching against the file