We have requirement to read the huge excel file in java. I am prefering Spring Batch for same ,as we have spring already in project. However, we have one bottleneck that this excel file can have different columns, user can change the order of columns . So, we have to identify which column has what data from first row / header.
Example :-
User 1 :
Name EmployeeId
Raj 1
Peter 2
User 2 :-
EmployeeId Name
5 Steven
6 Antony
But in spring Batch ,we need to tell column order while configuring job.
One way is that I will open file and read first row and configure Spring batch but that is not efficient.
There should be some inbuilt way, however I am not able to find it.
The Spring Batch Excel extension : https://github.com/mdeinum/spring-batch-extensions/tree/master/spring-batch-excel has a RowNumberColumnNameExtractor which fits your needs
i.e. it reads the row 0 as a column row and can then be used in conjunction with a BeanWrapperRowMapper to map to a java object
Related
Im developing a Spring Batch application, technology which by the way I'm new to.
I have made some tutoriais, and read some docs in order to prepare myself to this development.
Im already "confortable" with some of the most common APIs (ItemRead, ItemProcessor, ItemWriter, Setps, Tasklets, Jobs, Parameters...)
My requirement is simple.
1 - Read some data from CSV file.
2 - Fetch an Entity from database by each line of the CSV file.
3 - Update the state of the Entity.
4 - Export a new CSV file with some generated data from each Entity.
My problem is not how to fetch, how to update or how to export a csv file, but more conceptualy how to setup my JOB.
The way I see it I like to end up with a Job something like
1 - ItemRead -> to read the whole csv file.
2 - ItemProcessor -> to update the entity.
3 - ItemWriter -> to persist the entity.
4 - ItemWriter -> to export the new CSV file based on the entity state.
Does it make sense? There's a better way. Am I missisng some pitfalls?
updating line by line might not be best idea. Instead I suggest
Job 1
Read entire file
Write it to the tmp table
Here write update query that join original table and tmp table based on primary key.
After query execution, call second job
Job 2
Read records from table
Write it to the file
Finally clear tmp table for next Job sequence.
This Ans is based on my thought process. there might be other best approaches as well.
I need to bind a group of csv file in the format "YYYY-MM-DD hh:mm:ss.csv" that are present in the same folder with a unique table that contains all the data present in all the files.
I need to read the data from a Java EE application thus I would like to create a connection pool inside the application server. I found the CsvJdbc driver that allows the reading of multiple files as a single entity. A good starting point was this page in the section with this paragraph:
To read several files (for example, daily log files) as a single table, set the database connection property indexedFiles. The following example demonstrates how to do this.
The example could be fine for me but the problem is that I do not have a header word in the filename string. So the corresponding table becames an empty string that makes obviously impossible to query the table.
How can I tell the driver to map the pattern to a table that hasn't a header part?
P.S. I already tried to use hsqldb as a frontend to the csv files but it does not support multiple files.
Setup CsvJdbc to read several files as described in http://csvjdbc.sourceforge.net/doc.html and then use an empty table name in the SQL query because your CSV filenames do not have any header before the fileTailPattern regular expression. For example:
props.put("fileTailPattern", "(\\d+)-(\\d+)-(\\d+) (\\d+):(\\d+):(\\d+)");
props.put("fileTailParts", "Year,Month,Day,Hour,Minutes,Seconds");
...
ResultSet results = stmt.executeQuery("SELECT * FROM \"\" AS T1");
I have a csv file like
day,cost
20140101, 20
2014-01-5, 20
20140101, ab
so there are some invalid data and I want to load the valid data into table_normal, and invalid data into table_unnormal
so the final data should be
for table of table_normal
day,cost
20140101, 20
for table of table_unnormal
day,cost, reason
2014-01-5, 20, 'invalid day'
20140101, ab,'invalid cost'
I know how to get the reason in processor, but how could be job write to different tables?
I could suggest 3 ways to do this, none of which is very direct and easy.
a) Write you own CustomJDBCItemWriter - you can filter out any ways you want and you should be able to write some record in table table_normal and some record in table_unnormal
b) Use a CompositeItemWriter - both writers will get the "full record list" from the processor. You can then filter out the record needed in each writer. Very similar to (a)
c) If you can do 2 passes over the input - you can write you job in two steps.
Step 1 : Read Records --> Process only bad records --> write to table_unnormal
Step 2 : Read Records --> Process only good records --> write to table_normal
There isn't a good in-build feature to handle this scenario in spring batch directly (at-least none I am aware off)
Iam new to java i have a requirement that all the user logged in details are stored in database.The table looks like this
empId timestamp empDeparment empName
1 xxxx java xxxx
2 xxxx testing xxxxx
I need to get all these list of records and should store in text file based on some conditions.
1.I need to create new folder(directory) for each department for example "java_currentdate"
2.I need to create a text file the text file look like this
logindetails.txt
1timestamp3(no of employees in java department are 3 so 3 times body repeats) -----------Header
2empidtimestampempName
2 " -----------body
2 "
3 -----------------footer
All the employees should be sorted based on id and timestamp
Please help me explain to work on this.
I am using hibernate Orm tool and spring
Your question is too broad.
There are several resources that could be useful:
Spring Getting Starter guide: Accessing Relational Data using JDBC with Spring
Spring Data sample application tutorial: Spring Data Access Tutorial
Spring Data doc: Spring Data current
JPA Specification
A lot of examples on the web.
Now, it doesn't matter if you have to write/read on file or DB, as newbie, first of all you have to understand how Spring works to perform data access.
Hi I am creating table using schema file and loading table from data file through jdbc. I am doing batch upload using PreparedStatement and executeBatch. Data file contents look like the following structure:
key time rowid stream
X 11:40 1 A
Y 3:30 2 B
Now I am able to load successfully table in database. But I would like to test/verify that same table loaded into database against this same data file. how do I do it? How do compare table in database with data file? I am new to JDBC. Please guide. Thanks in advance.
Like Loki said, you can use a tool like DBUnit. Another option is to make a rudimentary integration test whereby your test generates a dump file of your table and compares this dump with the original "good" file.
You need DBunit . Check more details here : http://dbunit.sourceforge.net/howto.html
DB unit helps you to write test cases against data from database.