Currently I am working on a solution to read a 454 character/line based huge file (minimum 50000 rows) via Java.As per the requirement it is a positioned based file, we first need to read the file , then parse the position based values and need to insert into a table. (minimum 96 positions will be inserted into 96 columns of the table).
I took this concept after the parsing.
[ INSERT ALL INTO<TABLE NAME> [COL1,COL2,COL3] Values [VAL1,VAL2,VAL3]
INTO<TABLE NAME> [COL1,COL2,COL3] Values [VAL1,VAL2,VAL3]
SELECT * FROM DUAL;]
Here is my code:
try{
char[] line = new char[456];
while(br.read(line) > 0){
StringBuilder input = new StringBuilder(new String(line));
if(batchCounter>0){
int detailFileId = interfaceFileSequence();
sql.append(initSql+"(" +
detailFileId + "," + interfaceHeaderId + ", SYSDATE," +
interfaceRunId + "," + isSpace(input.substring(0, 2).trim()) + "," + "TO_DATE("+isSpace(input.substring(2, 12).trim())+",'YYYY-MM-DD')" +","+isSpace(input.substring(12, 22).trim()) + "," +
Double.parseDouble(input.substring(22, 35)+ "." + input.substring(35, 37)) + ", " +
Double.parseDouble(input.substring(22, 35)+ "." + input.substring(35, 37)) + ", " +
isSpace(input.substring(38, 44).trim()) + ","+isSpace(input.substring(38, 44).trim())+"," +isSpace(input.substring(38, 44).trim())+"," + isSpace(input.substring(44, 54).trim())+","+
isSpace(input.substring(54, 60).trim()) + "," + isSpace(input.substring(60, 68).trim()) + "," + isSpace(input.substring(68, 83).trim()) + "," +
isSpace(input.substring(83, 89).trim()) + "," + isSpace(input.substring(89, 94).trim()) + "," +
isSpace(input.substring(94, 102).trim()) + "," +
isSpace(input.substring(102, 103).trim()) + ","+"TO_DATE("+isSpace(input.substring(103,113).trim())+",'YYYY-MM-DD')"+"," +isSpace(input.substring(113, 125).trim()) + "," + isSpace(input.substring(125, 128).trim()) + "," +
isSpace(input.substring(131, 133).trim()) + "," + isSpace(input.substring(133, 135).trim()) + "," + isSpace(input.substring(135, 136).trim()) + "," +
isSpace(input.substring(136, 137).trim()) + "," + isSpace(input.substring(137, 142).trim()) + "," + isSpace(input.substring(142, 147).trim()) + "," +
isSpace(input.substring(147, 148).trim()) + "," + isSpace(input.substring(149, 159).trim()) + "," +
isSpace(input.substring(159, 160).trim()) + "," + isSpace(input.substring(160, 175).trim()) + "," + isSpace(input.substring(160, 175).trim()) + "," +
isSpace(input.substring(190, 220).trim()) +"," +"TO_DATE("+isSpace(input.substring(216, 220)+"-"+input.substring(220, 222)+"-"+input.substring(222, 224))+",'YYYY-MM-DD')"+","+
"TO_DATE("+isSpace(input.substring(216, 220)+"-"+input.substring(220, 222)+"-"+input.substring(222, 224))+",'YYYY-MM-DD')"+","+ isSpace(input.substring(226,227).trim()) + "," + isSpace(input.substring(231,236).trim()) + "," +
isSpace(input.substring(242, 245).trim()) + "," + isSpace(input.substring(245,275).trim()) + "," + isSpace(input.substring(275,280).trim()) + "," +
isSpace(input.substring(280, 290).trim()) + "," + isSpace(input.substring(290,293).trim()) + "," + isSpace(input.substring(293,303).trim()) + "," +
isSpace(input.substring(303, 314).trim()) + "," +
isSpace(input.substring(313,316).trim()) + //need check
"," + isSpace(input.substring(317,337).trim()) + "," +
isSpace(input.substring(337, 422).trim()) +
"," + isSpace(input.substring(422,433).trim()) + "," + isSpace(input.substring(433,443).trim())+","+isSpace(input.substring(22, 39).trim())+
")");
sql.append('\n');
}
/*if (batchCounter % 500 == 0) {
System.out.println("sql:::::::::::"+sql);
int executeUpdate = em.createNativeQuery(sql.toString()).executeUpdate();
System.out.println("executeUpdateexecu:::"+executeUpdate);
em.flush();
insertionCounter += executeUpdate;
System.out.println("insertionCounter::::"+insertionCounter);
sql.setLength(0);
System.out.println("SQL");
System.out.println("initSql"+initSql);
sql.append(initSql);
}*/
batchCounter++;
}
sql.append(" SELECT * FROM dual");
int executeUpdate = em.createNativeQuery(sql.toString()).executeUpdate();
em.flush();
insertionCounter += executeUpdate;
System.out.println("Check Rows in file::"+(batchCounter-1)+" Insertion counter::"+insertionCounter);
if((batchCounter-1)==insertionCounter){
detailFileObj = new DetailFileObject(FileName, "DETAIL", (batchCounter-1), "SUCCESS");
}
else {
detailFileObj = new DetailFileObject(FileName, "DETAIL", (batchCounter-1), "FAILED");
}
} catch (IOException e) {
e.printStackTrace();
}
But once I am running the code, if there are 40-50 lines, it is working fine.But it is more than that I am getting exception. Can anyone share me proper approach along with running code, so that I can use it.
Thanks
First, DO NOT use string concatenation to build a SQL statement with text values from outside. It leaves you susceptible to SQL Injection attacks, and may cause SQL syntax errors. Use a PreparedStatement with ? parameter markers instead.
Second, if you're inserting a lot of records, use JDBC batching.
Here is an example of how you would use it:
String sql = "INSERT INTO MyTable" +
" ( Col1, Col2, Col3, Col4, Col5, Col6, Col7 )" +
" VALUES (?,?,SYSDATE,?,?,?,?)";
try (PreparedStatement stmt = conn.prepareStatement(sql)) {
char[] line = new char[456];
int batchSize = 0;
while (br.read(line) > 0) {
String input = new String(line);
int detailFileId = interfaceFileSequence();
stmt.setInt (1, detailFileId);
stmt.setInt (2, interfaceHeaderId);
stmt.setInt (3, interfaceRunId);
stmt.setString(4, toString(input.substring(0, 2)));
stmt.setDate (5, toDate(input.substring(2, 12)));
stmt.setString(6, toString(input.substring(12, 22)));
// ...
stmt.addBatch();
if (++batchSize == 1000) {
stmt.executeBatch();
batchSize = 0;
}
}
if (batchSize != 0) {
stmt.executeBatch();
}
} catch (IOException e) {
e.printStackTrace();
}
The above code uses the following helper methods to keep code DRY:
private static String toString(String text) {
String trimmed = text.trim();
return (trimmed.isEmpty() ? null : trimmed);
}
private static java.sql.Date toDate(String text) {
String trimmed = text.trim();
return (trimmed.isEmpty() ? null : java.sql.Date.valueOf(LocalDate.parse(trimmed)));
}
I'm trying to get the print output and store it into an array of Strings in order to write the values in a CSV File. The problem that comes up is that when a run the code, from 100 lines printed, only the last line gets stored:
public void organize(String line) throws IOException {
CSVWriter writer = new CSVWriter(new FileWriter("out.csv"), '\t');
String[] token = line.split(",");
String[] DependencyItems = token[1].split(" ");
List<String> entriesList;
String[] entries;
String row = "";
for (int i = 0; i < DependencyItems.length; i++) {
row = token[0] + "," + DependencyItems[i] + "," + token[2] + "," + token[3] + "," + token[5]+ "," + token[6]
+ "," + token[7]+ "," + token[8]+ "," + token[9] + "," + token[10] + "," + token[11] + "," + token[12]
+ "," + token[13] + "," + token[14] + "," + token[15] + "," + token[16]+ "," + token[17]
+ "," + token[18]+ "," + token[19]+ "," + token[20]+ "," + token[21] + "," + token[22]
+ "," + token[23]+ "," + token[24]+ "," + token[25]+ "," + token[26]+ "," + token[27]
+ "," + token[28]+ "," + token[29];
}
System.out.println(row);
entriesList = Arrays.asList(row);
entries = entriesList.toArray(new String[0]);
writer.writeNext(entries);
writer.close();
}
What I am doing wrong?
row = <big long string>
You are always resetting row then all of the stuff after the loop is just using the last value of row. If you want to use all the values, perhaps the code after the loop should be in the loop? At the very least it looks like these 2 lines belong in the loop.
System.out.println(row);
entriesList = Arrays.asList(row);
Note that if the scope of row was limited to just be inside the loop, then this may help to identify this type of issue as it would not be valid outside the loop.
I have to display all of records in my blocks (text files), and do a split to "cover" the Fields Separators, but only the first record of my blocks are diplayed. What am I doing wrong?
enter code here
public static void listAllStudents() throws IOException {
File path = new File(Descriptor.getBlockPath());
for (int i = 0; i < path.listFiles().length; i++) {
try {
FileInputStream file = new FileInputStream(Descriptor.getBlockPath() + "BLK" + i + ".txt");
InputStreamReader entrada = new InputStreamReader(file);
BufferedReader buf= new BufferedReader(entrada);
String piece = " ";
System.out.println("\nBLOCO " + i + " ------------------------------------------------------ +");
do {
if (buf.ready()) {
piece = buf.readLine();
System.out.println("\n¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨");
String string = " ", field[] = piece.split(Descriptor.getFieldSeparator());
string = " ";
System.out.println("CPF: " + field[0]);
System.out.println("Name: " + field[1]);
System.out.println("Course: " + field[2]);
System.out.println("Age: " + field[3]);
System.out.println("Phone: " + field[4]);
System.out.println("Active: " + field[5]);
string = " ";
}
} while (buf.ready());
buf.close();
} catch (IOException e) {
System.out.println();
}
}
}
See the documentation for the BufferedReader.readLine() method:
or null if the end of the stream has been reached
Then change your code to read the file line by line:
while ((piece = buf.readLine()) != null) {
String field[] = piece.split(Descriptor.getFieldSeparator());
if (field.length >= 6) {
System.out.println("CPF: " + field[0]);
System.out.println("Name: " + field[1]);
System.out.println("Course: " + field[2]);
System.out.println("Age: " + field[3]);
System.out.println("Phone: " + field[4]);
System.out.println("Active: " + field[5]);
}
}
I'm trying to display the information from a text file in a JTextArea I've created in a GUI. I've figured out how to get the info from the file to the JTextArea, but it's only grabbing the last line of the file. I need to display all of the lines. I keep changing the loop around, but can't figure this one out. Any help would be greatly appreciated. Here's a look at my code:
public TextArea() {
initComponents();
try {
FileReader one = new FileReader ("info.txt");
BufferedReader buf = new BufferedReader(one);
String line = "";
StringTokenizer st = null;
int lineNumber = 0, tokenNumber = 0;
//textArea.setText(line);
while ((line = buf.readLine()) != null) {
lineNumber++;
//break comma separated line using ","
st = new StringTokenizer(line, ",");
while (st.hasMoreTokens()) {
//display csv values
tokenNumber++;
line = ("Title: " + st.nextToken()
+ "\n" + "Make:" + st.nextToken()
+ "\n" + "Model:" + st.nextToken()
+ "\n" + "Year:" + st.nextToken()
+ "\n" + "Price:" + st.nextToken()
+ "\n" + "Notes:" + st.nextToken()
+ "\n" + "Details:" + st.nextToken()
+ "\n");
textArea.setText(line);
}
//reset token number
tokenNumber = 0;
//textArea.setText(line);
}
} catch (FileNotFoundException e) {
JOptionPane.showMessageDialog(this, "File not found");
} catch (IOException e){
JOptionPane.showMessageDialog(this, "Data not read");
}
Look at your code:
while (st.hasMoreTokens()) {
//display csv values
tokenNumber++;
line = ("Title: " + st.nextToken()
+ "\n" + "Make:" + st.nextToken()
+ "\n" + "Model:" + st.nextToken()
+ "\n" + "Year:" + st.nextToken()
+ "\n" + "Price:" + st.nextToken()
+ "\n" + "Notes:" + st.nextToken()
+ "\n" + "Details:" + st.nextToken()
+ "\n");
textArea.setText(line);
}
Everytime you find a new token you set the textarea val to last token found.
So obviously text area will display only last line.
You can try something like:
textArea.setText(textArea.getText() + line);
I think You are overriding the line variable.
line+=...
Concatenate, and then set the value of the whole line concatenated outside the loop.
while (st.hasMoreTokens()) {
//display csv values
tokenNumber++;
line = line +"\n"+("Title: " + st.nextToken()
+ "\n" + "Make:" + st.nextToken()
+ "\n" + "Model:" + st.nextToken()
+ "\n" + "Year:" + st.nextToken()
+ "\n" + "Price:" + st.nextToken()
+ "\n" + "Notes:" + st.nextToken()
+ "\n" + "Details:" + st.nextToken()
+ "\n");
}
textArea.setText(line);