We Keep Coding

How CSV parsing can be utilized - JAVA

I am given a file that will read the following:
"String",int,int
"String",int,int
"String",int,int
...
Given an unknown number of variables, a while (scanner.hasNextLine()) can solve to the number of entries. My goal is to take these three pieces of data and store them into a Node. I am using the method BinaryTree.addNode(String, int, int) for this. My issue comes to when I am trying to read in the data. I am trying to remove the commas within the document and then attempting to re-read the data using the following:
Scanner firstpass = new Scanner(file);
String input = firstpass.nextLine().replaceAll(",", "");
Scanner secondpass = new Scanner(input);
String variable1 = secondpass.next();
int variable2 = secondpass.nextInt();
int variable3 = secondpass.nextInt();
This however is a very innefective way of going about this.
UPDATED
The compiling errors can be fixed with the following:
try {
Scanner scanner1 = new Scanner(file);
while (scanner1.hasNextLine()) {
String inventory = scanner1.nextLine().replaceAll(",", " ");
Scanner scanner2 = new Scanner(inventory);
while (scanner2.hasNext()){
String i = scanner2.next();
System.out.print(i);
}
scanner2.close();
}
scanner1.close();
}
catch (FileNotFoundException ex) {
ex.printStackTrace();
}
which gives me the output:
"String"intint"String"intint"String"intint...
So I know I am on the right track. However any (spaces) within the "String" variable are removed. So they would output "SomeString" instead of "Some String". Also I still don't know how to remove the "" from the strings.

The format you've shown matches the CSV (Comma-Separated Values) format, so your best option is to use a CSV parser, e.g. Apache Commons CSV ™.
If you don't want to add a third-party library, you could use Regular Expression to parse the line.
Reading lines from a file should not be done with a Scanner. Use a BufferedReader instead. See Scanner vs. BufferedReader.
try (BufferedReader in = new BufferedReader(new FileReader(file))) {
Pattern p = Pattern.compile("\"(.*?)\",(-?\\d+),(-?\\d+)");
for (String line; (line = in.readLine()) != null; ) {
Matcher m = p.matcher(line);
if (! m.matches())
throw new IOException("Invalid line: " + line);
String value1 = m.group(1);
int value2 = Integer.parseInt(m.group(2));
int value3 = Integer.parseInt(m.group(3));
// use values here
}
} catch (IOException | NumberFormatException ex) {
ex.printStackTrace();
}
Note that this will not work if the string contains escaped characters, e.g. if it contains embedded double-quotes. For that, you should use a parser library.
The code above will correctly handle embedded spaces and commas.

I would instead of using
String input = firstpass.nextLine().replaceAll(",", "");
Scanner secondpass = new Scanner(input);
String variable1 = secondpass.next();
int variable2 = secondpass.nextInt();
int variable3 = secondpass.nextInt();
Use the following approach
String line = firstpass.nextLine();
String[] temp = line.split(",");
String variable1 = temp[0];
int variable2 = Integer.parseInt(temp[1]);
int variable3 = Integer.parseInt(temp[2]);

Reading a set of values from a text file

{
"TEST":"189456",
"TEST1":"X_Y_Z",
"TEST2":"Y_Z_W",
"TEST3":"GGG ",
"TEST4":"32423423233322"
},
{
"TEST":"123456",
"TEST1":"X_E_Z",
"TEST2":"T_Z_W",
"TEST3":"EWE ",
"TEST4":"324234243234"
}
This is a .txt file I want to read and print only 189456,123456 from the above file.Can anyone help me in doing this.Please find the code for reference.Please post the easiest code.....
Pattern p = Pattern.compile("\"Test\"\\s*:\\s*\"(.*)\"", Pattern.CASE_INSENSITIVE);
while ( (line = bf.readLine()) != null) {
linecount++;
Matcher m = p.matcher(line);
// indicate all matches on the line
while (m.find()) {
System.out.println(m.group(1));
}
}

Another way to do it:
while ((line = br.readLine()) != null) {
if(line.contains("\"TEST:\"")){
String[] lineValues = line.split(":");
System.out.println(lineValues[1].replace("\"", "").replace(",",""));
}
}

As for a Regex solution :
(.*)\"TEST":\"(.*?)\"
Note the ? , it makes your regex to stop at the first match of ".
With spaces in between :
(.*)\"TEST"\s*:\s*\"(.*?)\"

With provided input, you should read it as json instead of raw text.
com.fasterxml.jackson.databind.ObjectMapper mapper = new com.fasterxml.jackson.databind.ObjectMapper();
List<TestObj> test = new ArrayList<TestObj>();
test = mapper.readValue(new File("c:\\YourFile.txt"), test.getClass());
Where TestObj is something like this:
class TestObj {
String test;
String test1; // You should use json annotation here because it does not match your json field name.
...
// getter setter methods
}

Hope I understood the question the right way :D
String saveData;
Pattern p = Pattern.compile("\"Test\"\\s*:\\s*\"(.*)\"", Pattern.CASE_INSENSITIVE);
while ( (line = bf.readLine()) != null) {
linecount++;
Matcher m = p.matcher(line);
// indicate all matches on the line
if(line.contains("189456") || line.contains("123456")) {
saveData = line;
}
}
if the String you get from readLine() contains the searched string it will save it in saveData

FileInputStream fstream = new FileInputStream("D:\\prac\\src\\test.txt");
BufferedReader br = new BufferedReader(new InputStreamReader(fstream));
String strLine;
while ((strLine = br.readLine()) != null) {
if(strLine.contains("\"TEST\":")){
System.out.println(strLine.split(":")[1].replaceAll("\"","").replace(",",""));
}
}
br.close();
}
Output:
189456
123456

How to apply regex to entire file, not just line after line?

I want to apply my regular expression not just to the first line of the text file, but to the all lines together.
Currently it matches only when the entire appropriate match is on one line. And if the appropriate match continues on the next line - it doesn't match at all.
class Parser {
public static void main(String[] args) throws IOException {
Pattern patt = Pattern.compile("(include|"
+ "integrate|"
+ "driven based on|"
+ "facilitate through|"
+ "contain|"
+ "using|"
+ "equipped"
+ "integrate|"
+ "implement|"
+ "utilized to facilitate|"
+ "comprise){1}"
+ "[\\s\\w\\,\\(\\)\\;\\:]*\\."); //Regex
BufferedReader r = new BufferedReader(new FileReader("E:/test/test.txt")); // read the file
String line;
PrintWriter pWriter = null;
while ((line = r.readLine()) != null) {
Matcher matcher = patt.matcher(line);
while (matcher.find()) {
try{
pWriter = new PrintWriter(new BufferedWriter(new FileWriter("E:/test/test1.txt", true)));//append any given input
pWriter.println(matcher.group()); //write the result of matcher to the new file
} catch (IOException ioe) {
ioe.printStackTrace();
} finally {
if (pWriter != null){
pWriter.flush();
pWriter.close();
}
}
System.out.println(matcher.group());
}
}
}
}

Change while ((line = r.readLine()) != null) to this:
String file = ""; // Basically, a conglomerate of all of the lines in the file
while ((line = r.readLine()) != null) {
file += line; // Append each line to the "file" string
}
Matcher matcher = patt.matcher(file);
while (matcher.find()) {
/* Blah blah blah, your outputting goes here. */
}
The reason why this happens is because you're doing each line individually. For what you want, you need to apply the regex to the file all at once.

Currently the matcher is applied per line, it needs to be applied to the whole file to work as intended.
Regex are greedy, you will match the whole String on the first match unless you have . (or other special characters) in your String:
...
+ "comprise){1}"
+ "[\\s\\w\\,\\(\\)\\;\\:]*\\."); //Regex
On the last line you match any whitespace and word, so pretty much anything but .. Also the {1} and most of the \ are superfluous (because in []):
...
+ "comprise)"
+ "[\\s\\w,();:]*\\."); //Regex
If you don't care about the newline characters just remove them first and it should work (I see no way around it if you have something like "com\nprise" and want to match that):
s = s.replaceAll("\\n+", "");

Java String Matching in a Sorted File and grouping similar data

i have sorted file and i need to do the following pattern match. I read the row and then compare or do patern match with the row just after it , if it matches then insert the string i used to match after a comma in that row and move on to the next row. I am new to Java and overwhelmed with options from Open CSV to BufferedReader. I intend to iterate through the file till it reaches the end. I may always have blanks and have a dated in quotes. The file size would be around 100 MBs.
My file has data like
ABCD
ABCD123
ABCD456, 123
XYZ
XYZ890
XYZ123, 890
and output is expected as
ABCD, ABCD
ABCD123, ABCD
ABCD456, 123, ABCD
XYZ, XYZ
XYZ890, XYZ
XYZ123, 890, XYZ
Not sure about the best method. Can you please help me.

To open a file, you can use File and FileReader classes:
File csvFile = new File("file.csv");
FileReader fileReader = null;
try {
fileReader = new FileReader(csvFile);
} catch (FileNotFoundException e) {
e.printStackTrace();
}
You can get a line of the file using Scanner:
Scanner reader = new Scanner(fileReader);
while(reader.hasNext()){
String line = reader.nextLine();
parseLine(line);
}
You want to parse this line. For it, you have to study Regex for using Pattern and Matcher classes:
private void parseLine(String line) {
Matcher matcher = Pattern.compile("(ABCD)").matcher(line);
if(matcher.find()){
System.out.println("find: " + matcher.group());
}
}
To find the next pattern of the same row, you can reuse matcher.find(). If some result was found, it will return true and you can get this result with matcher.groud();

Read line by line and use regex to replace it as per your need using String.replaceAll()
^([A-Z]+)([0-9]*)(, [0-9]+)?$
Replacement : $1$2$3, $1
Here is Online demo
Read more about Java Pattern
Sample code:
String regex = "^([A-Z]+)([0-9]*)(, [0-9]+)?$";
String replacement = "$1$2$3, $1";
String newLine = line.replaceAll(regex,replacement);
For better performance, read 100 or more lines at a time and store in a buffer and finally call String#replaceAll() single time to replace all at a time.
sample code:
String regex = "([A-Z]+)([0-9]*)(, [0-9]+)?(\r?\n|$)";
String replacement = "$1$2$3, $1$4";
StringBuilder builder = new StringBuilder();
int counter = 0;
String line = null;
try (BufferedReader reader = new BufferedReader(new FileReader("abc.csv"))) {
while ((line = reader.readLine()) != null) {
builder.append(line).append(System.lineSeparator());
if (counter++ % 100 == 0) { // 100 lines
String newLine = builder.toString().replaceAll(regex, replacement);
System.out.print(newLine);
builder.setLength(0); // reset the buffer
}
}
}
if (builder.length() > 0) {
String newLine = builder.toString().replaceAll(regex, replacement);
System.out.print(newLine);
}
Read more about Java 7 - The try-with-resources Statement

how do you get String Tokenizer to ignore text?

I have this code:
public void readTroops() {
File file = new File("resources/objects/troops.txt");
StringBuffer contents = new StringBuffer();
BufferedReader reader = null;
try {
reader = new BufferedReader(new FileReader(file));
String text = null;
// repeat until all lines is read
while ((text = reader.readLine()) != null) {
StringTokenizer troops = new StringTokenizer(text,"=");
String list = troops.nextToken();
String value = troops.nextToken();
}
and this file:
//this is a comment part of the text file//
Total=1
the problem is that 1) I cant get it to ignore everything within the //,// and can't get it to read with an 'ENTER' (line) in-between them. For example, this text works:
Total=1
So my question is what do I type into the delimiter area ie.
StringTokenizer troops = new StringTokenizer(text,"=","WHAT GOES HERE?");
So how can I get Tokenizer to ignore 'ENTER'/new line, and anything in-between // or something similar, thanks.
ps.I don't care if you use a String.split to answer my question.

Use the method countTokens to skip lines that don't have two tokens:
while ((text = reader.readLine()) != null) {
StringTokenizer troops = new StringTokenizer(text,"=");
if(troops.countTokens() == 2){
String list = troops.nextToken();
String value = troops.nextToken();
....
}else {
//ignore this line
}
}

Properties prop = new Properties();
prop.load(new FileInputStream("properties_file.txt"));
assertExuals("1",prop.getProperty("Total"));
ps. you might hold and close input stream.

Thinking out of the box, maybe you can use Properties instead of tokenizer (if you update your comments to start with #)?
Properties troops = new Properties();
InputStream inputStream = SomeClass.class.getResourceAsStream("troops.properties");
try {
props.load(inputStream);
} catch (IOException e) {
// Handle error
} finally {
// Close inputStream in a safe manner
}
troops.getProperty("Total"); // Returns "1"
Or if you are using Java 7:
Properties troops = new Properties();
try (InputStream inputStream = SomeClass.class.getResourceAsStream("troops.properties")) {
props.load(inputStream);
} catch (IOException e) {
// Handle error
}
troops.getProperty("Total"); // Returns "1"

If you are reading in the file a better way would be to use a StreamTokenizer. This then allows you to declare your own syntax of the tokenizer. I used this method to create a HTML rendering engine. This then allows you to parse direct from a reader, and also provides useful functions to identify numbers, which it seems you may use.
(I will post an example once my eclipse loads!)
public static String render(String file, HashMap vars){
// Create a stringbuffer to rebuild the string
StringBuffer renderedFile = new StringBuffer();
try{
FileReader in = new FileReader(file);
BufferedReader reader = new BufferedReader(in); // create your reader
StreamTokenizer tok;
tok = new StreamTokenizer(reader); //the tokenizer then takes in the reader as a builder
tok.resetSyntax();
tok.wordChars(0, 255); //sets all chars (inc spaces to be counted as words)
/*
* quoteChar allows you to set your comment char, for example $ hello $ means it will ignore hello
*/
tok.quoteChar('$');
while(tok.nextToken()!=StreamTokenizer.TT_EOF){ //while it is not at the end of file
String s = tok.sval;
if (vars.containsKey(s))
s =(String)vars.get(s);
renderedFile.append(s);
}
}
catch(Exception e){System.out.println("Error Loading Template");}
return renderedFile.toString();
}
Check this out for a good tutorial http://tutorials.jenkov.com/java-io/streamtokenizer.html