We Keep Coding

Processing large number of records from a file in Java

I have million records in CSV file which has 3 columns id,firstName,lastName. I have to process this file in java and validate that id should be unique, firstName should not be null. If there are scenarios where id is not unique and/or firstName is null then I have to write these records in an output file with a fourth column as the reason("id not unique"/"firstName is NULL"). Performance should be good. Please suggest the best effective way.

You can use a collection (ArrayList) to store all the ID's in it in a loop and check if it doesn't already exist. If it doest, write it in a file.
The code should be like this:
if(!idList.contains(id)){
idList.add(id);
}else{
writer.write(id);
}
The above code should work in a loop for all the records being read from the CSV file.

You can use OpenCsv jar for the purpose you have specified. It's under Apache 2.0 licence.
You can download the jar from
http://www.java2s.com/Code/Jar/o/Downloadopencsv22jar.htm
below is the code for the same
Reader reader = Files.newBufferedReader(Paths.get(INPUT_SAMPLE_CSV_FILE_PATH));
CSVReader csvReader = new CSVReader(reader);
Writer writer = Files.newBufferedReader(Paths.get(OUTPUT_SAMPLE_CSV_FILE_PATH));
CSVWriter csvWriter = new CSVWriter(writer);
List<String[]> list = csvReader.readAll();
for (String[] row : list) {
//assuming First column to be Id
String id = row[0];
//assuming name to be second column
String name = row[1];
//assuming lastName to be third column
String lastName = row[2];
//Put your pattern here
if(id==null || !id.matches("pattern") || name==null || !name.matches("pattern")){
String[] outPutData = new String[]{id, name , lastName, "Invalid Entry"};
csvWriter.writeNext(outPutData);
}
}
let me know if this works or you need further help or clarifications.

If you want a good performance algorithm, you should not use ArrayList.contains(element) as explained here, uses O(n) complexity. Instead I suggest you to use a HashSet as the HashSet.Contains(element) operation has an O(1) complexity. To make things short, with ArrayList you would make 1,000,000^2 operations, while with HashSet you would use 1,000,000 operations.
In pseudo-code (to not give away the full answer and make you find the answer on your own) I would do this:
File outputFile
String[] columns
HashSet<String> ids
for(line in file):
columns = line.split(',')
if(ids.contains(columns.id):
outputFile.append(columns.id + " is not unique")
continue
if(columns.name == null):
outputFile.append("first name is null!")
continue
ids.add(columns.id)

How do I extract variable data from a line of string with tags?

I'm trying to write Java code to go to a website, read the HTML code line-by-line, extract certain pieces of data, including an embedded URL to go to another website, and repeat the process 100 times.
I've been able to isolate most of the pieces of data I need using expressions like:
s.ranking = line.substring(line.indexOf(">")+1, line.length() -7);
But I'm having problems with the following line:
<strong>Writer:</strong> Dylan <br/><strong>Producer:</strong> Tom Wilson&nbsp <br/><strong>Released:</strong> July '65, Columbia<br/>12 weeks; No. 2</p>
I need to extract and save the Writer data (Dylan). The producer data (Tom Wilson) and the Release date data (July '65). Some of the pages will have multiple writers and will be labeled "Writers:", and some will have multiple producers, labeled "Producers:"
How do I capture "Dylan" ,"Tom Wilson" and "July '65" from the above line in Java?
Thank you very much!
DM

The best approach is to use HTML parser. But as i read your comment " I'm doing this for a class and am learning about finding, isolating and extracting data."
What you can do something like :
String producer = "Producer:";
String writer = "Writer:";
String released = "Released:";
String s = "<strong>Writer:</strong> Dylan <br/><strong>Producer:</strong> Tom Wilson&nbsp <br/><strong>Released:</strong> July '65, Columbia<br/>12 weeks; No. 2</p> ";
int writerIndex = s.lastIndexOf(writer);
int producerIndex = s.lastIndexOf(producer);
int realesedIndex = s.lastIndexOf(released);
String writerExtracted = s.substring(writerIndex + writer.length(),
producerIndex).replaceAll("\\<.*?>", "");
System.out.println(writerExtracted);
String producerExtracted = s.substring(
producerIndex + producer.length(), realesedIndex).replaceAll(
"\\<.*?>", "");
System.out.println(producerExtracted);
String releasedExtracted = s.substring(
realesedIndex + released.length(), s.length()).replaceAll(
"\\<.*?>", "");
System.out.println(releasedExtracted);
Output:
Dylan
Tom Wilson&nbsp
July '65, Columbia12 weeks; No. 2
NOTE: you can get rid of signs such as &#039 or &nbsp using another regex ...

using java select city and country from a given string that contains full address

I am writing a code in which I want user to provide a string of unknown length.. suppose he provided a string.. now I want to get city and country present in that string...
If anybody have any better idea, please share..

As your requirement, you have to build a case where you need to defined all the possibility city or country like Array city= new Array["America","England","China","Myanmar"]; after that now loop your array then read the user defined line from index 0 and each time move your character point +1(do in a loop too)(convert it in String) then search your city pattern to match with the character(String). Your program complexity will increase more and more due to your requirement, I think your complexity will raise up to O(n*n), it is not good for memory.
On my view of point, you should ask from user to get the actual requirement step by step like (Enter City :_ then Enter Country :__) it is better to handle the string.GOOD LUCK!

In the question you never specified the format of the input string but assuming the format is "city, country" then this works
String s = "the name of a city, the name of a country";
String city = s.substring(0, s.indexOf(", "));
String country = s.substring(s.indexOf(", ") + 2);
System.out.println("City = " + city);
System.out.println("Country = " + country);

Well, your questions are very interesting. The program you are writing now is depending on LOGIC and I think there is no such jar files available to get solution on it. It is better to get solution manually. Did you ever think about Dictionary program. Mostly Dictionary words are written in a text file and at run time, the program load that words into an array or some other Collections. This way you can also Load Your file at runtime into a 2D array or collection(mostly HashMap is used). So you can scan your file and load it.Suppose u want to read
Example:
Agra,India
London,England
Yangon,Myanmar
Tokyo,Japan
etc...
` String line;
FileInputStream fstream = new FileInputStream(dataFile);
//dataFile is your file directory
DataInputStream in = new DataInputStream(fstream);
BufferedReader br = new BufferedReader(new InputStreamReader(in));
HashMap<String,String> divideCityCountry =new HashMap<String,String>();
while((line=br.readLine())!=-1)
{
String[] lineSplit = line.split(",");//use the ',' as delimiter
divideCityCountry.put(lineSplit[0], lineSplit[1]);
} `
Now you can check or get the city and country in the divideCityCountry HashMap().
Hope this may helpful.Good Luck!

error during grouping files based on the date field

I have a large file which has 10,000 rows and each row has a date appended at the end. All the fields in a row are tab separated. There are 10 dates available and those 10 dates have randomly been assigned to all the 10,000 rows. I am now writing a java code to write all those rows with the same date into a separate file where each file has the corresponding rows with that date.
I am trying to do it using string manipulations, but when I am trying to sort the rows based on date, I am getting an error while mentioning the date and the error says the literal is out of range. Here is the code that I used. Please have a look at it let me know if this is the right approach, if not, kindly suggest a better approach. I tried changing the datatype to Long, but still the same error. The row in the file looks something like this:
Each field is tab separated and the fields are:
business id, category, city, biz.name, longitude, state, latitude, type, date
**
qarobAbxGSHI7ygf1f7a_Q ["Sandwiches","Restaurants"] Gilbert Jersey
Mike's Subs -111.8120071 AZ 3.5 33.3788385 business 06012010
**
The code is:
File f=new File(fn);
if(f.exists() && f.length()>0)
{
BufferedReader br=new BufferedReader(new FileReader(fn));
BufferedWriter bw = new BufferedWriter(new FileWriter("FilteredDate.txt"));
String s=null;
while((s=br.readLine())!=null){
String[] st=s.split("\t");
if(Integer.parseInt(st[13])==06012010){
Thanks a lot for your time..

Try this,
List<String> sampleList = new ArrayList<String>();
sampleList.add("06012012");
sampleList.add("06012013");
sampleList.add("06012014");
sampleList.add("06012015");
//
//
String[] sampleArray = s.split(" ");
if (sampleArray != null)
{
String sample = sampleArray[sampleArray.length - 1];
if (sampleList.contains(sample))
{
stringBuilder.append(sample + "\n");
}
}

i suggest not to use split, but rather use
String str = s.subtring(s.lastIndexOf('\t'));
in any case, you try to take st[13] when i see you only have 9 columns. might be you just need st[8]
one last thing, look at this post to learn what 06012010 really means

Parse specific output into variables Java

I am trying to find the best way how to parse specific output from jsch when connecting and executing commands on a c7000 HP enclosure.
What I have done so far is written a program that connects to a hp enclosure, executes a command, retrieves the output of the command and converts it from a stream to a String.
I end up with this String
Server Blade #1 Information:
Type: Server Blade
Manufacturer: HP
Product Name: ProLiant BL280c G6
Part Number: 507865-B21
System Board Spare Part Number: 531337-001
Serial Number: XZ73616G9Z
UUID: 38926035-5636-5D43-3330-359274423959
Server Name: SERVERONE
Asset Tag: [Unknown]
ROM Version: I25 02/01/2013
CPU 1: Intel(R) Xeon(R) CPU E5520 # 2.27GHz (4 cores)
CPU 2: Intel(R) Xeon(R) CPU E5520 # 2.27GHz (4 cores)
Memory: 49152 MB
Now I need to extract some information from this string and put it in a variable/variables. I have tried with regex but don't seem to hack it. What I need is to end up with for example, product name "Proliant BL280c G6" in a string variable that I later can use, same goes with serial number or any other info in there. What I do not need is the preceding text as Type: or Part Number:. I only need to extract what comes after that.
I am pretty new with Java, learning lots every day, can anyone here point me in the right direction of the best way of solving this?
EDIT: Thank you very much all for quick responses. I got a few ideas now on how to solve the problem. The biggest help goes to showing me how to use regex expressions correctly. What i missed there was the possibility of excluding string pieces not needed ex. (?<=Product\sName:).

String delims = "\n"; //this is the identifier of a line change in java
String[] lines = result.split(delims);
for (int i=0;i<lines.length;i++)
{
System.out.println(lines[i]);
}
This code will save (and print) the lines in that String (assuming that what you posted is saved as a java String).
You have several ways to do this thou. Sure they will be more reasonable methods to make this (regex, parsers,...), but you can do this to check if a String contains a Substring:
str1.toLowerCase().contains(str2.toLowerCase())
So, for example, if you want to know if a line of lines[i] contains the word Product Name, just make this:
subS = "Product Name";
if (lines[x].toLowerCase().contains(subS.toLowerCase()));
//x being a number between 0 and the total of lines
As stated, this is a very rustical method, but it will work.

You can use Asier's method of splitting the lines by the newline character ('\n') and then to get the value of each one you can do line.substring(line.indexOf(":")+1, line.length()). You can also get the name of the parameter with line.substring(0, line.indexOf(":")).
You could make a HashMap and access them by key:
HashMap<String, String> map = new HashMap<String, String>(); //or new HashMap<>() if you are on Java 7
String str = "your server stuff goes here";
String[] lines = str.split("\n");
for(int i = 0; i < lines.length; i++)
{
String curline = lines[i];
int index = curline.indexOf(":");
map.put(curline.substring(0, index).trim(), curline.substring(index+1, curline.length()).trim());
}

Assuming, that your response is in source.txt, we read the file
File file = new File("source.txt");
BufferedReader fileReader = new BufferedReader(new FileReader(file));
Reading line by line we match it against regexp with lookup behind:
String line;
String LOOKUP_BEHIND = "(?<=[a-zA-Z0-9 ]:)";
while((line = fileReader.readLine())!= null){
Matcher matcher1 = Pattern.compile(LOOKUP_BEHIND + ".+").matcher(line);
if (matcher1.find()){
System.out.println(matcher1.group(0));
}
}