using tokenizer to read a line - java

public void GrabData() throws IOException
{
try {
BufferedReader br = new BufferedReader(new FileReader("data/500.txt"));
String line = "";
int lineCounter = 0;
int TokenCounter = 1;
arrayList = new ArrayList < String > ();
while ((line = br.readLine()) != null) {
//lineCounter++;
StringTokenizer tk = new StringTokenizer(line, ",");
System.out.println(line);
while (tk.hasMoreTokens()) {
arrayList.add(tk.nextToken());
System.out.println("check");
TokenCounter++;
if (TokenCounter > 12) {
er = new DataRecord(arrayList);
DR.add(er);
arrayList.clear();
System.out.println("check2");
TokenCounter = 1;
}
}
}
} catch (FileNotFoundException ex) {
Logger.getLogger(Driver.class.getName()).log(Level.SEVERE, null, ex);
}
}
Hello , I am using a tokenizer to read the contents of a line and store it into an araylist. Here the GrabData class does that job.
The only problem is that the company name ( which is the third column in every line ) is in quotes and has a comma in it. I have included one line for your example. The tokenizer depends on the comma to separate the line into different tokens. But the company name throws it off i guess. If it weren't for the comma in the company column , everything goes as normal.
Example:-
Essie,Vaill,"Litronic , Industries",14225 Hancock Dr,Anchorage,Anchorage,AK,99515,907-345-0962,907-345-1215,essie#vaill.com,http://www.essievaill.com
Any ideas?

First of all StringTokenizer is considered to be legacy code. From Java doc:
StringTokenizer is a legacy class that is retained for compatibility reasons although its use is discouraged in new code. It is recommended that anyone seeking this functionality use the split method of String or the java.util.regex package instead.
Using the split() method you get an array of strings. While iterating through the array you can check if the current string starts with a quote and if that's the case check if the next one ends with a quote. If you meet these 2 conditions then you know you didn't split where you wanted and you can merge these 2 together, process it like you want and continue iterating through the array normally after that. In that pass you will probably do i+=2 instead of your regular i++ and it should go unnoticed.

You can accomplish this using Regular Expressions. The following code:
String s = "asd,asdasd,asd\"asdasdasd,asdasdasd\", asdasd, asd";
System.out.println(s);
s = s.replaceAll("(?<=\")([^\"]+?),([^\"]+?)(?=\")", "$1 $2");
s = s.replaceAll("\"", "");
System.out.println(s);
yields
asd,asdasd,asd, "asdasdasd,asdasdasd", asdasd, asd
asd,asdasd,asd, asdasdasd asdasdasd, asdasd, asd
which, from my understanding, is the preprocessing you require for your tokenizer-code to work. Hope this helps.

While StringTokenizer might not natively handle this for you, a couple lines of code will do it... probably not the most efficient, but should get the idea across...
while(tk.hasMoreTokens()) {
String token = tk.nextToken();
/* If the item is encapsulated in quotes, loop through all tokens to
* find closing quote
*/
if( token.startsWIth("\"") ){
while( tk.hasMoreTokens() && ! tk.endsWith("\"") ) {
// append our token with the next one. Don't forget to retain commas!
token += "," + tk.nextToken();
}
if( !token.endsWith("\"") ) {
// open quote found but no close quote. Error out.
throw new BadFormatException("Incomplete string:" + token);
}
// remove leading and trailing quotes
token = token.subString(1, token.length()-1);
}
}

As you can see, in the class description, the use of StringTokenizer is discouraged by Oracle.
Instead of using tokenizer I would use the String split() method
which you can use a regular expression as argument and significantly reduce your code.
String str = "Essie,Vaill,\"Litronic , Industries\",14225 Hancock Dr,Anchorage,Anchorage,AK,99515,907-345-0962,907-345-1215,essie#vaill.com,http://www.essievaill.com";
String[] strs = str.split("(?<! ),(?! )");
List<String> list = new ArrayList<String>(strs.length);
for(int i = 0; i < strs.length; i++) list.add(strs[i]);
Just pay attention to your regex, using this one you're assuming that the comma will be always between spaces.

Related

Convert String="one,two,three" to String='one','two','three'

Need to convert my String values "one,two,three" to 'one','two','three'
I have below code
String input = "One,two,three";
Need to send this input values for an query in hibernate.
So need to send as 'one','two','three' as single string, please provide me a solution easy way to do it
You can do the following:
String result = input.replace(",", "','").replaceAll("(.*)", "'$1'");
input.replace(",", "','") replaces each , with ',' so at this step your string will look like:
One','two','three
next we use a regex to surround the string with ', now it'll look
'One','two','three'
which is what you want.
Regex explanation: We catch the whole string, then we replace it with itself, but surrounded with single quote.
References:
String#replaceAll
String#replace
Regex tutorial
Using regex :
input.replaceAll("(\\w+)", "\'$1\'")
In regex, 'w' is the word character.
http://docs.oracle.com/javase/6/docs/api/java/util/regex/Pattern.html#sum
String[] split = input.split(",");
StringBuilder sb = new StringBuilder();
for (int i = 0 ; i < split.length ; i++ ) {
sb.append("'" + str + "'");
if ( i != split.length-1 ) {
sb.append(",");
}
}
sb.toString();
not smart, but easiest..

String Tokenizer missing 2 values off array

I am taking creating a StringTokenizer like so and populating an ArrayList using the tokens:
LogUtils.log("saved emails: " + savedString);
StringTokenizer st = new StringTokenizer(savedString, ",");
mListEmailAddresses = new ArrayList<String>();
for (int i = 0; i < st.countTokens(); i++) {
String strEmail = st.nextToken().toString();
mListEmailAddresses.add(strEmail);
}
LogUtils.log("mListEmailAddresses: emails: " + mListEmailAddresses.toString());
11-20 09:56:59.518: I/test(6794): saved emails: hdhdjdjdjd,rrfed,ggggt,tfcg,
11-20 09:56:59.518: I/test(6794): mListEmailAddresses: emails: [hdhdjdjdjd, rrfed]
As you can see mListEmailAddresses is missing 2 values off the end of the array. What should I do to fix this. From my eyes the code looks correct but maybe I am misunderstanding something.
Thanks.
using hasMoreTokens is the solution
while(st.hasMoreTokens()){
String strEmail = st.nextToken().toString();
mListEmailAddresses.add(strEmail);
}
Use the following while loop
StringTokenizer st = new StringTokenizer(savedString, ",");
mListEmailAddresses = new ArrayList<String>();
while (st.hasMoreTokens()) {
String strEmail = st.nextToken();
mListEmailAddresses.add(strEmail);
}
Note, you don't need to call toString, nextToken will return the string.
Alternatively, you could use the split method
String[] tokens = savedString.split(",");
mListEmailAddresses = new ArrayList<String>();
mListEmailAddresses.addAll(Arrays.asList(tokens));
Note, the API docs for StringTokenizer state:
StringTokenizer is a legacy class that is retained for compatibility
reasons although its use is discouraged in new code. It is recommended
that anyone seeking this functionality use the split method of String
or the java.util.regex package instead.
st.countTokens() method calculates the number of times that this tokenizer's nextToken() method can be called before it generates an exception. The current position is not advanced.
To get all elements in ArrayList you should use following code
while(st.hasMoreTokens()) {
String strEmail = st.nextToken().toString();
mListEmailAddresses.add(strEmail);
}

parse and capture the numbers that are at the end of strings

I am building a program to go through a log file that has entries like this:
en halo%20reach%20noble%20actual%20in%20theater 1 659
en Wazir_Khan_Mosque 2 77859
en Waziristan_War 3 285976
en Wazirpur_Upazila 1 364
I want to output the numbers that appear at the end of each string (ie 659, 77859, 285976, 285976, 364). As you can see the numbers have differing amounts of digits.
How can I grab the last numbers from these strings?
One possible solution is to split the String according to whitespaces:
String[] splitted = myStr.split("\\s+");
Then take the last element:
splitted[splitted.length - 1];
If you want to int value, you should use Integer#parseInt.
Another solution is using lastIndexOf and substring..
int pos = line.lastIndexOf(' ');
int value = Integer.parseInt(line.substr(pos+1));
If you are reading each line and assigning to a string like this
String line = "en halo%20reach%20noble%20actual%20in%20theater 1 659";
then doing this should give you the last number
String words[] = line.split("\\s");
System.out.println(words[words.length - 1]);
I usually don't recommend regular expressions as they are so often abused here on Stackoverflow ( especially when it comes to XML/HTML ), but this the perfect case to learn how to use them!
Splitting on whitespace, while that will work isn't as robust as this approach; which will continue to work if the whitespace varies, and allows you to capture all the other data in one operation, which you will probably want eventually:
^en\s+(.*)\s+(\d+)\s+(\d+)$ : Click for an explanation of how it works!
Then to use it:
final Pattern p = Pattern.compile("^en\\s+(.*)\\s+(\\d+)\\s+(\\d+)$");
final Matcher m = p.matches("en Wazirpur_Upazila 1 364");
final String g1 = m.group(1); // Wazirpur_Upazila
final String g2 = m.group(2); // 1
final String g3 = m.group(3); // 364
try this, may not be very good
public static void main(String[] args) throws FileNotFoundException, IOException {
BufferedReader br = new BufferedReader(new FileReader("log.txt"));
try {
String line = br.readLine();
List<String> stringList = new ArrayList<>();
while(line!=null) {
String[] strsplit = line.split(" ");
line = br.readLine();
for(int i=3;i<strsplit.length;i+=4) {
stringList.add(strsplit[i]);
}
}
System.out.println(stringList);
} finally {
br.close();
}

Filter words from string

I want to filter a string.
Basically when someone types a message, I want certain words to be filtered out, like this:
User types: hey guys lol omg -omg mkdj*Omg*ndid
I want the filter to run and:
Output: hey guys lol - mkdjndid
And I need the filtered words to be loaded from an ArrayList that contains several words to filter out. Now at the moment I am doing if(message.contains(omg)) but that doesn't work if someone types zomg or -omg or similar.
Use replaceAll with a regex built from the bad word:
message = message.replaceAll("(?i)\\b[^\\w -]*" + badWord + "[^\\w -]*\\b", "");
This passes your test case:
public static void main( String[] args ) {
List<String> badWords = Arrays.asList( "omg", "black", "white" );
String message = "hey guys lol omg -omg mkdj*Omg*ndid";
for ( String badWord : badWords ) {
message = message.replaceAll("(?i)\\b[^\\w -]*" + badWord + "[^\\w -]*\\b", "");
}
System.out.println( message );
}
try:
input.replaceAll("(\\*?)[oO][mM][gG](\\*?)", "").split(" ")
Dave gave you the answer already, but I will emphasize the statement here. You will face a problem if you implement your algorithm with a simple for-loop that just replaces the occurrence of the filtered word. As an example, if you filter the word ass in the word 'classic' and replace it with 'butt', the resultant word will be 'clbuttic' which doesn't make any sense. Thus, I would suggest using a word list,like the ones stored in Linux under /usr/share/dict/ directory, to check if the word is valid or it needs filtering.
I don't quite get what you are trying to do.
I ran into this same problem and solved it in the following way:
1) Have a google spreadsheet with all words that I want to filter out
2) Directly download the google spreadsheet into my code with the loadConfigs method (see below)
3) Replace all l33tsp33k characters with their respective alphabet letter
4) Replace all special characters but letters from the sentence
5) Run an algorithm that checks all the possible combinations of words within a string against the list efficiently, note that this part is key - you don't want to loop over your ENTIRE list every time to see if your word is in the list. In my case, I found every combination within the string input and checked it against a hashmap (O(1) runtime). This way the runtime grows relatively to the string input, not the list input.
6) Check if the word is not used in combination with a good word (e.g. bass contains *ss). This is also loaded through the spreadsheet
6) In our case we are also posting the filtered words to Slack, but you can remove that line obviously.
We are using this in our own games and it's working like a charm. Hope you guys enjoy.
https://pimdewitte.me/2016/05/28/filtering-combinations-of-bad-words-out-of-string-inputs/
public static HashMap<String, String[]> words = new HashMap<String, String[]>();
public static void loadConfigs() {
try {
BufferedReader reader = new BufferedReader(new InputStreamReader(new URL("https://docs.google.com/spreadsheets/d/1hIEi2YG3ydav1E06Bzf2mQbGZ12kh2fe4ISgLg_UBuM/export?format=csv").openConnection().getInputStream()));
String line = "";
int counter = 0;
while((line = reader.readLine()) != null) {
counter++;
String[] content = null;
try {
content = line.split(",");
if(content.length == 0) {
continue;
}
String word = content[0];
String[] ignore_in_combination_with_words = new String[]{};
if(content.length > 1) {
ignore_in_combination_with_words = content[1].split("_");
}
words.put(word.replaceAll(" ", ""), ignore_in_combination_with_words);
} catch(Exception e) {
e.printStackTrace();
}
}
System.out.println("Loaded " + counter + " words to filter out");
} catch (IOException e) {
e.printStackTrace();
}
}
/**
* Iterates over a String input and checks whether a cuss word was found in a list, then checks if the word should be ignored (e.g. bass contains the word *ss).
* #param input
* #return
*/
public static ArrayList<String> badWordsFound(String input) {
if(input == null) {
return new ArrayList<>();
}
// remove leetspeak
input = input.replaceAll("1","i");
input = input.replaceAll("!","i");
input = input.replaceAll("3","e");
input = input.replaceAll("4","a");
input = input.replaceAll("#","a");
input = input.replaceAll("5","s");
input = input.replaceAll("7","t");
input = input.replaceAll("0","o");
ArrayList<String> badWords = new ArrayList<>();
input = input.toLowerCase().replaceAll("[^a-zA-Z]", "");
for(int i = 0; i < input.length(); i++) {
for(int fromIOffset = 1; fromIOffset < (input.length()+1 - i); fromIOffset++) {
String wordToCheck = input.substring(i, i + fromIOffset);
if(words.containsKey(wordToCheck)) {
// for example, if you want to say the word bass, that should be possible.
String[] ignoreCheck = words.get(wordToCheck);
boolean ignore = false;
for(int s = 0; s < ignoreCheck.length; s++ ) {
if(input.contains(ignoreCheck[s])) {
ignore = true;
break;
}
}
if(!ignore) {
badWords.add(wordToCheck);
}
}
}
}
for(String s: badWords) {
Server.getSlackManager().queue(s + " qualified as a bad word in a username");
}
return badWords;
}

How can you parse the string which has a text qualifier

How can I parse a String str = "abc, \"def,ghi\"";
such that I get the output as
String[] strs = {"abc", "\"def,ghi\""}
i.e. an array of length 2.
Should I use regular expression or Is there any method in java api or anyother opensource
project which let me do this?
Edited
To give context about the problem, I am reading a text file which has a list of records one on each line. Each record has list of fields separated by delimiter(comma or semi-colon). Now I have a requirement where I have to support text qualifier some thing excel or open office supports. Suppose I have record
abc, "def,ghi"
In this , is my delimiter and " is my text qualifier such that when I parse this string I should get two fields abc and def,ghi not {abc,def,ghi}
Hope this clears my requirement.
Thanks
Shekhar
The basic algorithm is not too complicated:
public static List<String> customSplit(String input) {
List<String> elements = new ArrayList<String>();
StringBuilder elementBuilder = new StringBuilder();
boolean isQuoted = false;
for (char c : input.toCharArray()) {
if (c == '\"') {
isQuoted = !isQuoted;
// continue; // changed according to the OP comment - \" shall not be skipped
}
if (c == ',' && !isQuoted) {
elements.add(elementBuilder.toString().trim());
elementBuilder = new StringBuilder();
continue;
}
elementBuilder.append(c);
}
elements.add(elementBuilder.toString().trim());
return elements;
}
This question seems appropriate: Split a string ignoring quoted sections
Along that line, http://opencsv.sourceforge.net/ seems appropriate.
Try this -
String str = "abc, \"def,ghi\"";
String regex = "([,]) | (^[\"\\w*,\\w*\"])";
for(String s : str.split(regex)){
System.out.println(s);
}
Try:
List<String> res = new LinkedList<String>();
String[] chunks = str.split("\\\"");
if (chunks.length % 2 == 0) {
// Mismatched escaped quotes!
}
for (int i = 0; i < chunks.length; i++) {
if (i % 2 == 1) {
res.addAll(Array.asList(chunks[i].split(",")));
} else {
res.add(chunks[i]);
}
}
This will only split up the portions that are not between escaped quotes.
Call trim() if you want to get rid of the whitespace.

Categories