inputValue = "111,GOOG,20,475.0"
StringTokenizer tempToken = new StringTokenizer(inputValue, ",");
while(tempToken.hasMoreTokens() == true)
{
test = token.nextToken();
counterTest++;
}
It's giving me some invalid correct NULL character
I started to learn stringtokenizer and I'm not sure at this point what wen't wrong logicly I think it works out but am I forgetting something?
I see some typo in your code.
However,using StringTokenizer is discouraged in new code. From the javadocs:
StringTokenizer is a legacy class that is retained for compatibility
reasons although its use is discouraged in new code. It is recommended
that anyone seeking this functionality use the split method of String
or the java.util.regex package instead.
The recommended way is to use String#split.
Something like:
private void customSplit(String source) {
String[] tokens = source.split(";");
for (int i = 0; i < tokens; i++) {
System.out.println("Token" + i + "is: " + token[i]);
}
}
Your code snippet is working with some minor adjustments, maybe your missing something simple, so check the rewritten full example below:
public static void main(String[] args) throws Exception {
String inputValue = "111,GOOG,20,475.0";
StringTokenizer tempToken = new StringTokenizer(inputValue, ",");
int counterTest = 0;
while (tempToken.hasMoreTokens()) {
String test = tempToken.nextToken();
System.out.println(test);
counterTest++;
}
System.out.println("-------------------");
System.out.println("counterTest = " + counterTest);
}
Output:
111
GOOG
20
475.0
-------------------
counterTest = 4
Related
I am writing a small programming language for a game I am making, this language will be for allowing users to define their own spells for the wizard entity outside the internal game code. I have the language written down, but I'm not entirely sure how to change a string like
setSpellName("Fireball")
setSplashDamage(32,5)
into an array which would have the method name and the arguments after it, like
{"setSpellName","Fireball"}
{"setSplashDamage","32","5"}
How could I do this using java's String.split or string regex's?
Thanks in advance.
Since you're only interested in the function name and parameters I'd suggest scanning up to the first instance of ( and then to the last ) for the params, as so.
String input = "setSpellName(\"Fireball\")";
String functionName = input.substring(0, input.indexOf('('));
String[] params = input.substring(input.indexOf(')'), input.length - 1).split(",");
To capture the String
setSpellName("Fireball")
Do something like this:
String[] line = argument.split("(");
Gets you "setSpellName" at line[0] and "Fireball") at line[1]
Get rid of the last parentheses like this
line[1].replaceAll(")", " ").trim();
Build your JSON with the two "cleaned" Strings.
There's probably a better way with Regex, but this is the quick and dirty way.
With String.indexOf() and String.substring(), you can parse out the function and parameters. Once you parse them out, apply the quotes are around each of them. Then combine them all back together delimited by commas and wrapped in curly braces.
public static void main(String[] args) throws Exception {
List<String> commands = new ArrayList() {{
add("setSpellName(\"Fireball\")");
add("setSplashDamage(32,5)");
}};
for (String command : commands) {
int openParen = command.indexOf("(");
String function = String.format("\"%s\"", command.substring(0, openParen));
String[] parameters = command.substring(openParen + 1, command.indexOf(")")).split(",");
for (int i = 0; i < parameters.length; i++) {
// Surround parameter with double quotes
if (!parameters[i].startsWith("\"")) {
parameters[i] = String.format("\"%s\"", parameters[i]);
}
}
String combine = String.format("{%s,%s}", function, String.join(",", parameters));
System.out.println(combine);
}
}
Results:
{"setSpellName","Fireball"}
{"setSplashDamage","32","5"}
This is a solution using regex, use this Regex "([\\w]+)\\(\"?([\\w]+)\"?\\)":
String input = "setSpellName(\"Fireball\")";
String pattern = "([\\w]+)\\(\"?([\\w]+)\"?\\)";
Pattern r = Pattern.compile(pattern);
String[] matches;
Matcher m = r.matcher(input);
if (m.find()) {
System.out.println("Found value: " + m.group(1));
System.out.println("Found value: " + m.group(2));
String[] params = m.group(2).split(",");
if (params.length > 1) {
matches = new String[params.length + 1];
matches[0] = m.group(1);
System.out.println(params.length);
for (int i = 0; i < params.length; i++) {
matches[i + 1] = params[i];
}
System.out.println(String.join(" :: ", matches));
} else {
matches = new String[2];
matches[0] = m.group(1);
matches[1] = m.group(2);
System.out.println(String.join(", ", matches));
}
}
([\\w]+) is the first group to get the function name.
\\(\"?([\\w]+)\"?\\) is the second group to get the parameters.
This is a Working DEMO.
I am taking creating a StringTokenizer like so and populating an ArrayList using the tokens:
LogUtils.log("saved emails: " + savedString);
StringTokenizer st = new StringTokenizer(savedString, ",");
mListEmailAddresses = new ArrayList<String>();
for (int i = 0; i < st.countTokens(); i++) {
String strEmail = st.nextToken().toString();
mListEmailAddresses.add(strEmail);
}
LogUtils.log("mListEmailAddresses: emails: " + mListEmailAddresses.toString());
11-20 09:56:59.518: I/test(6794): saved emails: hdhdjdjdjd,rrfed,ggggt,tfcg,
11-20 09:56:59.518: I/test(6794): mListEmailAddresses: emails: [hdhdjdjdjd, rrfed]
As you can see mListEmailAddresses is missing 2 values off the end of the array. What should I do to fix this. From my eyes the code looks correct but maybe I am misunderstanding something.
Thanks.
using hasMoreTokens is the solution
while(st.hasMoreTokens()){
String strEmail = st.nextToken().toString();
mListEmailAddresses.add(strEmail);
}
Use the following while loop
StringTokenizer st = new StringTokenizer(savedString, ",");
mListEmailAddresses = new ArrayList<String>();
while (st.hasMoreTokens()) {
String strEmail = st.nextToken();
mListEmailAddresses.add(strEmail);
}
Note, you don't need to call toString, nextToken will return the string.
Alternatively, you could use the split method
String[] tokens = savedString.split(",");
mListEmailAddresses = new ArrayList<String>();
mListEmailAddresses.addAll(Arrays.asList(tokens));
Note, the API docs for StringTokenizer state:
StringTokenizer is a legacy class that is retained for compatibility
reasons although its use is discouraged in new code. It is recommended
that anyone seeking this functionality use the split method of String
or the java.util.regex package instead.
st.countTokens() method calculates the number of times that this tokenizer's nextToken() method can be called before it generates an exception. The current position is not advanced.
To get all elements in ArrayList you should use following code
while(st.hasMoreTokens()) {
String strEmail = st.nextToken().toString();
mListEmailAddresses.add(strEmail);
}
I am using a tab (/t) as delimiter and I know there are some empty fields in my data e.g.:
one->two->->three
Where -> equals the tab. As you can see an empty field is still correctly surrounded by tabs.
Data is collected using a loop :
while ((strLine = br.readLine()) != null) {
StringTokenizer st = new StringTokenizer(strLine, "\t");
String test = st.nextToken();
...
}
Yet Java ignores this "empty string" and skips the field.
Is there a way to circumvent this behaviour and force java to read in empty fields anyway?
There is a RFE in the Sun's bug database about this StringTokenizer issue with a status Will not fix.
The evaluation of this RFE states, I quote:
With the addition of the java.util.regex package in 1.4.0, we have
basically obsoleted the need for StringTokenizer. We won't remove the
class for compatibility reasons. But regex gives you simply what you need.
And then suggests using String#split(String) method.
Thank you at all. Due to the first comment I was able to find a solution:
Yes you are right, thank you for your reference:
Scanner s = new Scanner(new File("data.txt"));
while (s.hasNextLine()) {
String line = s.nextLine();
String[] items= line.split("\t", -1);
System.out.println(items[5]);
//System.out.println(Arrays.toString(cols));
}
You can use Apache Commons StringUtils.splitPreserveAllTokens(). It does exactly what you need.
I would use Guava's Splitter, which doesn't need all the big regex machinery, and is more well-behaved than String's split() method:
Iterable<String> parts = Splitter.on('\t').split(string);
As you can see in the Java Doc http://docs.oracle.com/javase/6/docs/api/java/util/StringTokenizer.html you can use the Constructor public StringTokenizer(String str, String delim, boolean returnDelims) with returnDelims true
So it returns each Delimiter as a seperate string!
Edit:
DON'T use this way, as #npe already typed out, StringTokenizer shouldn't be used any more! See JavaDoc:
StringTokenizer is a legacy class that is retained for compatibility
reasons although its use is discouraged in new code. It is recommended
that anyone seeking this functionality use the split method of String
or the java.util.regex package instead.
public class TestStringTokenStrict {
/**
* Strict implementation of StringTokenizer
*
* #param str
* #param delim
* #param strict
* true = include NULL Token
* #return
*/
static StringTokenizer getStringTokenizerStrict(String str, String delim, boolean strict) {
StringTokenizer st = new StringTokenizer(str, delim, strict);
StringBuffer sb = new StringBuffer();
while (st.hasMoreTokens()) {
String s = st.nextToken();
if (s.equals(delim)) {
sb.append(" ").append(delim);
} else {
sb.append(s).append(delim);
if (st.hasMoreTokens())
st.nextToken();
}
}
return (new StringTokenizer(sb.toString(), delim));
}
static void altStringTokenizer(StringTokenizer st) {
while (st.hasMoreTokens()) {
String type = st.nextToken();
String one = st.nextToken();
String two = st.nextToken();
String three = st.nextToken();
String four = st.nextToken();
String five = st.nextToken();
System.out.println(
"[" + type + "] [" + one + "] [" + two + "] [" + three + "] [" + four + "] [" + five + "]");
}
}
public static void main(String[] args) {
String input = "Record|One||Three||Five";
altStringTokenizer(getStringTokenizerStrict(input, "|", true));
}}
public void GrabData() throws IOException
{
try {
BufferedReader br = new BufferedReader(new FileReader("data/500.txt"));
String line = "";
int lineCounter = 0;
int TokenCounter = 1;
arrayList = new ArrayList < String > ();
while ((line = br.readLine()) != null) {
//lineCounter++;
StringTokenizer tk = new StringTokenizer(line, ",");
System.out.println(line);
while (tk.hasMoreTokens()) {
arrayList.add(tk.nextToken());
System.out.println("check");
TokenCounter++;
if (TokenCounter > 12) {
er = new DataRecord(arrayList);
DR.add(er);
arrayList.clear();
System.out.println("check2");
TokenCounter = 1;
}
}
}
} catch (FileNotFoundException ex) {
Logger.getLogger(Driver.class.getName()).log(Level.SEVERE, null, ex);
}
}
Hello , I am using a tokenizer to read the contents of a line and store it into an araylist. Here the GrabData class does that job.
The only problem is that the company name ( which is the third column in every line ) is in quotes and has a comma in it. I have included one line for your example. The tokenizer depends on the comma to separate the line into different tokens. But the company name throws it off i guess. If it weren't for the comma in the company column , everything goes as normal.
Example:-
Essie,Vaill,"Litronic , Industries",14225 Hancock Dr,Anchorage,Anchorage,AK,99515,907-345-0962,907-345-1215,essie#vaill.com,http://www.essievaill.com
Any ideas?
First of all StringTokenizer is considered to be legacy code. From Java doc:
StringTokenizer is a legacy class that is retained for compatibility reasons although its use is discouraged in new code. It is recommended that anyone seeking this functionality use the split method of String or the java.util.regex package instead.
Using the split() method you get an array of strings. While iterating through the array you can check if the current string starts with a quote and if that's the case check if the next one ends with a quote. If you meet these 2 conditions then you know you didn't split where you wanted and you can merge these 2 together, process it like you want and continue iterating through the array normally after that. In that pass you will probably do i+=2 instead of your regular i++ and it should go unnoticed.
You can accomplish this using Regular Expressions. The following code:
String s = "asd,asdasd,asd\"asdasdasd,asdasdasd\", asdasd, asd";
System.out.println(s);
s = s.replaceAll("(?<=\")([^\"]+?),([^\"]+?)(?=\")", "$1 $2");
s = s.replaceAll("\"", "");
System.out.println(s);
yields
asd,asdasd,asd, "asdasdasd,asdasdasd", asdasd, asd
asd,asdasd,asd, asdasdasd asdasdasd, asdasd, asd
which, from my understanding, is the preprocessing you require for your tokenizer-code to work. Hope this helps.
While StringTokenizer might not natively handle this for you, a couple lines of code will do it... probably not the most efficient, but should get the idea across...
while(tk.hasMoreTokens()) {
String token = tk.nextToken();
/* If the item is encapsulated in quotes, loop through all tokens to
* find closing quote
*/
if( token.startsWIth("\"") ){
while( tk.hasMoreTokens() && ! tk.endsWith("\"") ) {
// append our token with the next one. Don't forget to retain commas!
token += "," + tk.nextToken();
}
if( !token.endsWith("\"") ) {
// open quote found but no close quote. Error out.
throw new BadFormatException("Incomplete string:" + token);
}
// remove leading and trailing quotes
token = token.subString(1, token.length()-1);
}
}
As you can see, in the class description, the use of StringTokenizer is discouraged by Oracle.
Instead of using tokenizer I would use the String split() method
which you can use a regular expression as argument and significantly reduce your code.
String str = "Essie,Vaill,\"Litronic , Industries\",14225 Hancock Dr,Anchorage,Anchorage,AK,99515,907-345-0962,907-345-1215,essie#vaill.com,http://www.essievaill.com";
String[] strs = str.split("(?<! ),(?! )");
List<String> list = new ArrayList<String>(strs.length);
for(int i = 0; i < strs.length; i++) list.add(strs[i]);
Just pay attention to your regex, using this one you're assuming that the comma will be always between spaces.
I want to filter a string.
Basically when someone types a message, I want certain words to be filtered out, like this:
User types: hey guys lol omg -omg mkdj*Omg*ndid
I want the filter to run and:
Output: hey guys lol - mkdjndid
And I need the filtered words to be loaded from an ArrayList that contains several words to filter out. Now at the moment I am doing if(message.contains(omg)) but that doesn't work if someone types zomg or -omg or similar.
Use replaceAll with a regex built from the bad word:
message = message.replaceAll("(?i)\\b[^\\w -]*" + badWord + "[^\\w -]*\\b", "");
This passes your test case:
public static void main( String[] args ) {
List<String> badWords = Arrays.asList( "omg", "black", "white" );
String message = "hey guys lol omg -omg mkdj*Omg*ndid";
for ( String badWord : badWords ) {
message = message.replaceAll("(?i)\\b[^\\w -]*" + badWord + "[^\\w -]*\\b", "");
}
System.out.println( message );
}
try:
input.replaceAll("(\\*?)[oO][mM][gG](\\*?)", "").split(" ")
Dave gave you the answer already, but I will emphasize the statement here. You will face a problem if you implement your algorithm with a simple for-loop that just replaces the occurrence of the filtered word. As an example, if you filter the word ass in the word 'classic' and replace it with 'butt', the resultant word will be 'clbuttic' which doesn't make any sense. Thus, I would suggest using a word list,like the ones stored in Linux under /usr/share/dict/ directory, to check if the word is valid or it needs filtering.
I don't quite get what you are trying to do.
I ran into this same problem and solved it in the following way:
1) Have a google spreadsheet with all words that I want to filter out
2) Directly download the google spreadsheet into my code with the loadConfigs method (see below)
3) Replace all l33tsp33k characters with their respective alphabet letter
4) Replace all special characters but letters from the sentence
5) Run an algorithm that checks all the possible combinations of words within a string against the list efficiently, note that this part is key - you don't want to loop over your ENTIRE list every time to see if your word is in the list. In my case, I found every combination within the string input and checked it against a hashmap (O(1) runtime). This way the runtime grows relatively to the string input, not the list input.
6) Check if the word is not used in combination with a good word (e.g. bass contains *ss). This is also loaded through the spreadsheet
6) In our case we are also posting the filtered words to Slack, but you can remove that line obviously.
We are using this in our own games and it's working like a charm. Hope you guys enjoy.
https://pimdewitte.me/2016/05/28/filtering-combinations-of-bad-words-out-of-string-inputs/
public static HashMap<String, String[]> words = new HashMap<String, String[]>();
public static void loadConfigs() {
try {
BufferedReader reader = new BufferedReader(new InputStreamReader(new URL("https://docs.google.com/spreadsheets/d/1hIEi2YG3ydav1E06Bzf2mQbGZ12kh2fe4ISgLg_UBuM/export?format=csv").openConnection().getInputStream()));
String line = "";
int counter = 0;
while((line = reader.readLine()) != null) {
counter++;
String[] content = null;
try {
content = line.split(",");
if(content.length == 0) {
continue;
}
String word = content[0];
String[] ignore_in_combination_with_words = new String[]{};
if(content.length > 1) {
ignore_in_combination_with_words = content[1].split("_");
}
words.put(word.replaceAll(" ", ""), ignore_in_combination_with_words);
} catch(Exception e) {
e.printStackTrace();
}
}
System.out.println("Loaded " + counter + " words to filter out");
} catch (IOException e) {
e.printStackTrace();
}
}
/**
* Iterates over a String input and checks whether a cuss word was found in a list, then checks if the word should be ignored (e.g. bass contains the word *ss).
* #param input
* #return
*/
public static ArrayList<String> badWordsFound(String input) {
if(input == null) {
return new ArrayList<>();
}
// remove leetspeak
input = input.replaceAll("1","i");
input = input.replaceAll("!","i");
input = input.replaceAll("3","e");
input = input.replaceAll("4","a");
input = input.replaceAll("#","a");
input = input.replaceAll("5","s");
input = input.replaceAll("7","t");
input = input.replaceAll("0","o");
ArrayList<String> badWords = new ArrayList<>();
input = input.toLowerCase().replaceAll("[^a-zA-Z]", "");
for(int i = 0; i < input.length(); i++) {
for(int fromIOffset = 1; fromIOffset < (input.length()+1 - i); fromIOffset++) {
String wordToCheck = input.substring(i, i + fromIOffset);
if(words.containsKey(wordToCheck)) {
// for example, if you want to say the word bass, that should be possible.
String[] ignoreCheck = words.get(wordToCheck);
boolean ignore = false;
for(int s = 0; s < ignoreCheck.length; s++ ) {
if(input.contains(ignoreCheck[s])) {
ignore = true;
break;
}
}
if(!ignore) {
badWords.add(wordToCheck);
}
}
}
}
for(String s: badWords) {
Server.getSlackManager().queue(s + " qualified as a bad word in a username");
}
return badWords;
}