We Keep Coding

Java Scanner ignores first token if it is empty

I'm trying to read an InputStream of String tokens with a Scanner. Every token ends with a comma ,. An empty string "" is also a valid token. In that case the whole token is just the comma that ends it.
The InputStream is slowly read from another process, and any tokens should be handled as soon as they have been fully read. Therefore reading the whole InputStream to a String is out of the question.
An example input could look like this:
ab,,cde,fg,
If I set the delimiter of the Scanner to a comma, it seems to handle the job just fine.
InputStream input = slowlyArrivingStreamWithValues("ab,,cde,fg,");
Scanner scan = new Scanner(input);
scan.useDelimiter(Pattern.quote(","));
while (scan.hasNext()) {
System.out.println(scan.next());
}
output:
ab
cde
fg
However the problems appear when the stream begins with an empty token. For some reason Scanner just ignores the first token if it is empty.
/* begins with empty token */
InputStream input = slowlyArrivingStreamWithValues(",ab,,cde,fg,");
...
output:
ab
cde
fg
Why does Scanner ignore the first token? How can I include it?

Try using a lookbehind as the pattern:
(?<=,)
and then replace comma with empty string with each token that you match. Consider the following code:
String input = ",ab,,cde,fg,";
Scanner scan = new Scanner(input);
scan.useDelimiter("(?<=,)");
while (scan.hasNext()) {
System.out.println(scan.next().replaceAll(",", ""));
}
This outputs the following:
(empty line)
ab
cde
fg
Demo

It's easier if you write it yourself, without using Scanner:
static List<String> getValues(String source){
List<String> list = new ArrayList<String>();
for(int i = 0; i < source.length();i++){
String s = "";
while(source.charAt(i) != ','){
s+=source.charAt(i++);
if(i >= source.length()) break;
}
list.add(s);
}
return list;
}
For example, if source = ",a,,b,,c,d,e", the output will be "", "a", "", "c", "d", "e".

Java Delete Line from File

the code below is from a reference i saw online, so there might be some similarities i'm trying to implement the code to remove an entire line based on the 1st field in this instance it is (aaaa or bbbb) the file which has a delimiter "|", but it is not working. Hope someone can advise me on this. Do i need to split the line first? or my method is wrong?
data in player.dat (e.g)
bbbb|aaaaa|cccc
aaaa|bbbbbb|cccc
Code is below
public class testcode {
public static void main(String[] args)throws IOException
{
File inputFile = new File("players.dat");
File tempFile = new File ("temp.dat");
BufferedReader read = new BufferedReader(new FileReader(inputFile));
BufferedWriter write = new BufferedWriter(new FileWriter(tempFile));
Scanner UserInput = new Scanner(System.in);
System.out.println("Please Enter Username:");
String UserIn = UserInput.nextLine();
String lineToRemove = UserIn;
String currentLine;
while((currentLine = read.readLine()) != null) {
// trim newline when comparing with lineToRemove
String trimmedLine = currentLine.trim();
if(trimmedLine.equals(lineToRemove)) continue;
write.write(currentLine + System.getProperty("line.separator"));
}
write.close();
read.close();
boolean success = tempFile.renameTo(inputFile);
}
}

Your code compares the entire line it reads from the file to the user name the user enters, but you say in your question that you actually only want to compare to the first part up to the first pipe (|). Your code doesn't do that.
What you need to do is read the line from the file, get the part of the string up to the first pipe symbol (split the string) and skip the line based on comparing the first part of the split string to the lineToRemove variable.
To make it easier, you could also add the pipe symbol to the user input and then do this:
string lineToRemove = UserIn + "|";
...
if (trimmedLine.startsWith(lineToRemove)) continue;
This spares you from splitting the string.
I'm currently not sure whether UserInput.nextLine(); returns the newline character or not. To be safe here, you could change the above to:
string lineToRemove = UserIn.trim() + "|";

reading String with spaces java

I am trying to read from scanner with spaces, i want to read even the spaces.
for example "john smith" to be read "john smith".
my code is as follow:
when it gets to the space after john it just hangs and doesn't read any more.
any help would be appreciated.
Scanner in = new Scanner(new InputStreamReader(sock.getInputStream()));
String userName = "";
while (in.hasNext()) {
userName.concat(in.next());
}

Scanner.next() returns the next token, delimited by whitespace. If you would like to read the entire line, along with the spaces, use nextLine() instead:
String userName = in.nextLine();

Scanner scan = new Scanner(file);
scan.useDelimiter("\\Z");
String content = scan.next();
or
private String readFileAsString(String filePath) throws IOException {
StringBuffer fileData = new StringBuffer();
BufferedReader reader = new BufferedReader(
new FileReader(filePath));
char[] buf = new char[1024];
int numRead=0;
while((numRead=reader.read(buf)) != -1){
String readData = String.valueOf(buf, 0, numRead);
fileData.append(readData);
}
reader.close();
return fileData.toString();
}

When we use Scanner.next() to read token there is what we call a delimiter, the default delimiter used in by Scanner is \p{javaWhitespace}+ , you can get it by calling Scanner.delimiter(), which is any char that validate the Character.isWhitespace(char). you can use a customized delimiter for your Scanner using Scanner.useDelimiter().
If you want to take one line as a string so you can use nextLine() , if you already know what is the type of the next token in the input stream, scanner gives you a list of method next*() take convert the token to the specified type. see Scanner's doc here for more info.

How to determine the end of a line with a Scanner?

I have a scanner in my program that reads in parts of the file and formats them for HTML. When I am reading my file, I need to know how to make the scanner know that it is at the end of a line and start writing to the next line.
Here is the relevant part of my code, let me know if I left anything out :
//scanner object to read the input file
Scanner sc = new Scanner(file);
//filewriter object for writing to the output file
FileWriter fWrite = new FileWriter(outFile);
//Reads in the input file 1 word at a time and decides how to
////add it to the output file
while (sc.hasNext() == true)
{
String tempString = sc.next();
if (colorMap.containsKey(tempString) == true)
{
String word = tempString;
String color = colorMap.get(word);
String codeOut = colorize(word, color);
fWrite.write(codeOut + " ");
}
else
{
fWrite.write(tempString + " ");
}
}
//closes the files
reader.close();
fWrite.close();
sc.close();
I found out about sc.nextLine(), but I still don't know how to determine when I am at the end of a line.

If you want to use only Scanner, you need to create a temp string instantiate it to nextLine() of the grid of data (so it returns only the line it skipped) and a new Scanner object scanning the temp string. This way you're only using that line and hasNext() won't return a false positive (It isn't really a false positive because that's what it was meant to do, but in your situation it would technically be). You just keep nextLine()ing the first scanner and changing the temp string and the second scanner to scan each new line etc.

Lines are usually delimitted by \n or \r so if you need to check for it you can try doing it that way, though I'm not sure why you'd want to since you are already using nextLine() to read a whole line.
There is Scanner.hasNextLine() if you are worried about hasNext() not working for your specific case (not sure why it wouldn't though).

you can use the method hasNextLine to iterate the file line by line instead of word by word, then split the line by whitespaces and make your operations on the word
here is the same code using hasNextLine and split
//scanner object to read the input file
Scanner sc = new Scanner(file);
//filewriter object for writing to the output file
FileWriter fWrite = new FileWriter(outFile);
//get the line separator for the current platform
String newLine = System.getProperty("line.separator");
//Reads in the input file 1 word at a time and decides how to
////add it to the output file
while (sc.hasNextLine())
{
// split the line by whitespaces [ \t\n\x0B\f\r]
String[] words = sc.nextLine().split("\\s");
for(String word : words)
{
if (colorMap.containsKey(word))
{
String color = colorMap.get(word);
String codeOut = colorize(word, color);
fWrite.write(codeOut + " ");
}
else
{
fWrite.write(word + " ");
}
}
fWrite.write(newLine);
}
//closes the files
reader.close();
fWrite.close();
sc.close();

Wow I've been using java for 10 years and have never heard of scanner!
It appears to use white space delimiters by default so you can't tell when an end of line occurs.
Looks like you can change the delimiters of the scanner - see the example at Scanner Class:
String input = "1 fish 2 fish red fish blue fish";
Scanner s = new Scanner(input).useDelimiter("\\s*fish\\s*");
System.out.println(s.nextInt());
System.out.println(s.nextInt());
System.out.println(s.next());
System.out.println(s.next());
s.close();

string tokenizer wrong usage in java

I believe I am not using correctly String Tokenizer. Here is my code:
buffer = new byte[(int) (end - begin)];
fin.seek(begin);
fin.read(buffer, 0, (int) (end - begin));
StringTokenizer strk = new StringTokenizer(new String(buffer),
DELIMS,true);
As you can see I am reading a chunk of lines from a file(end and begin are line numbers) and I am transfering the data to a string tokenizer. My delimitators are:
DELIMS = "\r\n ";
because I want to separate words that have a space between them, or are on the next line.
However this code sometimes separates whole words also. What could be the explanation?? Is my DELIMS string conceived wrong?
Also I am passing "true" as an argument to the tokenizer because I want the delimitators to be treated as tokens as well.( I want this because I want to count the line I am currently at)
Could you please help me. Thanks a lot.

To start with, your method for converting bytes into a String is a bit suspect, and this overall method will be less-than-efficient, especially for a larger file.
Are you required to use StringTokenizer? If not, I'd strongly recommend using Scanner instead. I'd provide you with an example, but will ask that you just refer to the Javadocs instead, which are quite comprehensive and already contain good examples. That said, it accepts delimiters as well - but as Regular Expressions, so just be aware.

You could always wrap your input stream in a LineNumberReader. That will keep track of the line number for you. LineNumberReader extends BufferedReader, which has a readLine() method. With that, you could use a regular StringTokenizer to get your words as tokens. You could use regular expressions or Scanner, but for this case, StringTokenizer is simpler for beginners to understand and quicker.
You must have a RandomAccessFile. You didn't specify that, but I'm guessing based on the methods you used. Try something like:
byte [] buffer = ...; // you know how to get this.
ByteArrayInputStream stream = new ByteArrayInputStream(buffer);
// if you have java.util.Scanner
{
int lineNumber = 0;
Scanner s = new Scanner(stream);
while (s.hasNextLine()) {
lineNum++;
String line = s.nextLine();
System.out.format("I am on line %s%n", lineNum);
Scanner lineScanner = new Scanner(line);
while (lineScanner.hasNext()) {
String word = lineScanner.next();
// do whatever with word
}
}
}
// if you don't have java.util.Scanner, or want to use StringTokenizer
{
LineNumberReader reader = new LineNumberReader(
new InputStreamReader(stream));
String line = null;
while ((line = reader.nextLine()) != null) {
System.out.println("I am on line " + reader.getLineNumber());
StringTokenizer tok = new StringTokenizer(line);
while (tok.hasMoreTokens()) {
String word = tok.nextToken();
// do whatever with word
}
}
}