Java Scanner.nextLine() mistakenly parses unicode (emoji) as a new line - java

Easiest demonstrated with an example:
String test = "salut ð\u009F\u0098\u0085 test";
Scanner scan = new Scanner(test);
System.out.println("1:" + scan.nextLine());
System.out.println("2:" + scan.nextLine());
This was a string in user input so unfortunately I'm not 100% sure what that unicode is, but if I recall correctly, it was an emoji (I saw the message when it was sent).
The output is:
1:salut ð
2: test
My expected output is just 1 line (i.e. the example code should give a NoSuchElementException because the second nextLine() should fail.). Why is it parsing as two lines? What is a potential workaround?
When I open the file in a text editor it correctly does not treat that unicode as a new line.

Why is it parsing as two lines?
Although this is an uncommon codepoint, the unicode name of U+0085 is NEXT LINE [NEL], I guess it could be considered a new line character.
But is there a reason BufferedReader and text editors like Sublime Text don't parse it as an actual new line, while Scanner does?
If you look at the respective documentations of Scanner and BufferedReader:
Scanner.nextLine:
Advances this scanner past the current line and returns the input that was skipped. This method returns the rest of the current line, excluding any line separator at the end. The position is set to the beginning of the next line.
Since this method continues to search through the input looking for a line separator...
BufferedReader.readLine:
Reads a line of text. A line is considered to be terminated by any one of a line feed ('\n'), a carriage return ('\r'), or a carriage return followed immediately by a linefeed.
Scanner.nextLine just says "line separator" a very vague term (it certainly doesn't refer to the Unicode category "Line Separators", which only has one codepoint), whereas the BufferedReader.readLine documentation states exactly what a line is.
Considering how Scanner also handles localised number formats and stuff, my guess is that it is designed to be a "smarter" class than BufferedReader.
Looking at the source code of my version of the JDK, Scanner considers the following strings "line separators":
\r\n
\n
\r
\u2028
\u2029
\u0085
The reason why \u0085 is considered a new line character is apparently related to XML parsing.

Related

Reading in line with blank space at end using java scanner class

I'm helping my sisters with a simple java program and I'm stumped. They've only learned scanner classes to read file contents, so I think they're supposed to use the scanner class. Each line contains letters and potentially a blank space, and we're hoping to store each line in an array. This works fine and dandy until one of the lines contains something like:
abcde f (the blank space after f should be read in as part of the
line).
However, scanner.nextLine() seems to disregard this last blank space. I figured I could set my scanner delimiter to \n like so:
scanner.useDelimiter("\n")
and then use scanner.Next() from there, but this still doesn't seem to work. I've googled around and taken a look at a few stackoverflow questions. This question here seems to suggest this is not easily done with the scanner class: How to read whitespace with scanner.next()
Any ideas? I feel like there's an easy way I'm overlooking.
This is how I'm reading in the lines:
While(scanner.hasNextLine(){
String nextLine = scanner.nextLine();
Using the above example, my string would read abcde f. It will get rid of the empty space at the end.
I've also tried to use hasNext and next.
Pardon my formatting, I'm editing on a phone.
Save your text file as ANSI encoding and try again.
By right scanner.nextLine() will capture everything in the line, including whitespace.
scanner.next() will not capture whitespace as the delimiter is whitespace by default.

Difference between python readline() and Java's Scanner class nextLine()

What is the difference between Python's readline() and Java's Scanner class method nextLine()?
nextLine() looks for the next line separator character which could be something other than "\n" as written here:
http://docs.oracle.com/javase/7/docs/api/java/util/Scanner.html#nextLine()
Does the Python readline() method do the same? This is important because my file could have other line separators, but I need to look for specifically the new line character.
Any ideas?
You should test it by yourself.
I've tested it on the console using f.readline() and it reads until \n, even if I have a \r in the line.
>>> f.readline()
'This is a test\n'
>>> f.readline()
'Second line\rwith char\n'
>>> f.readline()
'Third line'
NOTE: Some weird things can happen if you simple print the read line on a python script. But if you use repr(str) you'll see all \n and \r.
First of all you are comparing apple to oranges. Scanner is not the Java equivalent of a python file object. BufferedReader is the equivalent, and in fact if you look at the nextLine method's documentation of BufferedReader:
Reads a line of text. A line is considered to be terminated by any one
of a line feed ('\n'), a carriage return ('\r'), or a carriage return
followed immediately by a linefeed.
Python does this too:
A manner of interpreting text streams in which all of the following
are recognized as ending a line: the Unix end-of-line convention '\n',
the Windows convention '\r\n', and the old Macintosh convention '\r'.
See PEP 278 and PEP 3116, as well as str.splitlines() for an
additional use.
AFAIK python does not provide a public equivalent of Java's Scanner. But there is an (undocumented) re.Scanner which could be used to achieve what you want.
You simply provide a "lexicon" when create an instance and then call the scan method.
Probably the easiest way of achieving what you want is to read the file in chunks, and split it using re.split.

java Scanner reads only first 2048 bytes

I'm using java.util.Scanner to read file contents from classpath with this code:
String path1 = getClass().getResource("/myfile.html").getFile();
System.out.println(new File(path1).length()); // 22244 (correct)
String file1 = new Scanner(new File(path1)).useDelimiter("\\Z").next();
System.out.println(file1.length()); // 2048 (first 2k only)
Code runs from idea with command (maven test)
/Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/bin/java -Dmaven.home=/usr/share/java/maven-3.0.4 -Dclassworlds.conf=/usr/share/java/maven-3.0.4/bin/m2.conf -Didea.launcher.port=7533 "-Didea.launcher.bin.path=/Applications/IntelliJ IDEA 12 CE.app/bin" -Dfile.encoding=UTF-8 -classpath "/usr/share/java/maven-3.0.4/boot/plexus-classworlds-2.4.jar:/Applications/IntelliJ IDEA 12 CE.app/lib/idea_rt.jar" com.intellij.rt.execution.application.AppMain org.codehaus.classworlds.Launcher --fail-fast --strict-checksums test
It was running perfectly on my win7 machine. But after I moved to mac same tests fail.
I tried to google but didn't find much =(
Why Scanner with delimiter \Z read my whole file into a string on win7 but won't do it on mac?
I know there're more ways to read a file, but I like this one-liner and want to understand why it's not working.
Thanks.
Here is some info from java about it
http://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html
\Z The end of the input but for the final terminator, if any
\z The end of the input
Line terminators
A line terminator is a one- or two-character sequence that marks the
end of a line of the input character sequence. The following are
recognized as line terminators:
A newline (line feed) character ('\n'), A carriage-return character
followed immediately by a newline character ("\r\n"), A standalone
carriage-return character ('\r'), A next-line character ('\u0085'), A
line-separator character ('\u2028'), or A paragraph-separator
character ('\u2029).
So use \z instead of \Z
There is a good article about this method of entirely reading file with Scanner:
http://closingbraces.net/2011/12/17/scanner-with-z-regex/
In brief:
Because a single read with “/z” as the delimiter should read
everything until “end of input”, it’s tempting to just do a single
read and leave it at that, as the examples listed above all do.
In most cases that’s OK, but I’ve found at least one situation where
reading to “end of input” doesn’t read the entire input – when the
input is a SequenceInputStream, each of the constituent InputStreams
appears to give a separate “end of input” of its own. As a result, if
you do a single read with a delimiter of “/z” it returns the content
of the first of the SequenceInputStream’s constituent streams, but
doesn’t read into the rest of the constituent streams.
Beware of using it. It will be better to read it line-by-line, or use hasNext() checking until it will be real false.
UPD: In other words, try this code:
StringBuilder file1 = new StringBuilder();
Scanner scanner = new Scanner(new File(path1)).useDelimiter("\\Z");
while (scanner.hasNext()) {
file1.append(scanner.next());
}
I encountered this as well when using nextLine() on Mac, Java 7 update 45. Worse, after the line that is longer than 2048 bytes, the rest of the file is ignored and the Scanner thinks that it is already the end of file.
I change it to explicitly tell Scanner to use larger buffer, and it works.
Scanner sc = new Scanner(new BufferedInputStream(new FileInputStream(nf), 20*1024*1024), "utf-8");

Java scanner to detect blank line?

I've been having lots of trouble trying to get either a scanner or a buffered reader to try and detect a blank line. For example if I have a file that contains:
there
cat
dog
(BLANK LINE)
If I do this:
while( scan.hasNextLine() )
{
String line = scan.nextLine();
...
...
}
The scanner doesn't pick up the blank line. I tried to use a buffered reader also but I run into this issue. Is there some way the scanner can just return a "" whenever it finds a blank line like that? Cheers
Your input has as many lines as it has \n characters. Given the input
"there\ncat\ndog\n"
the next-lines will be correctly divided as
"there\n"
"cat\n"
"dog\n"
(In other words, there is no fourth blank line, since it is not terminated by a \n.)
Put differently, after the "dog\n" has been read, the scanner (or buffered reader for that matter) has reached EOF and there's not even an empty line to return. (Note that when the lines are returned, the new-line character is stripped off.)
So, since this is the expected behavior, I don't know what the easiest fix is. I suspect that the best way to solve this is simply to append a \n to the input, so that the loop runs an extra iteration.

Reading next line InputStream Java

I have an InputStream that would read from a text file. I noticed that the input stream doesn't read from a blank next line.
Sample text file:
[This is
A test file
Here.]
The code:
while ((str = br.readLine())!= null) {
System.out.println(str);
}
Some text files would have multiple break lines in between. How do I get the input stream to accept break lines ?
As can be seen from the sample text file, '[This is' is followed by an empty line and then followed subsequently by 'A test file'. How do I read the empty line in between the two sets of strings ? (That is my definition of line break / break line)
Do you mean?
while ((str = br.readLine())!= null && str.trim().length()>0) {
System.out.println(str);
}
This gives you all the non blank lines.
BufferedReader will read all the lines in the file regardless of whether they're blank or not. It will look for line breaks: LF+CR on Windows and LF on GNU/Linux. According to the BufferedReader documentation:
A line is considered to be terminated
by any one of a line feed ('\n'), a
carriage return ('\r'), or a carriage
return followed immediately by a
linefeed.
So, it depends on what your text file really looks like. Does it really have carriage-returns and line-feeds between the lines or is it just displaying that way? You can find this out by looking at the file in a Hex editor (LF is 0x0A and CR is 0x0D). If so, then BufferedReader should be giving you those blank lines.
Sorry for the mistake, it's a BufferedReader taking in a InputStream. The BufferedReader did read every line including the next line / break line / line break. The problem was with my algorithm I wrote that simply missed the next line / break line / line break. Sorry for the hassle and trouble.
Thanks for all the answers.

Categories