I'm using java.util.Scanner to read file contents from classpath with this code:
String path1 = getClass().getResource("/myfile.html").getFile();
System.out.println(new File(path1).length()); // 22244 (correct)
String file1 = new Scanner(new File(path1)).useDelimiter("\\Z").next();
System.out.println(file1.length()); // 2048 (first 2k only)
Code runs from idea with command (maven test)
/Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/bin/java -Dmaven.home=/usr/share/java/maven-3.0.4 -Dclassworlds.conf=/usr/share/java/maven-3.0.4/bin/m2.conf -Didea.launcher.port=7533 "-Didea.launcher.bin.path=/Applications/IntelliJ IDEA 12 CE.app/bin" -Dfile.encoding=UTF-8 -classpath "/usr/share/java/maven-3.0.4/boot/plexus-classworlds-2.4.jar:/Applications/IntelliJ IDEA 12 CE.app/lib/idea_rt.jar" com.intellij.rt.execution.application.AppMain org.codehaus.classworlds.Launcher --fail-fast --strict-checksums test
It was running perfectly on my win7 machine. But after I moved to mac same tests fail.
I tried to google but didn't find much =(
Why Scanner with delimiter \Z read my whole file into a string on win7 but won't do it on mac?
I know there're more ways to read a file, but I like this one-liner and want to understand why it's not working.
Thanks.
Here is some info from java about it
http://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html
\Z The end of the input but for the final terminator, if any
\z The end of the input
Line terminators
A line terminator is a one- or two-character sequence that marks the
end of a line of the input character sequence. The following are
recognized as line terminators:
A newline (line feed) character ('\n'), A carriage-return character
followed immediately by a newline character ("\r\n"), A standalone
carriage-return character ('\r'), A next-line character ('\u0085'), A
line-separator character ('\u2028'), or A paragraph-separator
character ('\u2029).
So use \z instead of \Z
There is a good article about this method of entirely reading file with Scanner:
http://closingbraces.net/2011/12/17/scanner-with-z-regex/
In brief:
Because a single read with “/z” as the delimiter should read
everything until “end of input”, it’s tempting to just do a single
read and leave it at that, as the examples listed above all do.
In most cases that’s OK, but I’ve found at least one situation where
reading to “end of input” doesn’t read the entire input – when the
input is a SequenceInputStream, each of the constituent InputStreams
appears to give a separate “end of input” of its own. As a result, if
you do a single read with a delimiter of “/z” it returns the content
of the first of the SequenceInputStream’s constituent streams, but
doesn’t read into the rest of the constituent streams.
Beware of using it. It will be better to read it line-by-line, or use hasNext() checking until it will be real false.
UPD: In other words, try this code:
StringBuilder file1 = new StringBuilder();
Scanner scanner = new Scanner(new File(path1)).useDelimiter("\\Z");
while (scanner.hasNext()) {
file1.append(scanner.next());
}
I encountered this as well when using nextLine() on Mac, Java 7 update 45. Worse, after the line that is longer than 2048 bytes, the rest of the file is ignored and the Scanner thinks that it is already the end of file.
I change it to explicitly tell Scanner to use larger buffer, and it works.
Scanner sc = new Scanner(new BufferedInputStream(new FileInputStream(nf), 20*1024*1024), "utf-8");
Related
Easiest demonstrated with an example:
String test = "salut ð\u009F\u0098\u0085 test";
Scanner scan = new Scanner(test);
System.out.println("1:" + scan.nextLine());
System.out.println("2:" + scan.nextLine());
This was a string in user input so unfortunately I'm not 100% sure what that unicode is, but if I recall correctly, it was an emoji (I saw the message when it was sent).
The output is:
1:salut ð
2: test
My expected output is just 1 line (i.e. the example code should give a NoSuchElementException because the second nextLine() should fail.). Why is it parsing as two lines? What is a potential workaround?
When I open the file in a text editor it correctly does not treat that unicode as a new line.
Why is it parsing as two lines?
Although this is an uncommon codepoint, the unicode name of U+0085 is NEXT LINE [NEL], I guess it could be considered a new line character.
But is there a reason BufferedReader and text editors like Sublime Text don't parse it as an actual new line, while Scanner does?
If you look at the respective documentations of Scanner and BufferedReader:
Scanner.nextLine:
Advances this scanner past the current line and returns the input that was skipped. This method returns the rest of the current line, excluding any line separator at the end. The position is set to the beginning of the next line.
Since this method continues to search through the input looking for a line separator...
BufferedReader.readLine:
Reads a line of text. A line is considered to be terminated by any one of a line feed ('\n'), a carriage return ('\r'), or a carriage return followed immediately by a linefeed.
Scanner.nextLine just says "line separator" a very vague term (it certainly doesn't refer to the Unicode category "Line Separators", which only has one codepoint), whereas the BufferedReader.readLine documentation states exactly what a line is.
Considering how Scanner also handles localised number formats and stuff, my guess is that it is designed to be a "smarter" class than BufferedReader.
Looking at the source code of my version of the JDK, Scanner considers the following strings "line separators":
\r\n
\n
\r
\u2028
\u2029
\u0085
The reason why \u0085 is considered a new line character is apparently related to XML parsing.
I am wondering how you can use the split method with this example considering the fact that that there is a line break in the file.
g3,g3,g3,c4-,a3-,g4-,r,r,r,g3,g3,g3,c4-,a3-,a4,g4-,r,r,r,c4,c4,c4,e4,r
g4,r,a4,r,r,b4b,r,a4,f4,r,g4,r,r,g4#,r,g4,d4#,r,g4
I read the Pattern api and tutorials and think it should be like so.
line.split("(,\n)");
I also tried
line.split([,\n]);
and
line.split("[,\n]");
lines may separated using \r or \n both of them, or even some other characters. Since Java 8 you can use \\R to represent line separators (more info). So you could try using
String[] arr = yourText.split(",|\\R");
As Pshemo notes, the 3rd option str.split("[,\n]") should work assuming the file ends each line with \n and not \r\n.
Additionally, how you read the file may affect your split argument.
If you are reading the file in with a BufferedReader, then going line by line with the readLine() method will automatically exclude any line-termination characters.
What is the difference between Python's readline() and Java's Scanner class method nextLine()?
nextLine() looks for the next line separator character which could be something other than "\n" as written here:
http://docs.oracle.com/javase/7/docs/api/java/util/Scanner.html#nextLine()
Does the Python readline() method do the same? This is important because my file could have other line separators, but I need to look for specifically the new line character.
Any ideas?
You should test it by yourself.
I've tested it on the console using f.readline() and it reads until \n, even if I have a \r in the line.
>>> f.readline()
'This is a test\n'
>>> f.readline()
'Second line\rwith char\n'
>>> f.readline()
'Third line'
NOTE: Some weird things can happen if you simple print the read line on a python script. But if you use repr(str) you'll see all \n and \r.
First of all you are comparing apple to oranges. Scanner is not the Java equivalent of a python file object. BufferedReader is the equivalent, and in fact if you look at the nextLine method's documentation of BufferedReader:
Reads a line of text. A line is considered to be terminated by any one
of a line feed ('\n'), a carriage return ('\r'), or a carriage return
followed immediately by a linefeed.
Python does this too:
A manner of interpreting text streams in which all of the following
are recognized as ending a line: the Unix end-of-line convention '\n',
the Windows convention '\r\n', and the old Macintosh convention '\r'.
See PEP 278 and PEP 3116, as well as str.splitlines() for an
additional use.
AFAIK python does not provide a public equivalent of Java's Scanner. But there is an (undocumented) re.Scanner which could be used to achieve what you want.
You simply provide a "lexicon" when create an instance and then call the scan method.
Probably the easiest way of achieving what you want is to read the file in chunks, and split it using re.split.
I am working on a school project to build a pseudo terminal and file system. The terminal is scanning System.in and pass the string to controller.
Input to console: abc\r\nabc\r\nabc
Here is the code I tried
Scanner systemIn = Scanner(System.in);
input = systemIn.nextLine();
input = input.replaceAll("\\\\r\\\\n",System.getProperty("line.separator"));
System.out.print(input);
I want java to treat the \r\n I typed to console as a line separator, not actually \ and r.
What it does now is print the input as is.
Desired Ouput:
abc
abc
abc
UPDATE: I tried input = StringEscapeUtils.unescapeJava(input); and it solved the problem.
You need to double-escape the regexes in java (once for the regex backslash, once for the Java string). You dont want a linebreak (/\n/, "\\n"), but a backslash (/\\/) plus a "n": /\\n/, "\\\\n". So this should work:
input.replaceAll("(\\\\r)?\\\\n", System.getProperty("line.separator"));
For a more broad handling of escape sequences see How to unescape a Java string literal in Java?
If your input has the string '\r\n', try this
Scanner systemIn = Scanner(System.in);
input = systemIn.nextLine();
input = input.replaceAll("\\\\r\\\\n",System.getProperty("line.separator"))
For consistent behaviour I would replace \\r with \r and \\n with \n rather than replace \\r\\n with the newline as this will have different behaviour on different systems.
You can do
input = systemIn.nextLine().replaceAll("\\\\r", "\r").replaceAll("\\\\n", "\n");
nextLine() strips of the newline at the end. If you want to add a line separator you can do
input = systemIn.nextLine() + System.getProperty("line.separator");
if you are using println() you don't need to add it back.
System.out.println(systemIn.nextLine()); // prints a new line.
As it was mentioned by r0dney, the Bergi's solution doesn't work.
The ability to use some 3rd party libraries is good, however for a person who studies it is better to know the theory, because not for every problem exists some 3rd party library.
Overload project with tons of 3rd party libraries for tasks which can be solved in one line code makes project bulky and not easy maintainable. Anyway here is what's working:
content.replaceAll("(\\\\r)?\\\\n", System.getProperty("line.separator"));
Unless you are actually typing \ and r and \ and n into the console, you don't need to do this at all: instead you have a major misunderstanding. The CR character is represented in a String as \r but it consists of only one byte with the hex value 0xD. And if you are typing backslashes into the console, the simple answer is "don't". Just hit the Enter key: that's what it's for. It will transmit the CR byte into your code.
I'm trying to scan a file that has the DOS ^M as end-of-line using something like:
Scanner file = new Scanner(new File(saveToFilePath)).useDelimiter("(?=\^M)")
In other words, I want to read the text line by line but also keep the ^M that marks the end of the line. This would be easy with \n but I'm not good with regexes and the DOS end-of-line is driving me crazy.
After some research I finally got it. The following is the correct regex for finding and keeping ^M. I didn't know that it meant CTRL-M, so some of your responses helped with that. For some reason, the "M" is not included in the regex and I'm not sure why it works, but it does. This gives us a delimiter for lines that includes the delimiter (with a lookahead regex) when searching for the elusive "^M".
Scanner file = new Scanner(source).useDelimiter("(?=\p{Cntrl})")
Thank you, everyone.