TL;DR
What are the design decisions behind Matcher's API?
Background
Matcher has a behaviour that I didn't expect and for which I can't find a good reason. The API documentation says:
Once created, a matcher can be used to perform three different kinds of match operations:
[...]
Each of these methods returns a boolean indicating success or failure. More information about a successful match can be obtained by querying the state of the matcher.
What the API documentation further says is:
The explicit state of a matcher is initially undefined; attempting to query any part of it before a successful match will cause an IllegalStateException to be thrown.
Example
String s = "foo=23,bar=42";
Pattern p = Pattern.compile("foo=(?<foo>[0-9]*),bar=(?<bar>[0-9]*)");
Matcher matcher = p.matcher(s);
System.out.println(matcher.group("foo")); // (1)
System.out.println(matcher.group("bar"));
This code throws a
java.lang.IllegalStateException: No match found
at (1). To get around this, it is necessary to call matches() or other methods that bring the Matcher into a state that allows group(). The following works:
String s = "foo=23,bar=42";
Pattern p = Pattern.compile("foo=(?<foo>[0-9]*),bar=(?<bar>[0-9]*)");
Matcher matcher = p.matcher(s);
matcher.matches(); // (2)
System.out.println(matcher.group("foo"));
System.out.println(matcher.group("bar"));
Adding the call to matches() at (2) sets the Matcher into the proper state to call group().
Question, probably not constructive
Why is this API designed like this? Why not automatically match when the Matcher is build with Patter.matcher(String)?
Actually, you misunderstood the documentation. Take a 2nd look at the statement you quoted: -
attempting to query any part of it before a successful match will cause an
IllegalStateException to be thrown.
A matcher may throw IllegalStateException on accessing matcher.group() if no match was found.
So, you need to use following test, to actually initiate the matching process: -
- matcher.matches() //Or
- matcher.find()
The below code: -
Matcher matcher = pattern.matcher();
Just creates a matcher instance. This will not actually match a string. Even if there was a successful match.
So, you need to check the following condition, to check for successful matches: -
if (matcher.matches()) {
// Then use `matcher.group()`
}
And if the condition in the if returns false, that means nothing was matched. So, if you use matcher.group() without checking this condition, you will get IllegalStateException if the match was not found.
Suppose, if Matcher was designed the way you are saying, then you would have to do a null check to check whether a match was found or not, to call matcher.group(), like this: -
The way you think should have been done:-
// Suppose this returned the matched string
Matcher matcher = pattern.matcher(s);
// Need to check whether there was actually a match
if (matcher != null) { // Prints only the first match
System.out.println(matcher.group());
}
But, what if, you want to print any further matches, since a pattern can be matched multiple times in a String, for that, there should be a way to tell the matcher to find the next match. But the null check would not be able to do that. For that you would have to move your matcher forward to match the next String. So, there are various methods defined in Matcher class to serve the purpose. The matcher.find() method matches the String till all the matches is found.
There are other methods also, that match the string in a different way, that depends on you how you want to match. So its ultimately on Matcher class to do the matching against the string. Pattern class just creates a pattern to match against. If the Pattern.matcher() were to match the pattern, then there has to be some way to define various ways to match, as matching can be in different ways. So, there comes the need of Matcher class.
So, the way it actually is: -
Matcher matcher = pattern.matcher(s);
// Finds all the matches until found by moving the `matcher` forward
while(matcher.find()) {
System.out.println(matcher.group());
}
So, if there are 4 matches found in the string, your first way, would print only the first one, while the 2nd way will print all the matches, by moving the matcher forward to match the next pattern.
I Hope that makes it clear.
The documentation of Matcher class describes the use of the three methods it provides, which says: -
A matcher is created from a pattern by invoking the pattern's matcher
method. Once created, a matcher can be used to perform three different
kinds of match operations:
The matches method attempts to match the entire input sequence
against the pattern.
The lookingAt method attempts to match the input sequence, starting
at the beginning, against the pattern.
The find method scans the input sequence looking for the next
subsequence that matches the pattern.
Unfortunately, I have not been able find any other official sources, saying explicitly Why and How of this issue.
My answer is very similar to Rohit Jain's but includes some reasons why the 'extra' step is necessary.
java.util.regex implementation
The line:
Pattern p = Pattern.compile("foo=(?<foo>[0-9]*),bar=(?<bar>[0-9]*)");
causes a new Pattern object to be allocated, and it internally stores a structure representing the RE - information such as a choice of characters, groups, sequences, greedy vs. non-greedy, repeats and so on.
This pattern is stateless and immutable, so it can be reused, is multi-theadable and optimizes well.
The lines:
String s = "foo=23,bar=42";
Matcher matcher = p.matcher(s);
returns a new Matcher object for the Pattern and String - one that has not yet read the String. Matcher is really just a state machine's state, where the state machine is the Pattern.
The matching can be run by stepping the state machine through the matching process using the following API:
lookingAt(): Attempts to match the input sequence, starting at the beginning, against the pattern
find(): Scans the input sequence looking for the next subsequence that matches the pattern.
In both cases, the intermediate state can be read using the start(), end(), and group() methods.
Benefits of this approach
Why would anyone want to do step through the parsing?
Get values from groups that have quantification greater than 1 (i.e. groups that repeat and end up matching more than once). For example in the trivial RE below that parses variable assignments:
Pattern p = new Pattern("([a-z]=([0-9]+);)+");
Matcher m = p.matcher("a=1;b=2;x=3;");
m.matches();
System.out.println(m.group(2)); // Only matches value for x ('3') - not the other values
See the section on "Group name" in "Groups and capturing" the JavaDoc on Pattern
The developer can use the RE as a lexer and the developer can bind the lexed tokens to a parser. In practice, this would work for simple domain languages, but regular expressions are probably not the way to go for a full-blown computer language. EDIT This is partly related to the previous reason, but it can frequently be easier and more efficient to create the parse tree processing the text than lexing all the input first.
(For the brave-hearted) you can debug REs and find out which subsequence is failing to match (or incorrectly matching).
However, on most occasions you do not need to step the state machine through the matching, so there is a convenience method (matches) which runs the pattern matching to completion.
If a matcher would automatically match the input string, that would be wasted effort in case you wish to find the pattern.
A matcher can be used to check if the pattern matches() the input string, and it can be used to find() the pattern in the input string (even repeatedly to find all matching substrings). Until you call one of these two methods, the matcher does not know what test you want to perform, so it cannot give you any matched groups. Even if you do call one of these methods, the call may fail - the pattern is not found - and in that case a call to group must fail as well.
This is expected and documented.
The reason is that .matches() returns a boolean indicating if there was a match. If there was a match, then you can call .group(...) meaningfully. Otherwise, if there's no match, a call to .group(...) makes no sense. Therefore, you should not be allowed to call .group(...) before calling matches().
The correct way to use a matcher is something like the following:
Matcher m = p.matcher(s);
if (m.matches()) {
...println(matcher.group("foo"));
...
}
My guess is the design decision was based on having queries that had clear, well defined semantics that didn't conflate existence with match properties.
Consider this: what would you expect Matcher queries to return if the matcher has not successfully matched something?
Let's first consider group(). If we haven't successfully matched something, Matcher shouldn't return the empty string, as it hasn't matched the empty string. We could return null at this point.
Ok, now let's consider start() and end(). Each return int. What int value would be valid in this case? Certainly no positive number. What negative number would be appropriate? -1?
Given all this, a user is still going to have to check return values for every query to verify if a match occurred or not. Alternatively, you could check to see if it matches successfully outright, and if successful, the query semantics all have well-defined meaning. If not, the user gets consistent behaviour no matter which angle is queried.
I'll grant that re-using IllegalStateException may not have resulted in the best description of the error condition. But if we were to rename/subclass IllegalStateException to NoSuccessfulMatchException, one should be able to appreciate how the current design enforces query consistency and encourages the user to use queries that have semantics that are known to be defined at the time of asking.
TL;DR: What is value of asking the specific cause of death of a living organism?
You need to check the return value of matcher.matches(). It will return true when a match was found, false otherwise.
if (matcher.matches()) {
System.out.println(matcher.group("foo"));
System.out.println(matcher.group("bar"));
}
If matcher.matches() does not find a match and you call matcher.group(...), you'll still get an IllegalStateException. That's exactly what the documentation says:
The explicit state of a matcher is initially undefined; attempting to query any part of it before a successful match will cause an IllegalStateException to be thrown.
When matcher.match() returns false, no successful match has been found and it doesn't make a lot of sense to get information on the match by calling for example group().
Related
I noticed that when I match a regular expression like the following one on a text it is a lot slower than the one without preceeding and trailing (.*) parts. I did the same on perl and found that for perl it hardly makes a difference. Is there any way to optimize the original regular expression "(.*)someRegex(.*)" for java?
Pattern p = Pattern.compile("(.*)someRegex(.*)");
Matcher m = p.matcher("some text");
m.matches();
Pattern p = Pattern.compile("someRegex");
Matcher m = p.matcher("some text");
m.matches();
Edit:
Here is a concrete example:
(.*?)<b>\s*([^<]*)\s*<\/b>(.*)
Your best bet is to skip trying to match the front and end of the string at all. You must do that if you use the matches() method, but you don't if you use the find() method. That's probably what you want instead.
Pattern p = Pattern.compile("<b>\\s*([^<]*)\\s*<\\/b>");
Matcher m = p.matcher("some <b>text</b>");
m.find();
You can use start() and end() to find the indexes within the source string containing the match. You can use group() to find the contents of the () capture within the match (i.e., the text inside the bold tag.
In my experience, using regular expressions to process HTML is very fragile and works well in only the most trivial cases. You might have better luck using a full blown XML parser instead, but if this is one of those trivial cases, have at it.
Original Answer: Here is my original answer sharing why a .* at the beginning of a match will perform so badly.
The problem with using .* at the front is that it will cause lots of backtracking in your match. For example, consider the following:
Pattern p = Pattern.compile("(.*)ab(.*)");
Matcher m = p.matcher("aaabaaa");
m.matches();
The match will proceed like this:
The matcher will attempt to suck the whole string, "aaabaaa", into the first .*, but then tries to match a and fails.
The matcher will back up and match "aaabaa", then tries to match a and succeeds, but tries to match b and fails.
The matcher will back up and match "aaaba", then tries to match a and succeeds, but tries to match b and fails.
The matcher will back up and match "aaab", then tries to match a and succeeds, but tries to match b and fails.
The matcher will back up and match "aaa", then tries to match a and fails.
The matcher will back up and match "aa", then tries to match a and succeeds, tries b and succeeds, and then matches "aaa" to the final .*. Success.
You want to avoid a really broad match toward the beginning of your pattern matches whenever possible. Without knowing your actual problem, it would be very difficult to suggest something better.
Update: Anirudha suggests using (.*?)ab(.*) as a possible fix to avoid backtracking. This will short circuit backtracking to some extent, but at the cost of trying to apply the next match on each try. So now, consider the following:
Pattern p = Pattern.compile("(.*?)ab(.*)");
Matcher m = p.matcher("aaabaaa");
m.matches();
It will proceed like this:
The matcher will attempt to match nothing, "", into the first .*?, tries to match a and succeeds, but fails to match b.
The matcher will attempt to match the first letter, "a", into the first .*?, tries to match a and succeeds, but fails to match b.
The matcher will attempt to match the first two letters, "aa", into the first .*?, tries to match a and succeeds, tries to match b and succeeds, and then slurps up the rest into .*, "aaa". Success.
There aren't any backtracks this time, but we still have a more complicated matching process for each forward move within .*?. This may be a performance gain for a particular match or a loss if iterating through the match forward happens to be slower.
This also changes the way the match will proceed. The .* match is greedy and tries to match as much as possible where as .*? is more conservative.
For example, the string "aaabaaabaaa".
The first pattern, (.*)ab(.*) will match "aaabaa" to the first capture and "aaa" to the second.
The second pattern, (.*?)ab(.*) will match "aa" to the first capture and "aaabaaa" to the second.
Instead of doing "(.*)someRegex(.*)" , why not just split the string on "someRegex" and get the parts from the resulting array ? This will give you the same result, but much faster and simpler. Java supports splitting by regex if you need it - http://www.regular-expressions.info/java.html
. matches every character
instead of . try limiting your search by using classes like \w or \s.
But I dont' guarantee that it would run fast.
It all depends on the amount of text you are matching!
I'm doing a project which requires numerical pattern matching.
For example i want to know whether Value = 1331 is a part of 680+651 = 1331 or not, i.e. i want to match 1331 with 680+651 = 1331 or any other given string.
I'm trying pattern matching in java for the first time and i could not succeed. Below is my code snippet.
String REGEX1=s1; //s1 is '1331'
pattern = Pattern.compile(REGEX1);
matcher = pattern.matcher(line_out); //line_out is for ex. 680+651 = 1331
System.out.println("lookingAt(): "+matcher.lookingAt());
System.out.println("matches(): "+matcher.matches());
It is returning false all the times.
Pls help me.
matches() requires that the pattern be a complete match, not a partial.
You either need to change your pattern to something like .*= 1331$ or use the find() method which will do a partial match.
The matches method requires a perfect, full exact match. Since there is more text in 680+651=1331 than what is matched by the regex 1331, matches returns false.
As I pointed out in Brian's post, you need to be careful in your regex to ensure that a regex of 1331 does not match the number 213312 unless that is what you want.
matches() is the wrong method for this, use find().
http://download.oracle.com/javase/1.4.2/docs/api/java/util/regex/Matcher.html says:
public boolean matches()
Attempts to match the entire input sequence against the pattern.
and
public boolean find()
Attempts to find the next subsequence of the input sequence that matches the pattern.
I can't figure out why this regex doesn't work, I've tested it in php and other regex engines where it works fine and matches ",AA,".
Pattern p = Pattern.compile("(^|,)AA(,|$)");
Matcher m = p.matcher("A,B,AA,C,D");
//assigns as false
boolean matches = m.matches();
Side note: I have a split/array binary search method for doing an IN_SET / NOT_IN_SET search against the string. This is just an example I need to get working before implementing regex as another comparing option.
matches() validates the entire string. You want to use find() instead.
From the API:
matches()
Attempts to match the entire region against the pattern.
-- http://download.oracle.com/javase/6/docs/api/java/util/regex/Matcher.html#matches()
and:
find()
Attempts to find the next subsequence of the input sequence that matches the pattern.
-- http://download.oracle.com/javase/6/docs/api/java/util/regex/Matcher.html#find()
Matcher matches the entire region against the pattern. Use find().
I have been using the java.util.regex.* classes for Regular Expression in Java and all good so far. But today I have a different requirement. For example consider the pattern to be "aabb". Now if the input String is aa it will definitely not match, however there is still possibility that if I append bb it becomes aabb and it matches. However if I would have started with cc, no matter what I append it will never match.
I have explored the Pattern and Matcher class but didn't find any way of achieving this.
The input will come from user and system have to wait till pattern matches or it will never match irrespective of any input further.
Any clue?
Thanks.
You should have looked more closely at the Matcher API; the hitEnd() method works exactly as you described:
import java.util.regex.*;
public class Test
{
public static void main(String[] args) throws Exception
{
String[] ss = { "aabb", "aa", "cc", "aac" };
Pattern p = Pattern.compile("aabb");
Matcher m = p.matcher("");
for (String s : ss) {
m.reset(s);
if (m.matches()) {
System.out.printf("%-4s : match%n", s);
}
else if (m.hitEnd()) {
System.out.printf("%-4s : partial match%n", s);
}
else {
System.out.printf("%-4s : no match%n", s);
}
}
}
}
output:
aabb : match
aa : partial match
cc : no match
aac : no match
As far as I know, Java is the only language that exposes this functionality. There's also the requireEnd() method, which tells you if more input could turn a match into a non-match, but I don't think it's relevant in your case.
Both methods were added to support the Scanner class, so it can apply regexes to a stream without requiring the whole stream to be read into memory.
Pattern p = Pattern.compile(expr);
Matcher m = p.matcher(string);
m.find();
So you want to know not whether a String s matches the regex, but whether there might be a longer String starting with s that would match? Sorry, Regexes can't help you there because you get no access to the internal state of the matcher; you only get the boolean result and any groups you have defined, so you never know why a match failed.
If you're willing to hack the JDK libraries, you can extend (or probably fork) java.util.regex and give out more information about the matching process. If the match failed because the input was 'used up' the answer would be true; if it failed because of character discrimination or other checks it would be false. That seems like a lot of work though, because your problem is completely the opposite of what regexes are supposed to do.
Another option: maybe you can simply redefine the task so that you can treat the input as the regexp and match aabb against *aa.**? You have to be careful about regex metacharacters, though.
For the example you give you could try to use an anti-pattern to disqualify invalid results. For example "^[^a]" would tell you you're input "c..." can't match your example pattern of "aabb".
Depending on your pattern you may be able to break it up into smaller patterns to check and use multiple matchers and then set their bounds as one match occurs and you move to the next. This approach may work but if you're pattern is complex and can have variable length sub-parts you may end up reimplementing part of the matcher in your own code to adjust the possible bounds of the match to make it more or less greedy. A pseudo-code general idea of this would be:
boolean match(String input, Matcher[] subpatterns, int matchStart, int matchEnd){
matcher = next matcher in list;
int stop = matchend;
while(true){
if matcher.matches input from matchstart -> matchend{
if match(input, subpatterns, end of current match, end of string){
return true;
}else{
//make this match less greedy
stop--;
}
}else{
//no match
return false;
}
}
}
You could then merge this idea with the anti-patterns, and have anti-subpatterns and after each subpattern match you check the next anti-pattern, if it matches you know you have failed, otherwise continue the matching pattern. You would likely want to return something like an enum instead of a boolean (i.e. ALL_MATCHED, PARTIAL_MATCH, ANTI_PATTERN_MATCH, ...)
Again depending on the complexity of your actual pattern that you are trying to match writing the appropriate sub patterns / anti-pattern may be difficult if not impossible.
One way to do this is to parse your regex into a sequence of sub-regexes, and then reassemble them in a way that allows you to do partial matches; e.g. "abc" has 3 sub-regexes "a", "b" and "c" which you can then reassemble as "a(b*(c)?)?".
Things get more complicated when the input regex contains alternation and groups, but the same general approach should work.
The problem with this approach is that the resulting regex is more complicated, and could potentially lead to excessive backtracking for complex input regexes.
If you make each character of the regex optional and relax the multiplicity constraints, you kinda get what you want. Example if you have a matching pattern "aa(abc)+bbbb", you can have a 'possible match' pattern 'a?a?(a?b?c?)*b?b?b?b?'.
This mechanical way of producing possible-match pattern does not cover advanced constructs like forward and backward refs though.
You might be able to accomplish this with a state machine (http://en.wikipedia.org/wiki/State_machine). Have your states/transitions represent valid input and one error state. You can then feed the state machine one character (possibly substring depending on your data) at a time. At any point you can check if your state machine is in the error state. If it is not in the error state then you know that future input may still match. If it is in the error state then you know something previously failed and any future input will not make the string valid.
I've been struggling with doing some relatively straightforward regular expression matching in Java 1.4.2. I'm much more comfortable with the Perl way of doing things. Here's what's going on:
I am attempting to match /^<foo>/ from "<foo><bar>"
I try:
Pattern myPattern= Pattern.compile("^<foo>");
Matcher myMatcher= myPattern.matcher("<foo><bar>");
System.out.println(myMatcher.matches());
And I get "false"
I am used to saying:
print "<foo><bar>" =~ /^<foo>/;
which does indeed return true.
After much searching and experimentation, I discovered this which said:
"The String method further optimizes its search criteria by placing an invisible ^ before the pattern and a $ after it."
When I tried:
Pattern myPattern= Pattern.compile("^<foo>.*");
Matcher myMatcher= myPattern.matcher("<foo><bar>");
System.out.println(myMatcher.matches());
then it returns the expected true. I do not want that pattern though. The terminating .* should not be necessary.
Then I discovered the Matcher.useAnchoringBounds(boolean) method. I thought that expressly telling it to not use the anchoring bounds would work. It did not. I tried issuing a
myMatcher.reset();
in case I needed to flush it after turning the attribute off. No luck. Subsequently calling .matches() still returns false.
What have I overlooked?
Edit:
Well, that was easy, thanks.
Use the Matcher find method (instead of the matches method)
Matcher.useAnchoringBounds() was added in JDK1.5 so if you are using 1.4, I'm not sure that it would help you even if it did work (notice the #since 1.5 in the Javadocs).
The Javadocs for Matcher also state that the match() method:
Attempts to match the entire region against the pattern.
(emphasis mine)
Which explains why you only got .matches() == true when you changed the pattern to end with .*.
To match against the region starting at the beginning, but not necessarily requiring that the entire region be matched, use either the find() or lookingAt() methods.
If you examine the "match", what part of the input string do you expect to find?
In other words,
Matcher myMatcher= myPattern.matcher("<foo><bar>");
if (myMatcher.matches()) {
System.out.println(myMatcher.group(0));
}
… should print what?
If you are expecting it to print just "<foo>", use the find() method on Matcher instead of matches(). If you really want to find matches when the input starts with "<foo>", then you need to explicitly indicate that with a '^'.
If you are expecting it to match "<foo><bar>", you need to include the trailing ".*".