I am trying to improve the performance of some code. It looks something like this:
public boolean isImportant(String token) {
for (Pattern pattern : patterns) {
return pattern.matches(token).find();
}
}
What I noticed is that many of the Patterns seem to be simple string literals with no regular expression constructs. So I want to simply store these in a separate list (importantList) and do an equality test instead of performing a more expensive pattern match, such as follows:
public boolean isImportant(String token) {
if (importantList.contains(token)) return true;
for (Pattern pattern : patterns) {
return pattern.matches(token).find();
}
}
How do I programmatically determine if a particular string contains no regular expression constructs?
Edit:
I should add that the answer doesn't need to be performance-sensitive. (i.e. regular expressions can be used) I'm mainly concerned with the performance of isImportant() because it's called millions of times, while the initialzation of the patterns is only done once.
I normally hate answers that say this but...
Don't do that.
It probably won't make the code run faster, in fact it might even cause the program to take more time.
if you really need to optimize your code, there are likely much mush much more effective places where you can go.
It's going to be difficult. You can check for the non-presence of any regex metacharacters; that should be a good approximation:
Pattern regex = Pattern.compile("[$^()\\[\\]{}.*+?\\\\]");
Matcher regexMatcher = regex.matcher(subjectString);
regexIsLikely = regexMatcher.find();
Whether it's worth it is another question. Are you sure a regex match is slower than a list lookup (especially since you'll be doing a regex match after that in many cases anyway)? I'd bet it's much faster to just keep the regex match.
There is no way to determine it as every regex pattern is nothing else than a string. Furthermore there is nearly no performance difference as regex is smart nowadays and I'm pretty sure, if the pattern and source lengths are the same, equity check is the first that will be done
This is wrong
for (Pattern pattern : patterns)
you should create one big regex that ORs all patterns; then for each input you only match once.
Related
Given a String containing a comma delimited list representing a proper noun & category/description pair, what are the pros & cons of using String.split() versus Pattern & Matcher approach to find a particular proper noun and extract the associated category/description pair?
The haystack String format will not change. It will always contain comma delimited data in the form of
PROPER_NOUN|CATEGORY/DESCRIPTION
Common variables for both approaches:
String haystack="EARTH|PLANET/COMFORTABLE,MARS|PLANET/HARDTOBREATHE,PLUTO|DWARF_PLANET/FARAWAY";
String needle="PLUTO";
String result=null;
Using String.split():
for (String current : haystack.split(","))
if (current.contains(needle))
{
result=current.split("\\|")[1]);
break; // *edit* Not part of original code - added in response to comment from Pshemo
{
Using Pattern & Matcher:
Pattern pattern = pattern.compile("(" +needle+ "\|)(\w+/\w+)");
Matcher matches = pattern.matcher(haystack);
if (matches.find())
result=matches.group(2);
Both approaches provide the information I require.
I'm wondering if any reason exists to choose one over the other. I am not currently using Pattern & Matcher within my project so this approach will require imports from java.util.regex
And, of course, if there is an objectively 'better' way to parse the information I will welcome your input.
Thank you for your time!
Conclusion
I've opted for the Pattern/Matcher approach. While a little tricky to read w/the regex, it is faster than .split()/.contains()/.split() and, more importantly to me, captures the first match only.
For what it is worth, here are the results of my imperfect benchmark tests, in nanoseconds, after 100,000 iterations:
.split()/.contains()/.split
304,212,973
Pattern/Matcher w/ Pattern.compile() invoked for each iteration
230,511,000
Pattern/Matcher w/Pattern.compile() invoked prior to iteration
111,545,646
In a small case such as this, it won't matter that much. However, if you have extremely large strings, it may be beneficial to use Pattern/Matcher directly.
Most string functions that use regular expressions (such as matches(), split(), replaceAll(), etc.) makes use of Matcher/Pattern directly. Thus it will create a Matcher object every time, causing inefficiency when used in a large loop.
Thus if you really want speed, you can use Matcher/Pattern directly and ideally only create a single Matcher object.
There are no advantages to using pattern/matcher in cases where the manipulation to be done is as simple as this.
You can look at String.split() as a convenience method that leverages many of the same functionalities you use when you use a pattern/matcher directly.
When you need to do more complex matching/manipulation, use a pattern/matcher, but when String.split() meets your needs, the obvious advantage to using it is that it reduces code complexity considerably - and I can think of no good reason to pass this advantage up.
I would say that the split() version is much better here due to the following reasons:
The split() code is very clear, and it is easy to see what it does. The regex version demands much more analysis.
Regular expressions are more complex, and therefore the code becomes more error-prone.
I'm looking for how to create a regular expression, which is 100% equivalent to the "contains" method in the String class. Basically, I have thousands of phrases that I'm searching for, and from what I understand it is much better for performance reasons to compile the regular expression once and use it multiple times, vs calling "mystring.contains(testString)" over and over again on different "mystring" values, with the same testString values.
Edit: to expand on my question... I will have many thousands of "testString" values, and I don't want to have to convert those to a format that the regular expression mechanism understands. I just want to be able to directly pass in a phrase that users enter, and see if it is found in whatever value "mystring" happens to contain. "testString" will not change it's value ever, but there will be thousands of them so that is why I was thinking of creating the matcher object and re-using it over and over etc. (Obviously my regexp skills are not up to snuff)
You can use the LITERAL flag when compiling your pattern to tell the engine you're using a literal string, e.g.:
Pattern p = Pattern.compile(yourString, Pattern.LITERAL);
But are you really sure that doing that and then reusing the result is faster than just String#contains? Enough to make the complexity worth it?
Well you could use Pattern.quote to get a "piece of regular expression" for each input string. Do any of your terms contain line breaks? If so, that could at least make life slightly trickier, though far from impossible.
Anyway, you'd basically just join the quoted terms together as:
Pattern pattern = Pattern.compile("quoted1|quoted2|quoted3|...");
You might want to use Guava's Joiner to easily join the quoted strings together, although obviously it's not terribly hard to do manually.
However, I would try this and then test whether it's actually more efficient than just calling contains. Have you already got a benchmark which shows that contains is too slow?
My question is related to Regular Expressions in Java, and in particular, multiple matches for a given search pattern. All of the info i need to get is on 1 line and it contains an alias (e.g. SA) which maps to an IP address. Each one is separated by a comma. I need to extract each one.
SA "239.255.252.1", SB "239.255.252.2", SC "239.255.252.3", SD "239.255.252.4"
My Reg Ex looks like this:
Pattern alias = Pattern.compile("(\\S+)\\s+\"(\\d+\\.\\d+\\.\\d+\\.\\d+)\"");
Matcher match = alias.matcher(lineInFile)
while(match.find()) {
// do something
}
This works but I'm not totally happy with it because since introducing this small piece of code, my program has slowed down a bit (< 1 sec) but enough to notice a difference.
So my question is, am I going about this in the correct manner? Is there a more efficient or possibly lightweight solution without the need for a while(match) loop? and/or Pattern/Matcher classes?
If the line may not contain anything except that alias definition, then using .match() instead of .find() might speed up the searching on non-matches.
You can improve your regex to: "(\\S{2})\\s+\"((\\d{1,3}\\.){3}\\d{1,3})\"" by specifying an IP address more explicitly.
Try out the performance of using a StringTokenizer. It does not use regular expressions.
(If you are concerned about using a legacy class, then take a look at its source and see how it is done.)
StringTokenizer st = new StringTokenizer(lineInFile, " ,\"");
while(st.hasMoreTokens()){
String key = st.nextToken();
String ip = st.nextToken();
System.out.println(key + " ip: " + ip);
}
I don't know if this will yield a big performance benefit, but you could also first do
string.split(", ") // separate groups
and then
string.split(" ?\"") // separate alias from IP address
on the matches.
Precompiling and reusing the Pattern object is (IMO) likely to be the most effective optimization. Pattern compilation is potentially an expensive step.
Reusing the Matcher instance (e.g. using reset(CharSequence)) might help, but I doubt that it will make much difference.
The regex itself cannot be optimized significantly. One possible speedup would be to replace (\d+\.\d+\.\d+\.\d+) with ([0-9\.]+). This might help because it reduces the number of potential backtrack points ... but you'd need to do some experiments to be sure. And the obvious downside is that it matches character sequences that are not valid IP addresses.
If you`re noticing a difference of < 1 sec on that piece of code, then your input string must contain around a million (ot at least some 100k) of entries. I think that's a pretty fair performance and I cannot see how you could significantly optimize that without writing your own specialized parser.
I'm afraid your code looks pretty efficient already.
Here's my version:
Matcher match = Pattern
.compile("(\\w+)\\s+\"(\\d+\\.\\d+\\.\\d+\\.\\d+)\"")
.matcher(lineInFile);
while(match.find()) {
//do something
}
There are two micro-optimizations:
No need to keep pattern in an extra
variable, inlined that
For the alias, search for word
characters, not non-space characters
Actually, if you do a lot of processing like this and the pattern never changes, you should keep the compiled pattern in a constant:
private static final Pattern PATTERN = Pattern
.compile("(\\w+)\\s+\"(\\d+\\.\\d+\\.\\d+\\.\\d+)\"");
Matcher match = PATTERN.matcher(lineInFile);
while(match.find()) {
//do something
}
Update: I took some time on RegExr to come up with a much more specific pattern, which should only detect valid IP addresses as a bonus. I know it's ugly as hell, but my guess is that it's pretty efficient, as it eliminates most of the backtracking:
([A-Z]+)\s*\"((?:1[0-9]{2}|2(?:(?:5[0-5]|[0-9]{2})|[0-9]{1,2})\.)
{3}(?:1[0-9]{2}|2(?:5[0-5]|[0-9]{2})|[0-9]{1,2}))
(Wrapped for readability, all back-slashes need to be escaped in java, but you can test it on RegExr as it is with the OP's test string)
I am doing some fairly extensive string manipulations using regular expressions in Java. Currently, I have many blocks of code that look something like:
Matcher m = Pattern.compile("some pattern").matcher(text);
StringBuilder b = new StringBuilder();
int prevMatchIx = 0;
while (m.find()) {
b.append(text.substring(prevMatchIx, m.start()));
String matchingText = m.group(); //sometimes group(n)
//manipulate the matching text
b.append(matchingText);
prevMatchIx = m.end();
}
text = b.toString()+text.substring(prevMatchIx);
My question is which of the two alternatives is more efficient (primarily time, but space to some extent):
1) Keep many existing blocks as above (assuming there isn't a better way to handle such blocks -- I can't use a simple replaceAll() because the groups must be operated on).
2) Consolidate the blocks into one big block. Use a "some pattern" that is the combination of all the old blocks' patterns using the |/alternation operator. Then, use if/else if within the loop to handle each of the matching patterns.
Thank you for your help!
If the order in which the replacements are made matters, you would have to be careful when using technique #1. Allow me to give an example: If I want to format a String so it is suitable for inclusion in XML, I have to first replace all & with & and then make the other replacements (like < to <). Using technique #2, you would not have to worry about this because you are making all the replacements in one pass.
In terms of performance, I think #2 would be quicker because you would be doing less String concatenations. As always, you could implement both techniques and record their speed and memory consumption to find out for certain. :)
I'd suggest caching the patterns and having a method that uses the cache.
Patterns are expensive to compile so at least you will only compile them once and there is code reuse in using the same method for each instance. Shame about the lack of closures though as that would make things a lot cleaner.
private static Map<String, Pattern> patterns = new HashMap<String, Pattern>();
static Pattern findPattern(String patStr) {
if (! patterns.containsKey(patStr))
patterns.put(patStr, Pattern.compile(patStr));
return patterns.get(patStr);
}
public interface MatchProcessor {
public void process(String field);
}
public static void processMatches(String text, String pat, MatchProcessor processor) {
Matcher m = findPattern(pat).matcher(text);
int startInd = 0;
while (m.find(startInd)) {
processor.process(m.group());
startInd = m.end();
}
}
Last time I was in your position I used a product called jflex.
Java's regex doesn't provide the traditional O(N log M) performance guarantees of true regular expression engines (for input strings of length N, and patterns of length M). Instead it inherits from its perl roots exponential time for some patterns. Unfortunately these pathological patterns, while rare in normal use, are all too common when combining regexes as you propose to do (I can attest to this from personal experience).
Consequently, my advice is to either:
a) pre-compile your patterns as "static final Pattern" constants, so they will be initialized once during [cinit]; or
b) switch to a lexer package such as jflex, which will provide a more declarative, and far more readable, syntax to approach this sort of cascading/sequential regex processing; and
c) seriously consider using a parser generator package. My current favourite is Beaver, but CUP is also a good option. Both of these are excellent tools and I highly recommend both of them, and as they both sit on top of jflex you can add them as/when you need them.
That being said, if you haven't used a parser-generator before and you are in a hurry, it will be easier to get up to speed with JavaCC. Not as powerful as Beaver/CUP but its parsing model is easier to understand.
Whatever you do, please don't use Antlr. It is very fashionable, and has great cheerleaders, but its online documentation sucks, its syntax is awkward, its performance is poor, and its scannerless design makes several common simple cases painful to handle. You would be better off using an abomination like sablecc(v1).
Note: Yes I have used everything I have mentioned above, and more besides; so this advice comes from personal experience.
First, does this need to be efficient? If not, don't bother -- complexification won't help code maintainability.
Assuming it does, doing them separately is usually the most efficient. This is especially true if there are large blocks of text in the expressions: without alternation this can be used to speed up matching, with it can't help at all.
If performance is really critical, you can code it several ways and test with sample data.
Option #2 is almost certainly the better way to go, assuming it isn't too difficult to combine the regexes. And you don't have to implement it from scratch, either; the lower-level API that replaceAll() is built on (i.e., appendReplacement() and appendTail()), is also available for your use.
Taking the example that #mangst used, here's how you might process some text to be inserted into an XML document:
import java.util.regex.*;
public class Test
{
public static void main(String[] args)
{
String test_in = "One < two & four > three.";
Pattern p = Pattern.compile("(&)|(<)|(>)");
Matcher m = p.matcher(test_in);
StringBuffer sb = new StringBuffer(); // (1)
while (m.find())
{
String repl = m.start(1) != -1 ? "&" :
m.start(2) != -1 ? "<" :
m.start(3) != -1 ? ">" : "";
m.appendReplacement(sb, ""); // (2)
sb.append(repl);
}
m.appendTail(sb);
System.out.println(sb.toString());
}
}
In this very simple example, all I need to know about each match is which capture group participated in it, which I find out by means of the start(n) method. But you can use the group() or group(n) method to examine the matched text, as you mentioned in the question.
Note (1) As of JDK 1.6, we have to use a StringBuffer here because StringBuilder didn't exist yet when the Matcher class was written. JDK 1.7 will add support for StringBuilder, plus some other improvements.
Note (2) appendReplacement(StringBuffer, String) processes the String argument to replace any $n sequence with the contents of the n'th capture group. We don't want that to happen, so we pass it an empty string and then append() the replacement string ourselves.
Are there known difference(s) between String.replaceAll() and Matcher.replaceAll() (On a Matcher Object created from a Regex.Pattern) in terms of performance?
Also, what are the high-level API 'ish differences between the both? (Immutability, Handling NULLs, Handling empty strings, etc.)
According to the documentation for String.replaceAll, it has the following to say about calling the method:
An invocation of this method of the
form str.replaceAll(regex, repl)
yields exactly the same result as the
expression
Pattern.compile(regex).matcher(str).replaceAll(repl)
Therefore, it can be expected the performance between invoking the String.replaceAll, and explicitly creating a Matcher and Pattern should be the same.
Edit
As has been pointed out in the comments, the performance difference being non-existent would be true for a single call to replaceAll from String or Matcher, however, if one needs to perform multiple calls to replaceAll, one would expect it to be beneficial to hold onto a compiled Pattern, so the relatively expensive regular expression pattern compilation does not have to be performed every time.
Source code of String.replaceAll():
public String replaceAll(String regex, String replacement) {
return Pattern.compile(regex).matcher(this).replaceAll(replacement);
}
It has to compile the pattern first - if you're going to run it many times with the same pattern on short strings, performance will be much better if you reuse one compiled Pattern.
The main difference is that if you hold onto the Pattern used to produce the Matcher, you can avoid recompiling the regex every time you use it. Going through String, you don't get the ability to "cache" like this.
If you have a different regex every time, using the String class's replaceAll is fine. If you are applying the same regex to many strings, create one Pattern and reuse it.
Immutability / thread safety: compiled Patterns are immutable, Matchers are not. (see Is Java Regex Thread Safe?)
Handling empty strings: replaceAll should handle empty strings gracefully (it won't match an empty input string pattern)
Making coffee, etc.: last I heard, neither String nor Pattern nor Matcher had any API features for that.
edit: as for handling NULLs, the documentation for String and Pattern doesn't explicitly say so, but I suspect they'd throw a NullPointerException since they expect a String.
The implementation of String.replaceAll tells you everything you need to know:
return Pattern.compile(regex).matcher(this).replaceAll(replacement);
(And the docs say the same thing.)
While I haven't checked for caching, I'd certainly expect that compiling a pattern once and keeping a static reference to that would be more efficient than calling Pattern.compile with the same pattern each time. If there's a cache it'll be a small efficiency saving - if there isn't it could be a large one.
The difference is that String.replaceAll() compiles the regex each time it's called. There's no equivalent for .NET's static Regex.Replace() method, which automatically caches the compiled regex. Usually, replaceAll() is something you do only once, but if you're going to be calling it repeatedly with the same regex, especially in a loop, you should create a Pattern object and use the Matcher method.
You can create the Matcher ahead of time, too, and use its reset() method to retarget it for each use:
Matcher m = Pattern.compile(regex).matcher("");
for (String s : targets)
{
System.out.println(m.reset(s).replaceAll(repl));
}
The performance benefit of reusing the Matcher, of course, is nowhere as great as that of reusing the Pattern.
The other answers sufficiently cover the performance part of the OP, but another difference between Matcher::replaceAll and String::replaceAll is also a reason to compile your own Pattern. When you compile a Pattern yourself, there are options like flags to modify how the regex is applied. For example:
Pattern myPattern = Pattern.compile(myRegex, Pattern.CASE_INSENSITIVE);
The Matcher will apply all the flags you set when you call Matcher::replaceAll.
There are other flags you can set as well. Mostly I just wanted to point out that the Pattern and Matcher API has lots of options, and that's the primary reason to go beyond the simple String::replaceAll