We Keep Coding

Different Java Regex matching behavior when using UNICODE_CHARACTER_CLASS flag

I was testing the behavior of the Pattern.UNICODE_CHARACTER_CLASS flag for different punctuation characters and noticed that the matches for grave accent character (U+0060) ` occur differently depending on whether Pattern.UNICODE_CHARACTER_CLASS is used.
For example, see the below code:
public class GraceAccentTest {
public static void main(String args[]) {
Pattern p = Pattern.compile("\\p{Punct}");
Matcher m = p.matcher("`");
System.out.println(m.matches()); // returns true
Pattern p1 = Pattern.compile("\\p{Punct}", Pattern.UNICODE_CHARACTER_CLASS);
Matcher m1 = p1.matcher("`");
System.out.println(m1.matches()); // returns false
}
}
When I don't use Pattern.UNICODE_CHARACTER_CLASS flag grave accent character matches with \p{Punct} character class but when I use the flag it doesn't match. Can someone explain the reasoning for this ?

When you use Pattern p = Pattern.compile("\\p{Punct}");, then \p{Punct} refers to the following 32 characters:
!"#$%&'()*+,-./:;<=>?#[\]^_`{|}~
Reference: the Pattern class.
These 32 characters correspond to the ASCII character set characters 0x21 through 0x7e, excluding letters and digits. They also happen to represent all the non-letter and non-digit symbols on my standard U.S. keyboard (your keyboard may be different, of course).
The grave accent (also known as a backtick) is in that list and on my keyboard.
That is a simple example of a "predefined character class" - and explains why your m.matches() returns true.
When you add the Pattern.UNICODE_CHARACTER_CLASS flag things get more complicated.
As the documentation for this flag explains, it:
Enables the Unicode version of Predefined character classes and POSIX character classes.
and:
When this flag is specified then the (US-ASCII only) Predefined character classes and POSIX character classes are in conformance with Unicode Technical Standard #18: Unicode Regular Expressions Annex C: Compatibility Properties.
Looking at the Annex C referred to above, we find a table showing the "recommended assignments for compatibility property names".
For our property name (punct), the standard recommendation is to use characters defined by this:
\p{gc=Punctuation}
Here, "gc" stands for "general category". Unicode characters are assigned a "general category" value. In this case, that is Punctuation - also abbreviated to P and further broken down into various sub-categories such as Pc for connectors, Pd for dashes, and so on. There is also a catch-all Po for "other punctuation characters".
The grave character is assigned to the Symbol general category in Unicode - and to the Modifier subcategory. You can see that assignment to Sk here.
Contrast that with a character such as the ASCII exclamation mark (also part of our original \p{Punct} list, shown above). For that we can see that the general category assignment is Po.
That explains why the grave is no longer matched when we add the Pattern.UNICODE_CHARACTER_CLASS flag to our original pattern.
It is assigned to a different general category from the punctuation category we are using in our regex.
The obvious next question is why did the grave character not get included in the Unicode Po general category? Why is it in Sk instead?
I do not have a good answer for that - I'm sure there are "historical reasons". It's worth noting, however, that the Sk cateogry includes characters such as the acute accent, the cedilla, the diaeresis, and so on - and (as already noted) our grave accent.
All these are diacritics - typically used in combination with a base letter to alter the pronunciation. So maybe that is the underlying reason.
The grave is a bit of an oddity, perhaps, given it has a historical usage outside of being used as a diacritic.
It may be more relevant to ask how the grave ended up as part of the original ASCII character set, in the first place. Some background about this is provided in the Wikipedia page for the backtick.

Reading the documentation for UNICODE_CHARACTER_CLASS
When this flag is specified then the (US-ASCII only) Predefined
character classes and POSIX character classes are in conformance with
Unicode Technical Standard #18: Unicode Regular Expression Annex C:
Compatibility Properties.
So this is saying that is using US-ASCII only. So if you check the table of characters Punctuation you will check there is a lot of missing chars.
Tables :
https://www.fileformat.info/info/unicode/category/Po/list.htm
https://www.gaijin.at/en/infos/unicode-character-table-punctuation

How to type String content:encoded = "Hello"; in java?

How to type String content:encoded = "Hello"; in java ?
Eclipse keep telling me syntax error on tokens delete these tokens ?
setDescription(String content:encoded) {
_description = content:encoded;
}

Because content:encoded is a syntax error. Name in java only accept letters numbers $ and "_". The rule might allow some other characters but it should be pretty much it. Also a variable cannot start with a number.
To be clear, remove the : from the variable name because : is illegal in a name and might have an other meaning in the language.
Quote from the article below:
Variable names are case-sensitive. A variable's name can be any legal identifier — an unlimited-length sequence of Unicode letters and
digits, beginning with a letter, the dollar sign $, or the
underscore character _. The convention, however, is to always begin
your variable names with a letter, not $ or _. Additionally, the
dollar sign character, by convention, is never used at all. You may
find some situations where auto-generated names will contain the
dollar sign, but your variable names should always avoid using it. A
similar convention exists for the underscore character; while it's
technically legal to begin your variable's name with _, this
practice is discouraged. White space is not permitted.
Subsequent characters may be letters, digits, dollar signs, or underscore characters.
Here read more about it: http://docs.oracle.com/javase/tutorial/java/nutsandbolts/variables.html

if you are creating method setDescription then it whould be:
public void setDescription(String content_encoded) {
_description = content_encoded;
}
Here
public is modifier
void is return type
setDescription is method name
String is parameter type
content_encoded is Variable that is holding string value.

Are all names identifiers?

In the Java Language Specification 6.2 Link
Here is the following code example:
class Test {
public static void main(String[] args) {
Class c = System.out.getClass();
System.out.println(c.toString().length() +
args[0].length() + args.length);
}
}
And it states:
the identifiers Test, main, and the first occurrences of args and c are not names. Rather, they are used in declarations to specify the names of the declared entities. The names String, Class, System.out.getClass, System.out.println, c.toString, args, and args.length appear in the example.
But are the names like Class and String also identifiers? What is an identifier exactly?

An identifier is a type of a token. From the specification of the lexical structure of Java:
3.8. Identifiers
An identifier is an unlimited-length sequence of Java letters and Java digits, the first of which must be a Java letter.
Identifier:
IdentifierChars but not a Keyword or BooleanLiteral or NullLiteral
IdentifierChars:
JavaLetter
IdentifierChars JavaLetterOrDigit
JavaLetter:
any Unicode character that is a Java letter (see below)
JavaLetterOrDigit:
any Unicode character that is a Java letter-or-digit (see below)
A "Java letter" is a character for which the method
Character.isJavaIdentifierStart(int) returns true.
A "Java letter-or-digit" is a character for which the method
Character.isJavaIdentifierPart(int) returns true.
The "Java letters" include uppercase and lowercase ASCII Latin letters
A-Z (\u0041-\u005a), and a-z (\u0061-\u007a), and, for historical
reasons, the ASCII underscore (_, or \u005f) and dollar sign ($, or
\u0024). The $ character should be used only in mechanically generated
source code or, rarely, to access pre-existing names on legacy
systems.
The "Java digits" include the ASCII digits 0-9 (\u0030-\u0039).
Letters and digits may be drawn from the entire Unicode character set,
which supports most writing scripts in use in the world today,
including the large sets for Chinese, Japanese, and Korean. This
allows programmers to use identifiers in their programs that are
written in their native languages.
An identifier cannot have the same spelling (Unicode character
sequence) as a keyword (§3.9), boolean literal (§3.10.3), or the null
literal (§3.10.7), or a compile-time error occurs.

An identifier is a user defined symbol.
It allows the compiler to differentiate between bindings to objects of the same type in the symbol table.

This might answer your 2nd question:
http://www.cafeaulait.org/course/week2/08.html
Identifiers are the names of variables, methods, classes, packages and
interfaces. Unlike literals they are not the things themselves, just
ways of referring to them.

Messed up with Java Declaration

why java constant have strange behaviour (Unicode Character and normal representation).. I mean see below example.
Note : All code is in java language.
char a = '\u0061'; //This is correct
char 'a' = 'a'; //This gives compile time error
char \u0061 = 'a'; //this is correct no error
ch\u0061r a = 'a'; //This too works
ch'a'r a = 'a'; // This really is confusing compile time error
Why last declaration is not works whereas ch\u0061r a='a'; works?

You cannot put literals ('a') in the middle of identifiers.
The line
char 'a' = 'a';
Does not compile because there is no identifier, and you cannot assign one literal to another.
Unicode is permitted, however. It is just hard to read :-)

You can not put literal characters, 'a', in identifiers. You can use unicode, \u0061, though.

This isn't confusing at all. You're randomly scattering single quotes around and expecting them to be irrelevant. In the first case, you're assigning the value of the single character \u0061 to a char variable. Then you're trying to use a character literal as a variable name, which doesn't work. Then you're using a Unicode-formatted character (not quoted) as a variable name, which is okay. Perhaps you're confusing Java's quote rules with shell?

You can find the reason in specification of literals
Unicode composite characters are different from the decomposed characters.
Identifier:
IdentifierChars but not a Keyword or BooleanLiteral or NullLiteral
IdentifierChars:
JavaLetter
IdentifierChars JavaLetterOrDigit
JavaLetter:
any Unicode character that is a Java letter (see below)
JavaLetterOrDigit:
any Unicode character that is a Java letter-or-digit (see below)

What is significance of putting "$" sign before a parameter in Java / Android?

What is the difference or significance of putting $ sign before any variable or parameter..
e.g.
Suppose here is my class...
public class Vector3 {
public Vector3(float x, float y, float z){
//...
//... my ctor code
//...
}
}
and what will be the difference between these two declarations...
Declaration 1
Vector3 $vec = new Vector3(1f,1f,1f);
Declaration 2
Vector3 vec = new Vector3(1f,1f,1f);
If you can notice $ sign before "vec" in Declaration 1 and Declaration 2.
Any clues?
Also, declaring the same constructor as below,
public class Vector3 {
public Vector3(float $x, float $y, float $z){
//...
//... my ctor code
//...
}
}
What is difference between above constructor and the initial constructor ?
Thanks...
Edit: Thanks to all your responses, I did this with different combinations and there is no major significance :) I appreciate all your answers.

compiles and runs. kid you not.
package $;
public class $
{
int $;
int $(int $){ return $; }
void $$(int $){ this.$=$; }
$ _(){ return this; }
$ _;
{
$$($($));
_._()._._()._();
}
}

According to the Java Language Specification, it's permissible, but has no special meaning. Note that the specification goes on to recommend that it "only be used in mechanically generated source code, or rarely, to access pre-existing names in a legacy system".

While many people have pointed out that $ is legal in identifiers, I refute the use of $ (at least as an initial token) as it goes against the Java Naming Conventions:
Variable names should not start with underscore _ or dollar sign $ characters, even though both are allowed.
While this does not include the case of foo$bar I follow the JLS recommendation that the $ be used only in generated code. The Java compiler uses $ when it generates anonymous classes, for instance.
If nothing else, I would consider your code ugly :-)

I don't believe it is anything more than a valid symbol for use in Java variable names. From the Java tutorial on variables ( http://download.oracle.com/javase/tutorial/java/nutsandbolts/variables.html ):
Variable names are case-sensitive. A variable's name can be any legal identifier — an unlimited-length sequence of Unicode letters and digits, beginning with a letter, the dollar sign "$", or the underscore character "_". The convention, however, is to always begin your variable names with a letter, not "$" or "_". Additionally, the dollar sign character, by convention, is never used at all. You may find some situations where auto-generated names will contain the dollar sign, but your variable names should always avoid using it. A similar convention exists for the underscore character; while it's technically legal to begin your variable's name with "_", this practice is discouraged. White space is not permitted.

Variable names are case-sensitive. A variable's name can be any legal identifier — an unlimited-length sequence of Unicode letters and digits, beginning with a letter, the dollar sign "$", or the underscore character "". The convention, however, is to always begin your variable names with a letter, not "$" or "". Additionally, the dollar sign character, by convention, is never used at all. You may find some situations where auto-generated names will contain the dollar sign, but your variable names should always avoid using it. A similar convention exists for the underscore character; while it's technically legal to begin your variable's name with "_", this practice is discouraged. White space is not permitted.
Taken from : ttp://download.oracle.com/javase/tutorial/java/nutsandbolts/variables.html#naming