We Keep Coding

How to clean a file, replacing unwanted seperators, operators, string literals

I'm working on a concordance problem where I must: "Clean the file. For this, remove all string literals (anything enclosed
in double quotes, the second of which is not preceded by an odd number
of backslashes), remove all // comments, remove all separator characters
(look these up), and operators (look these up). Do not worry about ".class literals" (we will assume they will not appear in the input file)."
I think I know how the replaceAll() method works, but I don't know what's going to be in the file. For starters, how would I go about removing all string literals? Is there a way to replace everything within two double quotes? I.E. String someString = "I want to remove this from a file plz help me, thx";
I've currently put each line of text within an ArrayList of Strings.
Here's what I've got: http://pastebin.com/N84QdLqz

I think I've come up with a solution for your string literal regex. Something like:
inputLine.replaceAll("\"([^\\\\\"]*(\\\\\")*)*([\\\\]{2})*(\\\\\")*[^\"]*\"");
should do the trick. The regex is actually significantly more readable if you print it out to the console after Java has had a chance to escape all of the characters. So if you call System.out.println() with that String, you'll get:
"([^\\"]*(\\")*)*([\\]{2})*(\\")*[^"]*"
I'll break down the original regex to explain. First there's:
"\"([^\\\\\"]*(\\\\\")*)*
This says to match a quote character (") followed by 0 or more patterns of characters that are neither backslashes (\) nor quote characters (") which are followed by 0 or more escaped quotes (\"). As you can see, since \ is typically used as an escape character in Java, any regexes using them become pretty verbose.
([\\\\]{2})*
This says to next match 0 or more sets of 2 (i.e. even-numbered amounts) of backslashes.
(\\\\\")*
This says to match a single backslash followed by a quote character, and to find 0 or more of those together.
[^\"]*\"
This says to match anything that is not a quote character, 0 or more times, followed by a quote character.
I tested my regex with an example similar to what you were asking for:
string literals (anything enclosed in double quotes, the second of which is not preceded by an odd number of backslashes)
Emphasis mine. So by this statement, if the first quote in a literal has a backslash in front of it, it doesn't matter.
String s = "This is "a test\" + "So is this"
Applying the regex with replaceAll and a replacement of \"\", you'll get:
String s = ""a test\""So is this"
which should be correct. You can completely remove the matching literal's quotes, if you want, by calling replaceAll with a replacement of "":
String s = a test\So is this"
Alternately, using this regex on something much less contrived to cause headaches:
String s = "This is \"a test\\" + "So is this"
will return:
String s = +

Yo can do something like this:
private static final String REGEX = "(\"[\\w|\\s]*\")";
private static Pattern P;
private static Matcher M;
public static void main(String args[]){
P = Pattern.compile(REGEX);
//.... your code here ....
}
public static ArrayList<String> readStringsFromFile(String fileName) throws FileNotFoundException
{
Scanner scanner = null;
scanner = new Scanner(new File(fileName));
ArrayList<String> list = new ArrayList<>();
String str = new String();
try
{
while(scanner.hasNext())
{
str = scanner.nextLine();
str = cleanLine(str);//clean the line after read
list.add(str);
}
}
catch (InputMismatchException ex)
{
}
return list;
}
public static String cleanLine(String line) {
int index;
//remove comment lines
index = line.indexOf("//");
if (index != -1) {
line = line.substring(0, index);
}
//remove everything within two double quotes
M = P.matcher(line);
String tmp = "";
while(M.find()) {
tmp = line.substring(0,M.start());
tmp += line.substring(M.end());
line = tmp;
M = P.matcher(line);
}
return line;
}

Count length of String excluding linebreaks

I am writing out a String and i need it to break the lines every time there is 20 characters on a line.
However, it counts the
\n
as a character as well. How can i break the lines when there is 20 characters on the line, excluding the \n?
Edit: Every time it've counted 20 characters, it adds \n to the end of the line

Before printing your string, get rid of the containing line breaks like this:
yourString = yourString.replaceAll("\\n", "");
Here "\\n" is a regular expression matching the newline character. The string literal "\\n" is actually a string containing \n

You could first find and replace with an empty string all the occurences of "\n" into your source then execute your algorithm.
destinationSring = sourceString.replaceAll("\\n", "");

int count=0;
for (char c : someString.toCharArray()) {
if (c == '\n') continue;
System.out.print(c);
++count;
if (count % 20 == 0) System.out.println();
}

You can use the following
stringValue.replaceAll("[\n\r]+", "").length()

How to remove newlines from beginning and end of a string?

I have a string that contains some text followed by a blank line. What's the best way to keep the part with text, but remove the whitespace newline from the end?

Use String.trim() method to get rid of whitespaces (spaces, new lines etc.) from the beginning and end of the string.
String trimmedString = myString.trim();

String.replaceAll("[\n\r]", "");

This Java code does exactly what is asked in the title of the question, that is "remove newlines from beginning and end of a string-java":
String.replaceAll("^[\n\r]", "").replaceAll("[\n\r]$", "")
Remove newlines only from the end of the line:
String.replaceAll("[\n\r]$", "")
Remove newlines only from the beginning of the line:
String.replaceAll("^[\n\r]", "")

tl;dr
String cleanString = dirtyString.strip() ; // Call new `String::string` method.
String::strip…
The old String::trim method has a strange definition of whitespace.
As discussed here, Java 11 adds new strip… methods to the String class. These use a more Unicode-savvy definition of whitespace. See the rules of this definition in the class JavaDoc for Character::isWhitespace.
Example code.
String input = " some Thing ";
System.out.println("before->>"+input+"<<-");
input = input.strip();
System.out.println("after->>"+input+"<<-");
Or you can strip just the leading or just the trailing whitespace.
You do not mention exactly what code point(s) make up your newlines. I imagine your newline is likely included in this list of code points targeted by strip:
It is a Unicode space character (SPACE_SEPARATOR, LINE_SEPARATOR, or PARAGRAPH_SEPARATOR) but is not also a non-breaking space ('\u00A0', '\u2007', '\u202F').
It is '\t', U+0009 HORIZONTAL TABULATION.
It is '\n', U+000A LINE FEED.
It is '\u000B', U+000B VERTICAL TABULATION.
It is '\f', U+000C FORM FEED.
It is '\r', U+000D CARRIAGE RETURN.
It is '\u001C', U+001C FILE SEPARATOR.
It is '\u001D', U+001D GROUP SEPARATOR.
It is '\u001E', U+001E RECORD SEPARATOR.
It is '\u001F', U+0

If your string is potentially null, consider using StringUtils.trim() - the null-safe version of String.trim().

If you only want to remove line breaks (not spaces, tabs) at the beginning and end of a String (not inbetween), then you can use this approach:
Use a regular expressions to remove carriage returns (\\r) and line feeds (\\n) from the beginning (^) and ending ($) of a string:
s = s.replaceAll("(^[\\r\\n]+|[\\r\\n]+$)", "")
Complete Example:
public class RemoveLineBreaks {
public static void main(String[] args) {
var s = "\nHello world\nHello everyone\n";
System.out.println("before: >"+s+"<");
s = s.replaceAll("(^[\\r\\n]+|[\\r\\n]+$)", "");
System.out.println("after: >"+s+"<");
}
}
It outputs:
before: >
Hello world
Hello everyone
<
after: >Hello world
Hello everyone<

I'm going to add an answer to this as well because, while I had the same question, the provided answer did not suffice. Given some thought, I realized that this can be done very easily with a regular expression.
To remove newlines from the beginning:
// Trim left
String[] a = "\n\nfrom the beginning\n\n".split("^\\n+", 2);
System.out.println("-" + (a.length > 1 ? a[1] : a[0]) + "-");
and end of a string:
// Trim right
String z = "\n\nfrom the end\n\n";
System.out.println("-" + z.split("\\n+$", 2)[0] + "-");
I'm certain that this is not the most performance efficient way of trimming a string. But it does appear to be the cleanest and simplest way to inline such an operation.
Note that the same method can be done to trim any variation and combination of characters from either end as it's a simple regex.

Try this
function replaceNewLine(str) {
return str.replace(/[\n\r]/g, "");
}

String trimStartEnd = "\n TestString1 linebreak1\nlinebreak2\nlinebreak3\n TestString2 \n";
System.out.println("Original String : [" + trimStartEnd + "]");
System.out.println("-----------------------------");
System.out.println("Result String : [" + trimStartEnd.replaceAll("^(\\r\\n|[\\n\\x0B\\x0C\\r\\u0085\\u2028\\u2029])|(\\r\\n|[\\n\\x0B\\x0C\\r\\u0085\\u2028\\u2029])$", "") + "]");
Start of a string = ^ ,
End of a string = $ ,
regex combination = | ,
Linebreak = \r\n|[\n\x0B\x0C\r\u0085\u2028\u2029]

Another elegant solution.
String myString = "\nLogbasex\n";
myString = org.apache.commons.lang3.StringUtils.strip(myString, "\n");

For anyone else looking for answer to the question when dealing with different linebreaks:
string.replaceAll("(\n|\r|\r\n)$", ""); // Java 7
string.replaceAll("\\R$", ""); // Java 8
This should remove exactly the last line break and preserve all other whitespace from string and work with Unix (\n), Windows (\r\n) and old Mac (\r) line breaks: https://stackoverflow.com/a/20056634, https://stackoverflow.com/a/49791415. "\\R" is matcher introduced in Java 8 in Pattern class: https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html
This passes these tests:
// Windows:
value = "\r\n test \r\n value \r\n";
assertEquals("\r\n test \r\n value ", value.replaceAll("\\R$", ""));
// Unix:
value = "\n test \n value \n";
assertEquals("\n test \n value ", value.replaceAll("\\R$", ""));
// Old Mac:
value = "\r test \r value \r";
assertEquals("\r test \r value ", value.replaceAll("\\R$", ""));

String text = readFileAsString("textfile.txt");
text = text.replace("\n", "").replace("\r", "");

Regular expression troubles, escaped quotes

Basically, I'm being passed a string and I need to tokenise it in much the same manner as command line options are tokenised by a *nix shell
Say I have the following string
"Hello\" World" "Hello Universe" Hi
How could I turn it into a 3 element list
Hello" World
Hello Universe
Hi
The following is my first attempt, but it's got a number of problems
It leaves the quote characters
It doesn't catch the escaped quote
Code:
public void test() {
String str = "\"Hello\\\" World\" \"Hello Universe\" Hi";
List<String> list = split(str);
}
public static List<String> split(String str) {
Pattern pattern = Pattern.compile(
"\"[^\"]*\"" + /* double quoted token*/
"|'[^']*'" + /*single quoted token*/
"|[A-Za-z']+" /*everything else*/
);
List<String> opts = new ArrayList<String>();
Scanner scanner = new Scanner(str).useDelimiter(pattern);
String token;
while ((token = scanner.findInLine(pattern)) != null) {
opts.add(token);
}
return opts;
}
So the incorrect output of the following code is
"Hello\"
World
" "
Hello
Universe
Hi
EDIT I'm totally open to a non regex solution. It's just the first solution that came to mind

If you decide you want to forego regex, and do parsing instead, there are a couple of options. If you are willing to have just a double quote or a single quote (but not both) as your quote, then you can use StreamTokenizer to solve this easily:
public static List<String> tokenize(String s) throws IOException {
List<String> opts = new ArrayList<String>();
StreamTokenizer st = new StreamTokenizer(new StringReader(s));
st.quoteChar('\"');
while (st.nextToken() != StreamTokenizer.TT_EOF) {
opts.add(st.sval);
}
return opts;
}
If you must support both quotes, here is a naive implementation that should work (caveat that a string like '"blah \" blah"blah' will yield something like 'blah " blahblah'. If that isn't OK, you will need to make some changes):
public static List<String> splitSSV(String in) throws IOException {
ArrayList<String> out = new ArrayList<String>();
StringReader r = new StringReader(in);
StringBuilder b = new StringBuilder();
int inQuote = -1;
boolean escape = false;
int c;
// read each character
while ((c = r.read()) != -1) {
if (escape) { // if the previous char is escape, add the current char
b.append((char)c);
escape = false;
continue;
}
switch (c) {
case '\\': // deal with escape char
escape = true;
break;
case '\"':
case '\'': // deal with quote chars
if (c == '\"' || c == '\'') {
if (inQuote == -1) { // not in a quote
inQuote = c; // now we are
} else {
inQuote = -1; // we were in a quote and now we aren't
}
}
break;
case ' ':
if (inQuote == -1) { // if we aren't in a quote, then add token to list
out.add(b.toString());
b.setLength(0);
} else {
b.append((char)c); // else append space to current token
}
break;
default:
b.append((char)c); // append all other chars to current token
}
}
if (b.length() > 0) {
out.add(b.toString()); // add final token to list
}
return out;
}

I'm pretty sure you can't do this by just tokenising on a regex. If you need to deal with nested and escaped delimiters, you need to write a parser. See e.g. http://kore-nordmann.de/blog/do_NOT_parse_using_regexp.html
There will be open source parsers which can do what you want, although I don't know any. You should also check out the StreamTokenizer class.

To recap, you want to split on whitespace, except when surrounded by double quotes, which are not preceded by a backslash.
Step 1: tokenize the input: /([ \t]+)|(\\")|(")|([^ \t"]+)/
This gives you a sequence of SPACE, ESCAPED_QUOTE, QUOTE and TEXT tokens.
Step 2: build a finite state machine matching and reacting to the tokens:
State: START
SPACE -> return empty string
ESCAPED_QUOTE -> Error (?)
QUOTE -> State := WITHIN_QUOTES
TEXT -> return text
State: WITHIN_QUOTES
SPACE -> add value to accumulator
ESCAPED_QUOTE -> add quote to accumulator
QUOTE -> return and clear accumulator; State := START
TEXT -> add text to accumulator
Step 3: Profit!!

I think if you use pattern like this:
Pattern pattern = Pattern.compile("\".*?(?<!\\\\)\"|'.*?(?<!\\\\)'|[A-Za-z']+");
Then it will give you desired output. When I ran with your input data I got this list:
["Hello\" World", "Hello Universe", Hi]
I used [A-Za-z']+ from your own question but shouldn't it be just : [A-Za-z]+
EDIT
Change your opts.add(token); line to:
opts.add(token.replaceAll("^\"|\"$|^'|'$", ""));

The first thing you need to do is stop thinking of the job in terms of split(). split() is meant for breaking down simple strings like this/that/the other, where / is always a delimiter. But you're trying to split on whitespace, unless the whitespace is within quotes, except if the quotes are escaped with backslashes (and if backslashes escape quotes, they probably escape other things, like other backslashes).
With all those exceptions-to-exceptions, it's just not possible to create a regex to match all possible delimiters, not even with fancy gimmicks like lookarounds, conditionals, reluctant and possessive quantifiers. What you want to do is match the tokens, not the delimiters.
In the following code, a token that's enclosed in double-quotes or single-quotes may contain whitespace as well as the quote character if it's preceded by a backslash. Everything except the enclosing quotes is captured in group #1 (for double-quoted tokens) or group #2 (single-quoted). Any character may be escaped with a backslash, even in non-quoted tokens; the "escaping" backslashes are removed in a separate step.
public static void test()
{
String str = "\"Hello\\\" World\" 'Hello Universe' Hi";
List<String> commands = parseCommands(str);
for (String s : commands)
{
System.out.println(s);
}
}
public static List<String> parseCommands(String s)
{
String rgx = "\"((?:[^\"\\\\]++|\\\\.)*+)\"" // double-quoted
+ "|'((?:[^'\\\\]++|\\\\.)*+)'" // single-quoted
+ "|\\S+"; // not quoted
Pattern p = Pattern.compile(rgx);
Matcher m = p.matcher(s);
List<String> commands = new ArrayList<String>();
while (m.find())
{
String cmd = m.start(1) != -1 ? m.group(1) // strip double-quotes
: m.start(2) != -1 ? m.group(2) // strip single-quotes
: m.group();
cmd = cmd.replaceAll("\\\\(.)", "$1"); // remove escape characters
commands.add(cmd);
}
return commands;
}
output:
Hello" World
Hello Universe
Hi
This is about as simple as it gets for a regex-based solution--and it doesn't really deal with malformed input, like unbalanced quotes. If you're not fluent in regexes, you might be better off with a purely hand-coded solution or, even better, a dedicated command-line interpreter (CLI) library.

Java: Count empty lines in a text file/string

I am using the following code to count empty lines in Java, but this code returns a greater number of empty lines than there are.
int countEmptyLines(String s) {
int result=0;
Pattern regex = Pattern.compile("(?m)^\\s*$");
Matcher testMatcher = regex.matcher(s);
while (testMatcher.find())
{
result++;
}
return result;}
What am I doing wrong or is there a better way to do it?

Try this:
final BufferedReader br = new BufferedReader(new StringReader("hello\n\nworld\n"));
String line;
int empty = 0;
while ((line = br.readLine()) != null) {
if (line.trim().isEmpty()) {
empty++;
}
}
System.out.println(empty);

I found a way to fix my own regex while I was at lunch:
Pattern regex = Pattern.compile("(?m)^\\s*?$");
The '?' makes the \s* reluctant, meaning it will somehow not match the character that '$' will match.

\s matches any whitespace, which is either a space, a tab or a carriage return/linefeed.
The easiest way to do this is to count chains of successive EOL characters. I write EOL, because you need to determine which character denotes the end of line in your file. While under Windows, an end of line amounts to a Carriage Return and a Linefeed character. Under Unix, this is different, so for a file written under Unix your programm will have to be adjusted.
Then, count every the successive number of the end of line character(s) and each time add this number minus 1 to a count. At the end, you will have the empty line count.