We Keep Coding

grep adds new line if splitted by groovy?

My goal is to get the names of x .conf files in a directory called "conf". My code looks like this:
commandOutput = sshCommand remote: getRemote(), command: getPathCmd(type) + "cd conf; ls | grep '.conf' | grep -Eo '^[a-zA-Z0-9_-]+'"
def instances = commandOutput.split("\n") as String[]
return instances
If i print the variable commandOutput (with two files in the directory) it shows me:
name1
name2
But when i make a for loop to print every slot of the array, i get this:
name1
*empty line*
and
name2
I checked commandOutput[0] with .endsWith("\n") and returned false so i dont know where the new line is coming from.
Is there an obvious reason for that behaviour?
Edit: It also happens here:
command = getPathCmd(type) + getScript(type, "status") + "| grep -Eo '[0-9]{4,5}' | sort -u | grep -v '${params.xyID}\\|" + globalID + "'"
try{
commandOutput = sshCommand remote: getRemote(), command: command
def PIDs = commandOutput.split("\n") as String[]
return PIDs
}
catch(Exception e){
println("XY")
return "XY"
}
Print of commandOutput:
1234
5678
For loop print of PIDs:
1234
*empty line*
and
5678

stringXY = stringXY.replaceAll("[\\\r\\\n]+",""); did the trick. It replaces all new lines.
tr '\n' '\0' as suggested in this thread wouldn't work here, because it would be harder to find out where one name ends and the next one starts.

Rsync command not working in java

The following commands works directly:
rsync -rtuc --delete-after --exclude '.git*' --filter 'protect .git/**/*' ~/some/source/ ~/some/destination/
But when run via java:
private Boolean syncFiles() {
// Success flag default to true
Boolean success = true;
// Attempt sync repo
try {
ProcessBuilder runtimeProcessBuilder = new ProcessBuilder().inheritIO().command(new String[]{
"rsync", "-rtuc","--delete-after", "--exclude", "'.git*'", "--filter", "'protect .git/**/*'", "~/some/source/", "~/some/destination/"
});
// Wait until process terminates
int output = runtimeProcessBuilder.start().waitFor();
// Determine if successful
if (output == 0) {
System.out.println("Backup of " + getSource() + " to " + getDestination()
+ " was successful");
} else {
System.out.println("Error: rsync returned error code: " + output);
success = false;
}
} catch (Exception ex) {
success = false;
System.out.println("Error:");
System.out.println(ex.getMessage());
Logger.getLogger(Rsync.class.getName()).log(Level.SEVERE, null, ex);
}
return success;
}
I get the error:
Unknown filter rule: `'protect .git/**/*'' Error: rsync returned error
code: 1 rsync error: syntax or usage error (code 1) at exclude.c(902)
[client=3.1.2]

The shell handles quoting before passing the parameters to the command.
The comes into play with this part of your command line:
'protect .git/**/*'
The shell interprets this as the single parameter:
protect .git/**/*
If the single quotes had not been there in the first place, the shell would have:
interpreted it as two parameters (because of the space)
expanded glob characters like "*"
The solution is to pass:
"protect .git/**/"
as one of your Java parameters, rather than "'protect .git/**/*'".
You may have similar problems with ~, which the shell will expand to your home directory.

The answer to the solution is as follows:
The ProcessBuilder object needs to be initialised as follows:
ProcessBuilder runtimeProcessBuilder = new ProcessBuilder().inheritIO().command(new String[]{
"rsync", "-rtuc","--delete-after", "--filter", "protect .git", "--exclude", "'.git*'", "~/some/source/", "~/some/destination/"
});

Too Many Open Files (Selenium + PhantomJSDriver)

In my embedded Selenium/PhantomJSDriver driver it seems resources are not being cleaned up. Running the client synchronously causes millions of open files and eventually throws a "Too many files open" type exception.
Here is some output I gathered from lsof while the program is running for ~1 minute
$ lsof | awk '{ print $2; }' | uniq -c | sort -rn | head
1221966 12180
34790 29773
31260 12138
20955 8414
17940 10343
16665 32332
9512 27713
7275 19226
5496 7153
5040 14065
$ lsof -p 12180 | awk '{ print $2; }' | uniq -c | sort -rn | head
2859 12180
1 PID
$ lsof -p 12180 -Fn | sort -rn | uniq -c | sort -rn | head
1124 npipe
536 nanon_inode
4 nsocket
3 n/opt/jdk/jdk1.8.0_60/jre/lib/jce.jar
3 n/opt/jdk/jdk1.8.0_60/jre/lib/charsets.jar
3 n/dev/urandom
3 n/dev/random
3 n/dev/pts/20
2 n/usr/share/sbt-launcher-packaging/bin/sbt-launch.jar
2 n/usr/share/java/jayatana.jar
I don't understand why using the -p flag on lsof has a smaller result set. But it appears most of the entries are pipe and anon_inode.
The client is very simple at ~100 lines, and at the end of usage calls driver.close() and driver.quit(). I experimented with caching and reusing clients but it did not alleviate the open files
case class HeadlessClient(
country: String,
userAgent: String,
inheritSessionId: Option[Int] = None
) {
protected var numberOfRequests: Int = 0
protected val proxySessionId: Int = inheritSessionId.getOrElse(new Random().nextInt(Integer.MAX_VALUE))
protected val address = InetAddress.getByName("proxy.domain.com")
protected val host = address.getHostAddress
protected val login: String = HeadlessClient.username + proxySessionId
protected val windowSize = new org.openqa.selenium.Dimension(375, 667)
protected val (mobProxy, seleniumProxy) = {
val proxy = new BrowserMobProxyServer()
proxy.setTrustAllServers(true)
proxy.setChainedProxy(new InetSocketAddress(host, HeadlessClient.port))
proxy.chainedProxyAuthorization(login, HeadlessClient.password, AuthType.BASIC)
proxy.addLastHttpFilterFactory(new HttpFiltersSourceAdapter() {
override def filterRequest(originalRequest: HttpRequest): HttpFilters = {
new HttpFiltersAdapter(originalRequest) {
override def proxyToServerRequest(httpObject: HttpObject): io.netty.handler.codec.http.HttpResponse = {
httpObject match {
case req: HttpRequest => req.headers().remove(HttpHeaders.Names.VIA)
case _ =>
}
null
}
}
}
})
proxy.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT)
proxy.start(0)
val seleniumProxy = ClientUtil.createSeleniumProxy(proxy)
(proxy, seleniumProxy)
}
protected val driver: PhantomJSDriver = {
val capabilities: DesiredCapabilities = DesiredCapabilities.chrome()
val cliArgsCap = new util.ArrayList[String]
cliArgsCap.add("--webdriver-loglevel=NONE")
cliArgsCap.add("--ignore-ssl-errors=yes")
cliArgsCap.add("--load-images=no")
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy)
capabilities.setCapability("phantomjs.page.customHeaders.Referer", "")
capabilities.setCapability("phantomjs.page.settings.userAgent", userAgent)
capabilities.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS, cliArgsCap)
new PhantomJSDriver(capabilities)
}
driver.executePhantomJS(
"""
|var navigation = [];
|
|this.onNavigationRequested = function(url, type, willNavigate, main) {
| navigation.push(url)
| console.log('Trying to navigate to: ' + url);
|}
|
|this.onResourceRequested = function(request, net) {
| console.log("Requesting " + request.url);
| if (! (navigation.indexOf(request.url) > -1)) {
| console.log("Aborting " + request.url)
| net.abort();
| }
|};
""".stripMargin
)
driver.manage().window().setSize(windowSize)
def follow(url: String)(implicit ec: ExecutionContext): List[HarEntry] = {
try{
Await.result(Future{
mobProxy.newHar(url)
driver.get(url)
val entries = mobProxy.getHar.getLog.getEntries.asScala.toList
shutdown()
entries
}, 45.seconds)
} catch {
case e: Exception =>
try {
shutdown()
} catch {
case shutdown: Exception =>
throw new Exception(s"Error ${shutdown.getMessage} cleaning up after Exception: ${e.getMessage}")
}
throw e
}
}
def shutdown() = {
driver.close()
driver.quit()
}
}
I tried several versions of Selenium in case there was a bugfix. The build.sbt:
libraryDependencies += "org.seleniumhq.selenium" % "selenium-java" % "3.0.1"
libraryDependencies += "net.lightbody.bmp" % "browsermob-core" % "2.1.2"
Also, I tried PhantomJS 2.0.1, and 2.1.1:
$ phantomjs --version
2.0.1-development
$ phantomjs --version
2.1.1
Is this a PhantomJS or Selenium problem? Is my client using the API improperly?

The resource usage is caused by BrowserMob. To close the proxy and clean-up its resources, one must call stop().
For this client that means modifying the shutdown method
def shutdown() = {
mobProxy.stop()
driver.close()
driver.quit()
}
Another method, abort, offers immediate termination of the proxy server and does not wait for traffic to cease.

In my opinion it seems a problem of PhantomJS. You can try the following alternatives:
Use phantomjs 2.5.0-beta. It has been recently released. I'm not sure if this upgrade solves your problem, but at least it is worth to give a try. According to the changelog, the new features of this version are:
Upgrade QtWebKit to QtWebKitNG
Upgraded Qt to 5.7.1
Clean the phantomjs processes after closing webdriver. You can implement your own cleaner to force that phantomjs is actually closed after driver.close() (invoking killall -9 phantomjs or similar).

How to destroy the subprocess of Java Process

I use Runtime.getRuntime().exec to execute command tail -f filename | grep str which is based on OS pipe. I managed to achieve my business logic. But still there is a problem I must solve:
When using pipe, the Process will for another process for tail command:
$ ps -ef | grep test.log
admin 6953 32721 0 16:32 ? 00:00:00 /bin/sh -c tail -f /home/admin
/test.log | unbuffer -p grep '1444379575648'
admin 6957 6953 0 16:32 ? 00:00:00 tail -f /home/admin/test.log
Process.destroy() method destroys itself(pid:6953) only.How can I destroy its subprocess(pid:6957) in my Java program?

Don't use Runtime.getRuntime().exec(...), use 2 ProcessBuilders to explicitly build the individual processes, and connect their inputs and outputs together to do the equivalent of the piping.
Then you will a separate Process object for each, and can kill them as you please.

i found another way:
public static final String getPid() {
try {
RuntimeMXBean runtimeBean = ManagementFactory.getRuntimeMXBean();
String name = runtimeBean.getName();
int k = name.indexOf('#');
if (k > 0)
return name.substring(0, k);
} catch (Exception ex) {
}
return null;
}
This works on linux,win,mac.

Shell command not executing in Java [duplicate]

This question already has an answer here:
shell Command execution Failing when executed from Java
(1 answer)
Closed 9 years ago.
Normally when i do this:
printf 'select x from Y \ngo\n' | isql -Uxx -Pxxxxxx -Dxxxxx -w 65535 -s ','
on the command line it executes fine.
even with the below java code. it works fine
public class test
{
public static void main(String[] args)
{
String DB="|isql -Uxx -Pxxxxxx -Dxxxxx -w 65535 -s ','";
String qpfx="printf \'";
String qsfx=" \ngo\n\'";
if(args[0]!=null)
try {
String cmd=qpfx+args[0]+qsfx+DB;
System.out.println("argument query is:"+args[0]);
System.out.println("Command is:"+cmd);
Process p = Runtime.getRuntime().exec(new String[]{"sh","-c",cmd});
p.waitFor();
BufferedReader reader = new BufferedReader(new InputStreamReader(p.getInputStream()));
String line=reader.readLine();
while (line != null) {
System.out.println(line);
line = reader.readLine();
}
}
catch(IOException e1) {}
catch(InterruptedException e2) {}
}
}
I will run this above java code like below:
java test "select x from Y"
Probelm comes when i give the select query as "select * from Y" instead of column names.
That is when i wanted to execute like below:
java test "select * from Y"
The above jav code hangs.Can any body tell me the reason why?
below is the output where its hanging:
> java test "select * from Y"
argument query is:select * from Y
Command is:printf 'select * from Y
go
'|isql -Uxx -Pxxxxx -Dxxxxxx -w 65535 -s ','
I also tried:
java test "select \* from Y"
But it does not execute.

After severe debugging i found that the line that is hanging is p.waitFor();.
I had to read this complete article and then i finally found a solution.
I have shifted the p.waitFor(); after the while loop in the code and it worked.