We Keep Coding

Issue executing a piped ps unix command through Runtime.exec()

Issue:
When executing the following command through Runtime.exec(...), it fails with an unexpected EOF while looking for a matching quote character.
One oddity is that the error message has a grave character followed by two single quotes.
However, when I execute the command that prints out in the logs through putty, it works fine.
Command:
bin/sh -c 'ps -eo uname,pid,ppid,nlwp,pcpu,pmem,psr,start_time,tty,time,args | fgrep IAAPC | fgrep /f1/f2/a00-a/f3/server/server_1/env_1/javadriver | fgrep -v fgrep'
Resulting error:
-eo: -c: line 0: unexpected EOF while looking for matching `''
-eo: -c: line 1: syntax error: unexpected end of file
Java Code (Java 1.6 ... Don't Judge):
String driverHome = trimToEmpty(System.getProperty("batchdriver.home"));
String cmd = "/bin/sh -c 'ps -eo uname,pid,ppid,nlwp,pcpu,pmem,psr,start_time,tty,time,args | fgrep "+jobName+" | fgrep "+driverHome+" | fgrep -v fgrep'";
String out = null, err = null;
Process proc = null;
try {
proc = Runtime.getRuntime().exec(cmd);
out = fullyRead(proc.getInputStream());
err = fullyRead(proc.getErrorStream());
int exitVal = proc.waitFor();
if(logger.isDebugEnabled()) {
logger.debug("Process Information: "+out);
}
if (isNotEmpty(err)) {
logger.error(failedCommandMessage(cmd, out, err));
this.processId = null;
this.processDesc = PROCESS_NOT_FOUND;
return;
}
String[] processes = StringUtils.split(out, "\r?\n");
if (processes == null || processes.length == 0) {
this.processDesc = PROCESS_NOT_FOUND;
}
else if (processes.length == 1) {
String[] processInfo = processes[0].split("\\s+");
this.processId = processInfo[1];
if (!isNumeric(this.processId)) {
this.processId = null;
}
this.processDesc = out;
}
else {
this.processDesc = out;
}
if (logger.isDebugEnabled()) {
logger.debug("Call to the OS completed with exit value: " + exitVal);
}
} catch (Exception e) {
try {out = fullyRead(proc.getInputStream());} catch (Exception e1) {}
try {err = fullyRead(proc.getErrorStream());} catch (Exception e1) {}
this.processId = null;
this.processDesc = PROCESS_NOT_FOUND;
logger.error(failedCommandMessage(cmd, out, err), e);
}

Related but not quite dupe: Pass a string with multiple contiguous spaces as a parameter to a jar file using Windows command prompt called from a java program
The Runtime.exec methods that take a String break it into tokens at whitespace only so this actually runs the program /bin/sh (a shell) with the following arguments:
-c
'ps
-eo
uname,pid,ppid,nlwp,pcpu,pmem,psr,start_time,tty,time,args
|
fgrep
...
The shell interprets these arguments like this:
-c 'ps -- the script to run consists of the apostrophe character, p, s (and nothing more)
-eo -- the name of the command being run is -eo
uname,pid,.... -- the first argument to the script is this
| -- the second argument to the script is this
fgrep -- the third argument to the script is this
...
-- but the script ignores the arguments and doesn't use them
Thus you get
-eo: -c: unexpected EOF while looking for matching `''
# the script named -eo, with the option -c having value 'ps,
# tried to find a closing ' to match the opening ' and it's not there
This shell is apparently (GNU) bash; many GNU programs that put a data string in an error message surround it by backquote and apostrophe because these were sort of matching quotes in one interpretation of ASCII popular decades ago.
Instead use the String[] overload of exec to give the shell the two arguments that it gets when your above command line is parsed by a shell instead of StringTokenizer:
String[] cmdary = {"/bin/sh", "-c", "ps -eo stuff | fgrep this | fgrep that | fgrep -v fgrep"};
... Runtime.getRuntime().exec(cmdary);
But instead of running three fgrep's, you could just run the ps and read the inputstream as lines and test them in Java using String.contains or similar. Also most of the columns you ask ps for will never be used for either your matching nor result, so that's just a waste of effort and clutter.

Too Many Open Files (Selenium + PhantomJSDriver)

In my embedded Selenium/PhantomJSDriver driver it seems resources are not being cleaned up. Running the client synchronously causes millions of open files and eventually throws a "Too many files open" type exception.
Here is some output I gathered from lsof while the program is running for ~1 minute
$ lsof | awk '{ print $2; }' | uniq -c | sort -rn | head
1221966 12180
34790 29773
31260 12138
20955 8414
17940 10343
16665 32332
9512 27713
7275 19226
5496 7153
5040 14065
$ lsof -p 12180 | awk '{ print $2; }' | uniq -c | sort -rn | head
2859 12180
1 PID
$ lsof -p 12180 -Fn | sort -rn | uniq -c | sort -rn | head
1124 npipe
536 nanon_inode
4 nsocket
3 n/opt/jdk/jdk1.8.0_60/jre/lib/jce.jar
3 n/opt/jdk/jdk1.8.0_60/jre/lib/charsets.jar
3 n/dev/urandom
3 n/dev/random
3 n/dev/pts/20
2 n/usr/share/sbt-launcher-packaging/bin/sbt-launch.jar
2 n/usr/share/java/jayatana.jar
I don't understand why using the -p flag on lsof has a smaller result set. But it appears most of the entries are pipe and anon_inode.
The client is very simple at ~100 lines, and at the end of usage calls driver.close() and driver.quit(). I experimented with caching and reusing clients but it did not alleviate the open files
case class HeadlessClient(
country: String,
userAgent: String,
inheritSessionId: Option[Int] = None
) {
protected var numberOfRequests: Int = 0
protected val proxySessionId: Int = inheritSessionId.getOrElse(new Random().nextInt(Integer.MAX_VALUE))
protected val address = InetAddress.getByName("proxy.domain.com")
protected val host = address.getHostAddress
protected val login: String = HeadlessClient.username + proxySessionId
protected val windowSize = new org.openqa.selenium.Dimension(375, 667)
protected val (mobProxy, seleniumProxy) = {
val proxy = new BrowserMobProxyServer()
proxy.setTrustAllServers(true)
proxy.setChainedProxy(new InetSocketAddress(host, HeadlessClient.port))
proxy.chainedProxyAuthorization(login, HeadlessClient.password, AuthType.BASIC)
proxy.addLastHttpFilterFactory(new HttpFiltersSourceAdapter() {
override def filterRequest(originalRequest: HttpRequest): HttpFilters = {
new HttpFiltersAdapter(originalRequest) {
override def proxyToServerRequest(httpObject: HttpObject): io.netty.handler.codec.http.HttpResponse = {
httpObject match {
case req: HttpRequest => req.headers().remove(HttpHeaders.Names.VIA)
case _ =>
}
null
}
}
}
})
proxy.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT)
proxy.start(0)
val seleniumProxy = ClientUtil.createSeleniumProxy(proxy)
(proxy, seleniumProxy)
}
protected val driver: PhantomJSDriver = {
val capabilities: DesiredCapabilities = DesiredCapabilities.chrome()
val cliArgsCap = new util.ArrayList[String]
cliArgsCap.add("--webdriver-loglevel=NONE")
cliArgsCap.add("--ignore-ssl-errors=yes")
cliArgsCap.add("--load-images=no")
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy)
capabilities.setCapability("phantomjs.page.customHeaders.Referer", "")
capabilities.setCapability("phantomjs.page.settings.userAgent", userAgent)
capabilities.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS, cliArgsCap)
new PhantomJSDriver(capabilities)
}
driver.executePhantomJS(
"""
|var navigation = [];
|
|this.onNavigationRequested = function(url, type, willNavigate, main) {
| navigation.push(url)
| console.log('Trying to navigate to: ' + url);
|}
|
|this.onResourceRequested = function(request, net) {
| console.log("Requesting " + request.url);
| if (! (navigation.indexOf(request.url) > -1)) {
| console.log("Aborting " + request.url)
| net.abort();
| }
|};
""".stripMargin
)
driver.manage().window().setSize(windowSize)
def follow(url: String)(implicit ec: ExecutionContext): List[HarEntry] = {
try{
Await.result(Future{
mobProxy.newHar(url)
driver.get(url)
val entries = mobProxy.getHar.getLog.getEntries.asScala.toList
shutdown()
entries
}, 45.seconds)
} catch {
case e: Exception =>
try {
shutdown()
} catch {
case shutdown: Exception =>
throw new Exception(s"Error ${shutdown.getMessage} cleaning up after Exception: ${e.getMessage}")
}
throw e
}
}
def shutdown() = {
driver.close()
driver.quit()
}
}
I tried several versions of Selenium in case there was a bugfix. The build.sbt:
libraryDependencies += "org.seleniumhq.selenium" % "selenium-java" % "3.0.1"
libraryDependencies += "net.lightbody.bmp" % "browsermob-core" % "2.1.2"
Also, I tried PhantomJS 2.0.1, and 2.1.1:
$ phantomjs --version
2.0.1-development
$ phantomjs --version
2.1.1
Is this a PhantomJS or Selenium problem? Is my client using the API improperly?

The resource usage is caused by BrowserMob. To close the proxy and clean-up its resources, one must call stop().
For this client that means modifying the shutdown method
def shutdown() = {
mobProxy.stop()
driver.close()
driver.quit()
}
Another method, abort, offers immediate termination of the proxy server and does not wait for traffic to cease.

In my opinion it seems a problem of PhantomJS. You can try the following alternatives:
Use phantomjs 2.5.0-beta. It has been recently released. I'm not sure if this upgrade solves your problem, but at least it is worth to give a try. According to the changelog, the new features of this version are:
Upgrade QtWebKit to QtWebKitNG
Upgraded Qt to 5.7.1
Clean the phantomjs processes after closing webdriver. You can implement your own cleaner to force that phantomjs is actually closed after driver.close() (invoking killall -9 phantomjs or similar).

Sending table index in SNMP trap

I have implemented an SNMP agent using snmp4j, and am running into a bit of a snag with how to properly report SNMPv3 traps/notifications to an SNMP manager. The implementation is set up to manage a table of values indexed by instanceId. When you query the agent you receive the OIDs of the various fields suffixed by ".1", ".2", ".3", etc. based on the particular link instance associated with the OID value. So querying results in:
-----------------------------------------
| Entity | OID | Value |
-----------------------------------------
| Link1 | linkAlias.1 | Link 1 |
| Link2 | linkAlias.2 | Link 2 |
| Link1 | linkState.1 | 1 |
| Link2 | linkState.2 | 3 |
| Link1 | linkText.1 | UP |
| Link2 | linkText.2 | INITIALIZING |
-----------------------------------------
That works great. However, I need to be able to send traps in a similar way so that the index of the table is sent with the trap. That way alarms triggered from SNMP queries can be cleared when the link status changes I tried simply adding the instanceId as a varbind as seen below in my code block, but the entity is always reported as the generic "Link". Has anyone encountered this that could help me solve this? Anything is appreciated greatly.
public static void updateLinkStatus(int instanceId, LinkState status)
{
boolean varChanged = false;
Variable[] currentManagedObject = currentManagedObjects.get(instanceId);
if(currentManagedObject != null)
{
// If we are managing this object
if(((Integer32)currentManagedObject[1]).toInt() != status.value)
varChanged = true;
// Update the managed object Status Integer and Status Text
currentManagedObject[1] = new Integer32(status.value);
currentManagedObject[2] = new OctetString(status.getLinkStateText());
}
else
{
varChanged = true; // No previous record to check against
}
// Send trap now if not equal to previous value
if(varChanged)
{
OID trapOid = null;
int linkState = LinkState.UNKNOWN.value; // Will be overridden
String linkStateText = null;
if(status == LinkState.DOWN)
{
trapOid = oidLinkDown;
linkState = LinkState.DOWN.value;
linkStateText = LnkState.DOWN.getLinkStateText();
}
else if(status == LinkState.MISCONFIGURED)
{
trapOid = oidLinkMisconfigured;
linkState = LinkState.MISCONFIGURED.value;
linkStateText = LinkState.MISCONFIGURED.getLinkStateText();
}
else if(status == LinkState.UP)
{
trapOid = oidLinkUp;
linkState = LinkState.UP.value;
linkStateText = LinkState.UP.getLinkStateText();
}
else if(status == LinkState.INITIALIZING)
{
trapOid = oidLinkInitializing;
linkState = LinkState.INITIALIZING.value;
linkStateText = LinkState.INITIALIZING.getLinkStateText();
}
else
{
// Theoretically, should never change to LinkState.UNKNOWN - no trap available for it
linkState = LinkState.UNKNOWN.value;
linkStateText = LinkState.UNKNOWN.getLinkStateText();
}
// Create variable bindings for V3 trap
if(trapOid != null)
{
List<VariableBinding> variableBindings = new ArrayList<VariableBinding>();
variableBindings.add(new VariableBinding(oidLinkState, new Integer32(linkState)));
variableBindings.add(new VariableBinding(oidLinkText, new OctetString(linkStateText)));
variableBindings.add(new VariableBinding(oidLinkInstanceID, new Integer32(instanceId)));
//I've tried the below varbind too and it also doesn't work
//variableBindings.add(new VariableBinding(new OID(oidLinkInstanceID.toIntArray(), instanceId)));
agent.sendTrap_Version3(trapOid, variableBindings);
}
}
}
Edit:
Note: The links are dynamically configurable so I cannot simply define each link with a separate OID; I define the base OID in the MIB and need to add the index dynamically.

I discovered the solution. I attempted to simply add the instance index to the OID and send a NULLOBJ as the value in the varbind like so:
variableBindings.add(new VariableBinding(new OID(oidLinkInstanceID.toIntArray(), instanceId)));
But the manager rejected the message. So I added the instance index to the OID as well as the varbind value like so:
variableBindings.add(new VariableBinding(new OID(oidLinkInstanceID.toIntArray(), instanceId), new Integer32(instanceId)));
And the manager reports the entity as Link1, Link2, etc. consistent with the SNMP table.

How to destroy the subprocess of Java Process

I use Runtime.getRuntime().exec to execute command tail -f filename | grep str which is based on OS pipe. I managed to achieve my business logic. But still there is a problem I must solve:
When using pipe, the Process will for another process for tail command:
$ ps -ef | grep test.log
admin 6953 32721 0 16:32 ? 00:00:00 /bin/sh -c tail -f /home/admin
/test.log | unbuffer -p grep '1444379575648'
admin 6957 6953 0 16:32 ? 00:00:00 tail -f /home/admin/test.log
Process.destroy() method destroys itself(pid:6953) only.How can I destroy its subprocess(pid:6957) in my Java program?

Don't use Runtime.getRuntime().exec(...), use 2 ProcessBuilders to explicitly build the individual processes, and connect their inputs and outputs together to do the equivalent of the piping.
Then you will a separate Process object for each, and can kill them as you please.

i found another way:
public static final String getPid() {
try {
RuntimeMXBean runtimeBean = ManagementFactory.getRuntimeMXBean();
String name = runtimeBean.getName();
int k = name.indexOf('#');
if (k > 0)
return name.substring(0, k);
} catch (Exception ex) {
}
return null;
}
This works on linux,win,mac.

Powershell: Capturing standard out and error with Process object

I want to start a Java program from PowerShell and get the results printed on the console.
I have followed the instructions of this question:
Capturing standard out and error with Start-Process
But for me, this is not working as I expected. What I'm doing wrong?
This is the script:
$psi = New-object System.Diagnostics.ProcessStartInfo
$psi.CreateNoWindow = $true
$psi.UseShellExecute = $false
$psi.RedirectStandardOutput = $true
$psi.RedirectStandardError = $true
$psi.FileName = 'java.exe'
$psi.Arguments = #("-jar","tools\compiler.jar","--compilation_level", "ADVANCED_OPTIMIZATIONS", "--js", $BuildFile, "--js_output_file", $BuildMinFile)
$process = New-Object System.Diagnostics.Process
$process.StartInfo = $psi
$process.Start() | Out-Null
$process.WaitForExit()
$output = $process.StandardOutput.ReadToEnd()
$output
The $output variable is always empty (and nothing is printed on the console of course).

The docs on the RedirectStandardError property suggests that it is better to put the WaitForExit() call after the ReadToEnd() call. The following works correctly for me:
$psi = New-object System.Diagnostics.ProcessStartInfo
$psi.CreateNoWindow = $true
$psi.UseShellExecute = $false
$psi.RedirectStandardOutput = $true
$psi.RedirectStandardError = $true
$psi.FileName = 'ipconfig.exe'
$psi.Arguments = #("/a")
$process = New-Object System.Diagnostics.Process
$process.StartInfo = $psi
[void]$process.Start()
$output = $process.StandardOutput.ReadToEnd()
$process.WaitForExit()
$output

Small variation so that you can selectively print the output if needed. As in if your looking just for error or warning messages and by the way Keith you saved my bacon with your response...
$psi = New-object System.Diagnostics.ProcessStartInfo
$psi.CreateNoWindow = $true
$psi.UseShellExecute = $false
$psi.RedirectStandardOutput = $true
$psi.RedirectStandardError = $true
$psi.FileName = 'robocopy'
$psi.Arguments = #("$HomeDirectory $NewHomeDirectory /MIR /XF desktop.ini /XD VDI /R:0 /W:0 /s /v /np")
$process = New-Object System.Diagnostics.Process
$process.StartInfo = $psi
[void]$process.Start()
do
{
$process.StandardOutput.ReadLine()
}
while (!$process.HasExited)

Here is a modification to paul's answer, hopefully it addresses the truncated output. i did a test using a failure and did not see truncation.
function Start-ProcessWithOutput
{
param ([string]$Path,[string[]]$ArgumentList)
$Output = New-Object -TypeName System.Text.StringBuilder
$Error = New-Object -TypeName System.Text.StringBuilder
$psi = New-object System.Diagnostics.ProcessStartInfo
$psi.CreateNoWindow = $true
$psi.UseShellExecute = $false
$psi.RedirectStandardOutput = $true
$psi.RedirectStandardError = $true
$psi.FileName = $Path
if ($ArgumentList.Count -gt 0)
{
$psi.Arguments = $ArgumentList
}
$process = New-Object System.Diagnostics.Process
$process.StartInfo = $psi
[void]$process.Start()
do
{
if (!$process.StandardOutput.EndOfStream)
{
[void]$Output.AppendLine($process.StandardOutput.ReadLine())
}
if (!$process.StandardError.EndOfStream)
{
[void]$Error.AppendLine($process.StandardError.ReadLine())
}
Start-Sleep -Milliseconds 10
} while (!$process.HasExited)
#read remainder
while (!$process.StandardOutput.EndOfStream)
{
#write-verbose 'read remaining output'
[void]$Output.AppendLine($process.StandardOutput.ReadLine())
}
while (!$process.StandardError.EndOfStream)
{
#write-verbose 'read remaining error'
[void]$Error.AppendLine($process.StandardError.ReadLine())
}
return #{ExitCode = $process.ExitCode; Output = $Output.ToString(); Error = $Error.ToString(); ExitTime=$process.ExitTime}
}
$p = Start-ProcessWithOutput "C:\Program Files\7-Zip\7z.exe" -ArgumentList "x","-y","-oE:\PowershellModules",$NewModules.FullName -verbose
$p.ExitCode
$p.Output
$p.Error
the 10ms sleep is to avoid spinning cpu when nothing to read.

I was getting the deadlock scenario mentioned by Ash using Justin's solution. Modified the script accordingly to subscribe to async event handlers to get the output and error text which accomplishes the same thing but avoids the deadlock condition.
Seemed to resolve the deadlock issue in my testing without altering the return data.
# Define global variables used in the Start-ProcessWithOutput function.
$global:processOutputStringGlobal = ""
$global:processErrorStringGlobal = ""
# Launch an executable and return the exitcode, output text, and error text of the process.
function Start-ProcessWithOutput
{
# Function requires a path to an executable and an optional list of arguments
param (
[Parameter(Mandatory=$true)] [string]$ExecutablePath,
[Parameter(Mandatory=$false)] [string[]]$ArgumentList
)
# Reset our global variables to an empty string in the event this process is called multiple times.
$global:processOutputStringGlobal = ""
$global:processErrorStringGlobal = ""
# Create the Process Info object which contains details about the process. We tell it to
# redirect standard output and error output which will be collected and stored in a variable.
$ProcessStartInfoObject = New-object System.Diagnostics.ProcessStartInfo
$ProcessStartInfoObject.FileName = $ExecutablePath
$ProcessStartInfoObject.CreateNoWindow = $true
$ProcessStartInfoObject.UseShellExecute = $false
$ProcessStartInfoObject.RedirectStandardOutput = $true
$ProcessStartInfoObject.RedirectStandardError = $true
# Add the arguments to the process info object if any were provided
if ($ArgumentList.Count -gt 0)
{
$ProcessStartInfoObject.Arguments = $ArgumentList
}
# Create the object that will represent the process
$Process = New-Object System.Diagnostics.Process
$Process.StartInfo = $ProcessStartInfoObject
# Define actions for the event handlers we will subscribe to in a moment. These are checking whether
# any data was sent by the event handler and updating the global variable if it is not null or empty.
$ProcessOutputEventAction = {
if ($null -ne $EventArgs.Data -and $EventArgs.Data -ne ""){
$global:processOutputStringGlobal += "$($EventArgs.Data)`r`n"
}
}
$ProcessErrorEventAction = {
if ($null -ne $EventArgs.Data -and $EventArgs.Data -ne ""){
$global:processErrorStringGlobal += "$($EventArgs.Data)`r`n"
}
}
# We need to create an event handler for the Process object. This will call the action defined above
# anytime that event is triggered. We are looking for output and error data received by the process
# and appending the global variables with those values.
Register-ObjectEvent -InputObject $Process -EventName "OutputDataReceived" -Action $ProcessOutputEventAction
Register-ObjectEvent -InputObject $Process -EventName "ErrorDataReceived" -Action $ProcessErrorEventAction
# Process starts here
[void]$Process.Start()
# This sets up an asyncronous task to read the console output from the process, which triggers the appropriate
# event, which we setup handlers for just above.
$Process.BeginErrorReadLine()
$Process.BeginOutputReadLine()
# Wait for the process to exit.
$Process.WaitForExit()
# We need to wait just a moment so the async tasks that are reading the output of the process can catch
# up. Not having this sleep here can cause the return values to be empty or incomplete. In my testing,
# it seemed like half a second was enough time to always get the data, but you may need to adjust accordingly.
Start-Sleep -Milliseconds 500
# Return an object that contains the exit code, output text, and error text.
return #{
ExitCode = $Process.ExitCode;
OutputString = $global:processOutputStringGlobal;
ErrorString = $global:processErrorStringGlobal;
ExitTime = $Process.ExitTime
}
}