We Keep Coding

Different PowerShell script behavior in different console

So i write Windows update'r automat in java. For fetching needed data from windows servers i am using jPowerShell and i have stamble apon weird problem while execute this script
Java calling PowerShell Script
PowerShell ps = PowerShell.openSession();
PowerShellResponse response;
response = ps.executeScript("C:\\Users\\Prezes\\Desktop\\IsUpdateToInstal.ps1");
System.out.println(response.getCommandOutput());
PowerShell Script
$pw = ConvertTo-SecureString 'password' -AsPlainText -Force
$cred = New-Object -TypeName System.Management.Automation.PSCredential -argumentList domein\login, $pw
Enter-PSSession -ComputerName IP -Credential $cred
$UpdateSession = [activator]::CreateInstance([type]::GetTypeFromProgID("Microsoft.Update.Session"))
$UpdateSearcher = $UpdateSession.CreateUpdateSearcher()
$SearchResult = $UpdateSearcher.Search("IsAssigned=1 and IsHidden=0 and IsInstalled=0")
$Critical = $SearchResult.updates | where { $_.MsrcSeverity -eq "Critical" }
$important = $SearchResult.updates | where { $_.MsrcSeverity -eq "Important" }
$other = $SearchResult.updates | where { $_.MsrcSeverity -eq $nul}
$totalUpdates = $($SearchResult.updates.count)
if($totalUpdates -gt 0)
{
$updatesToInstall = $true
}
else { $updatesToInstall = $false }
$other
$totalUpdates
$updatesToInstall
if i execute this script line by line in PowerShell standard consol everyting is working fine and proper value are returned.
But when i run this script in PowerShell ISE line by line or run by Java i notice some problem with this line
$UpdateSession = [activator]::CreateInstance([type]::GetTypeFromProgID("Microsoft.Update.Session"))
while i enter this line and press enter i can see in ISE "already running a command, please wait" when i wait a faw minutes communicate is the same and nothing change but when i press enter secound time command pass through immediately. If from them now i run rest of script evertying i working well.
When i try to excecute full script in ISE i am geting this error
Exception form HRESULT: 0x80072EE2
At C:\Users\Prezes\Desktop\IsUpdateToInstal.ps1:6 char:1
+ $SearchResult = $UpdateSearcher.Search("IsAssigned=1 and IsHidden=0 a ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [], COMException
+ FullyQualifiedErrorId : System.Runtime.InteropServices.COMExceptio
Java gives me null saying that i can't run methond on null object reffering to $UpdateSearcher
I am very early beginer with PowerShell and script that i am using is pure form some example finded in google.

So i had not figure out what was the cause of weird behavior but i managed to write someting that started to work for me through PowerShell api in java and get returned value.
$pw = ConvertTo-SecureString 'PASSWORD' -AsPlainText -Force
$cred = New-Object -TypeName System.Management.Automation.PSCredential -argumentList 792\opmanager, $pw
$TotalUpdates = Invoke-Command -ComputerName IP -ScriptBlock{
$UpdateSession = [activator]::CreateInstance([type]::GetTypeFromProgID("Microsoft.Update.Session"))
$UpdateSearcher = $UpdateSession.CreateUpdateSearcher()
$SearchResult = $UpdateSearcher.Search("IsAssigned=1 and IsHidden=0 and IsInstalled=0")
$Critical = $SearchResult.updates | where { $_.MsrcSeverity -eq "Critical" }
$important = $SearchResult.updates | where { $_.MsrcSeverity -eq "Important" }
$other = $SearchResult.updates | where { $_.MsrcSeverity -eq $nul}
$totalUpdates = $($SearchResult.updates.count)
if($totalUpdates -gt 0)
{
$updatesToInstall = $true
}
else { $updatesToInstall = $false }
Return $totalUpdates
} -Credential $cred
$TotalUpdates

Too Many Open Files (Selenium + PhantomJSDriver)

In my embedded Selenium/PhantomJSDriver driver it seems resources are not being cleaned up. Running the client synchronously causes millions of open files and eventually throws a "Too many files open" type exception.
Here is some output I gathered from lsof while the program is running for ~1 minute
$ lsof | awk '{ print $2; }' | uniq -c | sort -rn | head
1221966 12180
34790 29773
31260 12138
20955 8414
17940 10343
16665 32332
9512 27713
7275 19226
5496 7153
5040 14065
$ lsof -p 12180 | awk '{ print $2; }' | uniq -c | sort -rn | head
2859 12180
1 PID
$ lsof -p 12180 -Fn | sort -rn | uniq -c | sort -rn | head
1124 npipe
536 nanon_inode
4 nsocket
3 n/opt/jdk/jdk1.8.0_60/jre/lib/jce.jar
3 n/opt/jdk/jdk1.8.0_60/jre/lib/charsets.jar
3 n/dev/urandom
3 n/dev/random
3 n/dev/pts/20
2 n/usr/share/sbt-launcher-packaging/bin/sbt-launch.jar
2 n/usr/share/java/jayatana.jar
I don't understand why using the -p flag on lsof has a smaller result set. But it appears most of the entries are pipe and anon_inode.
The client is very simple at ~100 lines, and at the end of usage calls driver.close() and driver.quit(). I experimented with caching and reusing clients but it did not alleviate the open files
case class HeadlessClient(
country: String,
userAgent: String,
inheritSessionId: Option[Int] = None
) {
protected var numberOfRequests: Int = 0
protected val proxySessionId: Int = inheritSessionId.getOrElse(new Random().nextInt(Integer.MAX_VALUE))
protected val address = InetAddress.getByName("proxy.domain.com")
protected val host = address.getHostAddress
protected val login: String = HeadlessClient.username + proxySessionId
protected val windowSize = new org.openqa.selenium.Dimension(375, 667)
protected val (mobProxy, seleniumProxy) = {
val proxy = new BrowserMobProxyServer()
proxy.setTrustAllServers(true)
proxy.setChainedProxy(new InetSocketAddress(host, HeadlessClient.port))
proxy.chainedProxyAuthorization(login, HeadlessClient.password, AuthType.BASIC)
proxy.addLastHttpFilterFactory(new HttpFiltersSourceAdapter() {
override def filterRequest(originalRequest: HttpRequest): HttpFilters = {
new HttpFiltersAdapter(originalRequest) {
override def proxyToServerRequest(httpObject: HttpObject): io.netty.handler.codec.http.HttpResponse = {
httpObject match {
case req: HttpRequest => req.headers().remove(HttpHeaders.Names.VIA)
case _ =>
}
null
}
}
}
})
proxy.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT)
proxy.start(0)
val seleniumProxy = ClientUtil.createSeleniumProxy(proxy)
(proxy, seleniumProxy)
}
protected val driver: PhantomJSDriver = {
val capabilities: DesiredCapabilities = DesiredCapabilities.chrome()
val cliArgsCap = new util.ArrayList[String]
cliArgsCap.add("--webdriver-loglevel=NONE")
cliArgsCap.add("--ignore-ssl-errors=yes")
cliArgsCap.add("--load-images=no")
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy)
capabilities.setCapability("phantomjs.page.customHeaders.Referer", "")
capabilities.setCapability("phantomjs.page.settings.userAgent", userAgent)
capabilities.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS, cliArgsCap)
new PhantomJSDriver(capabilities)
}
driver.executePhantomJS(
"""
|var navigation = [];
|
|this.onNavigationRequested = function(url, type, willNavigate, main) {
| navigation.push(url)
| console.log('Trying to navigate to: ' + url);
|}
|
|this.onResourceRequested = function(request, net) {
| console.log("Requesting " + request.url);
| if (! (navigation.indexOf(request.url) > -1)) {
| console.log("Aborting " + request.url)
| net.abort();
| }
|};
""".stripMargin
)
driver.manage().window().setSize(windowSize)
def follow(url: String)(implicit ec: ExecutionContext): List[HarEntry] = {
try{
Await.result(Future{
mobProxy.newHar(url)
driver.get(url)
val entries = mobProxy.getHar.getLog.getEntries.asScala.toList
shutdown()
entries
}, 45.seconds)
} catch {
case e: Exception =>
try {
shutdown()
} catch {
case shutdown: Exception =>
throw new Exception(s"Error ${shutdown.getMessage} cleaning up after Exception: ${e.getMessage}")
}
throw e
}
}
def shutdown() = {
driver.close()
driver.quit()
}
}
I tried several versions of Selenium in case there was a bugfix. The build.sbt:
libraryDependencies += "org.seleniumhq.selenium" % "selenium-java" % "3.0.1"
libraryDependencies += "net.lightbody.bmp" % "browsermob-core" % "2.1.2"
Also, I tried PhantomJS 2.0.1, and 2.1.1:
$ phantomjs --version
2.0.1-development
$ phantomjs --version
2.1.1
Is this a PhantomJS or Selenium problem? Is my client using the API improperly?

The resource usage is caused by BrowserMob. To close the proxy and clean-up its resources, one must call stop().
For this client that means modifying the shutdown method
def shutdown() = {
mobProxy.stop()
driver.close()
driver.quit()
}
Another method, abort, offers immediate termination of the proxy server and does not wait for traffic to cease.

In my opinion it seems a problem of PhantomJS. You can try the following alternatives:
Use phantomjs 2.5.0-beta. It has been recently released. I'm not sure if this upgrade solves your problem, but at least it is worth to give a try. According to the changelog, the new features of this version are:
Upgrade QtWebKit to QtWebKitNG
Upgraded Qt to 5.7.1
Clean the phantomjs processes after closing webdriver. You can implement your own cleaner to force that phantomjs is actually closed after driver.close() (invoking killall -9 phantomjs or similar).

PowerShell | System.Diagnostics.Process | Java | Hide Window

I try to start an Java program from a PowerShell script, but I don't want see any window. I have already tried all possible combinations of parameters for ProcessStartInfo and Process, but it still displays a empty shell window for java.exe. This is my current PowerShell code:
$pinfo = New-Object System.Diagnostics.ProcessStartInfo
$pinfo.FileName = "$jre\bin\java.exe"
$pinfo.RedirectStandardError = $true
$pinfo.RedirectStandardOutput = $true
$pinfo.CreateNoWindow = $true
$pinfo.WindowStyle = "Hidden"
$pinfo.UseShellExecute = $false
$pinfo.WorkingDirectory = $temp
if ($integrated) {
Write-Output "Connecting using integrated security ..."
# Start with different user context
$securePassword = ConvertTo-SecureString $pw -AsPlainText -Force
$credential = New-Object System.Management.Automation.PSCredential $user, $securePassword
$pinfo.UserName = $credential.GetNetworkCredential().UserName
$pinfo.Domain = $credential.GetNetworkCredential().Domain
$pinfo.Password = $credential.Password
$pinfo.Arguments = "-jar `"$jar`" validate -url `"$jdbc`" -i -dbVersion `"$msSqlVersion`""
} else {
Write-Output "Connecting ..."
$pinfo.Arguments = "-jar `"$jar`" validate -url `"$jdbc`" -u `"$user`" -p `"$pw`" -dbVersion `"$dbVersion`""
}
$p = New-Object System.Diagnostics.Process
$p.StartInfo = $pinfo
$p.Start()
$p.StandardOutput.ReadToEnd()
$p.StandardError.ReadToEnd()
$p.WaitForExit()
I hope anyone could help me with this issue.

Performance Issue: convert csv to excel using Powershell

I am using below power shell code to convert csv to excel. But the performance seems bad, for a 200mb csv file, I've been waiting for 20 mins. Anyone have idea how to improve the performance. Or do I have to do pragmatically, (using java using apache poi library)?
function convertToExcel {
param($fileName, $outputDir) # the two parameters.
#generate OutputFileName
$outputFilename = -join($fileName.BaseName, ".xlsx");
#prepare excel sheet
$excel = New-Object -ComObject Excel.Application
#$excel.visible = $true
$workbook = $excel.workbooks.add()
$sheets = $workbook.sheets
$sheetCount = $Sheets.count
$mySheet = 1
$mySheetName = "Sheet" + $mySheet
$s1 = $sheets | where{$_.name -eq $mySheetName}
$s1.Activate()
#Delete all other Sheets
If ($sheetCount -gt 1) {
$Sheets | ForEach {
$tmpSheetName = $_.Name
$tmpSheet = $_
If ($tmpSheetName -ne "Sheet1") {
$tmpSheet.Delete()
}
}
}
#copy paste data from csv
#$s1.Name = $fileName.BaseName
$s1.Name = "sheet1"
$s1.Activate()
$s1Data = Import-Csv $file.FullName -Delimiter '^'
$s1Data | ConvertTo-Csv -Delimiter "`t" -NoTypeInformation | Clip
$s1.cells.item(1,1).Select()
$s1.Paste()
$workbook.SaveAs("$outputDir\$outputFilename")
$excel.Quit()
}
#Read Input from Arguments
if ($args.length -ne 3){
}else{
$fileName = $Args[0] #read fileName
$inputFolder = $Args[1] #read inputFolder
$outputFolder = $Args[2] #read outputFolder
$files = dir -Path $inputFolder $fileName
ForEach ($file in $files) {
convertToExcel $file $outputFolder
}
}

Have you tried just opening a CSV file in Excel? This is probably the fastest method.
$app = New-Object -ComObject Excel.Application
$app.Visible = $true
$app.DisplayAlerts = $False
$app.Workbooks.Open("yourcsvfile.csv")
The other way would be reading the file by rows and adding entire rows, but it will be slower.
However, converting to PS object and writing cell by cell will always be slowest.
Edit: Here is a full function...
function ConvertToExcel {
param($fileName)
$outputFilename = -join($fileName.split('.')[0], ".xls")
$excel = New-Object -ComObject Excel.Application
#$excel.Visible = $true
$excel.DisplayAlerts = $False
$wb = $excel.Workbooks.Open($fileName)
$wb.SaveAs("$outputFilename", [Microsoft.Office.Interop.Excel.XlFileFormat]::xlExcel8)
Start-Sleep -Milliseconds 200
$excel.Quit()
[System.Runtime.Interopservices.Marshal]::ReleaseComObject($excel)
Remove-Variable excel
}
ConvertToExcel "D:\tst\serverlist.csv"

Powershell: Capturing standard out and error with Process object

I want to start a Java program from PowerShell and get the results printed on the console.
I have followed the instructions of this question:
Capturing standard out and error with Start-Process
But for me, this is not working as I expected. What I'm doing wrong?
This is the script:
$psi = New-object System.Diagnostics.ProcessStartInfo
$psi.CreateNoWindow = $true
$psi.UseShellExecute = $false
$psi.RedirectStandardOutput = $true
$psi.RedirectStandardError = $true
$psi.FileName = 'java.exe'
$psi.Arguments = #("-jar","tools\compiler.jar","--compilation_level", "ADVANCED_OPTIMIZATIONS", "--js", $BuildFile, "--js_output_file", $BuildMinFile)
$process = New-Object System.Diagnostics.Process
$process.StartInfo = $psi
$process.Start() | Out-Null
$process.WaitForExit()
$output = $process.StandardOutput.ReadToEnd()
$output
The $output variable is always empty (and nothing is printed on the console of course).

The docs on the RedirectStandardError property suggests that it is better to put the WaitForExit() call after the ReadToEnd() call. The following works correctly for me:
$psi = New-object System.Diagnostics.ProcessStartInfo
$psi.CreateNoWindow = $true
$psi.UseShellExecute = $false
$psi.RedirectStandardOutput = $true
$psi.RedirectStandardError = $true
$psi.FileName = 'ipconfig.exe'
$psi.Arguments = #("/a")
$process = New-Object System.Diagnostics.Process
$process.StartInfo = $psi
[void]$process.Start()
$output = $process.StandardOutput.ReadToEnd()
$process.WaitForExit()
$output

Small variation so that you can selectively print the output if needed. As in if your looking just for error or warning messages and by the way Keith you saved my bacon with your response...
$psi = New-object System.Diagnostics.ProcessStartInfo
$psi.CreateNoWindow = $true
$psi.UseShellExecute = $false
$psi.RedirectStandardOutput = $true
$psi.RedirectStandardError = $true
$psi.FileName = 'robocopy'
$psi.Arguments = #("$HomeDirectory $NewHomeDirectory /MIR /XF desktop.ini /XD VDI /R:0 /W:0 /s /v /np")
$process = New-Object System.Diagnostics.Process
$process.StartInfo = $psi
[void]$process.Start()
do
{
$process.StandardOutput.ReadLine()
}
while (!$process.HasExited)

Here is a modification to paul's answer, hopefully it addresses the truncated output. i did a test using a failure and did not see truncation.
function Start-ProcessWithOutput
{
param ([string]$Path,[string[]]$ArgumentList)
$Output = New-Object -TypeName System.Text.StringBuilder
$Error = New-Object -TypeName System.Text.StringBuilder
$psi = New-object System.Diagnostics.ProcessStartInfo
$psi.CreateNoWindow = $true
$psi.UseShellExecute = $false
$psi.RedirectStandardOutput = $true
$psi.RedirectStandardError = $true
$psi.FileName = $Path
if ($ArgumentList.Count -gt 0)
{
$psi.Arguments = $ArgumentList
}
$process = New-Object System.Diagnostics.Process
$process.StartInfo = $psi
[void]$process.Start()
do
{
if (!$process.StandardOutput.EndOfStream)
{
[void]$Output.AppendLine($process.StandardOutput.ReadLine())
}
if (!$process.StandardError.EndOfStream)
{
[void]$Error.AppendLine($process.StandardError.ReadLine())
}
Start-Sleep -Milliseconds 10
} while (!$process.HasExited)
#read remainder
while (!$process.StandardOutput.EndOfStream)
{
#write-verbose 'read remaining output'
[void]$Output.AppendLine($process.StandardOutput.ReadLine())
}
while (!$process.StandardError.EndOfStream)
{
#write-verbose 'read remaining error'
[void]$Error.AppendLine($process.StandardError.ReadLine())
}
return #{ExitCode = $process.ExitCode; Output = $Output.ToString(); Error = $Error.ToString(); ExitTime=$process.ExitTime}
}
$p = Start-ProcessWithOutput "C:\Program Files\7-Zip\7z.exe" -ArgumentList "x","-y","-oE:\PowershellModules",$NewModules.FullName -verbose
$p.ExitCode
$p.Output
$p.Error
the 10ms sleep is to avoid spinning cpu when nothing to read.

I was getting the deadlock scenario mentioned by Ash using Justin's solution. Modified the script accordingly to subscribe to async event handlers to get the output and error text which accomplishes the same thing but avoids the deadlock condition.
Seemed to resolve the deadlock issue in my testing without altering the return data.
# Define global variables used in the Start-ProcessWithOutput function.
$global:processOutputStringGlobal = ""
$global:processErrorStringGlobal = ""
# Launch an executable and return the exitcode, output text, and error text of the process.
function Start-ProcessWithOutput
{
# Function requires a path to an executable and an optional list of arguments
param (
[Parameter(Mandatory=$true)] [string]$ExecutablePath,
[Parameter(Mandatory=$false)] [string[]]$ArgumentList
)
# Reset our global variables to an empty string in the event this process is called multiple times.
$global:processOutputStringGlobal = ""
$global:processErrorStringGlobal = ""
# Create the Process Info object which contains details about the process. We tell it to
# redirect standard output and error output which will be collected and stored in a variable.
$ProcessStartInfoObject = New-object System.Diagnostics.ProcessStartInfo
$ProcessStartInfoObject.FileName = $ExecutablePath
$ProcessStartInfoObject.CreateNoWindow = $true
$ProcessStartInfoObject.UseShellExecute = $false
$ProcessStartInfoObject.RedirectStandardOutput = $true
$ProcessStartInfoObject.RedirectStandardError = $true
# Add the arguments to the process info object if any were provided
if ($ArgumentList.Count -gt 0)
{
$ProcessStartInfoObject.Arguments = $ArgumentList
}
# Create the object that will represent the process
$Process = New-Object System.Diagnostics.Process
$Process.StartInfo = $ProcessStartInfoObject
# Define actions for the event handlers we will subscribe to in a moment. These are checking whether
# any data was sent by the event handler and updating the global variable if it is not null or empty.
$ProcessOutputEventAction = {
if ($null -ne $EventArgs.Data -and $EventArgs.Data -ne ""){
$global:processOutputStringGlobal += "$($EventArgs.Data)`r`n"
}
}
$ProcessErrorEventAction = {
if ($null -ne $EventArgs.Data -and $EventArgs.Data -ne ""){
$global:processErrorStringGlobal += "$($EventArgs.Data)`r`n"
}
}
# We need to create an event handler for the Process object. This will call the action defined above
# anytime that event is triggered. We are looking for output and error data received by the process
# and appending the global variables with those values.
Register-ObjectEvent -InputObject $Process -EventName "OutputDataReceived" -Action $ProcessOutputEventAction
Register-ObjectEvent -InputObject $Process -EventName "ErrorDataReceived" -Action $ProcessErrorEventAction
# Process starts here
[void]$Process.Start()
# This sets up an asyncronous task to read the console output from the process, which triggers the appropriate
# event, which we setup handlers for just above.
$Process.BeginErrorReadLine()
$Process.BeginOutputReadLine()
# Wait for the process to exit.
$Process.WaitForExit()
# We need to wait just a moment so the async tasks that are reading the output of the process can catch
# up. Not having this sleep here can cause the return values to be empty or incomplete. In my testing,
# it seemed like half a second was enough time to always get the data, but you may need to adjust accordingly.
Start-Sleep -Milliseconds 500
# Return an object that contains the exit code, output text, and error text.
return #{
ExitCode = $Process.ExitCode;
OutputString = $global:processOutputStringGlobal;
ErrorString = $global:processErrorStringGlobal;
ExitTime = $Process.ExitTime
}
}