Performance Issue: convert csv to excel using Powershell - java

I am using below power shell code to convert csv to excel. But the performance seems bad, for a 200mb csv file, I've been waiting for 20 mins. Anyone have idea how to improve the performance. Or do I have to do pragmatically, (using java using apache poi library)?
function convertToExcel {
param($fileName, $outputDir) # the two parameters.
#generate OutputFileName
$outputFilename = -join($fileName.BaseName, ".xlsx");
#prepare excel sheet
$excel = New-Object -ComObject Excel.Application
#$excel.visible = $true
$workbook = $excel.workbooks.add()
$sheets = $workbook.sheets
$sheetCount = $Sheets.count
$mySheet = 1
$mySheetName = "Sheet" + $mySheet
$s1 = $sheets | where{$_.name -eq $mySheetName}
$s1.Activate()
#Delete all other Sheets
If ($sheetCount -gt 1) {
$Sheets | ForEach {
$tmpSheetName = $_.Name
$tmpSheet = $_
If ($tmpSheetName -ne "Sheet1") {
$tmpSheet.Delete()
}
}
}
#copy paste data from csv
#$s1.Name = $fileName.BaseName
$s1.Name = "sheet1"
$s1.Activate()
$s1Data = Import-Csv $file.FullName -Delimiter '^'
$s1Data | ConvertTo-Csv -Delimiter "`t" -NoTypeInformation | Clip
$s1.cells.item(1,1).Select()
$s1.Paste()
$workbook.SaveAs("$outputDir\$outputFilename")
$excel.Quit()
}
#Read Input from Arguments
if ($args.length -ne 3){
}else{
$fileName = $Args[0] #read fileName
$inputFolder = $Args[1] #read inputFolder
$outputFolder = $Args[2] #read outputFolder
$files = dir -Path $inputFolder $fileName
ForEach ($file in $files) {
convertToExcel $file $outputFolder
}
}

Have you tried just opening a CSV file in Excel? This is probably the fastest method.
$app = New-Object -ComObject Excel.Application
$app.Visible = $true
$app.DisplayAlerts = $False
$app.Workbooks.Open("yourcsvfile.csv")
The other way would be reading the file by rows and adding entire rows, but it will be slower.
However, converting to PS object and writing cell by cell will always be slowest.
Edit: Here is a full function...
function ConvertToExcel {
param($fileName)
$outputFilename = -join($fileName.split('.')[0], ".xls")
$excel = New-Object -ComObject Excel.Application
#$excel.Visible = $true
$excel.DisplayAlerts = $False
$wb = $excel.Workbooks.Open($fileName)
$wb.SaveAs("$outputFilename", [Microsoft.Office.Interop.Excel.XlFileFormat]::xlExcel8)
Start-Sleep -Milliseconds 200
$excel.Quit()
[System.Runtime.Interopservices.Marshal]::ReleaseComObject($excel)
Remove-Variable excel
}
ConvertToExcel "D:\tst\serverlist.csv"

Related

Different PowerShell script behavior in different console

So i write Windows update'r automat in java. For fetching needed data from windows servers i am using jPowerShell and i have stamble apon weird problem while execute this script
Java calling PowerShell Script
PowerShell ps = PowerShell.openSession();
PowerShellResponse response;
response = ps.executeScript("C:\\Users\\Prezes\\Desktop\\IsUpdateToInstal.ps1");
System.out.println(response.getCommandOutput());
PowerShell Script
$pw = ConvertTo-SecureString 'password' -AsPlainText -Force
$cred = New-Object -TypeName System.Management.Automation.PSCredential -argumentList domein\login, $pw
Enter-PSSession -ComputerName IP -Credential $cred
$UpdateSession = [activator]::CreateInstance([type]::GetTypeFromProgID("Microsoft.Update.Session"))
$UpdateSearcher = $UpdateSession.CreateUpdateSearcher()
$SearchResult = $UpdateSearcher.Search("IsAssigned=1 and IsHidden=0 and IsInstalled=0")
$Critical = $SearchResult.updates | where { $_.MsrcSeverity -eq "Critical" }
$important = $SearchResult.updates | where { $_.MsrcSeverity -eq "Important" }
$other = $SearchResult.updates | where { $_.MsrcSeverity -eq $nul}
$totalUpdates = $($SearchResult.updates.count)
if($totalUpdates -gt 0)
{
$updatesToInstall = $true
}
else { $updatesToInstall = $false }
$other
$totalUpdates
$updatesToInstall
if i execute this script line by line in PowerShell standard consol everyting is working fine and proper value are returned.
But when i run this script in PowerShell ISE line by line or run by Java i notice some problem with this line
$UpdateSession = [activator]::CreateInstance([type]::GetTypeFromProgID("Microsoft.Update.Session"))
while i enter this line and press enter i can see in ISE "already running a command, please wait" when i wait a faw minutes communicate is the same and nothing change but when i press enter secound time command pass through immediately. If from them now i run rest of script evertying i working well.
When i try to excecute full script in ISE i am geting this error
Exception form HRESULT: 0x80072EE2
At C:\Users\Prezes\Desktop\IsUpdateToInstal.ps1:6 char:1
+ $SearchResult = $UpdateSearcher.Search("IsAssigned=1 and IsHidden=0 a ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [], COMException
+ FullyQualifiedErrorId : System.Runtime.InteropServices.COMExceptio
Java gives me null saying that i can't run methond on null object reffering to $UpdateSearcher
I am very early beginer with PowerShell and script that i am using is pure form some example finded in google.
So i had not figure out what was the cause of weird behavior but i managed to write someting that started to work for me through PowerShell api in java and get returned value.
$pw = ConvertTo-SecureString 'PASSWORD' -AsPlainText -Force
$cred = New-Object -TypeName System.Management.Automation.PSCredential -argumentList 792\opmanager, $pw
$TotalUpdates = Invoke-Command -ComputerName IP -ScriptBlock{
$UpdateSession = [activator]::CreateInstance([type]::GetTypeFromProgID("Microsoft.Update.Session"))
$UpdateSearcher = $UpdateSession.CreateUpdateSearcher()
$SearchResult = $UpdateSearcher.Search("IsAssigned=1 and IsHidden=0 and IsInstalled=0")
$Critical = $SearchResult.updates | where { $_.MsrcSeverity -eq "Critical" }
$important = $SearchResult.updates | where { $_.MsrcSeverity -eq "Important" }
$other = $SearchResult.updates | where { $_.MsrcSeverity -eq $nul}
$totalUpdates = $($SearchResult.updates.count)
if($totalUpdates -gt 0)
{
$updatesToInstall = $true
}
else { $updatesToInstall = $false }
Return $totalUpdates
} -Credential $cred
$TotalUpdates

PowerShell | System.Diagnostics.Process | Java | Hide Window

I try to start an Java program from a PowerShell script, but I don't want see any window. I have already tried all possible combinations of parameters for ProcessStartInfo and Process, but it still displays a empty shell window for java.exe. This is my current PowerShell code:
$pinfo = New-Object System.Diagnostics.ProcessStartInfo
$pinfo.FileName = "$jre\bin\java.exe"
$pinfo.RedirectStandardError = $true
$pinfo.RedirectStandardOutput = $true
$pinfo.CreateNoWindow = $true
$pinfo.WindowStyle = "Hidden"
$pinfo.UseShellExecute = $false
$pinfo.WorkingDirectory = $temp
if ($integrated) {
Write-Output "Connecting using integrated security ..."
# Start with different user context
$securePassword = ConvertTo-SecureString $pw -AsPlainText -Force
$credential = New-Object System.Management.Automation.PSCredential $user, $securePassword
$pinfo.UserName = $credential.GetNetworkCredential().UserName
$pinfo.Domain = $credential.GetNetworkCredential().Domain
$pinfo.Password = $credential.Password
$pinfo.Arguments = "-jar `"$jar`" validate -url `"$jdbc`" -i -dbVersion `"$msSqlVersion`""
} else {
Write-Output "Connecting ..."
$pinfo.Arguments = "-jar `"$jar`" validate -url `"$jdbc`" -u `"$user`" -p `"$pw`" -dbVersion `"$dbVersion`""
}
$p = New-Object System.Diagnostics.Process
$p.StartInfo = $pinfo
$p.Start()
$p.StandardOutput.ReadToEnd()
$p.StandardError.ReadToEnd()
$p.WaitForExit()
I hope anyone could help me with this issue.

Getting output from Perl script called from Java code

I've read a lot of questions/examples on this issue, but unfortunately I have not been able to solve my problem. I need to call a Perl script (that I cannot change) from Java code, and then I need to get the output from that script.
The script is used to take student programming homework assignments, and check them for copying by comparing all of them together. The script can take ~45 seconds to run, but only requires the arguments to be properly formatted, there is no interactivity.
My issue is when I call the script from my Java code I get the first line of output from the script but nothing else. I'm using Runtime.exe() and then waitFor() to wait for the script to finish. However the waitFor() function returns before the script actually finishes. I don't know any Perl so I'm not sure if the script is doing something that 'confuses' the java Process object, or if there's an issue in my code.
Process run = Runtime.getRuntime().exec(cmd);
output = new BufferedReader(new InputStreamReader(run.getInputStream()));
run.waitFor();
String temp;
while((temp = output.readLine()) != null){
System.out.println(temp);
}
The Perl script..
use IO::Socket;
#
# As of the date this script was written, the following languages were supported. This script will work with
# languages added later however. Check the moss website for the full list of supported languages.
#
#languages = ("c", "cc", "java", "ml", "pascal", "ada", "lisp", "scheme", "haskell", "fortran", "ascii", "vhdl", "perl", "matlab", "python", "mips", "prolog", "spice", "vb", "csharp", "modula2", "a8086", "javascript", "plsql", "verilog");
$server = 'moss.stanford.edu';
$port = '7690';
$noreq = "Request not sent.";
$usage = "usage: moss [-x] [-l language] [-d] [-b basefile1] ... [-b basefilen] [-m #] [-c \"string\"] file1 file2 file3 ...";
#
# The userid is used to authenticate your queries to the server; don't change it!
#
$userid=[REDACTED];
#
# Process the command line options. This is done in a non-standard
# way to allow multiple -b's.
#
$opt_l = "c"; # default language is c
$opt_m = 10;
$opt_d = 0;
$opt_x = 0;
$opt_c = "";
$opt_n = 250;
$bindex = 0; # this becomes non-zero if we have any base files
while (#ARGV && ($_ = $ARGV[0]) =~ /^-(.)(.*)/) {
($first,$rest) = ($1,$2);
shift(#ARGV);
if ($first eq "d") {
$opt_d = 1;
next;
}
if ($first eq "b") {
if($rest eq '') {
die "No argument for option -b.\n" unless #ARGV;
$rest = shift(#ARGV);
}
$opt_b[$bindex++] = $rest;
next;
}
if ($first eq "l") {
if ($rest eq '') {
die "No argument for option -l.\n" unless #ARGV;
$rest = shift(#ARGV);
}
$opt_l = $rest;
next;
}
if ($first eq "m") {
if($rest eq '') {
die "No argument for option -m.\n" unless #ARGV;
$rest = shift(#ARGV);
}
$opt_m = $rest;
next;
}
if ($first eq "c") {
if($rest eq '') {
die "No argument for option -c.\n" unless #ARGV;
$rest = shift(#ARGV);
}
$opt_c = $rest;
next;
}
if ($first eq "n") {
if($rest eq '') {
die "No argument for option -n.\n" unless #ARGV;
$rest = shift(#ARGV);
}
$opt_n = $rest;
next;
}
if ($first eq "x") {
$opt_x = 1;
next;
}
#
# Override the name of the server. This is used for testing this script.
#
if ($first eq "s") {
$server = shift(#ARGV);
next;
}
#
# Override the port. This is used for testing this script.
#
if ($first eq "p") {
$port = shift(#ARGV);
next;
}
die "Unrecognized option -$first. $usage\n";
}
#
# Check a bunch of things first to ensure that the
# script will be able to run to completion.
#
#
# Make sure all the argument files exist and are readable.
#
print "Checking files . . . \n";
$i = 0;
while($i < $bindex)
{
die "Base file $opt_b[$i] does not exist. $noreq\n" unless -e "$opt_b[$i]";
die "Base file $opt_b[$i] is not readable. $noreq\n" unless -r "$opt_b[$i]";
die "Base file $opt_b is not a text file. $noreq\n" unless -T "$opt_b[$i]";
$i++;
}
foreach $file (#ARGV)
{
die "File $file does not exist. $noreq\n" unless -e "$file";
die "File $file is not readable. $noreq\n" unless -r "$file";
die "File $file is not a text file. $noreq\n" unless -T "$file";
}
if ("#ARGV" eq '') {
die "No files submitted.\n $usage";
}
print "OK\n";
#
# Now the real processing begins.
#
$sock = new IO::Socket::INET (
PeerAddr => $server,
PeerPort => $port,
Proto => 'tcp',
);
die "Could not connect to server $server: $!\n" unless $sock;
$sock->autoflush(1);
sub read_from_server {
$msg = <$sock>;
print $msg;
}
sub upload_file {
local ($file, $id, $lang) = #_;
#
# The stat function does not seem to give correct filesizes on windows, so
# we compute the size here via brute force.
#
open(F,$file);
$size = 0;
while (<F>) {
$size += length($_);
}
close(F);
print "Uploading $file ...";
open(F,$file);
$file =~s/\s/\_/g; # replace blanks in filename with underscores
print $sock "file $id $lang $size $file\n";
while (<F>) {
print $sock $_;
}
close(F);
print "done.\n";
}
print $sock "moss $userid\n"; # authenticate user
print $sock "directory $opt_d\n";
print $sock "X $opt_x\n";
print $sock "maxmatches $opt_m\n";
print $sock "show $opt_n\n";
#
# confirm that we have a supported languages
#
print $sock "language $opt_l\n";
$msg = <$sock>;
chop($msg);
if ($msg eq "no") {
print $sock "end\n";
die "Unrecognized language $opt_l.";
}
# upload any base files
$i = 0;
while($i < $bindex) {
&upload_file($opt_b[$i++],0,$opt_l);
}
$setid = 1;
foreach $file (#ARGV) {
&upload_file($file,$setid++,$opt_l);
}
print $sock "query 0 $opt_c\n";
print "Query submitted. Waiting for the server's response.\n";
&read_from_server();
print $sock "end\n";
close($sock);
Thank you for any input you may have on my problem.
You can use my native Java client for MOSS instead .
Don't mind the version number. I've been using it in production successfully.

getting video duration with ffmpeg, php function

I have FFMpeg installed and I know it's functional, but i'm trying to get the duration time from a flv video through PHP but when I use this code:
function mbmGetFLVDuration($file){
/*
* Determine video duration with ffmpeg
* ffmpeg should be installed on your server.
*/
//$time = 00:00:00.000 format
$ffmpeg = "../ffmpeg/ffmpeg";
$time = exec("$ffmpeg -i $file 2>&1 | grep 'Duration' | cut -d ' ' -f 4 | sed s/,//");
$duration = explode(":",$time);
$duration_in_seconds = $duration[0]*3600 + $duration[1]*60+ round($duration[2]);
return $duration_in_seconds;
}
and:
$duration = mbmGetFLVDuration('http://www.videoaddsite.com/videos/intro.flv');
echo $duration;
I get an output of 220. THe video is 3:40. Can any help me on what i'm doing wrong, or if there's something else I can use?
I dont see a problem. 220 seconds are 3:40.
To get minutes and seconds use this conversion:
<?php
$seconds = 220;
$minutes = $seconds/60;
$real_minutes = floor($minutes);
$real_seconds = round(($minutes-$real_minutes)*60);
?>
$real_minutes will be 3 and $real_seconds will be 40.
$ffmpeg = "../ffmpeg/ffmpeg";
$time = exec("$ffmpeg -i $file 2>&1 | grep 'Duration' | cut -d ' ' -f 4 | sed s/,//");
$duration = explode(":", $time);
$duration_in_seconds = ($duration[0] * 3600) + ($duration[1] * 60) + round($duration[2]);
return $duration_in_seconds;
The following code works for me
$file='http://techslides.com/demos/sample-videos/small.mp4';
$dur = shell_exec("ffmpeg -i ".$file." 2>&1");
if(preg_match("/: Invalid /", $dur)){
return false;
}
preg_match("/Duration: (.{2}):(.{2}):(.{2})/", $dur, $duration);
if(!isset($duration[1])){
return false;
}
$hours = $duration[1];
$minutes = $duration[2];
$seconds = $duration[3];
echo $seconds + ($minutes*60) + ($hours*60*60);
Reference

Powershell: Capturing standard out and error with Process object

I want to start a Java program from PowerShell and get the results printed on the console.
I have followed the instructions of this question:
Capturing standard out and error with Start-Process
But for me, this is not working as I expected. What I'm doing wrong?
This is the script:
$psi = New-object System.Diagnostics.ProcessStartInfo
$psi.CreateNoWindow = $true
$psi.UseShellExecute = $false
$psi.RedirectStandardOutput = $true
$psi.RedirectStandardError = $true
$psi.FileName = 'java.exe'
$psi.Arguments = #("-jar","tools\compiler.jar","--compilation_level", "ADVANCED_OPTIMIZATIONS", "--js", $BuildFile, "--js_output_file", $BuildMinFile)
$process = New-Object System.Diagnostics.Process
$process.StartInfo = $psi
$process.Start() | Out-Null
$process.WaitForExit()
$output = $process.StandardOutput.ReadToEnd()
$output
The $output variable is always empty (and nothing is printed on the console of course).
The docs on the RedirectStandardError property suggests that it is better to put the WaitForExit() call after the ReadToEnd() call. The following works correctly for me:
$psi = New-object System.Diagnostics.ProcessStartInfo
$psi.CreateNoWindow = $true
$psi.UseShellExecute = $false
$psi.RedirectStandardOutput = $true
$psi.RedirectStandardError = $true
$psi.FileName = 'ipconfig.exe'
$psi.Arguments = #("/a")
$process = New-Object System.Diagnostics.Process
$process.StartInfo = $psi
[void]$process.Start()
$output = $process.StandardOutput.ReadToEnd()
$process.WaitForExit()
$output
Small variation so that you can selectively print the output if needed. As in if your looking just for error or warning messages and by the way Keith you saved my bacon with your response...
$psi = New-object System.Diagnostics.ProcessStartInfo
$psi.CreateNoWindow = $true
$psi.UseShellExecute = $false
$psi.RedirectStandardOutput = $true
$psi.RedirectStandardError = $true
$psi.FileName = 'robocopy'
$psi.Arguments = #("$HomeDirectory $NewHomeDirectory /MIR /XF desktop.ini /XD VDI /R:0 /W:0 /s /v /np")
$process = New-Object System.Diagnostics.Process
$process.StartInfo = $psi
[void]$process.Start()
do
{
$process.StandardOutput.ReadLine()
}
while (!$process.HasExited)
Here is a modification to paul's answer, hopefully it addresses the truncated output. i did a test using a failure and did not see truncation.
function Start-ProcessWithOutput
{
param ([string]$Path,[string[]]$ArgumentList)
$Output = New-Object -TypeName System.Text.StringBuilder
$Error = New-Object -TypeName System.Text.StringBuilder
$psi = New-object System.Diagnostics.ProcessStartInfo
$psi.CreateNoWindow = $true
$psi.UseShellExecute = $false
$psi.RedirectStandardOutput = $true
$psi.RedirectStandardError = $true
$psi.FileName = $Path
if ($ArgumentList.Count -gt 0)
{
$psi.Arguments = $ArgumentList
}
$process = New-Object System.Diagnostics.Process
$process.StartInfo = $psi
[void]$process.Start()
do
{
if (!$process.StandardOutput.EndOfStream)
{
[void]$Output.AppendLine($process.StandardOutput.ReadLine())
}
if (!$process.StandardError.EndOfStream)
{
[void]$Error.AppendLine($process.StandardError.ReadLine())
}
Start-Sleep -Milliseconds 10
} while (!$process.HasExited)
#read remainder
while (!$process.StandardOutput.EndOfStream)
{
#write-verbose 'read remaining output'
[void]$Output.AppendLine($process.StandardOutput.ReadLine())
}
while (!$process.StandardError.EndOfStream)
{
#write-verbose 'read remaining error'
[void]$Error.AppendLine($process.StandardError.ReadLine())
}
return #{ExitCode = $process.ExitCode; Output = $Output.ToString(); Error = $Error.ToString(); ExitTime=$process.ExitTime}
}
$p = Start-ProcessWithOutput "C:\Program Files\7-Zip\7z.exe" -ArgumentList "x","-y","-oE:\PowershellModules",$NewModules.FullName -verbose
$p.ExitCode
$p.Output
$p.Error
the 10ms sleep is to avoid spinning cpu when nothing to read.
I was getting the deadlock scenario mentioned by Ash using Justin's solution. Modified the script accordingly to subscribe to async event handlers to get the output and error text which accomplishes the same thing but avoids the deadlock condition.
Seemed to resolve the deadlock issue in my testing without altering the return data.
# Define global variables used in the Start-ProcessWithOutput function.
$global:processOutputStringGlobal = ""
$global:processErrorStringGlobal = ""
# Launch an executable and return the exitcode, output text, and error text of the process.
function Start-ProcessWithOutput
{
# Function requires a path to an executable and an optional list of arguments
param (
[Parameter(Mandatory=$true)] [string]$ExecutablePath,
[Parameter(Mandatory=$false)] [string[]]$ArgumentList
)
# Reset our global variables to an empty string in the event this process is called multiple times.
$global:processOutputStringGlobal = ""
$global:processErrorStringGlobal = ""
# Create the Process Info object which contains details about the process. We tell it to
# redirect standard output and error output which will be collected and stored in a variable.
$ProcessStartInfoObject = New-object System.Diagnostics.ProcessStartInfo
$ProcessStartInfoObject.FileName = $ExecutablePath
$ProcessStartInfoObject.CreateNoWindow = $true
$ProcessStartInfoObject.UseShellExecute = $false
$ProcessStartInfoObject.RedirectStandardOutput = $true
$ProcessStartInfoObject.RedirectStandardError = $true
# Add the arguments to the process info object if any were provided
if ($ArgumentList.Count -gt 0)
{
$ProcessStartInfoObject.Arguments = $ArgumentList
}
# Create the object that will represent the process
$Process = New-Object System.Diagnostics.Process
$Process.StartInfo = $ProcessStartInfoObject
# Define actions for the event handlers we will subscribe to in a moment. These are checking whether
# any data was sent by the event handler and updating the global variable if it is not null or empty.
$ProcessOutputEventAction = {
if ($null -ne $EventArgs.Data -and $EventArgs.Data -ne ""){
$global:processOutputStringGlobal += "$($EventArgs.Data)`r`n"
}
}
$ProcessErrorEventAction = {
if ($null -ne $EventArgs.Data -and $EventArgs.Data -ne ""){
$global:processErrorStringGlobal += "$($EventArgs.Data)`r`n"
}
}
# We need to create an event handler for the Process object. This will call the action defined above
# anytime that event is triggered. We are looking for output and error data received by the process
# and appending the global variables with those values.
Register-ObjectEvent -InputObject $Process -EventName "OutputDataReceived" -Action $ProcessOutputEventAction
Register-ObjectEvent -InputObject $Process -EventName "ErrorDataReceived" -Action $ProcessErrorEventAction
# Process starts here
[void]$Process.Start()
# This sets up an asyncronous task to read the console output from the process, which triggers the appropriate
# event, which we setup handlers for just above.
$Process.BeginErrorReadLine()
$Process.BeginOutputReadLine()
# Wait for the process to exit.
$Process.WaitForExit()
# We need to wait just a moment so the async tasks that are reading the output of the process can catch
# up. Not having this sleep here can cause the return values to be empty or incomplete. In my testing,
# it seemed like half a second was enough time to always get the data, but you may need to adjust accordingly.
Start-Sleep -Milliseconds 500
# Return an object that contains the exit code, output text, and error text.
return #{
ExitCode = $Process.ExitCode;
OutputString = $global:processOutputStringGlobal;
ErrorString = $global:processErrorStringGlobal;
ExitTime = $Process.ExitTime
}
}

Categories