We Keep Coding

Need to filter, parse and sort multiple log files

I have a need to collect a subset of info from log files that reside on one-to-many log file servers. I have the following java code that does the initial data collection/filtering:
public String getLogServerInfo(String userName, String password, String hostNames, String id) throws Exception{
int timeout = 5;
String results = "";
String[] hostNameArray = hostNames.split("\\s*,\\s*");
for (String hostName : hostNameArray) {
SSHClient ssh = new SSHClient();
ssh.addHostKeyVerifier(new PromiscuousVerifier());
try {
Utils.writeStdOut("Parsing server: " + hostName);
ssh.connect(hostName);
ssh.authPassword(userName, password);
Session s = ssh.startSession();
try {
String sh1 = "cat /logs/en/event/event*.log | grep \"" + id + "\" | grep TYPE=ERROR";
Command cmd = s.exec(sh1);
results += IOUtils.readFully(cmd.getInputStream()).toString();
cmd.join(timeout, TimeUnit.SECONDS);
Utils.writeStdOut("\n** exit status: " + cmd.getExitStatus());
} finally {
s.close();
}
} finally {
ssh.disconnect();
ssh.close();
}
}
return results;
}
The results string variable looks something like this:
TYPE=ERROR, TIMESTAMP=10/03/2015 07:14:31 253 AM, HOST=server1, APPLICATION=app1, FUNCTION=function1, STATUS=null, GUID=null, etc. etc.
TYPE=ERROR, TIMESTAMP=10/03/2015 07:14:59 123 AM, HOST=server1, APPLICATION=app1, FUNCTION=function1, STATUS=null, GUID=null, etc. etc.
TYPE=ERROR, TIMESTAMP=10/03/2015 07:14:28 956 AM, HOST=server2, APPLICATION=app1, FUNCTION=function2, STATUS=null, GUID=null, etc. etc.
I need to accomplish the following:
What do I need to do to be able to sort results by TIMESTAMP? It is unsorted right now, because i am enumerating one to many files, and appending results to end of a string.
I only want a subset of "columns" returned, such as TYPE, TIMESTAMP, FUNCTION. I thought i could REGEX it in the grep, but maybe arrays would be better?
Results are simply being printed to console/report, as this is only printed for failed tests, and is there for troubleshooting purposes only.

I took the list of output that you provided and put it in a file, named test.txt, making sure that each "TYPE=ERROR etc. etc" was in a new line (I guess it's the same in your output, but it isn't clear).
Then I used cat test.txt | cut -d',' -f1,2,5 | sort -k2 to do what you want.
cut -d',' -f1,2,5 basically splits by comma and only reports tokens number 1,2,5 (TYPE,TIMESTAMP,FUNCTION). If you want more, you can add more numbers depending on what token you want
sort -k2 sorts according to the 2nd column (TIMESTAMP)
The output I get is:
TYPE=ERROR, TIMESTAMP=10/03/2015 07:14:28 956 AM, FUNCTION=function2
TYPE=ERROR, TIMESTAMP=10/03/2015 07:14:31 253 AM, FUNCTION=function1
TYPE=ERROR, TIMESTAMP=10/03/2015 07:14:59 123 AM, FUNCTION=function1
So what you should try and do, is to further pipe your command with |cut -d',' -f1,2,5 | sort -k2
I hope it helps.

After working on this some more, i come to find that one of the key/value pairs allows commas in the values, thus cut will not work. Here is the finished product:
My grep command stays the same, collecting data from all servers:
String sh1 = "cat /logs/en/event/event*.log | grep \"" + id + "\" | grep TYPE=ERROR";
Command cmd = s.exec(sh1);
results += IOUtils.readFully(cmd.getInputStream()).toString();
Put the string into an array, so i can process them line by line:
String lines[] = results.split("\r?\n");
I then used regex to get the data i needed, repeating the below for each line in the array, and for as many columns as needed. It's a bit of a hack, I probably could have done it better by simply replacing the comma in the offending key/value pair, then using SPLIT() and comma as delimeter, then looping for the fields i want.
lines2[i] = "";
Pattern p = Pattern.compile("TYPE=(.*?), APPLICATION=.*");
Matcher m = p.matcher(lines[i]);
if (m.find()) {
lines2[i] += ("TYPE=" + m.group(1));
}
Finally, this will sort by Timestamp, since it is 2nd column:
Arrays.sort(lines2);

Java: populating Scanner with default value on Scanner.nextLine();

I am writing a java program that runs a loop and keeps asking the user for input. The program then does a bunch of things with the string, and asks for another string and repeats.
The issue is that many strings are very similar, so i would like to populate the prompt with the input from the last time in the loop. For instance: If the user enters a value as follows:
Enter the SKU Number: APE-6603/A
... Then the next time it asks for an SKU, it will wait till the user presses enter as normal, but be ready with the last value before the user even types anything:
Enter the SKU Number: APE-6603/A
... And the user can make simple changes very fast like replace the /A with /B and press enter! If the string that holds the user input is called "lookFor", is there a way to populate the prompt with this value in Java? It would be VERY useful!
Thanks!

After discussing this idea with a few people, it seems that what i want is not possible. The way of input is too simple to allow something like this.
My only possible solutions involve not running this from my IDE. I can either elect to use my application, or change the application into a GUI based applet. Running from the console will open up the "Press up" option, as suggested by rchirino, and using a GUI would let the value entered sit there for editing later.
If anyone is looking to do what i posted above, the answer is "Java cant do it!". Sorry. :)

You might want to try something like this:
public String promptandgetWithShowDefault(String prompt, String supplied) {
String prmpt = prompt + " (press Enter for \"" + supplied + "\"):";
String tmpch = null;
System.out.print(prmpt);
tmpch = scanner.nextLine().trim();
if (tmpch == null || tmpch.equals("")) {
return supplied;
} else {
return tmpch;
}
}

If the goal is to get a simple binar answer from the user like:
Would you like to do that? ( y / n ) y
then the empty string returned by the user, in the answer from Dmv, will do the trick, except that when the user types "n" or attempts to delete the trailing "y", it won't disappear, so it would then be clearer to write the prompt like:
Would you like to do that? ( [ y ] / n )
But when the goal is to get a long string, like the original question or a file path for instance, that the user can edit to correct a typo or not to overwrite previous file .... then you definitely need something else which doesn't seem to be available in Java.
Well do it in C then!!! with the help of libreadline...
it's probably possible, easier and more portable to do the same trick in Python, but I have no idea how to code in Python.
Here is a simple Java MRE to illustrate it:
import java.io.BufferedReader;
import java.io.File;
import java.io.InputStreamReader;
public class Main
{
public static void main(String[] args)
{
String path = System.getProperty("user.home") + File.separatorChar + "Documents";
File file = null;
do {
path = askForString("Enter the filepath to open:", path );
if ( ( path == null) || ( path.isBlank())) break;
file = new File( path );
} while ( ! file.exists() );
System.out.println("Openning " + path + "....");
// ......
}
public static String askForString( String message, String defaultString)
{
String response = null;
System.out.println( message);
// any extra String in cmd[] will be added in readline history
String[] cmd = { "/path/to/executable/ask4stringWdefault", defaultString};
try
{
ProcessBuilder pb = new ProcessBuilder(cmd);
// Make sure the subprocess can print on console and capture keyboard events
pb.redirectInput(ProcessBuilder.Redirect.INHERIT);
pb.redirectOutput(ProcessBuilder.Redirect.INHERIT);
Process p = pb.start();
BufferedReader stderrBuffer = new BufferedReader(new InputStreamReader(p.getErrorStream()));
int retcode= p.waitFor();
if ( retcode != 0)
{
System.err.println("The process terminated with error code: " + retcode + "\n" + stderrBuffer.readLine());
return null;
}
response = stderrBuffer.readLine();
} catch( Exception e) {
e.printStackTrace();
}
return response;
}
}
To build the executable "ask4stringWdefault" you need first to get the GNU Readline Library utility and compile it, ideally cross-compile for any platform Java supports, to get a static library that you will link while compiling ( or cross-compiling ) the following C script:
#include <stdio.h>
#include <stdlib.h>
#include <readline/readline.h>
#include <readline/history.h>
const char *defstr;
int prefill(const char *txt, int i);
int main(int argc, const char * argv[])
{
if ( argc < 2)
{
fprintf(stderr, "You must provide a default value\n");
return -1;
} else if ( argc > 2) {
// * optional extra values can be passed to populate history * //
if ( argc > 255) argc = 255;
for ( unsigned char i=0; i < argc; i++)
{
add_history(argv[i]);
}
}
defstr = argv[1];
char *cbuffer;
rl_startup_hook = prefill;
if ((cbuffer = readline(NULL)) == NULL) /* if the user sends EOF, readline will return NULL */
return 1;
fprintf( stderr, "%s\n", cbuffer);
free(cbuffer);
return 0;
}
int prefill(const char *t, int i)
{
rl_insert_text(defstr);
return 0;
}
The result is printed on stderr as it is the only stream that Java can keep track of, stdout and stdin being under the control of the executable subprocess itself.
It works fine on a Mac with arm64 architecture, using Eclipse you can't actually edit the default provided, any character typed at the prompt will be append to default string, but just hitting return will send unchanged default value back, which can be enough for basic testing.

I think I understand what you want to do, but it's rather simple. If your program is a console application (command-line), which I'll assume, then you just need to press the UP key to populate the prompt with the last typed characters.
If you're working with GUI elements then you can check the API documentation for the particular class of object you're using and check out it's fields.
Hope this helps!

Batch file renaming – inserting text from a list (in Python or Java)

I'm finishing a business card production flow (excel > xml > indesign > single page pdfs) and I would like to insert the employees' names in the filenames.
What I have now:
BusinessCard_01_Blue.pdf
BusinessCard_02_Blue.pdf
BusinessCard_03_Blue.pdf (they are gonna go up to the hundreds)
What I need (I can manipulate the name list with regex easily):
BusinessCard_01_CarlosJorgeSantos_Blue.pdf
BusinessCard_02_TaniaMartins_Blue.pdf
BusinessCard_03_MarciaLima_Blue.pdf
I'm a Java and Python toddler. I've read the related questions, tried this in Automator (Mac) and Name Mangler, but couldn't get it to work.
Thanks in advance,
Gus

Granted you have a map where to look at the right name you could do something like this in Java:
List<Files> originalFiles = ...
for( File f : originalFiles ) {
f.renameTo( new File( getNameFor( f ) ) );
}
And define the getNameFor to something like:
public String getNameFor( File f ) {
Map<String,String> namesMap = ...
return namesMap.get( f.getName() );
}
In the map you'll have the associations:
BusinessCard_01_Blue.pdf => BusinessCard_01_CarlosJorgeSantos_Blue.pdf
Does it make sense?

In Python (tested):
#!/usr/bin/python
import sys, os, shutil, re
try:
pdfpath = sys.argv[1]
except IndexError:
pdfpath = os.curdir
employees = {1:'Bob', 2:'Joe', 3:'Sara'} # emp_id:'name'
files = [f for f in os.listdir(pdfpath) if re.match("BusinessCard_[0-9]+_Blue.pdf", f)]
idnumbers = [int(re.search("[0-9]+", f).group(0)) for f in files]
filenamemap = zip(files, [employees[i] for i in idnumbers])
newfiles = [re.sub('Blue.pdf', e + '_Blue.pdf', f) for f, e in filenamemap]
for old, new in zip(files, newfiles):
shutil.move(os.path.join(pdfpath, old), os.path.join(pdfpath, new))
EDIT: This now alters only those files that have not yet been altered.
Let me know if you want something that will build the the employees dictionary automatically.

If you have a list of names in the same order the files are produced, in Python it goes like this untested fragment:
#!/usr/bin/python
import os
f = open('list.txt', 'r')
for n, name in enumerate(f):
original_name = 'BusinessCard_%02d_Blue.pdf' % (n + 1)
new_name = 'BusinessCard_%02d_%s_Blue.pdf' % (
n, ''.join(name.title().split()))
if os.path.isfile(original_name):
print "Renaming %s to %s" % (original_name, new_name),
os.rename(original_name, new_name)
print "OK!"
else:
print "File %s not found." % original_name

Python:
Assuming you have implemented the naming logic already:
for f in os.listdir(<directory>):
try:
os.rename(f, new_name(f.name))
except OSError:
# fail
You will, of course, need to write a function new_name which takes the string "BusinessCard_01_Blue.pdf" and returns the string "BusinessCard_01_CarlosJorgeSantos_Blue.pdf".

How can I pipe input to a Java app with Perl?

I need to write a Perl script that pipes input into a Java program. This is related to this, but that didn't help me. My issue is that the Java app doesn't get the print statements until I close the handle. What I found online was that $| needs to be set to something greater than 0, in which case newline characters will flush the buffer. This still doesn't work.
This is the script:
#! /usr/bin/perl -w
use strict;
use File::Basename;
$|=1;
open(TP, "| java -jar test.jar") or die "fail";
sleep(2);
print TP "this is test 1\n";
print TP "this is test 2\n";
print "tests printed, waiting 5s\n";
sleep(5);
print "wait over. closing handle...\n";
close TP;
print "closed.\n";
print "sleeping for 5s...\n";
sleep(5);
print "script finished!\n";
exit
And here is a sample Java app:
import java.util.Scanner;
public class test{
public static void main( String[] args ){
Scanner sc = new Scanner( System.in );
int crashcount = 0;
while( true ){
try{
String input = sc.nextLine();
System.out.println( ":: INPUT: " + input );
if( "bananas".equals(input) ){
break;
}
} catch( Exception e ){
System.out.println( ":: EXCEPTION: " + e.toString() );
crashcount++;
if( crashcount == 5 ){
System.out.println( ":: Looks like stdin is broke" );
break;
}
}
}
System.out.println( ":: IT'S OVER!" );
return;
}
}
The Java app should respond to receiving the test prints immediately, but it doesn't until the close statement in the Perl script. What am I doing wrong?
Note: the fix can only be in the Perl script. The Java app can't be changed. Also, File::Basename is there because I'm using it in the real script.

I've grown rather fond of the IO::Handle derived modules. They make it easy to control flushing, reading data, binary mode, and many other aspects of a handle.
In this case we use IO::File.
use IO::File;
my $tp = IO::File->new( "| java -jar test.jar" )
or die "fail - $!";
# Manual print and flush
$tp->print( 'I am fond of cake' );
$tp->flush;
# print and flush in one method
$tp->printflush( 'I like pie' );
# Set autoflush ON
$tp->autoflush(1);
$tp->print( 'I still like pie' );
Also, since the file handle is lexically scoped, you don't have to close it manually. It will automatically close when it goes out of scope.
BTW, unless you are targeting a perl older than 5.6, you can use the warnings pragma instead of -w. See perllexwarn for more info.

$|=1 only works on the currently selected file handle (by default, STDOUT). To make your TP file handle hot you need to do this after opening it:
select(TP);
$| = 1;
select(STDOUT);

Execute jar file many times and analyse output

I'd like to test which of two implementation in java of a problem is the fastest.
I've 2 jar files with the two implementation I can execute from the terminal. I want to execute both about 100 times and analyse which one is the fastest to do that or that task.
In the output one of the line is "executing time : xx", I need to catch this xx to put in an array or something like that to analyse it later
While I'm executing the jar, I've also to give some input commands (like a name to search or a number).
I don't with which language is it the easiest to do it.
I know the basis in Bash and Python
thank you

Excuse me, but Why you dont make a jar that call n-times any jar?
For Example:
public static void main(String[] args) {
for(int i=0;i<1000;i++) {
params1 = randomNumber();
params2 = randomName();
...
paramsN = randomTime();
MYJAR1 test1 = new MYJAR1(params1,params2,...,paramsN);
timerStart();
test1.start();
timerEnd();
printTimer();
}
}
and make the same for the second jar.
I hope that my idea can help you.
Bye

If you use (say) Perl, you can spawn the process off, capture the output via a redirection and filter the times. e.g.
if (open(PROCESS, "myproc |")) {
while(<PROCESS>) {
if (/executing time : (\d+)/) {
# $1 has the time now
}
}
}
(not compiled or tested!)
Perhaps the simplest way of analysing the data is to redirect the above output to a file, and then load it into Excel. You can then use Excel to calculate averages/max/mins and std. devs (if you wish) trivially.

Ok I've found with 3 different scripts ^^
in the java code, for each function :
long time1 = System.currentTimeMillis();
// some code for function 1
long time2 = System.currentTimeMillis();
try {
OutputStreamWriter osw = new OutputStreamWriter(new FileOutputStream(new File("log.txt"),true));
osw.write("function1,"+(time2 - time1)+"\n");
osw.close();
} catch (FileNotFoundException ex) {
ex.printStackTrace();
} catch (IOException ex) {
ex.printStackTrace();
}
a bash code to run the 500 times the two algorithms
#!/bin/bash
i=0
while [ $i -lt 500 ]
do
./algo1.ex
./algo2.ex
i=$(( $i+1 ))
done
an (or two actually, one for each algorithm) expect code to do the command during the execution
#!/usr/bin/expect -f
spawn java -jar algo1.jar
expect "enter your choice" {send "1\n"}
expect "enter a name :" {send "Peju, M.\n"}
expect "enter your choice" {send "2\n"}
expect "enter a name :" {send "Creasy, R. J.\n"}
expect "enter your choice" {send "0\n"}
exit
As I didn't know how to do it in bash, to count I used a python code
#!/usr/bin/python
# -*- coding: utf8 -*-
import sys
if __name__ == "__main__":
if sys.argv[1:]:
arg = sys.argv[1]
filin = open(arg, 'r')
line = filin.readline()
res= 0, 0, 0
int n = 1
while line !='':
t = line.partition(',')
if t[0] == "function1":
res = (res[0] + int(t[2]), res[1], res[2])
if t[0] == "function2":
res = (res[0], res[1] + int(t[2]), res[2])
if t[0] == "function3":
res = (res[0], res[1], res[2] + int(t[2]))
ligne = filin.readline()
n = n+1
print res
print (res[0]/(n/3.0), res[1]/(n/3.0), res[2]/(n/3.0))
filin.close()
and it works
but thanks for your propositions