Are there any grep -like Unix/Linux command line tools that understand Java stacktraces in log files that are printed by log4j or logback? The tool should understand that a stacktrace consists of several lines.
Typical use case would be to filter out certain exceptions and corresponding stacktraces when viewing logs that are stored to files.
I'm using following sed one line program:
sed -nr ':main; /^[0-9 :,-]{23} ERROR / { :loop; p; n; /^[0-9 :,-]{23} / b main; b loop}'
The first [0-9 :,-]{23} recognizes log record start. Right after it, before the slash, you can write additional regexp to limit which records to print. The expression in {...} loops through the following lines until new record header is found.
My program works for logs where log records start with:
2015-08-25 12:49:34,906 ...
And prints all records with stack traces which has ERROR after record start. Example:
2015-08-25 12:49:34,906 ERROR [http-8080-89] [Error#112] NullPointerException:
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
...
The sed program explanation
The sed program expression /regexp/ command means: if the current line matches the regexp run command.
sed will read the input line and run the program. When the line matches /^[0-9 :,-]{23} ERROR / it runs the command block {...}, if not then since program ended, sed will not print the current line to output (option -n), then sed reads the next line and run the program again. This repeats until end of input.
{...} explanation:
p - print the current line
n - read next line
/^[0-9 :,-]{23} / b main - if the line matches the regexp continue at label :main - effectively rerunning the whole program on the current line without reading next line - to not miss next possible exception
continue at label :loop
So the regexps:
/^[0-9 :,-]{23} ERROR / matches lines which starts the log record
/^[0-9 :,-]{23} / matches line which is next log record
I don't know if this answers your question but when I want to get the stacktrace I use grep -A to get the lines right after the line I'm looking for.
For example:
grep -A 200 "2014-09-08/12:11:36.110" catalina.out
Related
I have pretty simple wrapper script which aquires parameters and passes them to java jar.
Unfortunatly, I experience very-very strange behaviour. Below is an example.
Command to execute script:
./wrapper http://localhost:8485/metrics 900 200
Script:
#/bin/sh
/usr/java/default/bin/java -jar /usr/plugins/checkmetrics.jar $#
Java code:
public static void main(String[] args) throws IOException {
String metricsUrl = args[0];
int heapWarnValue = Integer.parseInt(args[1]);
int threadWarnValue = Integer.parseInt(args[2]);
}
Which gives me NumberFormatException:
"xception in thread "main" java.lang.NumberFormatException: For input string: "200
But if I change command to following, everything works:
./wrapper http://localhost:8485/metrics 900 200" "
Breaks my brain, but I can't understand where I'm wrong. Could someone explain?
Thanks in advance
Is there an LF or CR character at the end of the script that is not being correctly processed (could happen if you have unix line endings in a windows environment or vice versa)?
the reason I mention this is that the error you mention says that it is
For input string: "200
I'm willing to bet that there is another quote mark at the start of the next line. If that's the case, it's trying to parse 200 and CR together as an integer. Sort out the line endings and all will be fine.
i am trying to run a batch file from my java code
this is the batch file line :
C:\Users\abdelk\workspace\Symmetrix>symconfigure -sid 13 -cmd "create dev count=16, size=139840, emulation=FBA , config=TDEV;" commit -nop >> out_file.txt
when i run the batch file from my code, "1" randomly appears before the ">>". so line in cmd becomes like that :
C:\Users\abdelk\workspace\Symmetrix>symconfigure -sid 13 -cmd "create dev count=16, size=139840, emulation=FBA , config=TDEV;" commit -nop **1>>** outfile.txt
I don't know how can i remove this random appearing "1"
this is how i run the batch file from my code
rt.exec("cmd.exe /c start "+functions_object.edit_host_name(current_host_name)+"_Meta.bat",null,new File("C:\\Users\\abdelk\\workspace\\Project"));
First, take a look on Microsoft's TechNet article using command redirection operators.
Numeric 1 is an equivalent for handle stdout (standard output).
In batch files numeric 1 is omitted on redirection stdout.
For example put those 2 lines into a batch file and run it
echo This is just a redirect test.>CapturedStandardOutput.txt
#pause
You will see that cmd.exe automatically inserts 1 (space and 1) left to the redirection operator >.
In general it is not advisable to add already in batch file 1 for stdout.
Why?
Look what is executed with:
echo This is just a redirect test.1>CapturedStandardOutput.txt
#pause
You see in console window:
echo This is just a redirect test.1 1>CapturedStandardOutput.txt
And the file CapturedStandardOutput.txt contains the line:
This is just a redirect test.1
The solution is to use in batch file:
echo This is just a redirect test. 1>CapturedStandardOutput.txt
This results in execution of the line:
echo This is just a redirect test. 1>CapturedStandardOutput.txt
And there is now the line below in file CapturedStandardOutput.txt:
This is just a redirect test.
What you can't see here in the browser window is that the line in the text file ends now with a trailing space in comparison to first example. Therefore best is to use > and >> always without 1 as otherwise it is not really simple to control what is written to the text file.
One more hint:
To redirect a text to a file which ends with 1, 2, ..., 9 it is necessary to escape the number with ^.
Execution of a batch file with
echo Number is ^1>CapturedStandardOutput.txt
#pause
results in executing the command line
echo Number is 1 1>CapturedStandardOutput.txt
and in file CapturedStandardOutput.txt the line
Number is 1
with no trailing space at end of the line.
0 left to > and >> must not be escaped to get number 0 written into a text file.
I am trying to use the program Trimmomatic to removed adapter sequences from an Illumina paired-end read over a computer cluster. While I can get the program to open, it will either not acknowledge the commands I enter or will return an error message. I have tried all kinds of permutations of the input commands without success. Examples of input code and error messages are below
Code:
java -classpath /*filepath*/Trimmomatic-0.32/trimmomatic-0.32.jar org.usadellab.trimmomatic.TrimmomaticPE \
-phred33 -trimlog /Results/log.txt \
~/*filepath*/data_R1.fq ~/*filepath*/data_R2.fq \
ILLUMINACLIP:/*filepath*/Trimmomatic-0.32/adapters/TruSeq3-PE-2.fa:2:30:10:3:"true"
Results: (the o/s seems to find and execute the software, but is not feeding in the command; I get the same result if I use the java -jar option for executing Trimmomatic)
TrimmomaticPE [-threads <threads>] [-phred33|-phred64] [-trimlog <trimLogFile>] [-basein <inputBase> | <inputFile1> <inputFile2>] [-baseout <outputBase> | <outputFile1P> <outputFile1U> <outputFile2P> <outputFile2U>] <trimmer1>...
Code: (If I add in the command PE immediately before all other commands, the program executes and can find the fasta file containing the adapter sequences, but then searches for and cannot fund a file called 'PE')
java -classpath /*filepath*/trimmomatic-0.32.jar org.usadellab.trimmomatic.TrimmomaticPE \
PE -phred33 -trimlog /Results/log.txt \
~/*filepath*/data_R1.fq ~/*filepath*/data_R2.fq \
ILLUMINACLIP:/*filepath*/Trimmomatic-0.32/adapters/TruSeq3-PE-2.fa:2:30:10:3:"true"
Results: (Programs rus and finds the fasta file of adapter sequences, but then fails to execute. Why is it looking for a PE file?)
TrimmomaticPE: Started with arguments: PE -phred33 -trimlog /Results/log.txt /*filepath*/data_R1.fq /*filepath*/data_R2.fq ILLUMINACLIP:/*filepath*/Trimmomatic-0.32/adapters/TruSeq3-PE-2.fa:2:30:10:3:true
Multiple cores found: Using 12 threads
Using PrefixPair: 'TACACTCTTTCCCTACACGACGCTCTTCCGATCT' and 'GTGACTGGAGTTCAGACGTGTGCTCTTCCGATCT'
Using Long Clipping Sequence: 'AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC'
Using Long Clipping Sequence: 'TACACTCTTTCCCTACACGACGCTCTTCCGATCT'
Using Long Clipping Sequence: 'GTGACTGGAGTTCAGACGTGTGCTCTTCCGATCT'
Using Long Clipping Sequence: 'AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTA'
ILLUMINACLIP: Using 1 prefix pairs, 4 forward/reverse sequences, 0 forward only sequences, 0 reverse only sequences
Exception in thread "main" java.io.FileNotFoundException: PE (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
at org.usadellab.trimmomatic.fastq.FastqParser.parse(FastqParser.java:127)
at org.usadellab.trimmomatic.TrimmomaticPE.process(TrimmomaticPE.java:251)
at org.usadellab.trimmomatic.TrimmomaticPE.run(TrimmomaticPE.java:498)
at org.usadellab.trimmomatic.TrimmomaticPE.main(TrimmomaticPE.java:506)
I've never used trimmomatic but it looks like you are passing in the incorrect parameters.
the trimmomatic webpage lists the usage from version 0.27+ as:
java -jar <path to trimmomatic.jar> PE [-threads <threads] [-phred33 | -phred64] [-trimlog <logFile>] <input 1> <input 2> <paired output 1> <unpaired output 1> <paired output 2> <unpaired output 2> <step 1> ...
or using the "old way"
java -classpath <path to trimmomatic jar> org.usadellab.trimmomatic.TrimmomaticPE [-threads <threads>] [-phred33 | -phred64] [-trimlog <logFile>] <input 1> <input 2> <paired output 1> <unpaired output 1> <paired output 2> <unpaired output 2> <step 1> ...
Where the only difference is the new way is specifying "PE" as the main class instead of a fully qualified path.
First, addressing your 2nd problem:
You look like you are doing both: specifying a fully qualified class name as well as the PE part. This makes trimmomatic think you have a fastq file named "PE" which doesn't exist.
If you get rid of the "PE" OR the qualfited class name; it will call the correct class. Which is what you do first in your first problem.
1st problem
I don't think you have the correct number of arguments listed in your first invocation so trimmomatic displays the usage to tell you what parameters are required. It would be nice if it told you what was wrong but it doesn't.
Solution
It looks like you are only providing 2 fastq files but trimmmoatic needs 6 file paths. You are missing the output paired and unpaired files paths for the read 1 and read 2 data which I assume get created by the program when it runs.
I guess your 2nd attempt got further along in the program since it saw enough parameters that you potentially had enough file paths specified (however, it turns out you had optional step parameters)
Following the advice of dkatzel below and user blakeoft on SeqAnswers (http://seqanswers.com/forums/showthread.php?t=45094), I dropped the PE flag and added individual file names for each output file and the program executed properly.
java -classpath /*filepath*/Trimmomatic-0.32/trimmomatic-0.32.jar org.usadellab.trimmomatic.TrimmomaticPE \
-phred33 \
~/refs/lec12/data_R1.fq ~/refs/lec12/data_R2.fq \
lane1_forward_paired.fq lane1_forward_unpaired.fq lane1_reverse_paired.fq lane1_reverse_unpaired.fq \
ILLUMINACLIP:/*filepath*/Trimmomatic-0.32/adapters/TruSeq3-PE-2.fa:2:30:10:3:true
NB: I also tried using the -baseout flag rather than a list of four files, and the program would open but not execute any commands
NB: The a log file could be generated using the flag -trimlog filename, but only if I first made a blank text file with the same name as the intended log file.
We have a hugely parallelized build process, so I frequently have to browse through large amounts of output from javac to find a build error.
To make this easier it would be nice if there were some tool that will colorize the output of javac to my terminal, highlighting errors in the code.
What tool can I use to colorize the output of javac?
using grep with the "--color" option ?
~$ javac Test.java 2>&1 | egrep --color "^|error"
I ended up using a tool called Generic Colorizer Tool and write my own configuration for colorizing the most important output. Works just fine. :)
Roll your own javac error colorizer by using any regex matcher to match your text and surround it with terminal color escape codes to apply a color:
Using a readfile and substitute ideology:
#1. Do your javac and pipe the result to a file:
javac whatever.java 2>/tmp/javac_errors.out;
#define the escape start and stop codes that your terminal
#uses to apply foreground and background color:
let redbackground = '\\e[48;5;196m'
let normalbackground = '\\e[0;0m'
#iterate the lines in the saved file:
for line in readfile("/tmp/javac_errors.out")
#Use sed, match, substitute or whatever to regex substitute
#the text with the text surrounded by the color escape codes
#find and replace the text 'error:' with the same surrounded by escape codes
let line = substitute(line,
'error:',
redbackground .
'error:' .
normalbackground, 'g')
#use echo -e flag to tell the terminal to interpret the escape codes:
echo -e line
endfor
Works for me:
This code is the same as above, but it uses a terminal line iterator and a sed replace ideology:
#run javac pipe to file
javac whatever.java 2>/tmp/errors.out
#Define terminal color codes
redbackground='\\e[48;5;196m'
normalbackground='\\e[0;0m'
#read the file and print out each line
filename="/tmp/errors.out"
while read -r line; do
#replace error surround with escape codes
line=`sed "s/error:/${redbackground}error:${normalbackground}/g" <<<"$line"`
echo -e "$line"
done < "$filename"
I need to run an .sh file and get its output. I need to see the setup of the file as well.
The .sh file simply runs a java app through terminal.
Any ideas? I'm truly stuck on this.....
Elijah
The server.sh file:
echo Starting Jarvis Program D.
ALICE_HOME=.
SERVLET_LIB=lib/servlet.jar
ALICE_LIB=lib/aliceserver.jar
JS_LIB=lib/js.jar
# Set SQL_LIB to the location of your database driver.
SQL_LIB=lib/mysql_comp.jar
# These are for Jetty; you will want to change these if you are using a different http server.
HTTP_SERVER_LIBS=lib/org.mortbay.jetty.jar
PROGRAMD_CLASSPATH=$SERVLET_LIB:$ALICE_LIB:$JS_LIB:$SQL_LIB:$HTTP_SERVER_LIBS
java -classpath $PROGRAMD_CLASSPATH -Xms64m -Xmx128m org.alicebot.server.net.AliceServer $1
My current code:
NSTask *server = [NSTask new];
[server setLaunchPath:#"/bin/sh"];
[server setArguments:[NSArray arrayWithObject:#"/applications/jarvis/brain/server.sh"]];
NSPipe *outputPipe = [NSPipe pipe];
[server setStandardInput:[NSPipe pipe]];
[server setStandardOutput:outputPipe];
[server launch];
NSMutableString *outputString = [NSMutableString string];
while ([outputString rangeOfString:#"Jarvis>"].location == NSNotFound) {
[outputString appendString:[[[NSString alloc] initWithData:[[outputPipe fileHandleForReading] readDataToEndOfFile] encoding:NSUTF8StringEncoding] autorelease]];
NSRunAlertPanel(#"", outputString, #"", #"", #"");
}
The NSRunAlertPanel is just for checking the output. Now my code is freezing and not even getting to the alertpanel.
See answer to this question.
There are a couple of things that should be fixed in your script:
The script should begin with a
shebang. Also make sure that the
script has its executable bit set.
Because the environment variables are set up relative to the shell script directory, you need to make sure that the script directory is the current directory.
You need to export the environment variables that should be visible to the Java process.
In the last line you can use exec to replace the shell process with the Java executable that runs Jetty.
Here is a revised version of your script:
#!/bin/sh
echo Starting Jarvis Program D.
cd "`dirname \"$0\"`"
export ALICE_HOME=.
export SERVLET_LIB=lib/servlet.jar
export ALICE_LIB=lib/aliceserver.jar
export JS_LIB=lib/js.jar
# Set SQL_LIB to the location of your database driver.
export SQL_LIB=lib/mysql_comp.jar
# These are for Jetty; you will want to change these if you are using a different http server.
export HTTP_SERVER_LIBS=lib/org.mortbay.jetty.jar
export PROGRAMD_CLASSPATH=$SERVLET_LIB:$ALICE_LIB:$JS_LIB:$SQL_LIB:$HTTP_SERVER_LIBS
exec java -classpath $PROGRAMD_CLASSPATH -Xms64m -Xmx128m org.alicebot.server.net.AliceServer $1
Invoking the shell script in Objective-C with multiple arguments:
NSTask *server = [NSTask new];
[server setLaunchPath:#"/bin/sh"];
[server setArguments:[NSArray arrayWithObjects:#"/applications/jarvis/brain/server.sh", #"argument", nil]];
...
Using AMShellWrapperTest.app you can filter (save, ...) the stdout stream of server.sh by modifying "- (void)appendOutput:(NSString *)output" in BannerController.m. (... but maybe there is a better way to do this ...)
/*
// output from stdout
- modified AMShellWrapper/AMShellWrapperTest/BannerController.m (http://www.harmless.de/cocoa-code.php)
to print server.sh setup information to "Error Messages:" text output field (or Console.app as an
alternative) and the Q & A dialog to the "Output:" text field
- use of default charliebot, http://sourceforge.net/projects/charliebot/, modified only to run server.sh
with complete path (here: ~/Desktop/charliebot/server.sh) in AMShellWrapperTest.app
*/
- (void)appendOutput:(NSString *)output
{
NSMutableString *outputString = [NSMutableString string];
if (
([output rangeOfString:#"Charlie>"].location != NSNotFound ) || \
([output rangeOfString:#"[Charlie] user>"].location != NSNotFound )
) {
[self write: output];
[self write: #"\n"];
} else {
[outputString appendString: output];
//[outputString writeToFile:#"/dev/console" atomically: NO]; // alternative
[errorOutlet setString:[[errorOutlet string] stringByAppendingString: outputString]];
}
}
yes, but why isn't my code (posted above) not working?
I guess your "Jarvis>" line is the first line of the server.sh ouput stream that expects some user input, which means that this line is incomplete without a terminating newline character "\n". If server.sh had been run in Terminal.app, the user would have to press the return key to let the dialog continue. The conditional code of the while loop (NSNotFound) cannot finish its job on this incomplete line (which would be to abort the while loop) and gets stuck.
You have to drop the while loop and use the 'readInBackgroundAndNotify' mode on NSFileHandle to get non-blocking I/O stdout stream behaviour!
See: NSTask/NSPipe STDIN hangs on large data, sometimes...
So, if you like, just transform the source code of AMShellWrapperTest.app into a pure command-line tool by removing the GUI code.