We Keep Coding

SLF4J/Java logging: Is it possible to add log arguments automatically?

Introduction
We're using SLF4J in combination with Logback in several Spring (Boot) applications and recently started using the logstash-logback-encoder to implement structured logging. As we also still have to support plain text logs, we wondered whether it was possible to append arguments automatically to log messages without having to add them manually to the message using the {} markers.
Example of the desired behaviour
To illustrate the desired behaviour this is what we'd wish for:
log.info("My message", kv("arg1", "firstArgument"), kv("arg2", "secondArgument"))
resulting in the following desired output, where the arguments are automatically appended in parentheses at the end of the message:
My message (arg1="firstArgument", arg2="secondArgument")
Or another example with both explicit arguments in the message and arguments at the end:
log.info("Status changed: {} => {}", v("from", "READY"), v("to", "UNAVAILABLE"), kv("service", "database"))
resulting in the following desired output:
Status changed: READY => UNAVAILABLE (from="READY", to="UNAVAILABLE", service="database")
Question
Is this possible with SLF4J/Logback? And if not, do you know other logging frameworks or ways to achieve this (in Java)?

I'm not aware of any log frameworks that let you do this, but you can trivially write your own. Because this really is just a simple API extension, and thus, all you need to duplicate is specifically the various log messages. For example, this one-liner would take care of it:
public static class LoggingExtensions {
#lombok.Value public static final class LogKeyValue {
String key, value;
}
public static LogKeyValue kv(String key, Object value) {
return new LogKeyValue(key, String.valueOf(value));
}
public static void info(Logger log, String message, Object... args) {
int extra = 0;
int len = args.length;
// Last arg could be a throwable, leave that alone.
if (len > 0 && args[len - 1] instanceof Throwable) len--;
for (int i = len - 1; i >= 0; i--) {
if (!(args[i] instanceof LogKeyValue)) break;
extra++;
}
if (extra > 0) {
StringBuilder sb = new StringBuilder(message.length() + 2 + (extra.size() - 1) * 2);
sb.append(message).append("({}");
for (int i = 1; i < extra; i++) sb.append(", {}");
message = sb.append(")").toString();
}
log.info(message, args);
}
}
This code tacks ({}, {} {}) at the end of the message, 1 for each 'kv' type. Note that most logging frameworks, including slf4j, do let you tack 1 exception at the end, even without a matching {} in the message, and this method would thus require that you first list all {} args, then any kv args, then 0 or 1 throwables.
Some caveats, though:
You'd have to change all your code to invoke these utility methods. You can make it look nice using static imports, but it does make your code less idiomatic, which is a downside.
Most log frameworks have an explosion of methods, because varargs cause the creation of arrays. In hotspotted code, the JDK will probably make that sufficiently efficient that it won't matter, but because log statements tend to be ubiquitous, you get a bit of a death-by-a-thousand cuts happening otherwise. It is unlikely that this call by logging frameworks to have a ton of methods to avoid the varargs penalty is a sensible move; generally logs end up on disk and are even fsynced, and the performance impact of that is many orders of magnitude larger. But, log frameworks have to cater to all scenarios, and logs which end up totally ignored due to log level config, in a tight loop, can see some performance improvement due to avoiding the varargs penalty. You can also try to optimize if it comes up that your log frameworks are impacting performance: You can ask the log handlers if the log level that is asked for is even relevant, and if not, just return; immediately. You can then also follow along and create this 'explosion'. See slf4j which has 10 methods for every log level, and many other frameworks have even more (they have a variant for 1, 2, 3, and sometimes even 4 arguments before resorting to varargs).

Get Nashorn script line number during interpretation

My code uses Nashorn to provide scripting functionality to the user, with many classes and functions implemented on Java exposed to the scripts through Nashorn.
One of the tasks of the program is of course to report any errors relating to the scripts to the user when encountered. This is very simple if a ScriptException occurs, as simply catching the exception and using the method getLineNumber() returns the correct value. However, occasionally an exception occurs not due to the syntax, but due to the way the Java-side code is called, for example due to a null parameter which was supposed to be a valid object. These cause other kinds of exceptions, which can still be caught around the call to the eval method, but since these do not have a getLineNumber() method its impossible to guess where the interpreter was left.
Is there a way I can get the last executed line somehow from the Nashorn engine?
The code roughly looks like this:
try {
engine.eval( script);
// successful
return -1;
} catch ( ScriptException e)
{
// the ScriptException reports the line number
return e.getLineNumber();
}
catch ( Exception e)
{
// is it possible to get the line number here too?
// ...
return lineNumber;
}

There is a standard Nashorn API to get StackTraceElement[] for "script frames" from a given arbitrary Throwable object.
jdk.nashorn.api.scripting.NashornException class has
public static StackTraceElement[] getScriptFrames(Throwable exception)
method
https://docs.oracle.com/javase/8/docs/jdk/api/nashorn/jdk/nashorn/api/scripting/NashornException.html#getScriptFrames-java.lang.Throwable-
You can pass arbitrary Throwable object and get back StackTraceElement array for the script frames. The top most script frame would be the zero'th element of the array and you can call getLineNumber on the StackTraceElement object.
This way, you can avoid dependency on nashorn internal package prefixes.

You could walk the stack of the exception, find the first StackTraceElement where the class name starts with jdk.nashorn.internal.scripts. and report its line number:
for(StackTraceElement ste: e.getStackTrace()) {
if (ste.getClassName().startsWith("jdk.nashorn.internal.scripts.")) {
return ste.getLineNumber();
}
}
return -1; // couldn't figure it out
You can also try to use ste.getFileName().endsWith(".js") if that's more robust for your situation.

How to suppress Cucumber/Junit assertion stack trace

I have a cucumber scenario and the step uses assertEquals. My results report shows the stack trace which is not end user friendly. How can I suppress it
Scenario: Add two numbers
Given I have two inputs "3" and "2"
When I add them
Then the output should be "15"

You're correct in observing that the default XML output (assuming you're not outputting to JSON or text, but you didn't say) from a Junit tests show stack traces for failed steps. This isn't actually a Cucumber thing. CucumberOptions won't help you here.
You can:
Use a different or custom Runner for your test and then setup a tag that controls what is included in the output, or what will be read by the CI software of your choosing. For example the Confulence API API for doing this tells how "debugger"
Same type of deal for Ant Scripts to tweak the output, so that is doesn't show the output. A good Tutorial for learning how to use Any scripts to fire off your Cucumber JUnit Test is here.
Other have build a custom formatter for JUnit by implementing XMLJUnitResultFormatter API, explained more here - How do I configure JUnit Ant task to only produce output on failures?
Hope that gives you what you need.

I was also facing same issue with my Cucumber-Selenium-Java project. In the cucumber reports, it was generating around 40 lines of stacktrace. Due to this, it was impacting look and feel of the report. And the end user/client was little concerned about it. Because he/she was not really able to figure out the actual use of this stacktrace. So, I came up with below idea/approach. It's little bit tricky but, it's worthy.
Few notes before starting:
We cannot completely disable stacktrace in in all the cases. But we can modify the stacktrace and then, re-throw the new exception with useful and shortened stacktrace.
You need to be aware about frequently faced exceptions, errors. So that, we can create custom exception depending on the exceptions.
In the stacktrace it will generate few line of code from wrapper APIs, few lines from Junit/TestNg, few lines for java and selenium and there will be only one or two lines in the stacktrace, where actually our issue occurred.
Our test classes must be in unique package. So that, we can filter the stacktrace trace with package name and get the class name, line number and method name of actual issue and we can use this information in throwing custom exception. Hence, it will be easy to figure out the actual line of issue occurred. In my case all the classes were in package named "page". If you have more than one packages for your classes, then you can accordingly add string conditions in below code.
We need to wrap the test code in try-catch block. And while catching, we need to use Throwable class not exception class. Because, if there is any assertion failure, then Exception class won't be able to handle the issue as you know all the assertions come under Error class and Throwable is the parent of Error and Exception.
If we throw the new exception in catch block, then, it will change the line number in stacktrace, where actual issue occurred. So it will be difficult to figure out the actual line of issue. In order to avoid it, we need to get the class name, line number, method name of actual issue and store it in StackTraceElement class and use it in throwing new exception.
Some exceptions like "NoSuchElementException" provides lot of information in their cause and most of it is not really required, So we need to modify the content of it's message by using substring(), indexOf() and replaceAll() methods of String class in Java. And then, provide the modified information in new exception.
Few important Java method from Throwable java class and their description: (i) getStackTrace(): This method will return us array of StackTraceElement class. StackTraceElement class will provide us the class name, method name, line number at which issue is occurred. (ii) setStackTrace(): This method is used to provide a custom stacktrace to new Exception. (iii) getCause(): This method will provide the issue message from cause of exception. But sometimes, it might return null. Because for some exceptions "cause" might not be specified. So this needs be surround in try catch block and here we need to use getMessage() method for getting the actual error message. (iv) getClass(): This method will return the actual exception class name. We will use this method for figuring out the exception class name and then, we will use it for providing specific implementation for different different exception classes. Note: "getClass()" method is not from "Throwable" class. It is from Object class.
You need to create a common method for handling all the exceptions and reuse this method in all the required classes. e.g.: I have named the method as "processException" and placed it in "ReusableMethod" class.
Note that, I am using package name "page" in below method (line#8), because all my test classes are placed in this package. In your case you need to update the package name as per your need. Also, I have written custom cases for two exceptions only: NoSuchElementException & AssertionError. You might need to write more cases as per your need.
public void processException(Throwable e) throws Exception {
StackTraceElement[] arr = e.getStackTrace();
String className = "";
String methodName = "";
int lineNumber = 0;
for (int i = 0; i < arr.length; i++) {
String localClassName = arr[i].getClassName();
if (localClassName.startsWith("page")) {
className = localClassName;
methodName = arr[i].getMethodName();
lineNumber = arr[i].getLineNumber();
break;
}
}
String cause = "";
try {
cause = e.getCause().toString();
} catch (NullPointerException e1) {
cause = e.getMessage();
}
StackTraceElement st = new StackTraceElement(className, methodName, "Line", lineNumber);
StackTraceElement[] sArr = { st };
if (e.getClass().getName().contains("NoSuchElementException")) {
String processedCause = cause.substring(cause.indexOf("Unable to locate"), cause.indexOf("(Session info: "))
.replaceAll("\\n", "");
Exception ex = new Exception("org.openqa.selenium.NoSuchElementException: " + processedCause);
ex.setStackTrace(sArr);
throw ex;
} else if (e.getClass().getName().contains("AssertionError")) {
AssertionError ae = new AssertionError(cause);
ae.setStackTrace(sArr);
throw ae;
} else {
Exception ex = new Exception(e.getClass() + ": " + cause);
ex.setStackTrace(sArr);
throw ex;
}
}
Below is the sample Method to showcase the usages of above method in Test Class methods. We are calling the above created method by using the class reference, which is "reuseMethod" in my case. And we are passing the caught Throwable reference "e" to the above method in catch block:
public void user_Navigates_To_Home_Page() throws Exception {
try {
//Certain lines of code as per your tests
//element.click();
} catch (Throwable e) {
reuseMethod.processException(e);
}
}
Here are few screenshots for implementation of NoSuchElementException:
Before Implementing this approach:
After Implementing this approach:

Java 7: throw exception without stack trace

EDIT2
#paradigmatic made a good point in suggesting to redirect rather than throw the exception; that solves the logging issue. The problem in Play 2 is that redirects need to occur within so-called Action scope, which is not always the case with date parser calls.
As a workaround, I went with Play's global interceptor, presumably the equivalent of a Java servlet filter.
val ymdMatcher = "\\d{8}".r // matcher for yyyyMMdd URI param
val ymdFormat = org.joda.time.format.DateTimeFormat.forPattern("yyyyMMdd")
def ymd2Date(ymd: String) = ymdFormat.parseDateTime(ymd)
override def onRouteRequest(r: RequestHeader): Option[Handler] = {
import play.api.i18n.Messages
ymdMatcher.findFirstIn(r.uri) map{ ymd=>
try { ymd2Date( ymd); super.onRouteRequest(r) }
catch { case e:Exception => // kick to "bad" action handler on invalid date
Some(controllers.Application.bad(Messages("bad.date.format")))
}
} getOrElse(super.onRouteRequest(r))
}
EDIT
Here 's a little context to work with:
// String "pimp": transforms ymdString.to_date call into JodaTime instance
class String2Date(ymd: String) {
def to_date = {
import play.api.i18n.Messages
try{ ymdFormat.parseDateTime(ymd) }
catch { case e:Exception => throw new NoTrace(Messages("bad.date.format")) }
}
val ymdFormat = org.joda.time.format.DateTimeFormat.forPattern("yyyyMMdd")
}
#inline implicit final def string2Date(ymd: String) = new String2Date(ymd)
and a test custom exception handler:
public class NoTrace extends Exception {
static final long serialVersionUID = -3387516993124229948L;
#Override
public Throwable fillInStackTrace() {
return null;
}
public NoTrace(String message) {
super(message);
}
}
Calling the date parser on an invalid yyyyMMdd string logs 30 line stack trace to the log (this occurs upstream by Play framework/Netty container, better than default 100 line trace):
"20120099".to_date
ORIGINAL
Have an issue where my application.log is getting filled with errors related to a uri date parser operation that should succeed given a valid yyyyMMdd uri date.
However, some users try to circumvent this by entering invalid dates in hopes of gaining free access to paid subscriber-only content. It's pointless, as it simply won't work, but regardless, I have MBs of these error traces in my application log.
Is there a way to throw a truly trimmed down Exception to the log? I found this SO answer, but in my application it looks like the container (Play framework on Netty) gets into the mix and logs its own 30 line stack trace to the log (30 lines is better than 100, but still 29 too many)
Similarly, I found this thread in regard to Java 7 and the new option to suppress stack trace; however, for some reason, despite being on Java 1.7, with Eclipse configured for Java 1.7, only the old 2 param method of Throwable is available (and I do see the 4 param method when I click through to the Throwable class; maybe a Scala 2.9.2 library issue?)
At any rate, ideally I can simply log a 1-line exception message and not the kitchen sink.

Simply override this method in your custom exception class:
#Override
public Throwable fillInStackTrace() {
return this;
}
after adding this method your trace method will not print

Your trouble is that although you can suppress the stacktrace of the exception your own code threw, there is nothing you can do about the exception it will be wrapped into by the framework. The only avenue I can see is not allowing the framework to catch your exception at all (doing your own top-level handling) or tweaking the logging configuration.

I think you have two options:
Control the logging to not save stack traces for some exceptions.
Write a post-processor that filters out the traces from the log file.
Unless you are in danger of running out of disk space, I think #2 is the better option, because if you do have a bug you can go back to the full log and have all the exception history.
The philosophy behind idea #2 is that disk space is cheap, but information can be precious during debug. Log a lot of data. Normally, use scripts to examine the log after it has been written to disk.
For example, if there is a type of log entry that you never expect to see, but that demands immediate action if it does appear, write a script that searches for it, and send you an e-mail if it finds one.
One of the most useful forms of script in this approach is one that drops stack trace lines. Usually, you only need to know what exceptions are happening, and the stack trace takes up a lot of screen space without telling you much. If you do need to investigate an exception, go back to the full log, find the exception line, and look at the stack trace and at what was happening immediately before the exception.
If there are too many of your date exceptions, have the script drop even the exception line. If you want to track how often they are happening, run a script that counts date exceptions per hour.
That sort of script typically costs a few minutes of programming in your favorite regex-capable script language.

How can I determine which class's `main` method was invoked at runtime?

I'd like to dynamically determine which class's main method was invoked, in order to allow for an easier to digest combined log file.
Currently, a single (rotated) log file aggregates all the log output from a number of daemons, but there is no obvious way to determine which daemon the log entry originated from, as all of the daemons use a shared code base, and loggers are created with log4j's getLogger(Something.class)
Since we're using a custom Layout class to begin with, actually outputting the information is not an issue, but finding it is.
One approach that could work as a fallback is defining a property at invocation time and reading that property.
java -cp ... -Dmain.program=<WHATEVER> MainProgram
However, there's no need to create a new convention if the ability already exists.
Update: For my purposes, the following seems to work fine:
import org.apache.log4j.PatternLayout;
import org.apache.log4j.spi.LoggingEvent;
public class MyLayout extends PatternLayout {
private static String _mainClass = null;
public String format( LoggingEvent event ) {
String mesg = super.format( event );
if (mesg.indexOf("$main") > -1) {
mesg = mesg.replaceAll("\\$main", getMainClass());
}
return mesg;
}
private static String getMainClass() {
if (_mainClass == null) {
StackTraceElement[] elem = new Exception().getStackTrace();
int offset = elem.length - 1;
if (elem[offset].getMethodName().equals("main")) {
_mainClass = elem[offset].getClassName();
}
else {
_mainClass = "<Unknown_Main_Class>";
}
}
return _mainClass;
}
}
Thanks for the suggestions!

If you only need to do it once you can walk the exception stack and look at the last class/method call.
StackTraceElement[] elem = new Exception().getStackTrace();
elem[elem.length - 1].getClassName();
But is error prone. If i load your class via reflection you will see a completely different method at the top.
You can try M. Jessup variant ( matching a main method signature ) but it will fail if I call a main method from the code too.

It is a bit hackish, but you could use the static method Thread.getAllStackTraces(). This will get you the stack trace for every live thread in the VM, and assuming the thread that started the application is still alive you could inspect the traces and look for a bottom element whose method signature matched main(String[] args).

Veera Sundar has written two articles on how to use Log4j's Mapped Diagnostic Context (source code for Servlet Filters) http://veerasundar.com/blog/2009/11/log4j-mdc-mapped-diagnostic-context-example-code/ that might be possible modify for your use case.
Let your application classes (with the main method) add its class name as a variable which can be written to the log files.
Is it less work than adding a property at runtime? No, not really but, more elegant