Given the Java argument evaluation mechanism, how does Log4j 2.x implement lazy evaluation when formatting the message with curly brackets "to avoid the cost of parameter construction" when log is disabled?
e.g.
logger.debug("Entry number: {} is {}", i, entry[i]);
I guess what Log4j means, is that with the curly brackets, they avoid constructing a string when its not necessary (e.g. the Level is not Debug):
With
logger.debug("Entry number: " + i + " is " + String.valueOf(entry[i]));
the complete message is always computed even when it will not be logged.
With
logger.debug("Entry number: {} is {}", i, entry[i]);
Log4j can check the Log-Level first and then decide if its worth to construct the message string. This saves resources needed for string concatenation. It will only call toString() on the objects supplied if the message is actually being created, saving further computation costs.
Log4j uses an internal class (org.apache.logging.log4j.message.ParameterFormatter) that replaces every {} with the arguments provided. It will also catch exceptions thrown by toString() and report these failures.
To avoid argument evaluation, just wrap it in lambda:
logger.debug(() -> {
"Entry number: " + i + " is " + String.valueOf(entry[i])
});
In this form, the supplier will be called for construction only before actual logging. Think of it like a function declaration.
In release 2.4, the Logger interface added support for lambda expressions. Such that, Log4j2 loggers can also take in a message supplier that will be lazily evaluated, here is the signature for the info one:
#Override
public void info(final Supplier<?> messageSupplier) {
logIfEnabled(FQCN, Level.INFO, null, messageSupplier, (Throwable) null);
}
This allows client code to lazily log messages without explicitly checking if the requested log level is enabled. For example, previously you would write:
// pre-Java 8 style optimization: explicitly check the log level
// to make sure the expensiveOperation() method is only called if necessary
if (logger.isTraceEnabled()) {
logger.trace("Some long-running operation returned {}", expensiveOperation());
}
With Java 8 and above you can achieve the same effect with a lambda expression. You no longer need to explicitly check the log level:
// Java-8 style optimization: no need to explicitly check the log level:
// the lambda expression is not evaluated if the TRACE level is not enabled
logger.trace("Some long-running operation returned {}", () -> expensiveOperation());
For Kotlin, there is a Kotlin logging facade based on Log4j 2 in which the signature is as follow:
fun info(supplier: () -> Any?) {
delegate.logIfEnabled(FQCN, Level.INFO, null, supplier.asLog4jSupplier(), null)
}
And here is a usage example:
logger.info { "This is a log event" }
Both will avoid the cost of parameter construction.
Related
Introduction
We're using SLF4J in combination with Logback in several Spring (Boot) applications and recently started using the logstash-logback-encoder to implement structured logging. As we also still have to support plain text logs, we wondered whether it was possible to append arguments automatically to log messages without having to add them manually to the message using the {} markers.
Example of the desired behaviour
To illustrate the desired behaviour this is what we'd wish for:
log.info("My message", kv("arg1", "firstArgument"), kv("arg2", "secondArgument"))
resulting in the following desired output, where the arguments are automatically appended in parentheses at the end of the message:
My message (arg1="firstArgument", arg2="secondArgument")
Or another example with both explicit arguments in the message and arguments at the end:
log.info("Status changed: {} => {}", v("from", "READY"), v("to", "UNAVAILABLE"), kv("service", "database"))
resulting in the following desired output:
Status changed: READY => UNAVAILABLE (from="READY", to="UNAVAILABLE", service="database")
Question
Is this possible with SLF4J/Logback? And if not, do you know other logging frameworks or ways to achieve this (in Java)?
I'm not aware of any log frameworks that let you do this, but you can trivially write your own. Because this really is just a simple API extension, and thus, all you need to duplicate is specifically the various log messages. For example, this one-liner would take care of it:
public static class LoggingExtensions {
#lombok.Value public static final class LogKeyValue {
String key, value;
}
public static LogKeyValue kv(String key, Object value) {
return new LogKeyValue(key, String.valueOf(value));
}
public static void info(Logger log, String message, Object... args) {
int extra = 0;
int len = args.length;
// Last arg could be a throwable, leave that alone.
if (len > 0 && args[len - 1] instanceof Throwable) len--;
for (int i = len - 1; i >= 0; i--) {
if (!(args[i] instanceof LogKeyValue)) break;
extra++;
}
if (extra > 0) {
StringBuilder sb = new StringBuilder(message.length() + 2 + (extra.size() - 1) * 2);
sb.append(message).append("({}");
for (int i = 1; i < extra; i++) sb.append(", {}");
message = sb.append(")").toString();
}
log.info(message, args);
}
}
This code tacks ({}, {} {}) at the end of the message, 1 for each 'kv' type. Note that most logging frameworks, including slf4j, do let you tack 1 exception at the end, even without a matching {} in the message, and this method would thus require that you first list all {} args, then any kv args, then 0 or 1 throwables.
Some caveats, though:
You'd have to change all your code to invoke these utility methods. You can make it look nice using static imports, but it does make your code less idiomatic, which is a downside.
Most log frameworks have an explosion of methods, because varargs cause the creation of arrays. In hotspotted code, the JDK will probably make that sufficiently efficient that it won't matter, but because log statements tend to be ubiquitous, you get a bit of a death-by-a-thousand cuts happening otherwise. It is unlikely that this call by logging frameworks to have a ton of methods to avoid the varargs penalty is a sensible move; generally logs end up on disk and are even fsynced, and the performance impact of that is many orders of magnitude larger. But, log frameworks have to cater to all scenarios, and logs which end up totally ignored due to log level config, in a tight loop, can see some performance improvement due to avoiding the varargs penalty. You can also try to optimize if it comes up that your log frameworks are impacting performance: You can ask the log handlers if the log level that is asked for is even relevant, and if not, just return; immediately. You can then also follow along and create this 'explosion'. See slf4j which has 10 methods for every log level, and many other frameworks have even more (they have a variant for 1, 2, 3, and sometimes even 4 arguments before resorting to varargs).
I'm new to loggers and I just wanted to ask if i will get an error if my log level is info, then i have something like:
logger.debug("hello " + object.getMethod());
And it is not inside an if block like below:
if(LOGGER.isDebugEnabled){...}
Also, will it have an impact on the time that an application may respond. If its not inside the if block statement?
All the log methods check for the level internally too. The reason we have isXYZEnabled methods if for performance.
Consider the following call:
logger.debug("Value of myObject is: " + hugeObject.toString());
and assume that hugeObject is, as its name suggests, huge. Constructing a string representation of it might be both time and memory consuming, and ultimately pointless, since after you created the string, nothing will be logged if the level isn't set to debug. Instead, you could explicitly check the level before performing this converstion, and save the time it could have taken:
if (logger.isDebugEnabled()) {
logger.debug("Value of myObject is: " + hugeObject.toString());
}
I have a Tomcat server running a large application. It has two classes similar to this example:
public abstract class ClassA {
private static final Logger LOGGER = Logger.getLogger(ClassA.class);
// ...
public File methodA(ICancellable cancel) {
URL request = new URL("an URL");
LOGGER.debug("Calling ClassB.methodB(type)");
File f = classB.methodB(request, "type", cancel);
LOGGER.debug("The call to ClassB.methodB(type)"
+ " returned the File==" + f);
// ...
}
}
public class ClassB {
private static final Logger LOGGER = Logger.getLogger(ClassB.class);
// ...
public static synchronized File methodB(URL url, String type,
ICancellable cancel)
{
final String thisMethodsName = "ClassB.methodB(url: " + url
+ ", type:" + type + ", cancel: " + cancel + ")";
LOGGER.debug("Entering method: " + thisMethodsName);
// ...
return f;
}
}
The application works properly, and ClassA.methodA() initially calls succesfully to ClassB.methodB(), as I can see in the log files:
[...]
14/02/2013 12:34:56 DEBUG ClassA:123 - Calling ClassB.methodB(type)
14/02/2013 12:34:56 DEBUG ClassB:456 - Entering method: ClassB.methodB(url: anURL, type: type, cancel: #1234);
[...]
14/02/2013 12:34:56 DEBUG ClassA:125 - The call to ClassB.methodB(type) returned the File=="aFile".
[...]
My problem is after the server is working for some time, it stops calling ClassB.methodB(). the application gets hung and it simply writes this to the log:
[...]
14/02/2013 12:34:56 DEBUG ClassA:123 - Calling ClassB.methodB(type)
That's the last line of the log file. ClassB.methodB() isn't actually called.
I suspected it could be due to opened resources that werent closed, but I'm trying to locate all the code that did that, and after fixsing so, it still happens.
What can be causing this? How can I continue searching for the cause?
JVM version: 1.6.0_13
Tomcat version: 6.0.18
Is it possible that there is a thread deadlock bug involving some code that you didn't paste? Your ClassB.methodB method is synchronized. You probably have some other thread that is holding and not releasing the synchronized lock on ClassB.class, preventing the thread that is doing the logging from ever acquiring that lock and entering the method.
Enable the debug mode and assign breakpoints in your code. Use IDE ( preferrably eclipse) to go step by step.
Eclipse has plugins like findbugs or PMD to scan the code and find out possible bugs.
Manually go through every conditions, method calls, loops to see flaws.
If you have written the complete app, then you would know where to look at. If not, it would still be good to check all that is written
What's the correct approach to log both an error message and an exception using SLF4J?
I've tried doing this but the exception stack trace is never printed:
logger.error("Unable to parse data {}", inputMessage, e);
In this case I want to populate {} with the inputMessage as well as logging out the exception stacktrace.
The only way I can see to do this would be to do this:
logger.error("Unable to parse data " + inputMessage, e);
which is not pretty.
As of SLF4J version 1.6, SLF4J will interpret the last parameter as you intended, i.e. as an exception. You must be using an older version of SLF4J API.
This feature is documented in a faq entry which is also referenced in the javadocs for Logger.
from http://www.slf4j.org/faq.html#paramException:
Yes, as of SLF4J 1.6.0, but not in previous versions. The SLF4J API supports parametrization in the presence of an exception, assuming the exception is the last parameter. Thus,
String s = "Hello world";
try {
Integer i = Integer.valueOf(s);
} catch (NumberFormatException e) {
logger.error("Failed to format {}", s, e);
}
will print the NumberFormatException with its stack trace as expected. The java compiler will invoke the error method taking a String and two Object arguments. SLF4J, in accordance with the programmer's most probable intention, will interpret NumberFormatException instance as a throwable instead of an unused Object parameter. In SLF4J versions prior to 1.6.0, the NumberFormatException instance was simply ignored.
If the exception is not the last argument, it will be treated as a plain object and its stack trace will NOT be printed. However, such situations should not occur in practice.
I got a recommendation to use this syntax when logging in java:
if (logger.isLoggable(Log.FINE))
{
logger.fine("bla"+" bla"+" bla");
}
The reason for this is to avoid the redundant construction of the parameter string incase the logging level is lower than "FINE". (in the example above - 5 redundant string object. (" bla"X3, " bla bla" and "bla bla bla").
I'd like to hear what others are doing about this or if you think that this is necessary at all.
Thanks!!
Some newer logging frameworks allow you to specify arguments as parameters, and won't evaluate them if there's no logging.
The example I found is LogBack, the successor to Log4j. Here's the info: http://www.infoq.com/news/2007/08/logback
This gives you the best of both worlds, so to speak. Elegant syntax yet good performance.
Log4j code example:
if( logger.isDebugEnabled() ) {
logger.debug( "User with account " +
user.getAccount() + " failed authentication; " +
"supplied crypted password " + user.crypt(password) +
" does not match." );
}
Equivalent LogBack code:
logger.debug( "User with account {} failed authentication; " +
"supplied crypted password {} does not match.",
user.getAccount(), user.crypt(password) );
This defers the cost of message assembly until LOGBack has ascertained whether or not this message will be viewed. It doesn't defer the cost of retrieving expensive parameters, such as the password crypting in the above example.
String objects are immutable, and repeated concatenation is therefore an expensive operation. It requires repeated memory allocation, object creation and iteration. Considering that some logging calls at the finer log levels can be invoked thousands or millions of times per minute, it might be a considerable performance gain to do as you illustrate. Though, for a smaller application, it might not be worth the extra effort.
As a side note: You can save even more performance, where this is truly critical by using a constant such as this:
public static final boolean DEBUG = false;
If you now wrap the logging code in an if-block such as this, the JVM will be able to completely optimize away the debug calls when running in product mode. This is as close as you get to a C #ifdef.
if (Globals.DEBUG) {
// Logging call
}
Absolutely necessary for debug type logging. It something like 10x quicker to check the log level first than create the string and throw it away.
This is an improvement (good) but it can be improved on a little.
Set up final flags for each logging level (FINE, etc) in a global object used as config, then use a StringBuffer to build up your debugging output -- you can even format numbers into the stream at the same time.
public class MyAppConfig {
public final boolean FINE=true;
// ... other fields
}
public class MyApp {
void someFunction() {
...
int imagesProcessed;
imagesProcessed = processImages();
if (MyAppConfig.FINE) logger.fine(new StringBuffer(35).
append("Count of images processed: ").append(imagesProcessed).toString());
...
}
}
Here the string buffer is set up with an 'initial capacity' of 35 characters. If you know how many characters are going to be generated you can provide hints to StringBuffer.