I got a recommendation to use this syntax when logging in java:
if (logger.isLoggable(Log.FINE))
{
logger.fine("bla"+" bla"+" bla");
}
The reason for this is to avoid the redundant construction of the parameter string incase the logging level is lower than "FINE". (in the example above - 5 redundant string object. (" bla"X3, " bla bla" and "bla bla bla").
I'd like to hear what others are doing about this or if you think that this is necessary at all.
Thanks!!
Some newer logging frameworks allow you to specify arguments as parameters, and won't evaluate them if there's no logging.
The example I found is LogBack, the successor to Log4j. Here's the info: http://www.infoq.com/news/2007/08/logback
This gives you the best of both worlds, so to speak. Elegant syntax yet good performance.
Log4j code example:
if( logger.isDebugEnabled() ) {
logger.debug( "User with account " +
user.getAccount() + " failed authentication; " +
"supplied crypted password " + user.crypt(password) +
" does not match." );
}
Equivalent LogBack code:
logger.debug( "User with account {} failed authentication; " +
"supplied crypted password {} does not match.",
user.getAccount(), user.crypt(password) );
This defers the cost of message assembly until LOGBack has ascertained whether or not this message will be viewed. It doesn't defer the cost of retrieving expensive parameters, such as the password crypting in the above example.
String objects are immutable, and repeated concatenation is therefore an expensive operation. It requires repeated memory allocation, object creation and iteration. Considering that some logging calls at the finer log levels can be invoked thousands or millions of times per minute, it might be a considerable performance gain to do as you illustrate. Though, for a smaller application, it might not be worth the extra effort.
As a side note: You can save even more performance, where this is truly critical by using a constant such as this:
public static final boolean DEBUG = false;
If you now wrap the logging code in an if-block such as this, the JVM will be able to completely optimize away the debug calls when running in product mode. This is as close as you get to a C #ifdef.
if (Globals.DEBUG) {
// Logging call
}
Absolutely necessary for debug type logging. It something like 10x quicker to check the log level first than create the string and throw it away.
This is an improvement (good) but it can be improved on a little.
Set up final flags for each logging level (FINE, etc) in a global object used as config, then use a StringBuffer to build up your debugging output -- you can even format numbers into the stream at the same time.
public class MyAppConfig {
public final boolean FINE=true;
// ... other fields
}
public class MyApp {
void someFunction() {
...
int imagesProcessed;
imagesProcessed = processImages();
if (MyAppConfig.FINE) logger.fine(new StringBuffer(35).
append("Count of images processed: ").append(imagesProcessed).toString());
...
}
}
Here the string buffer is set up with an 'initial capacity' of 35 characters. If you know how many characters are going to be generated you can provide hints to StringBuffer.
Related
Introduction
We're using SLF4J in combination with Logback in several Spring (Boot) applications and recently started using the logstash-logback-encoder to implement structured logging. As we also still have to support plain text logs, we wondered whether it was possible to append arguments automatically to log messages without having to add them manually to the message using the {} markers.
Example of the desired behaviour
To illustrate the desired behaviour this is what we'd wish for:
log.info("My message", kv("arg1", "firstArgument"), kv("arg2", "secondArgument"))
resulting in the following desired output, where the arguments are automatically appended in parentheses at the end of the message:
My message (arg1="firstArgument", arg2="secondArgument")
Or another example with both explicit arguments in the message and arguments at the end:
log.info("Status changed: {} => {}", v("from", "READY"), v("to", "UNAVAILABLE"), kv("service", "database"))
resulting in the following desired output:
Status changed: READY => UNAVAILABLE (from="READY", to="UNAVAILABLE", service="database")
Question
Is this possible with SLF4J/Logback? And if not, do you know other logging frameworks or ways to achieve this (in Java)?
I'm not aware of any log frameworks that let you do this, but you can trivially write your own. Because this really is just a simple API extension, and thus, all you need to duplicate is specifically the various log messages. For example, this one-liner would take care of it:
public static class LoggingExtensions {
#lombok.Value public static final class LogKeyValue {
String key, value;
}
public static LogKeyValue kv(String key, Object value) {
return new LogKeyValue(key, String.valueOf(value));
}
public static void info(Logger log, String message, Object... args) {
int extra = 0;
int len = args.length;
// Last arg could be a throwable, leave that alone.
if (len > 0 && args[len - 1] instanceof Throwable) len--;
for (int i = len - 1; i >= 0; i--) {
if (!(args[i] instanceof LogKeyValue)) break;
extra++;
}
if (extra > 0) {
StringBuilder sb = new StringBuilder(message.length() + 2 + (extra.size() - 1) * 2);
sb.append(message).append("({}");
for (int i = 1; i < extra; i++) sb.append(", {}");
message = sb.append(")").toString();
}
log.info(message, args);
}
}
This code tacks ({}, {} {}) at the end of the message, 1 for each 'kv' type. Note that most logging frameworks, including slf4j, do let you tack 1 exception at the end, even without a matching {} in the message, and this method would thus require that you first list all {} args, then any kv args, then 0 or 1 throwables.
Some caveats, though:
You'd have to change all your code to invoke these utility methods. You can make it look nice using static imports, but it does make your code less idiomatic, which is a downside.
Most log frameworks have an explosion of methods, because varargs cause the creation of arrays. In hotspotted code, the JDK will probably make that sufficiently efficient that it won't matter, but because log statements tend to be ubiquitous, you get a bit of a death-by-a-thousand cuts happening otherwise. It is unlikely that this call by logging frameworks to have a ton of methods to avoid the varargs penalty is a sensible move; generally logs end up on disk and are even fsynced, and the performance impact of that is many orders of magnitude larger. But, log frameworks have to cater to all scenarios, and logs which end up totally ignored due to log level config, in a tight loop, can see some performance improvement due to avoiding the varargs penalty. You can also try to optimize if it comes up that your log frameworks are impacting performance: You can ask the log handlers if the log level that is asked for is even relevant, and if not, just return; immediately. You can then also follow along and create this 'explosion'. See slf4j which has 10 methods for every log level, and many other frameworks have even more (they have a variant for 1, 2, 3, and sometimes even 4 arguments before resorting to varargs).
I have a Spring Boot application that uses a CredentialsService class to store credentials as GuardedStrings and return them when requested by other classes.
Where the problem arises is in the fact that we use Checkmarx to scan our code and catch potential issues. Where storage of the usernames/passwords are not a problem anymore, I still have to use a String variable to return the plain text credentials. Checkmarx doesn't like that - especially for passwords.
This is the abbreviated view of the CredentialsService:
#Component
public class CredentialsService {
final ExecutorService executor = Executors.newSingleThreadExecutor();
private GuardedString customerApiPassword;
. . .
private StringBuilder clearCustomerApiPassword;
public CredentialsService( . . .
#Value("${customerapi.pwd}") String customerApiPassword,. . .) {
setCustomerApiPassword(customerApiPassword);
. . .
}
private void setCustomerApiPassword(String customerApiPasswordString) {
this.customerApiPassword = new GuardedString(customerApiPasswordString.toCharArray());
this.customerApiPassword.makeReadOnly();
}
public String getCustomerApiPasswordNo() {
clearCustomerApiPassword = new StringBuilder();
customerApiPassword.access(new GuardedString.Accessor() {
#Override
public void access(final char[] clearChars) {
clearCustomerApiPassword.append(clearChars);
}
});
customerApiPassword.dispose();
System.out.println("DGC: clearCustomerApiPassword is " + clearCustomerApiPassword);
Runnable clearFromMemory = () -> {
clearCustomerApiPassword = null;
System.out.println("DGC: clearCustomerApiPassword is " + clearCustomerApiPassword);
};
executor.execute(clearFromMemory);
return clearCustomerApiPassword.toString();
}
And then a requester accesses the values it needs with:
IntegrationApiUtil.setBasicAuthKey(headers, credentialsService.getCustomerApiUsername(), credentialsService.getCustomerApiPassword());
However Checkmarx is still not happy. I use the same approach for storing the GuardedString usernames and passwords and the exact same approach to clearing the Strings that are returned. Checkmarx is fine with the usernames, but it still complains about the passwords:
Method clearCustomerApiPassword; at line 24 of
src/main/java/com/.../service/CredentialsService.java
defines clearCustomerApiPassword, which is designated to contain user passwords. However, while plaintext
passwords are later assigned to clearCustomerApiPassword, this variable is never cleared from memory.
I have tried all sorts of things - a finalize method to destroy the service after it is last used, a disposeAll method to explicitly set all variables to null and call the garbage collector. With the code above I am creating a separate thread in each get method to set the 'clear' variables to null as I return the value to the requester. While I can confirm that this latest approach does provide the requester with the correct values and also sets the variables to null, nothing seems to satisfy Checkmarx.
Does anyone have any ideas?
Thanks in advance.
D
Once you put sensitive data into an immutable object like String, the data will remain in the memory for a long time. You can release the variable, but even without a physical reference the value will still sit in the memory. You can run GC, it will still be there. The only thing that would help would be a creation of another variable using the same memory space and overriding the value.
Long story short: As long as you put your password in a String, Checkmarx will complain.
You have two things you can do:
you either rely on char[] only and clear the array once used,
or use a String value if you are forced to and request a special exception for your case.
Well, you are kinda throwing away all the value of storing passwords in GuardedString by returning/transporting them as regular String.
Don't know a lot about Checkmarx, but it's just a code scanning tool, so it's easy to fool. I suggest actually fixing the problems, instead of trying to sweep them under the rug.
Notice that GuardedString constructor accepts char[], not a String. That's the first problem - you should carry your password from the source up to this point as a char[] - more about it here.
Don't return String to your consumer - return the GuardedString or at least a char[].
Depends on what consumers you are targeting with this class/library, but try to provide a way for them to access the actual passwords for as short time as possible, and clearning the char[] after usage (in a way that consumer will not have to do that himself, since he can forget)
I'm new to loggers and I just wanted to ask if i will get an error if my log level is info, then i have something like:
logger.debug("hello " + object.getMethod());
And it is not inside an if block like below:
if(LOGGER.isDebugEnabled){...}
Also, will it have an impact on the time that an application may respond. If its not inside the if block statement?
All the log methods check for the level internally too. The reason we have isXYZEnabled methods if for performance.
Consider the following call:
logger.debug("Value of myObject is: " + hugeObject.toString());
and assume that hugeObject is, as its name suggests, huge. Constructing a string representation of it might be both time and memory consuming, and ultimately pointless, since after you created the string, nothing will be logged if the level isn't set to debug. Instead, you could explicitly check the level before performing this converstion, and save the time it could have taken:
if (logger.isDebugEnabled()) {
logger.debug("Value of myObject is: " + hugeObject.toString());
}
Given the Java argument evaluation mechanism, how does Log4j 2.x implement lazy evaluation when formatting the message with curly brackets "to avoid the cost of parameter construction" when log is disabled?
e.g.
logger.debug("Entry number: {} is {}", i, entry[i]);
I guess what Log4j means, is that with the curly brackets, they avoid constructing a string when its not necessary (e.g. the Level is not Debug):
With
logger.debug("Entry number: " + i + " is " + String.valueOf(entry[i]));
the complete message is always computed even when it will not be logged.
With
logger.debug("Entry number: {} is {}", i, entry[i]);
Log4j can check the Log-Level first and then decide if its worth to construct the message string. This saves resources needed for string concatenation. It will only call toString() on the objects supplied if the message is actually being created, saving further computation costs.
Log4j uses an internal class (org.apache.logging.log4j.message.ParameterFormatter) that replaces every {} with the arguments provided. It will also catch exceptions thrown by toString() and report these failures.
To avoid argument evaluation, just wrap it in lambda:
logger.debug(() -> {
"Entry number: " + i + " is " + String.valueOf(entry[i])
});
In this form, the supplier will be called for construction only before actual logging. Think of it like a function declaration.
In release 2.4, the Logger interface added support for lambda expressions. Such that, Log4j2 loggers can also take in a message supplier that will be lazily evaluated, here is the signature for the info one:
#Override
public void info(final Supplier<?> messageSupplier) {
logIfEnabled(FQCN, Level.INFO, null, messageSupplier, (Throwable) null);
}
This allows client code to lazily log messages without explicitly checking if the requested log level is enabled. For example, previously you would write:
// pre-Java 8 style optimization: explicitly check the log level
// to make sure the expensiveOperation() method is only called if necessary
if (logger.isTraceEnabled()) {
logger.trace("Some long-running operation returned {}", expensiveOperation());
}
With Java 8 and above you can achieve the same effect with a lambda expression. You no longer need to explicitly check the log level:
// Java-8 style optimization: no need to explicitly check the log level:
// the lambda expression is not evaluated if the TRACE level is not enabled
logger.trace("Some long-running operation returned {}", () -> expensiveOperation());
For Kotlin, there is a Kotlin logging facade based on Log4j 2 in which the signature is as follow:
fun info(supplier: () -> Any?) {
delegate.logIfEnabled(FQCN, Level.INFO, null, supplier.asLog4jSupplier(), null)
}
And here is a usage example:
logger.info { "This is a log event" }
Both will avoid the cost of parameter construction.
I would like to check with the community, if this is an accepted practice, to have multiple if conditions for each debug statement occurring in a row:
if (log.isDebugEnabled()) log.debug("rproductType = "+ producteType);
if (log.isDebugEnabled()) log.debug("rbundleFlag = " + bundleFrlag);
if (log.isDebugEnabled()) log.debug("rmrktSegment = " + mrktSeegment);
if (log.isDebugEnabled()) log.debug("rchannelTy = " + channelrTy);
if (log.isDebugEnabled()) log.debug("rcompanyPartner = " + coempanyPartner);
if (log.isDebugEnabled()) log.debug("rpreSaleDate = " + preSaleDaete);
if (log.isDebugEnabled()) log.debug("rportNC = " + portrNC);
if (log.isDebugEnabled()) log.debug("rLDC debug end");
I am personally supportive of have a single if condition to wrap up the entire log statements since they are appearing in a row. What are your inputs on this? Or do you see why the original author had wanted to have an if condition for each debug statement?
Thanks!
At best, it is messy. At worst, it performs absolutely redundant function calls.
The only potential difference in logic between sharing ifs is if the debugging option is somehow changed mid-call (possibly by a config reload). But capturing that extra half-a-call really isn't worth the wall of gross code.
Just change it. Don't Repeat Yourself
The reason the if is there at all is to avoid the overhead of building the debug strings if you aren't in debug mode; that part you should keep (Or not keep, if you find this is not a performance critical part of your application).
Edit FYI, by "change it", I mean do this instead:
if (log.isDebugEnabled())
{
log.debug("rproductType = "+ producteType);
log.debug("rbundleFlag = " + bundleFrlag);
// etc
}
The if condition is in there for increased speed.
It is intended to avoid the computational cost of the disabled debug statements. That is, if you have your log level set to ERROR, then there is no need to create the actual message.
http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/Category.html
My personal feeling is that the
if (log.isDebugEnabled())
statments are a micro-optimization that just makes the code harder to read.
I know this was once a best practice for log4j, but many of those checks are performed prior the log staments themselves ( you can see in the source code ) and the only operations you are saving yourself is the string concatenations. I would remove them.
I would do it like that:
if (log.isDebugEnabled()) {
StringBuilder builder = new StringBuilder();
builder.append("rproductType = ");
builder.append(producteType);
builder.append("rbundleFlag = ");
builder.append(bundleFrlag);
builder.append("rproductType = ");
builder.append(mrktSeegment);
builder.append("rchannelTy = ");
builder.append(channelrTy);
builder.append("rcompanyPartner = ");
builder.append(coempanyPartner);
builder.append("rpreSaleDate = ");
builder.append(preSaleDaete);
builder.append("rportNC = ");
builder.append(portrNC);
builder.append("rLDC debug end");
log.debug(builder.toString());
}
You have only 2 isDebugEnabled checks in this code: one at the beginning, one in log.debug. The first one prevents the creation of the builder and several short living objects (you heap will thank you)
The second one is obsolet in this case. But it is an pretty easy check, so I think the cost the builder would be higher. Given the fact that in most production system the debug level is off, I think this is the best option.
To summarize, I use isDebugEnabled when my debug statement - or anything else what I need to do for my debug message - becomes more complex then usual. In most cases this is when it comes to String concatenations.
Never use isDebugEnabled for single line log statements. As it has been already mentioned. the log.debug method calls it itself.
Cheers
Christian
Depends. Log4j does this very check at the beginning of every method—isDebugEnabled() before debug statement, isWarnEnabled() before warn, etc.,—by default.
This does not mean checks are not required. Checks can save procssing if any of the parameters passed invoke computation. E.g., LOGGER.debug(transformVars(a, b, c, d)); would result in unnecessary execution of transform(), if debug is not enabled!