We Keep Coding

SLF4J/Java logging: Is it possible to add log arguments automatically?

Introduction
We're using SLF4J in combination with Logback in several Spring (Boot) applications and recently started using the logstash-logback-encoder to implement structured logging. As we also still have to support plain text logs, we wondered whether it was possible to append arguments automatically to log messages without having to add them manually to the message using the {} markers.
Example of the desired behaviour
To illustrate the desired behaviour this is what we'd wish for:
log.info("My message", kv("arg1", "firstArgument"), kv("arg2", "secondArgument"))
resulting in the following desired output, where the arguments are automatically appended in parentheses at the end of the message:
My message (arg1="firstArgument", arg2="secondArgument")
Or another example with both explicit arguments in the message and arguments at the end:
log.info("Status changed: {} => {}", v("from", "READY"), v("to", "UNAVAILABLE"), kv("service", "database"))
resulting in the following desired output:
Status changed: READY => UNAVAILABLE (from="READY", to="UNAVAILABLE", service="database")
Question
Is this possible with SLF4J/Logback? And if not, do you know other logging frameworks or ways to achieve this (in Java)?

I'm not aware of any log frameworks that let you do this, but you can trivially write your own. Because this really is just a simple API extension, and thus, all you need to duplicate is specifically the various log messages. For example, this one-liner would take care of it:
public static class LoggingExtensions {
#lombok.Value public static final class LogKeyValue {
String key, value;
}
public static LogKeyValue kv(String key, Object value) {
return new LogKeyValue(key, String.valueOf(value));
}
public static void info(Logger log, String message, Object... args) {
int extra = 0;
int len = args.length;
// Last arg could be a throwable, leave that alone.
if (len > 0 && args[len - 1] instanceof Throwable) len--;
for (int i = len - 1; i >= 0; i--) {
if (!(args[i] instanceof LogKeyValue)) break;
extra++;
}
if (extra > 0) {
StringBuilder sb = new StringBuilder(message.length() + 2 + (extra.size() - 1) * 2);
sb.append(message).append("({}");
for (int i = 1; i < extra; i++) sb.append(", {}");
message = sb.append(")").toString();
}
log.info(message, args);
}
}
This code tacks ({}, {} {}) at the end of the message, 1 for each 'kv' type. Note that most logging frameworks, including slf4j, do let you tack 1 exception at the end, even without a matching {} in the message, and this method would thus require that you first list all {} args, then any kv args, then 0 or 1 throwables.
Some caveats, though:
You'd have to change all your code to invoke these utility methods. You can make it look nice using static imports, but it does make your code less idiomatic, which is a downside.
Most log frameworks have an explosion of methods, because varargs cause the creation of arrays. In hotspotted code, the JDK will probably make that sufficiently efficient that it won't matter, but because log statements tend to be ubiquitous, you get a bit of a death-by-a-thousand cuts happening otherwise. It is unlikely that this call by logging frameworks to have a ton of methods to avoid the varargs penalty is a sensible move; generally logs end up on disk and are even fsynced, and the performance impact of that is many orders of magnitude larger. But, log frameworks have to cater to all scenarios, and logs which end up totally ignored due to log level config, in a tight loop, can see some performance improvement due to avoiding the varargs penalty. You can also try to optimize if it comes up that your log frameworks are impacting performance: You can ask the log handlers if the log level that is asked for is even relevant, and if not, just return; immediately. You can then also follow along and create this 'explosion'. See slf4j which has 10 methods for every log level, and many other frameworks have even more (they have a variant for 1, 2, 3, and sometimes even 4 arguments before resorting to varargs).

Is it possible for logger.debug to produce error if logging level is info

I'm new to loggers and I just wanted to ask if i will get an error if my log level is info, then i have something like:
logger.debug("hello " + object.getMethod());
And it is not inside an if block like below:
if(LOGGER.isDebugEnabled){...}
Also, will it have an impact on the time that an application may respond. If its not inside the if block statement?

All the log methods check for the level internally too. The reason we have isXYZEnabled methods if for performance.
Consider the following call:
logger.debug("Value of myObject is: " + hugeObject.toString());
and assume that hugeObject is, as its name suggests, huge. Constructing a string representation of it might be both time and memory consuming, and ultimately pointless, since after you created the string, nothing will be logged if the level isn't set to debug. Instead, you could explicitly check the level before performing this converstion, and save the time it could have taken:
if (logger.isDebugEnabled()) {
logger.debug("Value of myObject is: " + hugeObject.toString());
}

Encog - EarlyStoppingStrategy with validation set

I would like to stop training a network once I see the error calculated from the validation set starts to increase. I'm using a BasicNetwork with RPROP as the training algorithm, and I have the following training iteration:
double validationError = 999.999;
while(!stop){
train.iteration(); //weights are updated here
System.out.println("Epoch #" + epoch + " Error : " + train.getError()) ;
//I'm just comparing to see if the error on the validation set increased or not
if (network.calculateError(validationSet) < validationError)
validationError = network.calculateError(validationSet);
else
//once the error increases I stop the training.
stop = true ;
System.out.println("Epoch #" + epoch + "Validation Error" + network.calculateError(validationSet));
epoch++;
}
train.finishTraining();
Obviously this isn't working because the weights have already been changed before figuring out if I need to stop training or not. Is there anyway I can take a step back and use the old weights?
I also see the EarlyStoppingStrategy class which is probably what I need to use by using the addStrategy() method. However, I really don't understand why the EarlyStoppingStrategy constructor takes both the validation set and the test set. I thought it would only need the validation set and the test set shouldn't be used at all until I test the output of the network.

Encog's EarlyStoppingStrategy class implements an early stopping strategy according to this paper:
Proben1 | A Set of Neural Network Benchmark Problems and Benchmarking Rules
(a full cite is included in the Javadoc)
If you just want to stop as soon as a validation set's error no longer improves you may just want to use the Encog SimpleEarlyStoppingStrategy, found here:
org.encog.ml.train.strategy.end.SimpleEarlyStoppingStrategy
Note, that SimpleEarlyStoppingStrategy requires Encog 3.3.

Dumping state of variables on Exception

I was wondering if there is a way to dump the state of all local variables when there is an exception, to get a better idea of the state of the environment that caused the exception. Below the variable idsToDump is unknown at run time and I want to find out the state at which value in the collection is causing the NPE.
Example:
public static void main(String[] args) {
HashMap<Integer, String> employees = new HashMap<Integer, String>();
employees.put(1, "James");
Integer[] idsToDump = new Integer[] { 1, 2, 3 };
for (Integer employeeId : idsToDump) {
String name = employees.get(employeeId).toLowerCase();
System.out.println(name + " is employee number: " + employeeId);
}
}
output:
james is employee number: 1
Exception in thread "main" java.lang.NullPointerException
Question:
Is there some JVM argument that I can pass to dump information about the current state of the local variables? ie we get
java.lang.NullPointerException
and (this is the part I'm after)
values: employeeId=2
I want to be able to do this on a client site, so no access to Eclipse or debugging tools, looking for just JVM arguments, can't make code changes either. I have looked through them but couldn't really find anything. In the meantime I'll keep searching there too ;)

I came across a commercial product that does this by simply using a startup -agentlib JVM agent param. I haven't used it yet but intend on giving it a try as it looks very promising.
https://www.takipi.com/product
Anyone have experience with this product?

Given all your restrictions, I can't recommend anything else apart from jdb. Fire that bad boy up and start stepping through the client code line by line. I know you said no debugging tools, but unless they are a JRE only environment you should have jdb installed already.

The easiest thing to do is to run this code in a debugger and step through it. If you can't debug the code then you can use a try/catch block and set employeeId outside of this like:
int debugEmployeeId = -1;
try {
Integer[] idsToDump = new Integer[] { 1, 2, 3 };
for (Integer employeeId : idsToDump) {
debugEmployeeId = employeeId;
...
} catch (Exception e) {
throw new Exception("Missing employeeId = " + Integer.toString(debugEmployeeId));
}

Although you have mentioned that you are not able to do any code changes, here a hint for the case that code changes are still possible: A handy tool to get some more information about the exception is the method "printStackTrace" of the thrown exception object. You may want to use something like this.
try {
...
}
catch ( Exception e ) {
System.out.println( "Exception occured! Reason: " + e.printStackTrace() );
}

Your best option, afaik, is the old manual way: use some sort of logging (be it Log4J, or just stdout), and to explicitly record the variables you're interested in via a catch block.

Like Thomas, I don't know of any current method that can dump variables. However, I believe you want to do so to aid you in your debugging. Here are some simple methods which I use to debug my code using just the console:
1.Learn to read the stacktrace when the exception occurs. It gives you a lot of information on what could be potentially causing the exception. If the particular line that the stacktrace is pointing to doesn't seem to have anything wrong, track back the code and see if it was a prior object that is causing the exception. For example (using some sample code and methods):
Book book = null;
Bookshelf bookshelf = new Bookshelf();
bookshelf.add(book);
bookshelf.getBooks();
Something like that will cause the NPE stacktrace to point to bookshelf, but actually it is book that is causing the NPE.
2.Print out the variables that you suspect are causing the NPE.
for (Integer employeeId : idsToDump) {
System.out.println(employeeId);
String name = employees.get(employeeId).toLowerCase();
System.out.println(name + " is employee number: " + employeeId);
}
Your output will be:
1
2
And then you'll know that 2 is causing the NPE.
3.Comment out the suspicious code and systematically uncomment out the code until the exception occurs, or vice-versa.
Although you may find it tedious and frustrating at times, it really does help in your foundation because with enough practice, you'll soon be able to intuitively spot where the error usually occurs (and hence spend less time on future debugging).

Sorrounding Logger with an If clause to avoid redundant String construction

I got a recommendation to use this syntax when logging in java:
if (logger.isLoggable(Log.FINE))
{
logger.fine("bla"+" bla"+" bla");
}
The reason for this is to avoid the redundant construction of the parameter string incase the logging level is lower than "FINE". (in the example above - 5 redundant string object. (" bla"X3, " bla bla" and "bla bla bla").
I'd like to hear what others are doing about this or if you think that this is necessary at all.
Thanks!!

Some newer logging frameworks allow you to specify arguments as parameters, and won't evaluate them if there's no logging.
The example I found is LogBack, the successor to Log4j. Here's the info: http://www.infoq.com/news/2007/08/logback
This gives you the best of both worlds, so to speak. Elegant syntax yet good performance.
Log4j code example:
if( logger.isDebugEnabled() ) {
logger.debug( "User with account " +
user.getAccount() + " failed authentication; " +
"supplied crypted password " + user.crypt(password) +
" does not match." );
}
Equivalent LogBack code:
logger.debug( "User with account {} failed authentication; " +
"supplied crypted password {} does not match.",
user.getAccount(), user.crypt(password) );
This defers the cost of message assembly until LOGBack has ascertained whether or not this message will be viewed. It doesn't defer the cost of retrieving expensive parameters, such as the password crypting in the above example.

String objects are immutable, and repeated concatenation is therefore an expensive operation. It requires repeated memory allocation, object creation and iteration. Considering that some logging calls at the finer log levels can be invoked thousands or millions of times per minute, it might be a considerable performance gain to do as you illustrate. Though, for a smaller application, it might not be worth the extra effort.
As a side note: You can save even more performance, where this is truly critical by using a constant such as this:
public static final boolean DEBUG = false;
If you now wrap the logging code in an if-block such as this, the JVM will be able to completely optimize away the debug calls when running in product mode. This is as close as you get to a C #ifdef.
if (Globals.DEBUG) {
// Logging call
}

Absolutely necessary for debug type logging. It something like 10x quicker to check the log level first than create the string and throw it away.

This is an improvement (good) but it can be improved on a little.
Set up final flags for each logging level (FINE, etc) in a global object used as config, then use a StringBuffer to build up your debugging output -- you can even format numbers into the stream at the same time.
public class MyAppConfig {
public final boolean FINE=true;
// ... other fields
}
public class MyApp {
void someFunction() {
...
int imagesProcessed;
imagesProcessed = processImages();
if (MyAppConfig.FINE) logger.fine(new StringBuffer(35).
append("Count of images processed: ").append(imagesProcessed).toString());
...
}
}
Here the string buffer is set up with an 'initial capacity' of 35 characters. If you know how many characters are going to be generated you can provide hints to StringBuffer.