Introduction
We're using SLF4J in combination with Logback in several Spring (Boot) applications and recently started using the logstash-logback-encoder to implement structured logging. As we also still have to support plain text logs, we wondered whether it was possible to append arguments automatically to log messages without having to add them manually to the message using the {} markers.
Example of the desired behaviour
To illustrate the desired behaviour this is what we'd wish for:
log.info("My message", kv("arg1", "firstArgument"), kv("arg2", "secondArgument"))
resulting in the following desired output, where the arguments are automatically appended in parentheses at the end of the message:
My message (arg1="firstArgument", arg2="secondArgument")
Or another example with both explicit arguments in the message and arguments at the end:
log.info("Status changed: {} => {}", v("from", "READY"), v("to", "UNAVAILABLE"), kv("service", "database"))
resulting in the following desired output:
Status changed: READY => UNAVAILABLE (from="READY", to="UNAVAILABLE", service="database")
Question
Is this possible with SLF4J/Logback? And if not, do you know other logging frameworks or ways to achieve this (in Java)?
I'm not aware of any log frameworks that let you do this, but you can trivially write your own. Because this really is just a simple API extension, and thus, all you need to duplicate is specifically the various log messages. For example, this one-liner would take care of it:
public static class LoggingExtensions {
#lombok.Value public static final class LogKeyValue {
String key, value;
}
public static LogKeyValue kv(String key, Object value) {
return new LogKeyValue(key, String.valueOf(value));
}
public static void info(Logger log, String message, Object... args) {
int extra = 0;
int len = args.length;
// Last arg could be a throwable, leave that alone.
if (len > 0 && args[len - 1] instanceof Throwable) len--;
for (int i = len - 1; i >= 0; i--) {
if (!(args[i] instanceof LogKeyValue)) break;
extra++;
}
if (extra > 0) {
StringBuilder sb = new StringBuilder(message.length() + 2 + (extra.size() - 1) * 2);
sb.append(message).append("({}");
for (int i = 1; i < extra; i++) sb.append(", {}");
message = sb.append(")").toString();
}
log.info(message, args);
}
}
This code tacks ({}, {} {}) at the end of the message, 1 for each 'kv' type. Note that most logging frameworks, including slf4j, do let you tack 1 exception at the end, even without a matching {} in the message, and this method would thus require that you first list all {} args, then any kv args, then 0 or 1 throwables.
Some caveats, though:
You'd have to change all your code to invoke these utility methods. You can make it look nice using static imports, but it does make your code less idiomatic, which is a downside.
Most log frameworks have an explosion of methods, because varargs cause the creation of arrays. In hotspotted code, the JDK will probably make that sufficiently efficient that it won't matter, but because log statements tend to be ubiquitous, you get a bit of a death-by-a-thousand cuts happening otherwise. It is unlikely that this call by logging frameworks to have a ton of methods to avoid the varargs penalty is a sensible move; generally logs end up on disk and are even fsynced, and the performance impact of that is many orders of magnitude larger. But, log frameworks have to cater to all scenarios, and logs which end up totally ignored due to log level config, in a tight loop, can see some performance improvement due to avoiding the varargs penalty. You can also try to optimize if it comes up that your log frameworks are impacting performance: You can ask the log handlers if the log level that is asked for is even relevant, and if not, just return; immediately. You can then also follow along and create this 'explosion'. See slf4j which has 10 methods for every log level, and many other frameworks have even more (they have a variant for 1, 2, 3, and sometimes even 4 arguments before resorting to varargs).
Related
In Cucumber 7.4.1+ TypeRegistry is deprecated in favour of annotations.
Indeed, as of today, I have never used anything but #ParameterType to define my ParameterTypes. Searching for alternatives, TypeRegistry is the only one I have found - but if it is "on the way out", of course I'd rather not start using it now.
Given a construct like this I cannot use annotations because those cannot take static parameters:
enum SpecialDate implements Supplier<Date> {
TODAY { #Override public Date get() { return Date(); } },
// YESTERDAY, etc.
;
static String typeSafeRegEx() {
return Arrays.stream(Zeitpunkt.values())
.map(SpecialDate::specName)
.collect(Collectors.joining("|"));
}
static SpecialDate from(final String specName) {
return valueOf(StringUtils.upperCase(specName));
}
String specName() {
return StringUtils.capitalize(StringUtils.lowerCase(this.name()));
}
}
public class ParameterTypes {
// does not compile: "Attribute value must be constant"
#ParameterType("(" + SpecialDate.typeSafeRegEx() + ")")
public Date specialDate(final String specName) {
return SpecialDate.from(specName).get();
}
}
A so-specified regEx is nice, because it will only match values guaranteed to be mappable, so I need no additional error handling code beyond Cucumber's own. The list of allowed values is also maintenance-free (compared to a "classic" switch which would silently grow incorrect when adding new values).
The alternative would be to use an unsafe switch + default: throw, strictly worse because it has to be maintained manually.
(Or, I guess, to just valueOf whatever + wrap into a more specific exception, when it eventually fails.)
To me, Cucumber's native UndefinedStepException appears to be the best outcome on a mismatch, because everyone familiar with Cucumber will immediately recognise it, unlike a project-specific one.
I see that e.g. the ParameterType class is not deprecated but cannot seem to find information how to use it without TypeRegistry.
FWIW:
Updating the libraries or Java would not be an issue. (But downgrading is sadly not viable.)
Business Specialists want to write examples like [Today], [Today + 1], [Yesterday - 3], etc. If this can be realised more elegantly using a different approach, X/Y answers would also be welcome.
An example step looks like this:
And I enter into the field 'Start of Implementation' the <begin>
Examples:
| begin
| [Today]
| [Today + 1]
For logging purposes, I created a method logTitle() that prints out the calling method name for our TestNG tests. Sample code is below.
#Test
public void test1() throws Exception {
method1();
}
public static void method1() throws Exception {
Utils.logTitle(2);
}
...
public static void logTitle(Integer level) throws Exception {
// Gets calling method name
String method = Thread.currentThread().getStackTrace()[2].getMethodName();
// This would get current method name
switch (level) {
case 1:
logger.info("=======================================================");
logger.info(method);
logger.info("=======================================================");
break;
case 2:
logger.info("------------------------------------");
logger.info(method);
logger.info("------------------------------------");
break;
case 3:
logger.info("---------------------");
logger.info(method);
logger.info("---------------------");
break;
case 4:
logger.info("--------- " + method + " ------------");
break;
default:
logger.info(method);
}
}
The problem is I am getting different results for logTitle() on two different machines.
Everyone's laptop returns correctly:
2016-06-20 14:22:06 INFO - ------------------------------------
2016-06-20 14:22:06 INFO - method1
2016-06-20 14:22:06 INFO - ------------------------------------
Our dev unix box returns differently:
2016-06-20 14:42:26 INFO - ------------------------------------
2016-06-20 14:42:26 INFO - logTitle
2016-06-20 14:42:26 INFO - ------------------------------------
This works correctly on everyone else's laptop, just not the dev unix box. I think the dev unix box is using IBM's version of Java, while everyone else is using Oracle's version of Java, but not sure if that is the culprit or not.
Any ideas?
From Javadoc:
Some virtual machines may, under some circumstances, omit one or more stack frames from the stack trace. In the extreme case, a virtual machine that has no stack trace information concerning this throwable is permitted to return a zero-length array from this method.
So, the only guaranteed way to do this is to use aspects, or collect stack trace with some other custom way. But you can combine this approach with fallback to some way of getting current method's name (for case when your logTitle method will be inlined). It can be found here, for example. Again, no guarantee, but better chance.
The simpler way to have a test method name is by using a #BeforeMethod and injecting the Method. See TestNG's documentation, here.
Just store the name somewhere and use it in your log (why not in a #AfterMethod ?)
My guess, and as mentioned by MeBigFatGuy. This can happen because of the different in implementation/defaults of the JIT compiler of the IBM/Oracle JVM when doing method inlining optimization.
I suggest running the code in the dev unix box with
-Xjit:disableInlining
and see if the issue disappear.
If this will work for you it may be fine for testing, but as mentioned in Alexey Adamovskiy answer we cannot trust java to be consist in the stack frames.
See also:
Would Java inline method(s) during optimization?
Selectively disabling the JIT compiler
Java method inlining
performance considerations
I guess the behavior is JVM specific. In the past I came up with this solution:
// find first stack trace entry that is not in this class
Optional<StackTraceElement> ste = Iterables.tryFind(
Arrays.asList(new RuntimeException().getStackTrace()),
new Predicate<StackTraceElement>() {
#Override
public boolean apply(StackTraceElement input) {
return !input.getClassName().equals(PutYourClassHere.class.getName());
}
});
if (ste.isPresent()) {
LOG.trace("Method called by: {}.{}", ste.get().getClassName(), ste.get().getMethodName());
}
The snippet is using Google Guava because this is for Java 7. If you have Java 8, you can use Streams API and lambdas. I made the ste.isPresent() check because I encountered an empty stack trace once. As far as I remember the Oracle JVM is skipping stack traces when the same exception is thrown over and over again.
EDIT: Java 8 way
Optional<StackTraceElement> ste = Arrays.stream(new RuntimeException().getStackTrace())
.filter(x -> !x.getClassName().equals(Utils.class.getName()))
.findFirst();
I think its the specific depth that is causing the issue which is 2 in your scenario.
So, instead of writing
String method = Thread.currentThread().getStackTrace()[2].getMethodName();
if you write
StackTraceElement[] ste = Thread.currentThread().getStackTrace();
String method = null;
boolean doNext = false;
for (StackTraceElement s : ste) {
if (doNext) {
method = s.getMethodName();
return;
}
doNext = s.getMethodName().equals("getStackTrace");
}
It will work only for JDK 1.5+
The other option is as below:
String method = new Object(){}.getClass().getEnclosingMethod().getName();
Or a slower option will be :
String method = new Exception().getStackTrace()[0].getMethodName();
As this will create an instance of Exception everytime.
Hope that helps you out.
Log4J finds the method name by searching down the stack trace until it finds the target class name which must be passed in, then reads the method name.
In your code, you could use a similar technique - instead of a static method Utils you could create an instance in your test, passing in the class of the test:
Utils utils = new Utils(MyTest.class);
Then use the previously mentioned search technique in the Utils.logTitle() method.
Utils.logTitle() would search forwards through stack trace elements of a newly created Throwable until it finds the first element with the desired target class.
Log4j 2 uses the fully qualified class name of the Logger to locate the class and method from which the Logger was called. The code to find the location follows below. Feel free to use it.
Note that the loop starts from the bottom of the stacktrace; this is necessary to detect exceptional cases where the logger is called recursively (perhaps from the toString() method of an object that was logged). In such cases we want to report the first class/method that called the Logger, not the last one, so we have no choice but to walk the stack trace from the bottom up.
public static StackTraceElement calcLocation(final String fqcnOfLogger) {
if (fqcnOfLogger == null) {
return null;
}
// LOG4J2-1029 new Throwable().getStackTrace is faster
// than Thread.currentThread().getStackTrace().
final StackTraceElement[] stackTrace = new Throwable().getStackTrace();
StackTraceElement last = null;
for (int i = stackTrace.length - 1; i > 0; i--) {
final String className = stackTrace[i].getClassName();
if (fqcnOfLogger.equals(className)) {
return last;
}
last = stackTrace[i];
}
return null;
}
Context:
I have created a small (java) multithread server for a game. Despite my best efforts of following the best practices it turned out that some methods that were intended to be called from just one thread got called from 2 or more threads. After debugging and analysis I have managed to "fix" my design but I was wondering:
The question:
Is there a tool (or if not - is it possible (and how) to be developed) that allows you to mark some methods with annotations like #SingleThread or #ThreadCount(2) or #ThreadNameLike("my_fancy_thread_group*") which counts/monitors/logs access to these methods like:
#SingleThread - checks if this method is always accessed by only thread
#ThreadCount(2) - is accessed by two threads exactly
#ThreadNameLike - is accessed only by threads with name matching the pattern(s)
The idea is to do a TEST run of the program and get at least log record that annotated condition is violated.
I was thinking that probably AspectJ can do the job to some extend with it's pointcuts but then I realized that some approach similar to Dagger / Dagger2 will be better, i.e. when you want to test your server you will have to turn on an annotation processor (let's called it hypothetically "SafetyFirst") which will generate adapter (wrapper?) classes which contain the monitoring code. Then you will run the server, run some load tests and then check the logs for violations (or in an ideal world - get a report file).
I fully realize that such tool:
will not provide 100% coverage of all potential cases;
will mask/trigger heisenbugs
will slow down the examined program
but at very least it can serve as an early warning system that will clearly advertise violations of the intended design.
I used a similar test with AspectJ load time weaving for intended printing of all function calls within my package.
Best way of load time weaving is you dont dirty your classes like compile time wevaing. When you remove -javaagent:<path to aspectj lib> and your custom astpect lib classpath entry from your run command. Then all all gone, clear.
I made some changes and implemented a test covering #ThreadCount functionality you asked. You need to download and install AspectJ.
Please see code snippets:
aspect Profile {
private static Map<String, AtomicInteger> counterMap = new HashMap<String, AtomicInteger>();
pointcut threadCountPc(test.ThreadCount tc) : execution(* test..*(..)) && #annotation(tc);
Object around(test.ThreadCount tc) : threadCountPc(tc) {
String signature = thisJoinPointStaticPart.getSignature().toString();
AtomicInteger counter = getCounter(signature);
int currentValue = counter.incrementAndGet();
if (currentValue >= tc.value()){
System.out.println("[Thread Name:" + Thread.currentThread().getName() +
"] Method Name:" + signature + ", threadCount:" + currentValue + " exceeds " + tc.value());
}
try{
return proceed(tc);
}finally{
counter.decrementAndGet();
}
}
private static AtomicInteger getCounter(String methodName){
AtomicInteger value = counterMap.get(methodName);
if (value == null){
synchronized (counterMap){
value = counterMap.get(methodName);
if (value == null){
value = new AtomicInteger(0);
counterMap.put(methodName, value);
}
}
}
return value;
}
}
Compiling: "C:/aspectj1.8/bin/ajc.bat" profile_ft.aj -cp C:/aspectj1.8/lib/aspectjrt.jar;../test/. -1.6 -outxml -outjar profile_ft.jar
Running: java -javaagent:C:/aspectj1.8/lib/aspectjweaver.jar -cp aspectj/profile_ft.jar;test test.Test1
EDIT2
#paradigmatic made a good point in suggesting to redirect rather than throw the exception; that solves the logging issue. The problem in Play 2 is that redirects need to occur within so-called Action scope, which is not always the case with date parser calls.
As a workaround, I went with Play's global interceptor, presumably the equivalent of a Java servlet filter.
val ymdMatcher = "\\d{8}".r // matcher for yyyyMMdd URI param
val ymdFormat = org.joda.time.format.DateTimeFormat.forPattern("yyyyMMdd")
def ymd2Date(ymd: String) = ymdFormat.parseDateTime(ymd)
override def onRouteRequest(r: RequestHeader): Option[Handler] = {
import play.api.i18n.Messages
ymdMatcher.findFirstIn(r.uri) map{ ymd=>
try { ymd2Date( ymd); super.onRouteRequest(r) }
catch { case e:Exception => // kick to "bad" action handler on invalid date
Some(controllers.Application.bad(Messages("bad.date.format")))
}
} getOrElse(super.onRouteRequest(r))
}
EDIT
Here 's a little context to work with:
// String "pimp": transforms ymdString.to_date call into JodaTime instance
class String2Date(ymd: String) {
def to_date = {
import play.api.i18n.Messages
try{ ymdFormat.parseDateTime(ymd) }
catch { case e:Exception => throw new NoTrace(Messages("bad.date.format")) }
}
val ymdFormat = org.joda.time.format.DateTimeFormat.forPattern("yyyyMMdd")
}
#inline implicit final def string2Date(ymd: String) = new String2Date(ymd)
and a test custom exception handler:
public class NoTrace extends Exception {
static final long serialVersionUID = -3387516993124229948L;
#Override
public Throwable fillInStackTrace() {
return null;
}
public NoTrace(String message) {
super(message);
}
}
Calling the date parser on an invalid yyyyMMdd string logs 30 line stack trace to the log (this occurs upstream by Play framework/Netty container, better than default 100 line trace):
"20120099".to_date
ORIGINAL
Have an issue where my application.log is getting filled with errors related to a uri date parser operation that should succeed given a valid yyyyMMdd uri date.
However, some users try to circumvent this by entering invalid dates in hopes of gaining free access to paid subscriber-only content. It's pointless, as it simply won't work, but regardless, I have MBs of these error traces in my application log.
Is there a way to throw a truly trimmed down Exception to the log? I found this SO answer, but in my application it looks like the container (Play framework on Netty) gets into the mix and logs its own 30 line stack trace to the log (30 lines is better than 100, but still 29 too many)
Similarly, I found this thread in regard to Java 7 and the new option to suppress stack trace; however, for some reason, despite being on Java 1.7, with Eclipse configured for Java 1.7, only the old 2 param method of Throwable is available (and I do see the 4 param method when I click through to the Throwable class; maybe a Scala 2.9.2 library issue?)
At any rate, ideally I can simply log a 1-line exception message and not the kitchen sink.
Simply override this method in your custom exception class:
#Override
public Throwable fillInStackTrace() {
return this;
}
after adding this method your trace method will not print
Your trouble is that although you can suppress the stacktrace of the exception your own code threw, there is nothing you can do about the exception it will be wrapped into by the framework. The only avenue I can see is not allowing the framework to catch your exception at all (doing your own top-level handling) or tweaking the logging configuration.
I think you have two options:
Control the logging to not save stack traces for some exceptions.
Write a post-processor that filters out the traces from the log file.
Unless you are in danger of running out of disk space, I think #2 is the better option, because if you do have a bug you can go back to the full log and have all the exception history.
The philosophy behind idea #2 is that disk space is cheap, but information can be precious during debug. Log a lot of data. Normally, use scripts to examine the log after it has been written to disk.
For example, if there is a type of log entry that you never expect to see, but that demands immediate action if it does appear, write a script that searches for it, and send you an e-mail if it finds one.
One of the most useful forms of script in this approach is one that drops stack trace lines. Usually, you only need to know what exceptions are happening, and the stack trace takes up a lot of screen space without telling you much. If you do need to investigate an exception, go back to the full log, find the exception line, and look at the stack trace and at what was happening immediately before the exception.
If there are too many of your date exceptions, have the script drop even the exception line. If you want to track how often they are happening, run a script that counts date exceptions per hour.
That sort of script typically costs a few minutes of programming in your favorite regex-capable script language.
I got a recommendation to use this syntax when logging in java:
if (logger.isLoggable(Log.FINE))
{
logger.fine("bla"+" bla"+" bla");
}
The reason for this is to avoid the redundant construction of the parameter string incase the logging level is lower than "FINE". (in the example above - 5 redundant string object. (" bla"X3, " bla bla" and "bla bla bla").
I'd like to hear what others are doing about this or if you think that this is necessary at all.
Thanks!!
Some newer logging frameworks allow you to specify arguments as parameters, and won't evaluate them if there's no logging.
The example I found is LogBack, the successor to Log4j. Here's the info: http://www.infoq.com/news/2007/08/logback
This gives you the best of both worlds, so to speak. Elegant syntax yet good performance.
Log4j code example:
if( logger.isDebugEnabled() ) {
logger.debug( "User with account " +
user.getAccount() + " failed authentication; " +
"supplied crypted password " + user.crypt(password) +
" does not match." );
}
Equivalent LogBack code:
logger.debug( "User with account {} failed authentication; " +
"supplied crypted password {} does not match.",
user.getAccount(), user.crypt(password) );
This defers the cost of message assembly until LOGBack has ascertained whether or not this message will be viewed. It doesn't defer the cost of retrieving expensive parameters, such as the password crypting in the above example.
String objects are immutable, and repeated concatenation is therefore an expensive operation. It requires repeated memory allocation, object creation and iteration. Considering that some logging calls at the finer log levels can be invoked thousands or millions of times per minute, it might be a considerable performance gain to do as you illustrate. Though, for a smaller application, it might not be worth the extra effort.
As a side note: You can save even more performance, where this is truly critical by using a constant such as this:
public static final boolean DEBUG = false;
If you now wrap the logging code in an if-block such as this, the JVM will be able to completely optimize away the debug calls when running in product mode. This is as close as you get to a C #ifdef.
if (Globals.DEBUG) {
// Logging call
}
Absolutely necessary for debug type logging. It something like 10x quicker to check the log level first than create the string and throw it away.
This is an improvement (good) but it can be improved on a little.
Set up final flags for each logging level (FINE, etc) in a global object used as config, then use a StringBuffer to build up your debugging output -- you can even format numbers into the stream at the same time.
public class MyAppConfig {
public final boolean FINE=true;
// ... other fields
}
public class MyApp {
void someFunction() {
...
int imagesProcessed;
imagesProcessed = processImages();
if (MyAppConfig.FINE) logger.fine(new StringBuffer(35).
append("Count of images processed: ").append(imagesProcessed).toString());
...
}
}
Here the string buffer is set up with an 'initial capacity' of 35 characters. If you know how many characters are going to be generated you can provide hints to StringBuffer.