We Keep Coding

How to add dynamic metadata to a singleton SLF4J logger in Java?

This is a Java question.
I have around 100 static functions that use a slf4j logger.
I want to add some metadata to standardise the logs - assume it's some kind of preamble that is a function of what processing is currently going on.
How can I get the logger to print that metadata without going in to each of the static functions and changing them to explicitly add in the metadata.
e.g.
static Logger logger; ...
void mainProcessing() {
String file = "current_file.txt";
int line = 3;
...
func1();
func2();
...
}
void func1() {
...
logger.warn("some warning");
}
I'd like to see "WARN: File current_file.txt, line 3, msg: some warning" in the logs.
Any ideas?
(Prefer not to have to change each of the func1() functions obviously, if possible)
Thanks in advance.

You need to specify print format. However beware, that obtaining line number and/or file will greatly decrease your application performance. This is not C++, Logback will probably create a Throwable object for each line, to retrieve the stacktrace and extract the line number and file name.
Regarding line:
L / line Outputs the line number from where the logging request was
issued.
Generating the line number information is not particularly fast. Thus,
its use should be avoided unless execution speed is not an issue.
Regarding file:
F / file Outputs the file name of the Java source file where the
logging request was issued.
Generating the file information is not particularly fast. Thus, its
use should be avoided unless execution speed is not an issue.
http://logback.qos.ch/manual/layouts.html#file
Sample logback.xml configuration would be:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="console" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{"yy-MM-dd HH:mm:ss,SSS"}: %-5p %F:%L [%t] %c{0}: %M - %m%n</pattern>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="console" />
</root>
</configuration>
Better to redesign your code, that messages are more verbose and not unique, containing the context and required data to debug it by a person that does not know the code very well (your future co-worker)

Add empty line in slf4j logs

Do any one knows how to add a empty line in slf4j logs without its formatting.
To get empty log I have added empty string as log.
log.error("");
I need to get it with out formatting, Just empty line, Please help.

In your logback.xml, declare a "minimal" appender that doesn't embellish the log message with timestamps or log levels or anything like so:
<appender name="minimal" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%m%n</pattern>
</encoder>
</appender>
Then declare a logger that uses that appender like so:
<logger name="blank.line.logger" level="ALL" additivity="false">
<appender-ref ref="minimal"/>
</logger>
In your Java code where you'd like to produce empty log lines, you'll need a reference to that logger separate from whatever other logger you're using for regular log messages, something like
private static final Logger blankLineLogger = LoggerFactory.getLogger("blank.line.logger");
Then you can get your desired blank line in the log output with
blankLineLogger.info("");

The accepted answser is a bit tricky, for me I just use a simple log.error("\n") , or append a \n in the previous log line.

Log4j2 Rolling appender - fileName "sliding" according to pattern

I am looking for rollover strategy where current log (active output target in manual's terminology) file name is not fixed but specified by a pattern, or - more precisely - same pattern as in filePattern attribute.
I want to achieve daily rollover where today's log is, say, log-2015-05-05.log and on midnight framework just stops writing it and starts writing into log-2015-05-06.log. However, AFAIK, current configuration allows only
<RollingFile name="ROLFILE"
fileName="log.log"
filePattern="log-%d{yyyy-MM-dd}.log"
>
Specifying same value into fileName attribute doesn't work (leads to file with sensitive characters literally interpreted). I noticed no example or SO question with such a dynamic value of fileName. Note the fileName="log-${date:yyyy-MM-dd}.log" doesn't solve problem since expression is evaluated only at startup and events are still sent into file even if their timestamp doesn't match the expression.
I am migrating from Log4j 1.2 to Log4j 2.2. In old version, required behavior was possible using
<appender name="ROLFILE" class="org.apache.log4j.rolling.RollingFileAppender">
<rollingPolicy class="org.apache.log4j.rolling.TimeBasedRollingPolicy">
<param name="FileNamePattern" value="log-%d{yyyy-MM-dd}.log" />
</rollingPolicy>
...
I prefer to preserve current way since some log analyzing tools rely on it.
Is it possible in Log4j2?
Thanks.

Note sure if this works, but you can try using a double $$ in the date lookup. This allows the variable to be resolved at runtime.
<RollingFile name="ROLFILE"
fileName="log-$${date:yyyy-MM-dd}.log"
filePattern="oldlog-%d{yyyy-MM-dd}.log"
>
You may need to be careful to ensure that the active output target file name is different from the after-rollover file name. (I used 'oldlog' in the snippet above.)

Finally, I wrote my own rollover strategy which generates different set of rollover actions. Instead renaming active file the active file name is simply replaced inside RollingFileManager. Yes, it's ugly reflection hack and also appender must be initialized with fileName constant corresponding with current date and having same pattern, e.g.
<RollingFile name="ROLFILE"
fileName="log-${date:yyyy-MM-dd}.log"
filePattern="log-%d{yyyy-MM-dd}.log"
>
<SlidingFilenameRolloverStrategy />
...
yet for me this solution is worth doing it despite these small drawbacks.
(Initial fileName stays forever as a key in AbstractManager registry MAP even if in manager itself it has been changed - seems it doesn't mind, I also tried replacing manager in registry for new one but it's impossible to collect all parameters necessary for its construction.)
I hope this hack shouldn't have been so ugly if RollingFileManager API made it possible normal way. I got some hope seeing this javadoc but framework AFAIK never utilizes this field, let alone for mutating RollingFileAppender.

I think it would work just fine using:
fileName="log-${date:yyyy-MM-dd}.log"
filePattern="log-%d{yyyy-MM-dd}.log"
I use it with log4j2 version 2.5

This has been implemented in Log4j 2.8 (see issue LOG4J2-1101).
Currently it only works with a RollingFile appender by omitting the filename parameter and using a DirectWriteRolloverStrategy.
Also, this feature seems to have some issues with the TimeBasedTriggeringPolicy (the first rollover doesn't happen so every logfile is offset by one interval); CronTriggeringPolicy works properly.
Example config:
<RollingRandomAccessFile name="MyLogger"
filePattern="logs/application.%d{yyyy-MM-dd}.log">
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
<Policies>
<CronTriggeringPolicy schedule="0 * * * * ?" evaluateOnStartup="true"/>
</Policies>
<DirectWriteRolloverStrategy/>
</RollingRandomAccessFile>
Support for RollingRandomAccessFile appender is requested in issue LOG4J2-1878.
Edit: Changed to CronTriggeringPolicy policy after finding TimeBasedTriggeringPolicy had issues.

Log4J2 sync loggers faster than mixed async/sync loggers

Based on https://logging.apache.org/log4j/2.x/manual/async.html I want to use the Mixing Synchronous and Asynchronous Loggers approach in order to benefit from some performance improvement over the all synchronous loggers.
The benchmark code:
public static void main(String[] args) {
org.apache.logging.log4j.Logger log4j2Logger = org.apache.logging.log4j.LogManager
.getLogger("com.foo.Bar");
long start = System.currentTimeMillis();
int nbLogMessages = 1 * 1000 * 1000;
for (int i = 0; i < nbLogMessages; i++) {
log4j2Logger.info("Log Message");
}
org.apache.logging.log4j.core.Logger coreLogger = (org.apache.logging.log4j.core.Logger) log4j2Logger;
org.apache.logging.log4j.core.LoggerContext context = (org.apache.logging.log4j.core.LoggerContext) coreLogger
.getContext();
Map<String, org.apache.logging.log4j.core.Appender> appenders = context
.getConfiguration().getAppenders();
for (org.apache.logging.log4j.core.Appender appender : appenders
.values()) {
appender.stop();
}
long elapsed = System.currentTimeMillis() - start;
System.out.println("Elapsed " + elapsed + "ms "
+ (nbLogMessages * 1000 / elapsed) + "logs/sec.");
}
The Log4j2 configuration is exactly the one in the doc (https://logging.apache.org/log4j/2.x/manual/async.html):
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
<Appenders>
<RandomAccessFile name="RandomAccessFile" fileName="/tmp/asyncWithLocation.log" immediateFlush="false" append="false">
<PatternLayout>
<Pattern>%d %p %class{1.} [%t] %location %m %ex%n</Pattern>
</PatternLayout>
</RandomAccessFile>
</Appenders>
<Loggers>
<!-- pattern layout actually uses location, so we need to include it -->
<AsyncLogger name="com.foo.Bar" level="trace" includeLocation="true">
<AppenderRef ref="RandomAccessFile"/>
</AsyncLogger>
<Root level="info" includeLocation="true">
<AppenderRef ref="RandomAccessFile"/>
</Root>
</Loggers>
</Configuration>
With that mixed sync/async logger config I get around 33,400 logs/sec.
Now if I replace the AsyncLogger "com.foo.Bar" with a regular Logger I get around 35,300 logs/sec.
How come the synchronous logger strategy is faster, according to the graphs it should have a way higher throughput no?
I've tried various other things like "warming up" the JVM before doing the benchmark like they recommend but it didn't help.
Note that if I set the property Log4jContextSelector to "org.apache.logging.log4j.core.async.AsyncLoggerContextSelector" to activate the "all async" loggers, I get around 86,400 logs/seconds. Unfortunately I can't use that option for other reasons.
Using log4j2-2.3, disruptor-3.3.2.
OS is Ubuntu, 8 cores.

Before you start measuring, I would recommend that you log a smaller number of messages, say 100,000 or so, then sleep for one second before starting the measured loop. (You don't want to measure log4j's initialization or the JVM optimization here.)
What are your performance results if you measure until after the logging loop finishes (before trying to stop the appenders)?
Also try setting includeLocation="false" in your config. This should make a big difference.
I don't think you should manually stop the appenders (see my answer to your other question).

Using MDC in log4j to dynamically name the log file

Is it possible some how to use MDC to name the log file at run time.
I have a single web application which is being called by different names at the same time using tomcat docbase. So i need to have separate log files for each of them.

This can be accomplished in Logback, the successor to Log4J.
Logback is intended as a successor to the popular log4j project, picking up where log4j leaves off.
See the documentation for Sifting Appender
The SiftingAppender is unique in its capacity to reference and configure nested appenders. In the above example, within the SiftingAppender there will be nested FileAppender instances, each instance identified by the value associated with the "userid" MDC key. Whenever the "userid" MDC key is assigned a new value, a new FileAppender instance will be built from scratch. The SiftingAppender keeps track of the appenders it creates. Appenders unused for 30 minutes will be automatically closed and discarded.
In the example, they generate a separate log file for each user based on an MDC value.
Other MDC values could be used depending on your needs.

This is also possible with log4j. You can do this by implementing your own appender. I guess the easiest way is to
subclass AppenderSkeleton.
All logging events end up in the append(LoggingEvent event) method you have to implement.
In that method you can access the MDC by event.getMDC("nameOfTheKeyToLookFor");
Then you could use this information to open the file to write to.
It may be helpful to have a look at the implementation of the standard appenders like RollingFileAppender to figure out the rest.
I used this approach myself in an application to separate the logs of different threads into different log files and it worked very well.

I struggled for a while to find SiftingAppender-like functionality in log4j (we couldn't switch to logback because of some dependencies), and ended up with a programmatic solution that works pretty well, using an MDC and appending loggers at runtime:
// this can be any thread-specific string
String processID = request.getProcessID();
Logger logger = Logger.getRootLogger();
// append a new file logger if no logger exists for this tag
if(logger.getAppender(processID) == null){
try{
String pattern = "%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n";
String logfile = "log/"+processID+".log";
FileAppender fileAppender = new FileAppender(
new PatternLayout(pattern), logfile, true);
fileAppender.setName(processID);
// add a filter so we can ignore any logs from other threads
fileAppender.addFilter(new ProcessIDFilter(processID));
logger.addAppender(fileAppender);
}catch(Exception e){
throw new RuntimeException(e);
}
}
// tag all child threads with this process-id so we can separate out log output
MDC.put("process-id", processID);
//whatever you want to do in the thread
LOG.info("This message will only end up in "+processID+".log!");
MDC.remove("process-id");
The filter appended above just checks for a specific process id:
public class RunIdFilter extends Filter {
private final String runId;
public RunIdFilter(String runId) {
this.runId = runId;
}
#Override
public int decide(LoggingEvent event) {
Object mdc = event.getMDC("run-id");
if (runId.equals(mdc)) {
return Filter.ACCEPT;
}
return Filter.DENY;
}
}
Hope this helps a bit.

As of 20-01-2020, this is now a default functionality of Log4j.
To achieve that you just need to use a RoutingAppender with MDC.
Example:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN" monitorInterval="30">
<Appenders>
<Routing name="Analytics" ignoreExceptions="false">
<Routes>
<Script name="RoutingInit" language="JavaScript"><![CDATA[
// This script must return a route name
//
// Example from https://logging.apache.org/log4j/2.x/manual/appenders.html#RoutingAppender
// on how to get a MDC value
// logEvent.getContextMap().get("event_type");
//
// but as we use only one route with dynamic name, we return 1
1
]]>
</Script>
<Route>
<RollingFile
name="analytics-${ctx:event_type}"
fileName="logs/analytics/${ctx:event_type}.jsonl"
filePattern="logs/analytics/$${date:yyyy-MM}/analytics-${ctx:event_type}-%d{yyyy-dd-MM-}-%i.jsonl.gz">
<PatternLayout>
<pattern>%m%n</pattern>
</PatternLayout>
<Policies>
<TimeBasedTriggeringPolicy/>
<SizeBasedTriggeringPolicy size="250 MB"/>
</Policies>
</RollingFile>
</Route>
</Routes>
<!-- Created appender TTL -->
<IdlePurgePolicy timeToLive="15" timeUnit="minutes"/>
</Routing>
</Appenders>
<Loggers>
<Logger name="net.bytle.api.http.AnalyticsLogger" level="debug" additivity="false">
<AppenderRef ref="Analytics"/>
</Logger>
</Loggers>
</Configuration>
To known more, see Log4j - How to route message to log file created dynamically.