We Keep Coding

How to add dynamic metadata to a singleton SLF4J logger in Java?

This is a Java question.
I have around 100 static functions that use a slf4j logger.
I want to add some metadata to standardise the logs - assume it's some kind of preamble that is a function of what processing is currently going on.
How can I get the logger to print that metadata without going in to each of the static functions and changing them to explicitly add in the metadata.
e.g.
static Logger logger; ...
void mainProcessing() {
String file = "current_file.txt";
int line = 3;
...
func1();
func2();
...
}
void func1() {
...
logger.warn("some warning");
}
I'd like to see "WARN: File current_file.txt, line 3, msg: some warning" in the logs.
Any ideas?
(Prefer not to have to change each of the func1() functions obviously, if possible)
Thanks in advance.

You need to specify print format. However beware, that obtaining line number and/or file will greatly decrease your application performance. This is not C++, Logback will probably create a Throwable object for each line, to retrieve the stacktrace and extract the line number and file name.
Regarding line:
L / line Outputs the line number from where the logging request was
issued.
Generating the line number information is not particularly fast. Thus,
its use should be avoided unless execution speed is not an issue.
Regarding file:
F / file Outputs the file name of the Java source file where the
logging request was issued.
Generating the file information is not particularly fast. Thus, its
use should be avoided unless execution speed is not an issue.
http://logback.qos.ch/manual/layouts.html#file
Sample logback.xml configuration would be:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="console" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{"yy-MM-dd HH:mm:ss,SSS"}: %-5p %F:%L [%t] %c{0}: %M - %m%n</pattern>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="console" />
</root>
</configuration>
Better to redesign your code, that messages are more verbose and not unique, containing the context and required data to debug it by a person that does not know the code very well (your future co-worker)

logback how to log only the classname not also the package path?

I have an appender in my logback xml config:
<encoder>
<pattern>%d{"yyyy-MM-dd HH:mm:ss"} [%thread] %-5level %logger{5} - %msg %n</pattern>
</encoder>
but it prints out the abbreviated package along with the class, even if I set it to %logger{1} like:
2019-12-19 10:26:16 [main] INFO o.f.d.d.u.Myclass - my message
I want it to just log the class MyClass like log4j does. How?

looks like the number you specify for %logger is some kind of special case and doesnt' exactly dictate size. Except it does dictate size, but never less than a minimum size which is the entire package path with the full class name. Unless you set it to "0" which is a special case to mean "don't include the package".
So if I set it to %logger{0} then I get just the class name, see http://logback.qos.ch/manual/layouts.html#conversionWord
2019-12-19 10:28:45 [main] INFO MyClass - my message
In log4j the equivalent would have been %c{1} or %logger{1} so I guess it's different.

log4j - StringMatchFilter still logging unmatched logs

Hope anyone could help me with this issue, so basically I am using log4j-1.2.17 and sorry, I can't update it to log4j2.
As the title suggests, I need to do the following,
I am trying to create an audit.log file which only logs information that has a certain string element (e.g., USER ACTION) and doesn't log anything else.
I have another operations.log that will log everything but not the one with String USER ACTION
My log4j.properties file looks something like this,
log4j.logger.com.ssl=INFO, Output
log4j.logger.com.ssl=INFO, Audit
log4j.appender.Output=org.apache.log4j.DailyRollingFileAppender
log4j.appender.Output.DatePattern='.'yyyy-MM-dd
log4j.appender.Output.File=/var/log/operational/operations.log
log4j.appender.Output.layout=org.apache.log4j.PatternLayout
log4j.appender.Output.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p [%t] %m (%c)%n
log4j.appender.Output.Threshold=TRACE
log4j.appender.Output.filter.1=org.apache.log4j.varia.StringMatchFilter
log4j.appender.Output.filter.1.StringToMatch=USER ACTION
log4j.appender.Output.filter.1.AcceptOnMatch=false
log4j.appender.Output.filter=org.apache.log4j.varia.DenyAllFilter
log4j.appender.Audit=org.apache.log4j.DailyRollingFileAppender
log4j.appender.Audit.DatePattern='.'yyyy-MM-dd
log4j.appender.Audit.File=/var/log/operational/audit.log
log4j.appender.Audit.Threshold=TRACE
log4j.appender.Audit.layout=org.apache.log4j.PatternLayout
log4j.appender.Audit.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p [%t] %m (%c)%n
log4j.appender.Audit.filter.1=org.apache.log4j.varia.StringMatchFilter
log4j.appender.Audit.filter.1.StringToMatch=USER ACTION
log4j.appender.Audit.filter.1.AcceptOnMatch=true
log4j.appender.Audit.filter=org.apache.log4j.varia.DenyAllFilter
Inside the Java Code, I have something like this,
private static final String AUDIT_LOG_PREFIX = "USER ACTION";
logger.info("Just testing if this appears before the one below it");
logger.info("[" + AUDIT_LOG_PREFIX + "] - User Details Updated by User : "+ r.getUserName());
The log output I get is as follows,
operational.log
2018-11-22 10:55:31 INFO [http-nio-8080-exec-179] Just testing if this appears before the one below it (com.ssl.operational.action.EditDetailsActionBean)
audit.log
2018-11-22 10:58:51 INFO [http-nio-8080-exec-192] Just testing if this appears before the one below it (com.ssl.operational.action.EditDetailsActionBean)
2018-11-22 10:58:51 INFO [http-nio-8080-exec-192] [USER ACTION] - User Details Updated by User : adminuser
So really hoping that audit.log doesn't display the first log entry and just displays the one with USER ACTION string.
Any help or advice will be greatly appreciated, Thank you

Log4j2 Rolling appender - fileName "sliding" according to pattern

I am looking for rollover strategy where current log (active output target in manual's terminology) file name is not fixed but specified by a pattern, or - more precisely - same pattern as in filePattern attribute.
I want to achieve daily rollover where today's log is, say, log-2015-05-05.log and on midnight framework just stops writing it and starts writing into log-2015-05-06.log. However, AFAIK, current configuration allows only
<RollingFile name="ROLFILE"
fileName="log.log"
filePattern="log-%d{yyyy-MM-dd}.log"
>
Specifying same value into fileName attribute doesn't work (leads to file with sensitive characters literally interpreted). I noticed no example or SO question with such a dynamic value of fileName. Note the fileName="log-${date:yyyy-MM-dd}.log" doesn't solve problem since expression is evaluated only at startup and events are still sent into file even if their timestamp doesn't match the expression.
I am migrating from Log4j 1.2 to Log4j 2.2. In old version, required behavior was possible using
<appender name="ROLFILE" class="org.apache.log4j.rolling.RollingFileAppender">
<rollingPolicy class="org.apache.log4j.rolling.TimeBasedRollingPolicy">
<param name="FileNamePattern" value="log-%d{yyyy-MM-dd}.log" />
</rollingPolicy>
...
I prefer to preserve current way since some log analyzing tools rely on it.
Is it possible in Log4j2?
Thanks.

Note sure if this works, but you can try using a double $$ in the date lookup. This allows the variable to be resolved at runtime.
<RollingFile name="ROLFILE"
fileName="log-$${date:yyyy-MM-dd}.log"
filePattern="oldlog-%d{yyyy-MM-dd}.log"
>
You may need to be careful to ensure that the active output target file name is different from the after-rollover file name. (I used 'oldlog' in the snippet above.)

Finally, I wrote my own rollover strategy which generates different set of rollover actions. Instead renaming active file the active file name is simply replaced inside RollingFileManager. Yes, it's ugly reflection hack and also appender must be initialized with fileName constant corresponding with current date and having same pattern, e.g.
<RollingFile name="ROLFILE"
fileName="log-${date:yyyy-MM-dd}.log"
filePattern="log-%d{yyyy-MM-dd}.log"
>
<SlidingFilenameRolloverStrategy />
...
yet for me this solution is worth doing it despite these small drawbacks.
(Initial fileName stays forever as a key in AbstractManager registry MAP even if in manager itself it has been changed - seems it doesn't mind, I also tried replacing manager in registry for new one but it's impossible to collect all parameters necessary for its construction.)
I hope this hack shouldn't have been so ugly if RollingFileManager API made it possible normal way. I got some hope seeing this javadoc but framework AFAIK never utilizes this field, let alone for mutating RollingFileAppender.

I think it would work just fine using:
fileName="log-${date:yyyy-MM-dd}.log"
filePattern="log-%d{yyyy-MM-dd}.log"
I use it with log4j2 version 2.5

This has been implemented in Log4j 2.8 (see issue LOG4J2-1101).
Currently it only works with a RollingFile appender by omitting the filename parameter and using a DirectWriteRolloverStrategy.
Also, this feature seems to have some issues with the TimeBasedTriggeringPolicy (the first rollover doesn't happen so every logfile is offset by one interval); CronTriggeringPolicy works properly.
Example config:
<RollingRandomAccessFile name="MyLogger"
filePattern="logs/application.%d{yyyy-MM-dd}.log">
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
<Policies>
<CronTriggeringPolicy schedule="0 * * * * ?" evaluateOnStartup="true"/>
</Policies>
<DirectWriteRolloverStrategy/>
</RollingRandomAccessFile>
Support for RollingRandomAccessFile appender is requested in issue LOG4J2-1878.
Edit: Changed to CronTriggeringPolicy policy after finding TimeBasedTriggeringPolicy had issues.

Using MDC in log4j to dynamically name the log file

Is it possible some how to use MDC to name the log file at run time.
I have a single web application which is being called by different names at the same time using tomcat docbase. So i need to have separate log files for each of them.

This can be accomplished in Logback, the successor to Log4J.
Logback is intended as a successor to the popular log4j project, picking up where log4j leaves off.
See the documentation for Sifting Appender
The SiftingAppender is unique in its capacity to reference and configure nested appenders. In the above example, within the SiftingAppender there will be nested FileAppender instances, each instance identified by the value associated with the "userid" MDC key. Whenever the "userid" MDC key is assigned a new value, a new FileAppender instance will be built from scratch. The SiftingAppender keeps track of the appenders it creates. Appenders unused for 30 minutes will be automatically closed and discarded.
In the example, they generate a separate log file for each user based on an MDC value.
Other MDC values could be used depending on your needs.

This is also possible with log4j. You can do this by implementing your own appender. I guess the easiest way is to
subclass AppenderSkeleton.
All logging events end up in the append(LoggingEvent event) method you have to implement.
In that method you can access the MDC by event.getMDC("nameOfTheKeyToLookFor");
Then you could use this information to open the file to write to.
It may be helpful to have a look at the implementation of the standard appenders like RollingFileAppender to figure out the rest.
I used this approach myself in an application to separate the logs of different threads into different log files and it worked very well.

I struggled for a while to find SiftingAppender-like functionality in log4j (we couldn't switch to logback because of some dependencies), and ended up with a programmatic solution that works pretty well, using an MDC and appending loggers at runtime:
// this can be any thread-specific string
String processID = request.getProcessID();
Logger logger = Logger.getRootLogger();
// append a new file logger if no logger exists for this tag
if(logger.getAppender(processID) == null){
try{
String pattern = "%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n";
String logfile = "log/"+processID+".log";
FileAppender fileAppender = new FileAppender(
new PatternLayout(pattern), logfile, true);
fileAppender.setName(processID);
// add a filter so we can ignore any logs from other threads
fileAppender.addFilter(new ProcessIDFilter(processID));
logger.addAppender(fileAppender);
}catch(Exception e){
throw new RuntimeException(e);
}
}
// tag all child threads with this process-id so we can separate out log output
MDC.put("process-id", processID);
//whatever you want to do in the thread
LOG.info("This message will only end up in "+processID+".log!");
MDC.remove("process-id");
The filter appended above just checks for a specific process id:
public class RunIdFilter extends Filter {
private final String runId;
public RunIdFilter(String runId) {
this.runId = runId;
}
#Override
public int decide(LoggingEvent event) {
Object mdc = event.getMDC("run-id");
if (runId.equals(mdc)) {
return Filter.ACCEPT;
}
return Filter.DENY;
}
}
Hope this helps a bit.

As of 20-01-2020, this is now a default functionality of Log4j.
To achieve that you just need to use a RoutingAppender with MDC.
Example:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN" monitorInterval="30">
<Appenders>
<Routing name="Analytics" ignoreExceptions="false">
<Routes>
<Script name="RoutingInit" language="JavaScript"><![CDATA[
// This script must return a route name
//
// Example from https://logging.apache.org/log4j/2.x/manual/appenders.html#RoutingAppender
// on how to get a MDC value
// logEvent.getContextMap().get("event_type");
//
// but as we use only one route with dynamic name, we return 1
1
]]>
</Script>
<Route>
<RollingFile
name="analytics-${ctx:event_type}"
fileName="logs/analytics/${ctx:event_type}.jsonl"
filePattern="logs/analytics/$${date:yyyy-MM}/analytics-${ctx:event_type}-%d{yyyy-dd-MM-}-%i.jsonl.gz">
<PatternLayout>
<pattern>%m%n</pattern>
</PatternLayout>
<Policies>
<TimeBasedTriggeringPolicy/>
<SizeBasedTriggeringPolicy size="250 MB"/>
</Policies>
</RollingFile>
</Route>
</Routes>
<!-- Created appender TTL -->
<IdlePurgePolicy timeToLive="15" timeUnit="minutes"/>
</Routing>
</Appenders>
<Loggers>
<Logger name="net.bytle.api.http.AnalyticsLogger" level="debug" additivity="false">
<AppenderRef ref="Analytics"/>
</Logger>
</Loggers>
</Configuration>
To known more, see Log4j - How to route message to log file created dynamically.