We Keep Coding

How to add a PatternLayout to the root logger at runtime?

I am using logback as the backend for Slf4j. Currently, I configure the logger using a logback.xml file. My issue is that sensitive information is being logged (outside of my control) and I want to mask this sensitive information. To mask the information, I have wrote a custom PatternLayout class that essentially does:
#Override
public String doLayout(ILoggingEvent event) {
String message = super.doLayout(event);
Matcher matcher = sesnsitiveInfoPattern.matcher(message);
if (matcher.find()) {
message = matcher.replaceAll("XXX");
}
return message;
}
My issue is that I need to tell logback to use this custom pattern layout. I don't want to add this to the XML configuration however. My current configuration looks like this:
<configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<layout class="com.my.MaskingPatternLayout"> <!-- here -->
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
</layout>
</encoder>
</appender>
<root level="info">
<appender-ref ref="STDOUT" />
</root>
</configuration>
In XML, my desired configuration would look like this (but I don't want to use XML):

Hello Max I hope you are using Log4j 2.x because this solution uses the plugins approache introduced in log4j 2.x . first you should create a package where you are going to put your plugins classes and you put there these two classes :
my.log4j.pluggins.CustomConfigurationFactory :
#Plugin(name = "CustomConfigurationFactory", category = ConfigurationFactory.CATEGORY)
#Order(value = 0)
public class CustomConfigurationFactory extends ConfigurationFactory {
private Configuration createConfiguration(final String name,
ConfigurationBuilder<BuiltConfiguration> builder) {
System.out.println("init logger");
builder.setConfigurationName(name);
builder.setStatusLevel(Level.INFO);
builder.setPackages("my.log4j.pluggins");
AppenderComponentBuilder appenderBuilder = builder.newAppender(
"Stdout", "CONSOLE").addAttribute("target",
ConsoleAppender.Target.SYSTEM_OUT);
appenderBuilder
.add(builder
.newLayout("PatternLayout")
.addAttribute("pattern", "%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %myMsg%n"));
builder.add(appenderBuilder);
builder.add(builder.newRootLogger(Level.TRACE).add(
builder.newAppenderRef("Stdout")));
return builder.build();
}
#Override
protected String[] getSupportedTypes() {
String[] supportedExt = { "*" };
return supportedExt;
}
#Override
public Configuration getConfiguration(ConfigurationSource source) {
ConfigurationBuilder<BuiltConfiguration> builder = newConfigurationBuilder();
return createConfiguration(source.toString(), builder);
}
#Override
public Configuration getConfiguration(String name, URI configLocation) {
ConfigurationBuilder<BuiltConfiguration> builder = newConfigurationBuilder();
return createConfiguration(name, builder);
}
}
my.log4j.pluggins.SampleLayout :
#Plugin(name = "CustomConverter", category = "Converter")
#ConverterKeys({"myMsg"})
public class SampleLayout extends LogEventPatternConverter {
protected SampleLayout(String name, String style) {
super(name, style);
}
public static SampleLayout newInstance(){
return new SampleLayout("custConv", "custConv");
}
#Override
public void format(LogEvent event, StringBuilder stringBuilder) {
//replace the %myMsg by XXXXX if sensitive
if (sensitive()){
stringBuilder.append("XXXX");}
else {
stringBuilder.append(event.getMessage().getFormattedMessage());}
}
}
the CustomConfiguration class is responsable for creating the configuration of log4j and the line 9 where 'builder.setPackages("my.log4j.pluggins")' is important in order to scan that package and pick up the converter pluggin wich is SampleLayout.
the second class will be responsible for formatting the new key '%myMsg' in the pattern that contains my sensitive message, this Converter class checks if that message is sensitive and actes accordingly.
Before you start logging you should configure your log4j like this
ConfigurationFactory.setConfigurationFactory(new CustomConfigurationFactory());

Logback: gracefully split messages at 64k

We have a log collecting service that automatically splits messages at 64KB, but the split is not elegant at all. We are printing individual log messages as json blobs with some additional metadata. Sometimes these include large stack traces that we want to preserve in full.
So I was looking into writing a custom logger or appender wrapper that would take the message and split it into smaller chunks and re-log it but that's looking non-trivial.
Is there a simple way to configure logback to split up its messages into multiple separate messages if the size of the message is greater than some value?
Here is the appender configuration:
<!-- Sumo optimized rolling log file -->
<appender name="file" class="ch.qos.logback.core.rolling.RollingFileAppender">
<Append>true</Append>
<file>${log.dir}/${service.name}-sumo.log</file>
<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
<providers>
<timestamp>
<fieldName>t</fieldName>
<pattern>yyyy-MM-dd'T'HH:mm:ss.SSS'Z'</pattern>
<timeZone>UTC</timeZone>
</timestamp>
<message/>
<loggerName/>
<threadName/>
<logLevel/>
<stackTrace>
<if condition='isDefined("throwable.converter")'>
<then>
<throwableConverter class="${throwable.converter}"/>
</then>
</if>
</stackTrace>
<mdc/>
<tags/>
</providers>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<maxIndex>1</maxIndex>
<FileNamePattern>${log.dir}/${service.name}-sumo.log.%i</FileNamePattern>
</rollingPolicy>
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<MaxFileSize>256MB</MaxFileSize>
</triggeringPolicy>
</appender>
<appender name="sumo" class="ch.qos.logback.classic.AsyncAppender">
<queueSize>500</queueSize>
<discardingThreshold>0</discardingThreshold>
<appender-ref ref="file" />
</appender>

The solution I came up with is simply to wrap my logger in something that splits up messages nicely. Note that I'm primarily interested in splitting messages with Throwables since those are what causes long messages.
Written with lambdas for Java 8
Also note this code is not fully tested, I'll update it if I find any bugs.
public class MessageSplittingLogger extends MarkerIgnoringBase {
//Target size is 64k for split. UTF-8 nominally has 1 byte characters, but some characters will use > 1 byte so leave some wiggle room
//Also leave room for additional messages
private static final int MAX_CHARS_BEFORE_SPLIT = 56000;
private static final String ENCODING = "UTF-8";
private Logger LOGGER;
public MessageSplittingLogger(Class<?> clazz) {
this.LOGGER = LoggerFactory.getLogger(clazz);
}
private void splitMessageAndLog(String msg, Throwable t, Consumer<String> logLambda) {
String combinedMsg = msg + (t != null ? "\nStack Trace:\n" + printStackTraceToString(t) : "");
int totalMessages = combinedMsg.length() / MAX_CHARS_BEFORE_SPLIT;
if(combinedMsg.length() % MAX_CHARS_BEFORE_SPLIT > 0){
totalMessages++;
}
int index = 0;
int msgNumber = 1;
while (index < combinedMsg.length()) {
String messageNumber = totalMessages > 1 ? "(" + msgNumber++ + " of " + totalMessages + ")\n" : "";
logLambda.accept(messageNumber + combinedMsg.substring(index, Math.min(index + MAX_CHARS_BEFORE_SPLIT, combinedMsg.length())));
index += MAX_CHARS_BEFORE_SPLIT;
}
}
/**
* Get the stack trace as a String
*/
private String printStackTraceToString(Throwable t) {
try {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
PrintStream ps = new PrintStream(baos, true, ENCODING);
t.printStackTrace(ps);
return baos.toString(ENCODING);
} catch (UnsupportedEncodingException e) {
return "Exception printing stack trace: " + e.getMessage();
}
}
#Override
public String getName() {
return LOGGER.getName();
}
#Override
public boolean isTraceEnabled() {
return LOGGER.isTraceEnabled();
}
#Override
public void trace(String msg) {
LOGGER.trace(msg);
}
#Override
public void trace(String format, Object arg) {
LOGGER.trace(format, arg);
}
#Override
public void trace(String format, Object arg1, Object arg2) {
LOGGER.trace(format, arg1, arg2);
}
#Override
public void trace(String format, Object... arguments) {
LOGGER.trace(format, arguments);
}
#Override
public void trace(String msg, Throwable t) {
splitMessageAndLog(msg, t, LOGGER::trace);
}
//... Similarly wrap calls to debug/info/error
}

Logback - Logging multiple levels to one file

I need to create a file that only logs TRACE and DEBUG events, but I'm unable to make it work (don't know if it's even possible).
Already tried searching for an option to invert the ThresholdFilter or a way to specify multiple levels on a LevelFilter, but with no success.
The only way that I'm thinking to make this work is to create 2 appende, exactly the same from one to another, but with the LevelFilter in one specified to TRACE and the other to DEBUG.
Is there any other way to accomplish this? Because I'm not a big fan of duplicating code/configuration.

Simple approach might be to create custom filter as below.
public class CustomFilter extends Filter<ILoggingEvent> {
private String levels;
public String getLevels() {
return levels;
}
public void setLevels(String levels) {
this.levels = levels;
}
private Level[] level;
#Override
public FilterReply decide(ILoggingEvent arg0) {
if (level == null && levels != null) {
setLevels();
}
if (level != null) {
for (Level lev : level) {
if (lev == arg0.getLevel()) {
return FilterReply.ACCEPT;
}
}
}
return FilterReply.DENY;
}
private void setLevels() {
if (!levels.isEmpty()) {
level = new Level[levels.split("\\|").length];
int i = 0;
for (String str : levels.split("\\|")) {
level[i] = Level.valueOf(str);
i++;
}
}
}
}
Add filter in logback.xml
<configuration>
<appender name="fileAppender1" class="ch.qos.logback.core.FileAppender">
<filter class="com.swasthik.kp.logback.filters.CustomFilter">
<levels>TRACE|DEBUG</levels>
</filter>
<file>c:/logs/kplogback.log</file>
<append>true</append>
<encoder>
<pattern>%d [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>
</appender>
<root level="TRACE">
<appender-ref ref="fileAppender1" />
</root>
</configuration>

Redirect Log4j to specific file

In a batch application that read and parse multiple files, the specifications ask me to output logs for each file separately.
How can I do this?
Example:
for(File f : allFiles) {
//TODO after this line all log should be output to "<f.getName()>.log"
LOGGER.debug('Start processing '+f.getName());
// process the file by calling librairies (JPA, spring, whatever ...)
LOGGER.debug('End processing '+f.getName());
}
So that, if I have 3 files to process, in the end, I want to have 3 log files.
What I have done so far is the following class.
import org.apache.log4j.ConsoleAppender;
import org.apache.log4j.FileAppender;
import org.apache.log4j.Level;
import org.apache.log4j.Logger;
import org.apache.log4j.PatternLayout;
public final class LoggerHelper {
/**
* Functional logger
*/
private static final Logger LOGGER = Logger.getLogger("BATCH_LOGGER");
/**
* Pattern for the layout of the logger
*/
private static final String PATTERN_LAYOUT = "%d{yyyy-MM-dd [HH:mm:ss]} %m%n";
/**
* Constructor
*/
private LoggerHelper() {
}
/**
* Initialize the loggers
*
* #param filename
* the name of the file where the logs will be written
* #throws IOException
* if a problem occur when instantiate a file appender
*/
public static void initLoggers(String filename) throws IOException {
// change functional appender
LOGGER.removeAllAppenders();
LOGGER.addAppender(new FileAppender(new PatternLayout(PATTERN_LAYOUT), filename));
LOGGER.setLevel(Level.DEBUG);
}
/**
* Get the batch logger
*
* #return the batch Logger
*/
public static Logger getLogger() {
return LOGGER;
}
}
But I have to replace all LOGGER calls with LoggerHelper.getLogger().debug(...).
And with this solution, I can't log frameworks logs.
for(File f : allFiles) {
//TODO after this line all log should be output to "<f.getName()>.log"
LoggerHelper.initLoggers(f.getName());
LoggerHelper.getLogger().debug('Start processing '+f.getName());
// process the file by calling librairies (JPA, spring, whatever ...)
LoggerHelper.getLogger().debug('End processing '+f.getName());
}
How can I do this?

You are already on a good track. I guess your misstake is to create new loggers. The solution might be to add different appenders to the same logger. So your logger helper just have to replace the appender (as you already did at your code):
private static final class LoggerHelper {
private static final String PATTERN_LAYOUT = "%d{yyyy-MM-dd [HH:mm:ss]} %m%n";
private static final Layout LAYOUT = new PatternLayout(PATTERN_LAYOUT);
public static final void setFileOutputOfLogger(Logger log, String fileName) throws IOException {
log.removeAllAppenders();
log.addAppender(new FileAppender(LAYOUT, fileName));
}
}
That is something you can call once within your loop.
Logger log = Logger.getLogger(FileStuff.class);
for(File f : allFiles) {
LoggerHelper.setFileOutputOfLogger(log, f.getName());
All the framework output will not be touched.

That's the solution I finally implemented.
I share it here, if this can help others...
First, the helper class that reload the log4j configuration.
Note that it (re)set some System properties. Those properties will be used in the log4j file directly.
import org.apache.log4j.xml.DOMConfigurator;
import org.springframework.core.io.ClassPathResource;
import org.springframework.util.Assert;
import org.springframework.util.StringUtils;
public final class LogHelper {
private final static String LOG4J_XML_FILENAME = "log4j.xml";
private final static String LOG_APPLI_DIRECTORY = "LOG_APPLI_DIRECTORY";
private final static String FILENAME = "FILENAME";
public static void initLogsForCurrentFile(String currentFile, String logDir) {
Assert.hasLength(currentFile);
Assert.doesNotContain(currentFile, File.pathSeparator);
ClassPathResource log4jxml = new ClassPathResource(LOG4J_XML_FILENAME);
if (!log4jxml.exists()) {
throw new IllegalArgumentException(
"The [log4j.xml] configuration file has not been found on the classpath.");
}
// TODO Define variables that could be used inside the log4j
// configuration file
System.setProperty(FILENAME, FileUtils.removeExtension(currentFile));
System.setProperty(LOG_APPLI_DIRECTORY, logDir);
// Reload the log4j configuration
try {
DOMConfigurator.configure(log4jxml.getURL());
} catch (Exception e) {
throw new IllegalArgumentException(
"A problem occured while loading the log4j configuration.",
e);
}
}
}
And the corresponding log4j file:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
<!-- This log4j file will be reloaded multiple times -->
<!-- so that each files processed by the applicatin will have their own log file -->
<!-- ${LOG_APPLI_DIRECTORY} = the log directory -->
<!-- ${FILENAME} = the basename of the current file processed by the batch -->
<appender name="batch-appender" class="org.apache.log4j.RollingFileAppender">
<param name="file"
value="${LOG_APPLI_DIRECTORY}/batch-${FILENAME}.log" />
<param name="MaxFileSize" value="1MB" />
<param name="MaxBackupIndex" value="3" />
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{ISO8601} %-5p %-40.40c{1} - %m%n" />
</layout>
</appender>
<!-- ================ -->
<!-- Root logger -->
<!-- ================ -->
<root>
<priority value="info" />
<appender-ref ref="batch-appender" />
</root>
</log4j:configuration>
With such solution, we stay as close as possible to what we usually do to configure log4j.
And moreover this solution keeps the configuration in the log4j file with no configuration in the java source code.

how to create a log4j custom appender and control the filename

My company uses a software package that reads in log files from our servers, parses them, and spits performance data into a database. We dont have access / permission to modify the source code for the app that reads the files but we do have access to the code that writes the files. I need to change the way the log files are being written and I would like to use log4j (so I can use an AsyncAppender). The program expects a few things:
1). There should be 10 log files that roll and each log file will be one day of logs. The files need to be named 0 through 9 and I need to be able to programatically set the file name and when they roll based on the server time.
2). Essentially when generating the 11th log file it should delete the oldest one and start writing to that one.
3). When a new log file is generated I need to be able to insert a timestamp as the first line of the file (System.currentTimeMillis()).
Is it possible to meet the above requirements with a custom log4j file appender? Ive looked at DailyRollingFileAppender but cant seem to figure out how to control the file names exactly like I need to. Also I cant seem to figure out how to write the first line in the log when it is generated (for example is there some callback function I can register when a new log file gets rolled in)?

I think you can achieve first 2 with
using RollingFileAppender and specifying FixedWindowRollingPolicy for RollingPolicy
as for the #3 you can always write your own handler

For the sake of posterity. I used the below class as my custom rolling policy
import org.apache.log4j.rolling.RollingPolicyBase;
import org.apache.log4j.rolling.RolloverDescription;
import org.apache.log4j.rolling.RolloverDescriptionImpl;
import org.apache.log4j.rolling.TriggeringPolicy;
import org.apache.log4j.Appender;
import org.apache.log4j.spi.LoggingEvent;
public final class CustomRollingPolicy extends RollingPolicyBase
implements TriggeringPolicy
{
private short curFileId = -1;
private String lastFileName = null;
static private final long FILETIMEINTERVAL = 86400000l;
static private final int NUM_FILES = 10;//86400000l;
public String folderName = "";
public String getFolderName() {
return folderName;
}
public void setFolderName(String folderName) {
this.folderName = folderName;
}
private short calculateID(long startTime) {
return (short) ((startTime / FILETIMEINTERVAL) % NUM_FILES);
}
public String getCurrentFileName()
{
StringBuffer buf = new StringBuffer();
buf.append(folderName);
buf.append(calculateID(System.currentTimeMillis()));
return buf.toString();
}
public void activateOptions()
{
super.activateOptions();
this.lastFileName = getCurrentFileName();
}
public RolloverDescription initialize(String currentActiveFile, boolean append)
{
curFileId = this.calculateID(System.currentTimeMillis());
lastFileName = getCurrentFileName();
String fileToUse = activeFileName != null? activeFileName: currentActiveFile != null?currentActiveFile:lastFileName;
return new RolloverDescriptionImpl(fileToUse, append, null, null);
}
public RolloverDescription rollover(String currentActiveFile)
{
curFileId = this.calculateID(System.currentTimeMillis());
String newFileName = getCurrentFileName();
if (newFileName.equals(this.lastFileName))
{
return null;
}
String lastBaseName = this.lastFileName;
String nextActiveFile = newFileName;
if (!currentActiveFile.equals(lastBaseName))
{
nextActiveFile = currentActiveFile;
}
this.lastFileName = newFileName;
return new RolloverDescriptionImpl(nextActiveFile, false, null, null);
}
public boolean isTriggeringEvent(Appender appender, LoggingEvent event, String filename, long fileLength)
{
short fileIdForCurrentServerTime = this.calculateID(System.currentTimeMillis());
return curFileId != fileIdForCurrentServerTime;
}
}
And here is the appender config in my log4j xml file:
<!-- ROLLING FILE APPENDER FOR RUM LOGS -->
<appender name="rumRollingFileAppender" class="org.apache.log4j.rolling.RollingFileAppender">
<rollingPolicy class="com.ntrs.wpa.util.CustomRollingPolicy">
<param name="folderName" value="C:/bea-portal-10.3.2/logs/"/>
<param name="FileNamePattern" value="C:/bea-portal-10.3.2/logs/foo.%d{yyyy-MM}.gz"/>
</rollingPolicy>
<layout class="com.ntrs.els.log4j.AppServerPatternLayout">
<param name="ConversionPattern" value="%m%n" />
</layout>
</appender>