I am currently struggling with masking the data available in the logs intercepted at the SOAP client. I have taken the approach to writing customized PatternLayout:
public class PatternMaskingLayout extends ch.qos.logback.classic.PatternLayout {
private Pattern multilinePattern;
private final List<String> maskPatterns = new ArrayList<>();
public void addMaskPattern(String maskPattern) {
maskPatterns.add(maskPattern);
multilinePattern = Pattern.compile(
String.join("|", maskPatterns),
Pattern.MULTILINE
);
}
#Override
public String doLayout(ILoggingEvent event) {
return maskMessage(super.doLayout(event)); // calling superclass method is required
}
private String maskMessage(String message) {
if (multilinePattern == null) {
return message;
}
StringBuilder sb = new StringBuilder(message);
Matcher matcher = multilinePattern.matcher(sb);
while (matcher.find()) {
IntStream.rangeClosed(1, matcher.groupCount()).forEach(group -> {
if (matcher.group(group) != null) {
IntStream.range(matcher.start(group), matcher.end(group))
.forEach(i -> sb.setCharAt(i, '*')); // replace each character with asterisk
}
});
}
return sb.toString();
}
}
My logback-spring.xml appenders looks like:
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<layout class="app.example.monitoring.tracing.PatternMaskingLayout">
<maskPattern>\"username\"\s*:\s*\"(.*?)\"</maskPattern>
<pattern>
${LOGBACK_LOGGING_PATTERN:-%d{yyyy-MM-dd HH:mm:ss.SSS} ${LOG_LEVEL_PATTERN:-%5p} ${PID:- } --- [%15.15t] %logger{36} : %msg %replace(%ex){'\n','\\u000a'}%nopex%n}
</pattern>
</layout>
</appender>
I still can not get my username masked. The XML field looks like <xa2:username>John</xa2:username>|
Have anyone have some experience with this?
Related
I have added a custom filter class to my spring boot project and configured it in the logback.xml file. I am getting the below error. please let me know how to resolve the below error.
Exception in thread "main" java.lang.IllegalStateException: Logback configuration error detected:
ERROR in ch.qos.logback.core.joran.spi.Interpreter#14:11 - no applicable action for [level], current ElementPath is [[configuration][appender][filter][level]]
ERROR in ch.qos.logback.core.joran.spi.Interpreter#15:12 - no applicable action for [append], current ElementPath is [[configuration][appender][filter][append]]
Sample code: logback.xml
<appender name="RollingAppender" class="ch.qos.logback.core.rolling.RollingFileAppender">
<param name="MaxFileSize" value="50KB" />
<filter class="com.filter.CustomFilter">
<level>INFO|DEBUG|ERROR</level>
<append>true</append>
</filter>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<pattern>[%d{yyyy-MM-dd'T'HH:mm:ss.sss'Z'}] [%C] [%M] [%L] [%t] [%-5p] %replace(%m){"Bearer.*'", "Bearerxxx'"}%n</pattern>
</encoder>
</rollingPolicy>
</appender>
Java Code:
#Component
public class CustomFilter extends Filter<ILoggingEvent> {
private String levels;
public String getLevels() {
return levels;
}
public void setLevels(String levels) {
this.levels = levels;
}
private Level[] level;
#Override
public FilterReply decide(ILoggingEvent loginEvent) {
if (level == null && levels != null) {
setLevels();
}
if (level != null) {
for (Level lev : level) {
if (lev == loginEvent.getLevel()) {
return FilterReply.ACCEPT;
}
}
}
return FilterReply.DENY;
}
private void setLevels() {
if (!levels.isEmpty()) {
level = new Level[levels.split("\\|").length];
int i = 0;
for (String str : levels.split("\\|")) {
level[i] = Level.valueOf(str);
i++;
}
}
}
}
I am using logback as the backend for Slf4j. Currently, I configure the logger using a logback.xml file. My issue is that sensitive information is being logged (outside of my control) and I want to mask this sensitive information. To mask the information, I have wrote a custom PatternLayout class that essentially does:
#Override
public String doLayout(ILoggingEvent event) {
String message = super.doLayout(event);
Matcher matcher = sesnsitiveInfoPattern.matcher(message);
if (matcher.find()) {
message = matcher.replaceAll("XXX");
}
return message;
}
My issue is that I need to tell logback to use this custom pattern layout. I don't want to add this to the XML configuration however. My current configuration looks like this:
<configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<layout class="com.my.MaskingPatternLayout"> <!-- here -->
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
</layout>
</encoder>
</appender>
<root level="info">
<appender-ref ref="STDOUT" />
</root>
</configuration>
In XML, my desired configuration would look like this (but I don't want to use XML):
Hello Max I hope you are using Log4j 2.x because this solution uses the plugins approache introduced in log4j 2.x . first you should create a package where you are going to put your plugins classes and you put there these two classes :
my.log4j.pluggins.CustomConfigurationFactory :
#Plugin(name = "CustomConfigurationFactory", category = ConfigurationFactory.CATEGORY)
#Order(value = 0)
public class CustomConfigurationFactory extends ConfigurationFactory {
private Configuration createConfiguration(final String name,
ConfigurationBuilder<BuiltConfiguration> builder) {
System.out.println("init logger");
builder.setConfigurationName(name);
builder.setStatusLevel(Level.INFO);
builder.setPackages("my.log4j.pluggins");
AppenderComponentBuilder appenderBuilder = builder.newAppender(
"Stdout", "CONSOLE").addAttribute("target",
ConsoleAppender.Target.SYSTEM_OUT);
appenderBuilder
.add(builder
.newLayout("PatternLayout")
.addAttribute("pattern", "%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %myMsg%n"));
builder.add(appenderBuilder);
builder.add(builder.newRootLogger(Level.TRACE).add(
builder.newAppenderRef("Stdout")));
return builder.build();
}
#Override
protected String[] getSupportedTypes() {
String[] supportedExt = { "*" };
return supportedExt;
}
#Override
public Configuration getConfiguration(ConfigurationSource source) {
ConfigurationBuilder<BuiltConfiguration> builder = newConfigurationBuilder();
return createConfiguration(source.toString(), builder);
}
#Override
public Configuration getConfiguration(String name, URI configLocation) {
ConfigurationBuilder<BuiltConfiguration> builder = newConfigurationBuilder();
return createConfiguration(name, builder);
}
}
my.log4j.pluggins.SampleLayout :
#Plugin(name = "CustomConverter", category = "Converter")
#ConverterKeys({"myMsg"})
public class SampleLayout extends LogEventPatternConverter {
protected SampleLayout(String name, String style) {
super(name, style);
}
public static SampleLayout newInstance(){
return new SampleLayout("custConv", "custConv");
}
#Override
public void format(LogEvent event, StringBuilder stringBuilder) {
//replace the %myMsg by XXXXX if sensitive
if (sensitive()){
stringBuilder.append("XXXX");}
else {
stringBuilder.append(event.getMessage().getFormattedMessage());}
}
}
the CustomConfiguration class is responsable for creating the configuration of log4j and the line 9 where 'builder.setPackages("my.log4j.pluggins")' is important in order to scan that package and pick up the converter pluggin wich is SampleLayout.
the second class will be responsible for formatting the new key '%myMsg' in the pattern that contains my sensitive message, this Converter class checks if that message is sensitive and actes accordingly.
Before you start logging you should configure your log4j like this
ConfigurationFactory.setConfigurationFactory(new CustomConfigurationFactory());
We have a log collecting service that automatically splits messages at 64KB, but the split is not elegant at all. We are printing individual log messages as json blobs with some additional metadata. Sometimes these include large stack traces that we want to preserve in full.
So I was looking into writing a custom logger or appender wrapper that would take the message and split it into smaller chunks and re-log it but that's looking non-trivial.
Is there a simple way to configure logback to split up its messages into multiple separate messages if the size of the message is greater than some value?
Here is the appender configuration:
<!-- Sumo optimized rolling log file -->
<appender name="file" class="ch.qos.logback.core.rolling.RollingFileAppender">
<Append>true</Append>
<file>${log.dir}/${service.name}-sumo.log</file>
<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
<providers>
<timestamp>
<fieldName>t</fieldName>
<pattern>yyyy-MM-dd'T'HH:mm:ss.SSS'Z'</pattern>
<timeZone>UTC</timeZone>
</timestamp>
<message/>
<loggerName/>
<threadName/>
<logLevel/>
<stackTrace>
<if condition='isDefined("throwable.converter")'>
<then>
<throwableConverter class="${throwable.converter}"/>
</then>
</if>
</stackTrace>
<mdc/>
<tags/>
</providers>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<maxIndex>1</maxIndex>
<FileNamePattern>${log.dir}/${service.name}-sumo.log.%i</FileNamePattern>
</rollingPolicy>
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<MaxFileSize>256MB</MaxFileSize>
</triggeringPolicy>
</appender>
<appender name="sumo" class="ch.qos.logback.classic.AsyncAppender">
<queueSize>500</queueSize>
<discardingThreshold>0</discardingThreshold>
<appender-ref ref="file" />
</appender>
The solution I came up with is simply to wrap my logger in something that splits up messages nicely. Note that I'm primarily interested in splitting messages with Throwables since those are what causes long messages.
Written with lambdas for Java 8
Also note this code is not fully tested, I'll update it if I find any bugs.
public class MessageSplittingLogger extends MarkerIgnoringBase {
//Target size is 64k for split. UTF-8 nominally has 1 byte characters, but some characters will use > 1 byte so leave some wiggle room
//Also leave room for additional messages
private static final int MAX_CHARS_BEFORE_SPLIT = 56000;
private static final String ENCODING = "UTF-8";
private Logger LOGGER;
public MessageSplittingLogger(Class<?> clazz) {
this.LOGGER = LoggerFactory.getLogger(clazz);
}
private void splitMessageAndLog(String msg, Throwable t, Consumer<String> logLambda) {
String combinedMsg = msg + (t != null ? "\nStack Trace:\n" + printStackTraceToString(t) : "");
int totalMessages = combinedMsg.length() / MAX_CHARS_BEFORE_SPLIT;
if(combinedMsg.length() % MAX_CHARS_BEFORE_SPLIT > 0){
totalMessages++;
}
int index = 0;
int msgNumber = 1;
while (index < combinedMsg.length()) {
String messageNumber = totalMessages > 1 ? "(" + msgNumber++ + " of " + totalMessages + ")\n" : "";
logLambda.accept(messageNumber + combinedMsg.substring(index, Math.min(index + MAX_CHARS_BEFORE_SPLIT, combinedMsg.length())));
index += MAX_CHARS_BEFORE_SPLIT;
}
}
/**
* Get the stack trace as a String
*/
private String printStackTraceToString(Throwable t) {
try {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
PrintStream ps = new PrintStream(baos, true, ENCODING);
t.printStackTrace(ps);
return baos.toString(ENCODING);
} catch (UnsupportedEncodingException e) {
return "Exception printing stack trace: " + e.getMessage();
}
}
#Override
public String getName() {
return LOGGER.getName();
}
#Override
public boolean isTraceEnabled() {
return LOGGER.isTraceEnabled();
}
#Override
public void trace(String msg) {
LOGGER.trace(msg);
}
#Override
public void trace(String format, Object arg) {
LOGGER.trace(format, arg);
}
#Override
public void trace(String format, Object arg1, Object arg2) {
LOGGER.trace(format, arg1, arg2);
}
#Override
public void trace(String format, Object... arguments) {
LOGGER.trace(format, arguments);
}
#Override
public void trace(String msg, Throwable t) {
splitMessageAndLog(msg, t, LOGGER::trace);
}
//... Similarly wrap calls to debug/info/error
}
I am migrating my application from log4j to log4j2 API. While migration, I found custom patternlayouts, patternparsers and patternconverters are used. I am not aware of how to implement these changes using log4j2 plugins. Can anyone help me on how to convert this custom layout TestPatternLayout to log4j2. Many thanks.
PFB the complete details on how custom pattern layout is implemented using log4j.
TestPatternLayout:
public class TestPatternLayout extends PatternLayout {
#Override
protected PatternParser createPatternParser(String pattern) {
return new TestPatternParser(pattern);
}
}
TestPatternParser:
public class TestPatternParser extends PatternParser {
private static final char Test_CHAR = 'e';
private static final char DATETIME_CHAR = 'd';
public TestPatternParser(String pattern) {
super(pattern);
}
#Override
protected void finalizeConverter(char c) {
switch (c) {
case Test_CHAR:
currentLiteral.setLength(0);
addConverter(new TestPatternConverter());
break;
default:
super.finalizeConverter(c);
}
}
}
TestPatternConverter:
public class TestPatternConverter extends PatternConverter {
#Override
protected String convert(LoggingEvent event) {
String testID = ObjectUtils.EMPTY_STRING;
if(TestLogHandler.isTestLogEnabled()) {
TestContextHolder contextHolder = TestLogHandler.getLatestContextHolderFromStack(event.getThreadName());
if(contextHolder != null) {
testID = contextHolder.getTestIDForThread(event.getThreadName());
}
else{
testID = TestContextHolder.getTestIDForThread(event.getThreadName());
}
}
return testID;
}
}
Layout definition in log4j.xml:
<appender name="TEST_LOG_FILE" class="org.apache.log4j.RollingFileAppender">
...
<layout class="com.test.it.logging.TestPatternLayout">
<param name="ConversionPattern" value="%d %-5p [%c{1}] [TestId: %e] [%t] %m%n"/>
</layout>
...
</appender>
You just need to create a new Plugin and extend LogEventPatternConverter:
#Plugin(name = "TestPatternConverter", category = PatternConverter.CATEGORY)
#ConverterKeys({"e"})
public final class TestPatternConverter extends LogEventPatternConverter {
/**
* Private constructor.
* #param options options, may be null.
*/
private TestPatternConverter(final String[] options) {
super("TestId", "testId");
}
/**
* Obtains an instance of pattern converter.
*
* #param options options, may be null.
* #return instance of pattern converter.
*/
public static TestPatternConverter newInstance(final String[] options) {
return new TestPatternConverter(options);
}
/**
* {#inheritDoc}
*/
#Override
public void format(final LogEvent event, final StringBuilder toAppendTo) {
String testID = ObjectUtils.EMPTY_STRING;
if(TestLogHandler.isTestLogEnabled()) {
TestContextHolder contextHolder = TestLogHandler.getLatestContextHolderFromStack(event.getThreadName());
if(contextHolder != null) {
testID = contextHolder.getTestIDForThread(event.getThreadName());
}
else{
testID = TestContextHolder.getTestIDForThread(event.getThreadName());
}
}
toAppendTo.append(testID);
}
}
and update your configuration:
<Appender type="RollingFile" name="TEST_LOG_FILE" fileName="${filename}">
<Layout type="PatternLayout">
<Pattern>%d %-5p [%c{1}] [TestId: %e] [%t] %m%n</Pattern>
</Layout>
</Appender>
That should be all. Please see Extending log4j2
for more details.
I need to create a file that only logs TRACE and DEBUG events, but I'm unable to make it work (don't know if it's even possible).
Already tried searching for an option to invert the ThresholdFilter or a way to specify multiple levels on a LevelFilter, but with no success.
The only way that I'm thinking to make this work is to create 2 appende, exactly the same from one to another, but with the LevelFilter in one specified to TRACE and the other to DEBUG.
Is there any other way to accomplish this? Because I'm not a big fan of duplicating code/configuration.
Simple approach might be to create custom filter as below.
public class CustomFilter extends Filter<ILoggingEvent> {
private String levels;
public String getLevels() {
return levels;
}
public void setLevels(String levels) {
this.levels = levels;
}
private Level[] level;
#Override
public FilterReply decide(ILoggingEvent arg0) {
if (level == null && levels != null) {
setLevels();
}
if (level != null) {
for (Level lev : level) {
if (lev == arg0.getLevel()) {
return FilterReply.ACCEPT;
}
}
}
return FilterReply.DENY;
}
private void setLevels() {
if (!levels.isEmpty()) {
level = new Level[levels.split("\\|").length];
int i = 0;
for (String str : levels.split("\\|")) {
level[i] = Level.valueOf(str);
i++;
}
}
}
}
Add filter in logback.xml
<configuration>
<appender name="fileAppender1" class="ch.qos.logback.core.FileAppender">
<filter class="com.swasthik.kp.logback.filters.CustomFilter">
<levels>TRACE|DEBUG</levels>
</filter>
<file>c:/logs/kplogback.log</file>
<append>true</append>
<encoder>
<pattern>%d [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>
</appender>
<root level="TRACE">
<appender-ref ref="fileAppender1" />
</root>
</configuration>