We Keep Coding

I have added custom filter to the logback.xml, spring boot project through the exception

I have added a custom filter class to my spring boot project and configured it in the logback.xml file. I am getting the below error. please let me know how to resolve the below error.
Exception in thread "main" java.lang.IllegalStateException: Logback configuration error detected:
ERROR in ch.qos.logback.core.joran.spi.Interpreter#14:11 - no applicable action for [level], current ElementPath is [[configuration][appender][filter][level]]
ERROR in ch.qos.logback.core.joran.spi.Interpreter#15:12 - no applicable action for [append], current ElementPath is [[configuration][appender][filter][append]]
Sample code: logback.xml
<appender name="RollingAppender" class="ch.qos.logback.core.rolling.RollingFileAppender">
<param name="MaxFileSize" value="50KB" />
<filter class="com.filter.CustomFilter">
<level>INFO|DEBUG|ERROR</level>
<append>true</append>
</filter>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<pattern>[%d{yyyy-MM-dd'T'HH:mm:ss.sss'Z'}] [%C] [%M] [%L] [%t] [%-5p] %replace(%m){"Bearer.*'", "Bearerxxx'"}%n</pattern>
</encoder>
</rollingPolicy>
</appender>
Java Code:
#Component
public class CustomFilter extends Filter<ILoggingEvent> {
private String levels;
public String getLevels() {
return levels;
}
public void setLevels(String levels) {
this.levels = levels;
}
private Level[] level;
#Override
public FilterReply decide(ILoggingEvent loginEvent) {
if (level == null && levels != null) {
setLevels();
}
if (level != null) {
for (Level lev : level) {
if (lev == loginEvent.getLevel()) {
return FilterReply.ACCEPT;
}
}
}
return FilterReply.DENY;
}
private void setLevels() {
if (!levels.isEmpty()) {
level = new Level[levels.split("\\|").length];
int i = 0;
for (String str : levels.split("\\|")) {
level[i] = Level.valueOf(str);
i++;
}
}
}
}

Masking sensitive data at SOAP envelop with logback-spring.xml

I am currently struggling with masking the data available in the logs intercepted at the SOAP client. I have taken the approach to writing customized PatternLayout:
public class PatternMaskingLayout extends ch.qos.logback.classic.PatternLayout {
private Pattern multilinePattern;
private final List<String> maskPatterns = new ArrayList<>();
public void addMaskPattern(String maskPattern) {
maskPatterns.add(maskPattern);
multilinePattern = Pattern.compile(
String.join("|", maskPatterns),
Pattern.MULTILINE
);
}
#Override
public String doLayout(ILoggingEvent event) {
return maskMessage(super.doLayout(event)); // calling superclass method is required
}
private String maskMessage(String message) {
if (multilinePattern == null) {
return message;
}
StringBuilder sb = new StringBuilder(message);
Matcher matcher = multilinePattern.matcher(sb);
while (matcher.find()) {
IntStream.rangeClosed(1, matcher.groupCount()).forEach(group -> {
if (matcher.group(group) != null) {
IntStream.range(matcher.start(group), matcher.end(group))
.forEach(i -> sb.setCharAt(i, '*')); // replace each character with asterisk
}
});
}
return sb.toString();
}
}
My logback-spring.xml appenders looks like:
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<layout class="app.example.monitoring.tracing.PatternMaskingLayout">
<maskPattern>\"username\"\s*:\s*\"(.*?)\"</maskPattern>
<pattern>
${LOGBACK_LOGGING_PATTERN:-%d{yyyy-MM-dd HH:mm:ss.SSS} ${LOG_LEVEL_PATTERN:-%5p} ${PID:- } --- [%15.15t] %logger{36} : %msg %replace(%ex){'\n','\\u000a'}%nopex%n}
</pattern>
</layout>
</appender>
I still can not get my username masked. The XML field looks like <xa2:username>John</xa2:username>|
Have anyone have some experience with this?

GKE & Stackdriver: Java logback logging format?

I have a project running Java in a docker image on Kubernetes. Logs are automatically ingested by the fluentd agent and end up in Stackdriver.
However, the format of the logs is wrong: Multiline logs get put into separate log lines in Stackdriver, and all logs have "INFO" log level, even though they are really warning, or error.
I have been searching for information on how to configure logback to output the correct format for this to work properly, but I can find no such guide in the google Stackdriver or GKE documentation.
My guess is that I should be outputting JSON of some form, but where do I find information on the format, or even a guide on how to properly set up this pipeline.
Thanks!

This answer contained most of the information I needed: https://stackoverflow.com/a/39779646
I have adapted the answer to fit my exact question, and to fix some weird imports and code that seems to have been deprecated.
logback.xml:
<configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="my.package.logging.GCPCloudLoggingJSONLayout">
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg</pattern>
</layout>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="STDOUT"/>
</root>
</configuration>
GCPCloudLoggingJSONLayout:
import ch.qos.logback.classic.Level;
import ch.qos.logback.classic.PatternLayout;
import ch.qos.logback.classic.spi.ILoggingEvent;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.Map;
import static ch.qos.logback.classic.Level.DEBUG_INT;
import static ch.qos.logback.classic.Level.ERROR_INT;
import static ch.qos.logback.classic.Level.INFO_INT;
import static ch.qos.logback.classic.Level.TRACE_INT;
import static ch.qos.logback.classic.Level.WARN_INT;
/**
* GKE fluentd ingestion detective work:
* https://cloud.google.com/error-reporting/docs/formatting-error-messages#json_representation
* http://google-cloud-python.readthedocs.io/en/latest/logging-handlers-container-engine.html
* http://google-cloud-python.readthedocs.io/en/latest/_modules/google/cloud/logging/handlers/container_engine.html#ContainerEngineHandler.format
* https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/logging/google/cloud/logging/handlers/_helpers.py
* https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
*/
public class GCPCloudLoggingJSONLayout extends PatternLayout {
private static final ObjectMapper objectMapper = new ObjectMapper();
#Override
public String doLayout(ILoggingEvent event) {
String formattedMessage = super.doLayout(event);
return doLayoutInternal(formattedMessage, event);
}
/**
* For testing without having to deal wth the complexity of super.doLayout()
* Uses formattedMessage instead of event.getMessage()
*/
private String doLayoutInternal(String formattedMessage, ILoggingEvent event) {
GCPCloudLoggingEvent gcpLogEvent =
new GCPCloudLoggingEvent(formattedMessage, convertTimestampToGCPLogTimestamp(event.getTimeStamp()),
mapLevelToGCPLevel(event.getLevel()), event.getThreadName());
try {
// Add a newline so that each JSON log entry is on its own line.
// Note that it is also important that the JSON log entry does not span multiple lines.
return objectMapper.writeValueAsString(gcpLogEvent) + "\n";
} catch (JsonProcessingException e) {
return "";
}
}
private static GCPCloudLoggingEvent.GCPCloudLoggingTimestamp convertTimestampToGCPLogTimestamp(
long millisSinceEpoch) {
int nanos =
((int) (millisSinceEpoch % 1000)) * 1_000_000; // strip out just the milliseconds and convert to nanoseconds
long seconds = millisSinceEpoch / 1000L; // remove the milliseconds
return new GCPCloudLoggingEvent.GCPCloudLoggingTimestamp(seconds, nanos);
}
private static String mapLevelToGCPLevel(Level level) {
switch (level.toInt()) {
case TRACE_INT:
return "TRACE";
case DEBUG_INT:
return "DEBUG";
case INFO_INT:
return "INFO";
case WARN_INT:
return "WARN";
case ERROR_INT:
return "ERROR";
default:
return null; /* This should map to no level in GCP Cloud Logging */
}
}
/* Must be public for Jackson JSON conversion */
public static class GCPCloudLoggingEvent {
private String message;
private GCPCloudLoggingTimestamp timestamp;
private String thread;
private String severity;
public GCPCloudLoggingEvent(String message, GCPCloudLoggingTimestamp timestamp, String severity,
String thread) {
super();
this.message = message;
this.timestamp = timestamp;
this.thread = thread;
this.severity = severity;
}
public String getMessage() {
return message;
}
public void setMessage(String message) {
this.message = message;
}
public GCPCloudLoggingTimestamp getTimestamp() {
return timestamp;
}
public void setTimestamp(GCPCloudLoggingTimestamp timestamp) {
this.timestamp = timestamp;
}
public String getThread() {
return thread;
}
public void setThread(String thread) {
this.thread = thread;
}
public String getSeverity() {
return severity;
}
public void setSeverity(String severity) {
this.severity = severity;
}
/* Must be public for JSON marshalling logic */
public static class GCPCloudLoggingTimestamp {
private long seconds;
private int nanos;
public GCPCloudLoggingTimestamp(long seconds, int nanos) {
super();
this.seconds = seconds;
this.nanos = nanos;
}
public long getSeconds() {
return seconds;
}
public void setSeconds(long seconds) {
this.seconds = seconds;
}
public int getNanos() {
return nanos;
}
public void setNanos(int nanos) {
this.nanos = nanos;
}
}
}
#Override
public Map<String, String> getDefaultConverterMap() {
return PatternLayout.defaultConverterMap;
}
}
As I said earlier, the code was originally from another answer, I have just cleaned up the code slightly to fit my use-case better.

Google has provided a logback appender for Stackdriver, I have enhanced it abit to include the thread name in the logging label, so that it can be searched more easily.
pom.xml
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-logging-logback</artifactId>
<version>0.116.0-alpha</version>
</dependency>
logback-spring.xml
<springProfile name="prod-gae">
<appender name="CLOUD"
class="com.google.cloud.logging.logback.LoggingAppender">
<log>clicktrade.log</log>
<loggingEventEnhancer>com.jimmy.clicktrade.arch.CommonEnhancer</loggingEventEnhancer>
<flushLevel>WARN</flushLevel>
</appender>
<root level="info">
<appender-ref ref="CLOUD" />
</root>
</springProfile>
CommonEnhancer.java
public class CommonEnhancer implements LoggingEventEnhancer {
#Override
public void enhanceLogEntry(Builder builder, ILoggingEvent e) {
builder.addLabel("thread", e.getThreadName());
}
}
Surprisingly, the logback appender in MVN repository doesn't align with the github repo source code. I need to dig into the JAR source code for that. The latest version is 0.116.0-alpha, it seems it has version 0.120 in Github
https://github.com/googleapis/google-cloud-java/tree/master/google-cloud-clients/google-cloud-contrib/google-cloud-logging-logback

You can use the glogging library.
Simply add it as a dependency and use provided layout in logback.xml:
<configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="io.github.aaabramov.glogging.GoogleLayout">
<!-- You have a choice which JSON encoder to use. Or create your own via implementing JsonEncoder interface -->
<json>io.github.aaabramov.glogging.JacksonEncoder</json>
<!-- OR -->
<!-- <json>io.github.aaabramov.glogging.GsonEncoder</json> -->
<!-- Optionally append "${prefix}/loggerName" labels -->
<appendLoggerName>true</appendLoggerName>
<!-- Optionally configure prefix for labels -->
<prefix>com.yourcompany</prefix>
<!-- Provide message pattern you like. -->
<!-- Note: there is no need anymore to log timestamps & levels to the message. Google will pick them up from specific fields. -->
<pattern>%message %xException{10}</pattern>
</layout>
</encoder>
</appender>
<appender name="ASYNCSTDOUT" class="ch.qos.logback.classic.AsyncAppender">
<appender-ref ref="STDOUT"/>
</appender>
<!-- Configure logging levels -->
<logger name="com.github" level="DEBUG"/>
<root level="DEBUG">
<appender-ref ref="ASYNCSTDOUT"/>
</root>
</configuration>
It will produce messages in the format that GSL will gladly accept:
{"timestamp":{"seconds":1629642099,"nanos":659000000},"severity":"DEBUG","message":"debug","labels":{"io.github.aaabramov/name":"Andrii","io.github.aaabramov/loggerName":"io.github.aaabramov.glogging.App"}}
Disclaimer: I am the author of this library.
Credits: https://stackoverflow.com/a/44168383/5091346

Extending FileAppender in logback - advanced file naming

Problem:
I'm converting from log4j to logback and having problems with my custom FileAppender. I'm trying to replicate the log4j behavior I had, where the FileAppender created a new file named request_req#_req_type_date.log
Currently, the logged data is correct, however it's appending to the same file request.log rather than creating a new log file for each travel request.
Question:
What do I need to change so new request log files get created as opposed to appending to a single log file?
This is my logback.xml configuration:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="FORM_REQ" class="com.app.package.FormRequestAppender">
<file>${catalina.base}/APPlogs/formrequests/request.xml</file>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<Pattern>%msg%n</Pattern>
</encoder>
</appender>
<logger name="formRequestLogger" level="DEBUG" additivity="false">
<appender-ref ref="FORM_REQ" />
</logger>
<root level="INFO">
<appender-ref ref="FILE" />
<appender-ref ref="STDOUT" />
</root>
</configuration>
This is my custom FileAppender (FormRequestAppender):
import org.apache.commons.lang3.StringUtils;
import ch.qos.logback.core.FileAppender;
import ch.qos.logback.classic.spi.LoggingEvent;
import ch.qos.logback.classic.spi.*;
public class FormRequestAppender<E> extends FileAppender<E> {
private static String path = null;
protected void subAppend(LoggingEvent event) {
this.closeOutputStream();
setFileName();
this.setFile(fileName);
this.subAppend(event);
}
private void setFileName() {
if (fileName != null) {
try {
String paths[] = fileName.split("\\/");
StringBuilder sb = new StringBuilder();
if (path == null) {
for (int i = 0; i < paths.length - 1; i++) {
sb.append(paths[i] + "//");
}
path = sb.toString();
} else {
sb.append(path);
}
sb.append("request");
String reqNum = getAttribute(TRAVEL_REQUEST_NUMBER);
String approvalStatus = getAttribute(APPROVAL_STATUS);
String postSubmitType = getAttribute(POST_SUBMIT_APP_TYPE);
if (!StringUtils.isEmpty(reqNum)) {
sb.append("_");
sb.append(reqNum);
}
if (!StringUtils.isEmpty(approvalStatus)) {
sb.append("_");
sb.append(approvalStatus);
}
if (!StringUtils.isEmpty(postSubmitType)) {
sb.append("_");
sb.append(postSubmitType);
}
sb.append("_");
sb.append(DateTimeUtil.getDateTimeString(new Date()));
sb.append(".xml");
fileName = sb.toString();
} catch (Exception e) {
addError("An exception on FormRequestAppender.setFileName - " + e.getMessage());
}
}
}
}
And this is the only class that uses it when a new request has been submitted:
public class SubmissionService {
private static final Logger formRequestLogger = LoggerFactory.getLogger("formRequestLogger");

I have used this subAppend method (try to use #Override annotation to be sure that you are overriding super class method. Is possible to be overloading without intention and with that annotation you won't be able to compile if method is not overriding):
#Override
protected void subAppend(E event) {
setFileName();
this.setFile(fileName);
start();
super.subAppend(event);
}
rest of code is same as yours. Is not needed for append property to be specified in logback.xml.
Hope it helps.

How to add a PatternLayout to the root logger at runtime?

I am using logback as the backend for Slf4j. Currently, I configure the logger using a logback.xml file. My issue is that sensitive information is being logged (outside of my control) and I want to mask this sensitive information. To mask the information, I have wrote a custom PatternLayout class that essentially does:
#Override
public String doLayout(ILoggingEvent event) {
String message = super.doLayout(event);
Matcher matcher = sesnsitiveInfoPattern.matcher(message);
if (matcher.find()) {
message = matcher.replaceAll("XXX");
}
return message;
}
My issue is that I need to tell logback to use this custom pattern layout. I don't want to add this to the XML configuration however. My current configuration looks like this:
<configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<layout class="com.my.MaskingPatternLayout"> <!-- here -->
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
</layout>
</encoder>
</appender>
<root level="info">
<appender-ref ref="STDOUT" />
</root>
</configuration>
In XML, my desired configuration would look like this (but I don't want to use XML):

Hello Max I hope you are using Log4j 2.x because this solution uses the plugins approache introduced in log4j 2.x . first you should create a package where you are going to put your plugins classes and you put there these two classes :
my.log4j.pluggins.CustomConfigurationFactory :
#Plugin(name = "CustomConfigurationFactory", category = ConfigurationFactory.CATEGORY)
#Order(value = 0)
public class CustomConfigurationFactory extends ConfigurationFactory {
private Configuration createConfiguration(final String name,
ConfigurationBuilder<BuiltConfiguration> builder) {
System.out.println("init logger");
builder.setConfigurationName(name);
builder.setStatusLevel(Level.INFO);
builder.setPackages("my.log4j.pluggins");
AppenderComponentBuilder appenderBuilder = builder.newAppender(
"Stdout", "CONSOLE").addAttribute("target",
ConsoleAppender.Target.SYSTEM_OUT);
appenderBuilder
.add(builder
.newLayout("PatternLayout")
.addAttribute("pattern", "%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %myMsg%n"));
builder.add(appenderBuilder);
builder.add(builder.newRootLogger(Level.TRACE).add(
builder.newAppenderRef("Stdout")));
return builder.build();
}
#Override
protected String[] getSupportedTypes() {
String[] supportedExt = { "*" };
return supportedExt;
}
#Override
public Configuration getConfiguration(ConfigurationSource source) {
ConfigurationBuilder<BuiltConfiguration> builder = newConfigurationBuilder();
return createConfiguration(source.toString(), builder);
}
#Override
public Configuration getConfiguration(String name, URI configLocation) {
ConfigurationBuilder<BuiltConfiguration> builder = newConfigurationBuilder();
return createConfiguration(name, builder);
}
}
my.log4j.pluggins.SampleLayout :
#Plugin(name = "CustomConverter", category = "Converter")
#ConverterKeys({"myMsg"})
public class SampleLayout extends LogEventPatternConverter {
protected SampleLayout(String name, String style) {
super(name, style);
}
public static SampleLayout newInstance(){
return new SampleLayout("custConv", "custConv");
}
#Override
public void format(LogEvent event, StringBuilder stringBuilder) {
//replace the %myMsg by XXXXX if sensitive
if (sensitive()){
stringBuilder.append("XXXX");}
else {
stringBuilder.append(event.getMessage().getFormattedMessage());}
}
}
the CustomConfiguration class is responsable for creating the configuration of log4j and the line 9 where 'builder.setPackages("my.log4j.pluggins")' is important in order to scan that package and pick up the converter pluggin wich is SampleLayout.
the second class will be responsible for formatting the new key '%myMsg' in the pattern that contains my sensitive message, this Converter class checks if that message is sensitive and actes accordingly.
Before you start logging you should configure your log4j like this
ConfigurationFactory.setConfigurationFactory(new CustomConfigurationFactory());