We Keep Coding

Log4j2 fatjar programmatically config

I'm trying to get log4j2 work within a fatjar.
When running in intellij every works fine, but when building my fatjar an running it like
java -jar dx.jar
only errors are logged to console, and nothing to the rollingfile. It seams ignoring my config.
Configuration:
String logFile = logPath + "dxpos.log";
String logFilePattern = logPath + "dxpos-%d{yyyy-MM-dd}.log";
LogManager.shutdown();
var builder = ConfigurationBuilderFactory.newConfigurationBuilder();
var layout = builder.newLayout("PatternLayout")
.addAttribute("pattern", "%d{yyyy-MM-dd HH:mm:ss} %-5p [%t] %c: %m%n");
var console = builder.newAppender("console", "Console")
.addAttribute("target", "SYSTEM_OUT")
.addAttribute("immediateFlush", true)
.add(layout);
var triggeringPolicy = builder.newComponent("Policies")
.addComponent(builder.newComponent("TimeBasedTriggeringPolicy"))
.addComponent(builder.newComponent("SizeBasedTriggeringPolicy").addAttribute("size", "100M"));
var rollingFile = builder.newAppender("rollingFile", "RollingFile")
.addAttribute("fileName", logFile)
.addAttribute("filePattern", logFilePattern)
.addAttribute("immediateFlush", true)
.addAttribute("append", true)
.addComponent(triggeringPolicy)
.add(layout);
var rootLogger = builder.newRootLogger(Level.INFO)
.add(builder.newAppenderRef("console"))
.add(builder.newAppenderRef("rollingFile"));
builder.add(console);
builder.add(rollingFile);
builder.add(rootLogger);
Configurator.initialize(builder.build());

After searching a file I a switch from programmatically config method to XML
To getting it to work you have to declare the logger
public static Logger logger = null;
Add some variables like that
MDC.put("logFile", logFile);
MDC.put("LogFilePattern", logFilePattern);
And create the logger
logger = LoggerFactory.getLogger(MainForm.class);

Masking sensitive data at SOAP envelop with logback-spring.xml

I am currently struggling with masking the data available in the logs intercepted at the SOAP client. I have taken the approach to writing customized PatternLayout:
public class PatternMaskingLayout extends ch.qos.logback.classic.PatternLayout {
private Pattern multilinePattern;
private final List<String> maskPatterns = new ArrayList<>();
public void addMaskPattern(String maskPattern) {
maskPatterns.add(maskPattern);
multilinePattern = Pattern.compile(
String.join("|", maskPatterns),
Pattern.MULTILINE
);
}
#Override
public String doLayout(ILoggingEvent event) {
return maskMessage(super.doLayout(event)); // calling superclass method is required
}
private String maskMessage(String message) {
if (multilinePattern == null) {
return message;
}
StringBuilder sb = new StringBuilder(message);
Matcher matcher = multilinePattern.matcher(sb);
while (matcher.find()) {
IntStream.rangeClosed(1, matcher.groupCount()).forEach(group -> {
if (matcher.group(group) != null) {
IntStream.range(matcher.start(group), matcher.end(group))
.forEach(i -> sb.setCharAt(i, '*')); // replace each character with asterisk
}
});
}
return sb.toString();
}
}
My logback-spring.xml appenders looks like:
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<layout class="app.example.monitoring.tracing.PatternMaskingLayout">
<maskPattern>\"username\"\s*:\s*\"(.*?)\"</maskPattern>
<pattern>
${LOGBACK_LOGGING_PATTERN:-%d{yyyy-MM-dd HH:mm:ss.SSS} ${LOG_LEVEL_PATTERN:-%5p} ${PID:- } --- [%15.15t] %logger{36} : %msg %replace(%ex){'\n','\\u000a'}%nopex%n}
</pattern>
</layout>
</appender>
I still can not get my username masked. The XML field looks like <xa2:username>John</xa2:username>|
Have anyone have some experience with this?

How to create multiple log file programatically in log4j2?

I am developing a java application which communicates with lots of devices. For each device I need to create a different log file to log it's communication with device. This is the wrapper class I developed. It creates two log files but the data is written to only the first one. The second file is created but nothing is written to it. The output that should go to second file goes to console. If I uncomment createRootLogger() in constructor nothing is written to both the files, everything goes to console. I have gone through log4j2 documentation but it is poorly written with very few code samples. Here is my wrapper class, where is the error? I am using log4j-api-2.9.0.jar and log4j-core-2.9.0.jar.
package xyz;
import org.apache.logging.log4j.Level;
import org.apache.logging.log4j.core.Logger;
import org.apache.logging.log4j.core.LoggerContext;
import org.apache.logging.log4j.core.appender.ConsoleAppender;
import org.apache.logging.log4j.core.config.Configuration;
import org.apache.logging.log4j.core.config.Configurator;
import org.apache.logging.log4j.core.config.builder.api.*;
import java.util.Hashtable;
public class LogManager
{
static protected LogManager m_clsInstance = null;
protected Hashtable<String, Logger> m_clsLoggers = new Hashtable<String, Logger>();
private LogManager()
{
//createRootLogger();
}
/**
* getInstance is used to get reference to the singalton class obj ......
*/
static synchronized public LogManager getInstance()
{
try
{
if (m_clsInstance == null)
{
m_clsInstance = new LogManager();
//Configurator.setRootLevel(Level.TRACE);
}
}
catch (Exception xcpE)
{
System.err.println(xcpE);
}
return m_clsInstance;
}
static public Logger getLogger(String sLogger)
{
try
{
return getInstance().m_clsLoggers.get(sLogger);
}
catch (Exception xcpE)
{
System.err.println(xcpE);
}
return null;
}
public Logger createLogger(String strName, String sPath, int nBackupSize, long lngMaxSize, String strPattern, String strLevel)
{
try
{
ConfigurationBuilder builder = ConfigurationBuilderFactory.newConfigurationBuilder();
builder.setStatusLevel(Level.getLevel(strLevel));
builder.setConfigurationName("RollingBuilder"+strName);
// create a console appender
AppenderComponentBuilder appenderBuilder = builder.newAppender("Stdout", "CONSOLE").addAttribute("target",
ConsoleAppender.Target.SYSTEM_OUT);
appenderBuilder.add(builder.newLayout("PatternLayout")
.addAttribute("pattern", strPattern));
builder.add( appenderBuilder );
// create a rolling file appender
LayoutComponentBuilder layoutBuilder = builder.newLayout("PatternLayout")
.addAttribute("pattern", strPattern);
ComponentBuilder triggeringPolicy = builder.newComponent("Policies")
// .addComponent(builder.newComponent("CronTriggeringPolicy").addAttribute("schedule", "0 0 0 * * ?"))
.addComponent(builder.newComponent("SizeBasedTriggeringPolicy").addAttribute("size", lngMaxSize));
appenderBuilder = builder.newAppender("rolling"+strName, "RollingFile")
.addAttribute("fileName", sPath)
.addAttribute("filePattern", "d:\\trash\\archive\\rolling-%d{MM-dd-yy}.log.gz")
.add(layoutBuilder)
.addComponent(triggeringPolicy);
builder.add(appenderBuilder);
// create the new logger
builder.add( builder.newLogger( strName, Level.getLevel(strLevel) )
.add( builder.newAppenderRef( "rolling"+strName ) )
.addAttribute( "additivity", false ) );
Configuration clsCnfg = (Configuration) builder.build();
LoggerContext ctx = Configurator.initialize(clsCnfg);
Logger clsLogger = ctx.getLogger(strName);
m_clsLoggers.put(strName, clsLogger);
return clsLogger;
}
catch (Exception xcpE)
{
System.err.println(xcpE);
}
return null;
}
protected void createRootLogger()
{
try
{
ConfigurationBuilder builder = ConfigurationBuilderFactory.newConfigurationBuilder();
builder.setStatusLevel(Level.getLevel("TRACE"));
builder.setConfigurationName("rootConfig");
// create a console appender
AppenderComponentBuilder appenderBuilder = builder.newAppender("Stdout", "CONSOLE").addAttribute("target",
ConsoleAppender.Target.SYSTEM_OUT);
appenderBuilder.add(builder.newLayout("PatternLayout")
.addAttribute("pattern", "[%d{yyyy-MMM-dd HH:mm:ss:SSS}][%-5p %l][%t] %m%n"));
builder.add( appenderBuilder );
builder.add( builder.newRootLogger( Level.getLevel("TRACE"))
.add( builder.newAppenderRef( "Stdout") ) );
Configuration clsCnfg = (Configuration) builder.build();
LoggerContext ctx = Configurator.initialize(clsCnfg);
Logger clsLogger = ctx.getRootLogger();
m_clsLoggers.put("root", clsLogger);
}
catch (Exception xcpE)
{
System.err.println(xcpE);
}
}
static public void main(String args[])
{
//Logger clsLogger = setLogger();
Logger clsLogger = Emflex.LogManager.getInstance().createLogger(
"AnsiAmrController_" + 5555,
"d:\\trash\\LogManagerTest5555.log",
10,
100000000,
"[%d{yyyy-MMM-dd HH:mm:ss:SSS}][%-5p %l][%t] %m%n",
"TRACE"
);
Logger clsLogger2 = Emflex.LogManager.getInstance().createLogger(
"AnsiAmrController_" + 6666,
"d:\\trash\\LogManagerTest6666.log",
10,
100000000,
"[%d{yyyy-MMM-dd HH:mm:ss:SSS}][%-5p %l][%t] %m%n",
"TRACE"
);
for (int i=0;i<100;i++)
{
clsLogger.error("Testing - ["+i+"]");
clsLogger2.error("Testing - ["+(i*i)+"]");
}
}
}

You said your objective is:
For each device I need to create a different log file to log it's communication with device.
There are many different ways to accomplish this without programmatic configuration. Programmatic configuration is bad because it forces you to depend on the logging implementation rather than the public interface.
For example you could use a context map key in conjunction with a Routing Appender to separate your logs, similar to the example I gave in another answer. Note that in the other answer I used the variable as the folder where the log is stored but you can use it for the log name if you wish.
Another way to do what you want would be to use a MapMessage as shown in the log4j2 manual.
Yet another way would be to use markers in combination with a RoutingAppender. Here is some example code for this approach:
package example;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.Marker;
import org.apache.logging.log4j.MarkerManager;
public class LogLvlByMarkerMain {
private static final Logger log = LogManager.getLogger();
private static final Marker DEVICE1 = MarkerManager.getMarker("DEVICE1");
private static final Marker DEVICE2 = MarkerManager.getMarker("DEVICE2");
public static void main(String[] args) {
log.info(DEVICE1, "The first device got some input");
log.info(DEVICE2, "The second device now has input");
}
}
Configuration:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
<Appenders>
<Routing name="MyRoutingAppender">
<Routes pattern="$${marker:}">
<Route>
<File
fileName="logs/${marker:}.txt"
name="appender-${marker:}">
<PatternLayout>
<Pattern>[%date{ISO8601}][%-5level][%t] %m%n</Pattern>
</PatternLayout>
</File>
</Route>
</Routes>
</Routing>
<Console name="STDOUT" target="SYSTEM_OUT">
<PatternLayout pattern="[%date{ISO8601}][%-5level][%t] %m%n" />
</Console>
</Appenders>
<Loggers>
<Logger name="example" level="TRACE" additivity="false">
<AppenderRef ref="STDOUT" />
<AppenderRef ref="MyRoutingAppender" />
</Logger>
<Root level="WARN">
<AppenderRef ref="STDOUT" />
</Root>
</Loggers>
</Configuration>
Output:
This will generate 2 log files - DEVICE1.txt and DEVICE2.txt as shown in the image below.
The first log will contain only messages that were marked as DEVICE1 and the second will contain only DEVICE2 logs.
I.e. the first log contains:
[2017-09-21T09:52:04,171][INFO ][main] The first device got some input
and the second contains:
[2017-09-21T09:52:04,176][INFO ][main] The second device now has input

The approach log4j2 is initialize programmatically and later configuration is modified is different. And you you trying to add dynamic appender and logger using initialization approach.
So, first you should initialize your RootLogger using initialization approach that seems correct in your code.
After that, add dynamic appender and logger using approach mentioned here

adding on D.B answer:
I had trouble making this write to file. (and yes I tried using log4j2 version 2.8.1 but still didn't work)
To make it work I edited this part
<Root level="WARN">
<AppenderRef ref="STDOUT" />
</Root>
to this:
<Root level="WARN">
<AppenderRef ref="STDOUT" />
<AppenderRef ref="MyRoutingAppender" />
</Root>
And since the Debug level is set to WARN
<Configuration status="WARN">
and we trying to log info
log.info(DEVICE$, "The $ device now has input");
the info log wont be written (WARN will only print: warn, error, fatal check this link log4j logging level)
you can simply change
log.info() --> log.warn()
just as a proof of concept.

How do I programmatically set conversionRule for logback?

I have a logback XML configuration file where I use my custom tag converter to tag my log messages.
<configuration scan="true">
<conversionRule conversionWord="tag"
converterClass="com.foo.MyCustomTagConverter" />
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<!-- encoders are by default assigned the type
ch.qos.logback.classic.encoder.PatternLayoutEncoder -->
<encoder>
<pattern>%tag - %m%n</pattern>
</encoder>
</appender>
...
</configuration>
How can I do this programmatically via Java?

public class loggerutils {
static Logger foo = createLoggerFor("foo", "foo.log");
public static void main(String[] args) {
foo.info("test");
}
private static Logger createLoggerFor(String string, String file) {
String conversionWord = "tag";
String converterClass = "com.foo.MyCustomTagConverter";
LoggerContext lc = (LoggerContext) LoggerFactory.getILoggerFactory();
Map<String, String> ruleRegistry = (Map) lc
.getObject(CoreConstants.PATTERN_RULE_REGISTRY);
if (ruleRegistry == null) {
ruleRegistry = new HashMap<String, String>();
}
lc.putObject(CoreConstants.PATTERN_RULE_REGISTRY, ruleRegistry);
ruleRegistry.put(conversionWord, converterClass);
PatternLayoutEncoder ple = new PatternLayoutEncoder();
ple.setPattern("%-20tag %-30(%d{HH:mm:ss.SSS} [%thread]) - %-15(%M) - %-5level -%logger{32} - %msg%n ");
ple.setContext(lc);
ple.start();
ConsoleAppender<ILoggingEvent> consoleAppender = new ConsoleAppender<ILoggingEvent>();
consoleAppender.setEncoder(ple);
consoleAppender.setContext(lc);
consoleAppender.start();
Logger logger = (Logger) LoggerFactory.getLogger(string);
logger.addAppender(fileAppender);
logger.setLevel(Level.DEBUG);
logger.setAdditive(false); /* set to true if root should log too */
logger.addAppender(consoleAppender);
logger.setLevel(Level.DEBUG);
logger.setAdditive(false);
return logger;
}
}

How to add a PatternLayout to the root logger at runtime?

I am using logback as the backend for Slf4j. Currently, I configure the logger using a logback.xml file. My issue is that sensitive information is being logged (outside of my control) and I want to mask this sensitive information. To mask the information, I have wrote a custom PatternLayout class that essentially does:
#Override
public String doLayout(ILoggingEvent event) {
String message = super.doLayout(event);
Matcher matcher = sesnsitiveInfoPattern.matcher(message);
if (matcher.find()) {
message = matcher.replaceAll("XXX");
}
return message;
}
My issue is that I need to tell logback to use this custom pattern layout. I don't want to add this to the XML configuration however. My current configuration looks like this:
<configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<layout class="com.my.MaskingPatternLayout"> <!-- here -->
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
</layout>
</encoder>
</appender>
<root level="info">
<appender-ref ref="STDOUT" />
</root>
</configuration>
In XML, my desired configuration would look like this (but I don't want to use XML):

Hello Max I hope you are using Log4j 2.x because this solution uses the plugins approache introduced in log4j 2.x . first you should create a package where you are going to put your plugins classes and you put there these two classes :
my.log4j.pluggins.CustomConfigurationFactory :
#Plugin(name = "CustomConfigurationFactory", category = ConfigurationFactory.CATEGORY)
#Order(value = 0)
public class CustomConfigurationFactory extends ConfigurationFactory {
private Configuration createConfiguration(final String name,
ConfigurationBuilder<BuiltConfiguration> builder) {
System.out.println("init logger");
builder.setConfigurationName(name);
builder.setStatusLevel(Level.INFO);
builder.setPackages("my.log4j.pluggins");
AppenderComponentBuilder appenderBuilder = builder.newAppender(
"Stdout", "CONSOLE").addAttribute("target",
ConsoleAppender.Target.SYSTEM_OUT);
appenderBuilder
.add(builder
.newLayout("PatternLayout")
.addAttribute("pattern", "%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %myMsg%n"));
builder.add(appenderBuilder);
builder.add(builder.newRootLogger(Level.TRACE).add(
builder.newAppenderRef("Stdout")));
return builder.build();
}
#Override
protected String[] getSupportedTypes() {
String[] supportedExt = { "*" };
return supportedExt;
}
#Override
public Configuration getConfiguration(ConfigurationSource source) {
ConfigurationBuilder<BuiltConfiguration> builder = newConfigurationBuilder();
return createConfiguration(source.toString(), builder);
}
#Override
public Configuration getConfiguration(String name, URI configLocation) {
ConfigurationBuilder<BuiltConfiguration> builder = newConfigurationBuilder();
return createConfiguration(name, builder);
}
}
my.log4j.pluggins.SampleLayout :
#Plugin(name = "CustomConverter", category = "Converter")
#ConverterKeys({"myMsg"})
public class SampleLayout extends LogEventPatternConverter {
protected SampleLayout(String name, String style) {
super(name, style);
}
public static SampleLayout newInstance(){
return new SampleLayout("custConv", "custConv");
}
#Override
public void format(LogEvent event, StringBuilder stringBuilder) {
//replace the %myMsg by XXXXX if sensitive
if (sensitive()){
stringBuilder.append("XXXX");}
else {
stringBuilder.append(event.getMessage().getFormattedMessage());}
}
}
the CustomConfiguration class is responsable for creating the configuration of log4j and the line 9 where 'builder.setPackages("my.log4j.pluggins")' is important in order to scan that package and pick up the converter pluggin wich is SampleLayout.
the second class will be responsible for formatting the new key '%myMsg' in the pattern that contains my sensitive message, this Converter class checks if that message is sensitive and actes accordingly.
Before you start logging you should configure your log4j like this
ConfigurationFactory.setConfigurationFactory(new CustomConfigurationFactory());