We Keep Coding

logback: add a new appender programmatically and GLOBALLY as in logback.xml

As other posts show (e.g. this here) it's possible to add an appender via code instead of defining it in logback.xml
Problem: the program, as usual, uses a lot different Logger instances in several java classes. All of them instanciated like:
static final Logger logger = (Logger) LoggerFactory.getLogger(SomeClassName.class);
If I now add a new appender canonically (is it canonically?) like....
Logger rootLogger = (Logger) LoggerFactory.getLogger(Logger.ROOT_LOGGER_NAME);
LoggerContext lc = rootLogger.getLoggerContext();
PatternLayoutEncoder ple = new PatternLayoutEncoder();
ple.setPattern("%date{ISO8601} %level ${PID} %-32c{32} [%12t]: %message%n");
ple.setContext(lc);
ple.start();
SizeAndTimeBasedRollingPolicy<ILoggingEvent> rp = new SizeAndTimeBasedRollingPolicy<>();
rp.setContext(lc);
rp.setMaxHistory(25);
rp.setMaxFileSize(FileSize.valueOf("200KB"));
rp.setTotalSizeCap(FileSize.valueOf("1GB"));
rp.setFileNamePattern(
"logs/archived/" + Zeb2JiraRESTLoader.class.getSimpleName() + ".%d{yyyy-MM-dd}.%i.log.zip");
RollingFileAppender<ILoggingEvent> rfa = new RollingFileAppender<>();
rfa.setFile("logs/" + Zeb2JiraRESTLoader.class.getSimpleName() + ".log");
rfa.setName("FILE_APPENDER");
rfa.setContext(lc);
rfa.setEncoder(ple);
rfa.setRollingPolicy(rp);
rp.setParent(rfa);
rp.start();
rfa.start();
rootLogger.addAppender(rfa);
rootLogger.setLevel(Level.INFO);
rootLogger.setAdditive(true);
rootLogger.iteratorForAppenders().forEachRemaining(System.out::println);
// leads to console output:
// ch.qos.logback.core.ConsoleAppender[STDOUT] <-- defined in logback.xml
// ch.qos.logback.core.rolling.RollingFileAppender[FILE_APPENDER]
// logger is the logger of this class
logger.iteratorForAppenders().forEachRemaining(System.out::println);
// leads to NO console output at all. List is empty
...the unwanted behaviour is:
All Logger instances log into the Appender[STDOUT] defined in the logback.xml file
Only logs from the very class, where the new Appender has been added (see code above), log into the newly defined appender.
Wanted behaviour: All Logger instances shall log into both Appenders, the first one defined in the logback.xml file and the second one, defined by the java code above.

log4j2 with pax-logging: can't use values from StructuredDataMessage

I'm using pax-logging-api along with pax-logging-log4j2 for logging from my OSGi bundles. I would like to utilize Log4J2's StructuredDataMessage (using EventLogger) to write some messages to a database. However, I'm unable to read the values I put into the StructuredDataMessage from the appenders when using Pax Logging.
The following works in a non-OSGi project using the Log4J2 libraries directly:
log4j2.properties:
appender.console.type = Console
appender.console.name = STDOUT
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %m%n
appender.event.type = Console
appender.event.name = event
appender.event.layout.type = PatternLayout
appender.event.layout.pattern = %marker ${sd:id} ${sd:testKey} %n %m%n
rootLogger.level = debug
rootLogger.appenderRef.console.ref = STDOUT
logger.event.name = EventLogger
logger.event.level = debug
logger.event.appenderRef.console.ref = event
logger.event.additivity = false
Test.java:
public class Test {
private static final Logger LOGGER = LogManager.getLogger(Test.class);
public static void main(String[] args) {
StructuredDataMessage msg = new StructuredDataMessage("1", "message", "event");
msg.put("testKey", "testValue");
LOGGER.info(msg);
EventLogger.logEvent(msg);
}
}
Output:
1 testValue event [1 testKey="testValue"] message
EVENT 1 testValue
event [1 testKey="testValue"] message
Note that the event appender properly dereferenced the sd keys from the StructuredDataMessage.
However, the following does not work in OSGi with pax-logging:
org.ops4j.pax.logging.cfg:
log4j2.appender.console.type = Console
log4j2.appender.console.name = STDOUT
log4j2.appender.console.layout.type = PatternLayout
log4j2.appender.console.layout.pattern = %m%n
log4j2.appender.event.type = Console
log4j2.appender.event.name = event
log4j2.appender.event.layout.type = PatternLayout
log4j2.appender.event.layout.pattern = %marker \$\\\{sd:id\} \$\\\{sd:testKey\} %n %m%n
log4j2.rootLogger.level = debug
log4j2.rootLogger.appenderRef.console.ref = STDOUT
log4j2.logger.event.name = EventLogger
log4j2.logger.event.level = debug
log4j2.logger.event.appenderRef.console.ref = event
log4j2.logger.event.additivity = false
Test.java:
public class Test implements BundleActivator {
private static final Logger LOGGER = LogManager.getLogger(Test.class);
#Override
public void start(BundleContext context) throws Exception {
StructuredDataMessage msg = new StructuredDataMessage("1", "message", "event");
msg.put("testKey", "testValue");
LOGGER.info(msg);
EventLogger.logEvent(msg, Level.INFO);
}
#Override
public void stop(BundleContext context) throws Exception {
}
}
Output:
event [1 testKey="testValue"] message
EVENT ${sd:id} ${sd:testKey}
event [1 testKey="testValue"] message
Is there a trick to getting this to work in pax-logging? I am able to access values from the MDC using \$\\\{ctx:key\} when applicable, so I'm assuming the syntax is similar. I've also tried using the lookups in patterns for RoutingAppender, FileAppender, etc. to no avail.
Thanks in advance!
Edit: I'm using the latest version of pax-logging-api and pax-logging-log4j2 (1.11.3)

OK, it's not yet a definitive answer - simply comment is too short to describe what happens.
The stack trace with your call is:
"pipe-restart 238#10666" prio=5 tid=0xc3 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at org.ops4j.pax.logging.log4j2.internal.PaxLoggerImpl.doLog0(PaxLoggerImpl.java:354)
at org.ops4j.pax.logging.log4j2.internal.PaxLoggerImpl.doLog(PaxLoggerImpl.java:337)
at org.ops4j.pax.logging.log4j2.internal.PaxLoggerImpl.inform(PaxLoggerImpl.java:233)
at org.ops4j.pax.logging.internal.TrackingLogger.inform(TrackingLogger.java:209)
at org.ops4j.pax.logging.log4jv2.Log4jv2Logger.logMessage(Log4jv2Logger.java:162)
at org.apache.logging.log4j.spi.AbstractLogger.log(AbstractLogger.java:2102)
at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2190)
at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2144)
at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2127)
at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1828)
at org.apache.logging.log4j.EventLogger.logEvent(EventLogger.java:56)
at grgr.test.ActivatorLogging.start(ActivatorLogging.java:39)
...
org.ops4j.pax.logging.log4jv2.Log4jv2Logger.logMessage() is bridge between logging facade and logging backend.
Remember - with pax-logging, you can, say, use Commons Logging facade with Log4J1 backend, or Log4j2 facade (and that's what you're doing) with e.g., Logback backend.
That's why org.ops4j.pax.logging.log4jv2.Log4jv2Logger.logMessage() does this:
} else if (level.intLevel() >= Level.INFO.intLevel()) {
m_delegate.inform(paxMarker, message.getFormattedMessage(), t, fqcn);
and your structured message is changed into String event [1 testKey="testValue"] message.
Only then configured appenders are called - and the appender with layout that extract structured data can't find it, because the structured message is already converted to plain String.
These 3 lines:
at org.ops4j.pax.logging.log4j2.internal.PaxLoggerImpl.inform(PaxLoggerImpl.java:233)
at org.ops4j.pax.logging.internal.TrackingLogger.inform(TrackingLogger.java:209)
at org.ops4j.pax.logging.log4jv2.Log4jv2Logger.logMessage(Log4jv2Logger.java:162)
do the crossing from pax-logging-api (facade) through TrackingLogger (bridge) to pax-logging-log4j2 (backend) losing structured information in between.
I've created https://ops4j1.jira.com/browse/PAXLOGGING-302 and hope to do something about it soon.
EDIT1
The key is that in org.apache.logging.log4j.core.lookup.StructuredDataLookup#lookup(), this condition is true:
if (event == null || !(event.getMessage() instanceof StructuredDataMessage)) {
return null;
}
EDIT2
I've just fixed https://ops4j1.jira.com/browse/PAXLOGGING-302 and this test proves it works:
Logger logger = LogManager.getLogger("my.logger");
logger.info(new StructuredDataMessage("1", "hello!", "typeX").with("key1", "sd1"));
logger.info(new StringMapMessage().with("key1", "map1"));
List<String> lines = readLines();
assertTrue(lines.contains("my.logger/org.ops4j.pax.logging.it.Log4J2MessagesIntegrationTest typeX/sd1 [INFO] typeX [1 key1=\"sd1\"] hello!"));
assertTrue(lines.contains("my.logger/org.ops4j.pax.logging.it.Log4J2MessagesIntegrationTest ${sd:type}/map1 [INFO] key1=\"map1\""));
the configuration is:
log4j2.appender.console.type = Console
log4j2.appender.console.name = console
log4j2.appender.console.layout.type = PatternLayout
log4j2.appender.console.layout.pattern = %c/%C ${sd:type}/${map:key1} [%p] %m%n
log4j2.rootLogger.level = info
log4j2.rootLogger.appenderRef.file.ref = console
(you have to use the escape sequences you've used, if configuring via etc/org.ops4j.pax.logging.cfg in Karaf).

log4j => add caller file and line number to log

I use following pattern [%file:%line] %msg%n to output file + number to my log.
I as well use a simple wrapper class, that I call L.java. Now it does not make sense to output [L.java:74] Message... to my log. Instead, I would like to output the calling file name and line number...
Is that somehow possible with log4j?

The PatternLayout are slighty different between log4j 1.x and 2.x, and I do not know which version are you using, but I think you can't achieve this by configuration in neither versions.
You can achieve that programmatically (but this is going to affect your performance), I think in your L.java method you will have to use a method like:
private Logger logger = getYourLoggerAsYouAreCurrentlyDoing();
public enum LogLevel { INFO,DEBUG, ERROR, ETC }
void log(String msg, LogLevel level) {
StackTraceElement[] stackTraceElements = Thread.currentThread().getStackTrace();
String callerClass = stackTraceElements[1].getClassName();
String callerLine = "" + stackTraceElements[1].getLineNumber();
String msg = callerClass + ":" + callerLine + "-" + msg;
switch(LogLevel) {
case INFO: logger.info(msg); break;
case DEBUG: logger.debug(msg); break;
//etc.
}
}
And in case another method with the Throwable argument to log stacktraces.

GROOVY + Slf4j + rotation and backup log files

the following groovy script generate the C:\tmp\groovy.log log file by the Slf4j
groovy script works fine and log increased each time we use log.info/log.debug ... etc
but the problem is when the log file became with huge capacity and each day the log become more larger
I wonder how to add to my script the capability of the rotation mechanism
for example what I need is when the log file become 100K , then log will backup like zip file ( it will decrease the capacity for example to 10K )
is it possible ?
if yes please advice how to change the groovy script in order to give the backup capability
#Grab('org.slf4j:slf4j-api:1.6.1')
#Grab('ch.qos.logback:logback-classic:0.9.28')
import org.slf4j.*
import groovy.util.logging.Slf4j
import ch.qos.logback.core.*
import ch.qos.logback.classic.encoder.*
// Use annotation to inject log field into the class.
#Slf4j
class Family {
static {
new FileAppender().with {
name = 'file appender'
file = 'C:\\tmp\\groovy.log'
context = LoggerFactory.getILoggerFactory()
encoder = new PatternLayoutEncoder().with {
context = LoggerFactory.getILoggerFactory()
pattern = "%date{HH:mm:ss.SSS} [%thread] %-5level %logger{35} - %msg%n"
start()
it
}
start()
log.addAppender(it)
}
}
def father() {
log.debug 'car engine is hot'
log.error 'my car is stuck'
}
def mother() {
log.debug 'dont have a water in the kitchen'
log.error 'Cant make a cake'
}
}
def helloWorld = new Family()
helloWorld.father()
helloWorld.mother()

You can try with RollingFileAppender instead of FileAppender and set the RollingPolicy you need. i.e you can try with:
...
#Slf4j
class Family {
static {
new RollingFileAppender().with {
name = 'file appender'
file = 'C:\\tmp\\groovy.log'
// the policy to roll files
rollingPolicy = new TimeBasedRollingPolicy().with{
context = LoggerFactory.getILoggerFactory()
// file name pattern for the rolled files
fileNamePattern = 'C:\\tmp\\groovy.%date{yyyy-MM-dd}.%i.log'
// the maximum number of files to be keeped.
maxHistory = 10
timeBasedFileNamingAndTriggeringPolicy = new SizeAndTimeBasedFNATP().with{
context = LoggerFactory.getILoggerFactory()
// the max size of each rolled file
maxFileSize = '3MB'
}
}
context = LoggerFactory.getILoggerFactory()
encoder = new PatternLayoutEncoder().with {
context = LoggerFactory.getILoggerFactory()
pattern = "%date{HH:mm:ss.SSS} [%thread] %-5level %logger{35} - %msg%n"
start()
it
}
start()
log.addAppender(it)
}
}
...
Hope this helps,

Log4j : Creating/Modifying appenders at runtime, log file recreated and not appended

I want to create and enable an appender for a particular method call MyMethod(), whose log output is supposed to go to a file present at "logFilePath".
I do not want to include this appender in the xml configuration file, so I thought to create it at run time.
First, I tried to modify the logger properties at runtime and then calling activateOptions, eg. setting level to DEBUG before and setting it to Off in the finally block, so that output is logged only while the method is in use. That didn't work.
My problem here is that appender recreates a file everytime, and does not append to the same file. This is inspite of setAppend being true.
I am not very familiar with log4j, so please feel free to suggest an alternative approach.
The following is sample code to explain what I am trying.
private static FileAppender createNewAppender(String logFilePath) {
FileAppender appender = new FileAppender();
appender.setName("MyFileAppender");
appender.setLayout(new PatternLayout("%d %-5p [%c{1}] %m%n"));
appender.setFile(logFilePath);
appender.setAppend(true);
appender.setThreshold(Level.INFO);
appender.activateOptions();
Logger.getRootLogger().addAppender(appender);
return appender;
}
private static void removeAppender() {
Logger.getRootLogger().removeAppender(fileAppender) ; // ("MyFileAppender");
}
I call the above methods in the following way:
private static FileAppender fileAppender = null;
private static void myMethod(String logFilePath) {
try {
fileAppender = createNewAppender();
someOperation();
}
finally {
removeAppender();
fileAppender=null;
}
}

very easy just create a method and add this
String targetLog="where ever you want your log"
FileAppender apndr = new FileAppender(new PatternLayout("%d %-5p [%c{1}] %m%n"),targetLog,true);
logger.addAppender(apndr);
logger.setLevel((Level) Level.ALL);
then in any method you need to log just do this:
logger.error("your error here");

I do the following from scala (basically the same):
Set my root logging level to TRACE but set the threshold on my global appenders to info.
# Root logger option
log4j.rootLogger=TRACE, file, stdout
# log messages to a log file
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=./log.log
log4j.appender.file.MaxFileSize=100MB
log4j.appender.file.MaxBackupIndex=1
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{HH:mm:ss} %m%n
log4j.appender.file.Threshold=INFO
# log messages to stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{HH:mm:ss} %m%n
log4j.appender.stdout.Threshold=INFO
Then in the class I want to log:
private def set_debug_level(debug: String) {
import org.apache.log4j._
def create_appender(level: Level) {
val console_appender = new ConsoleAppender()
val pattern = "%d %p [%c,%C{1}] %m%n"
console_appender.setLayout(new PatternLayout(pattern))
console_appender.setThreshold(level)
console_appender.activateOptions()
Logger.getRootLogger().addAppender(console_appender)
}
debug match {
case "TRACE" => create_appender(Level.TRACE)
case "DEBUG" => create_appender(Level.DEBUG)
case _ => // just ignore other levels
}
}
So basically, since I set the threshold of my new appender to TRACE or DEBUG it will actually append. If I change the root to another level it will not log a lower level.