We Keep Coding

log4j2 with pax-logging: can't use values from StructuredDataMessage

I'm using pax-logging-api along with pax-logging-log4j2 for logging from my OSGi bundles. I would like to utilize Log4J2's StructuredDataMessage (using EventLogger) to write some messages to a database. However, I'm unable to read the values I put into the StructuredDataMessage from the appenders when using Pax Logging.
The following works in a non-OSGi project using the Log4J2 libraries directly:
log4j2.properties:
appender.console.type = Console
appender.console.name = STDOUT
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %m%n
appender.event.type = Console
appender.event.name = event
appender.event.layout.type = PatternLayout
appender.event.layout.pattern = %marker ${sd:id} ${sd:testKey} %n %m%n
rootLogger.level = debug
rootLogger.appenderRef.console.ref = STDOUT
logger.event.name = EventLogger
logger.event.level = debug
logger.event.appenderRef.console.ref = event
logger.event.additivity = false
Test.java:
public class Test {
private static final Logger LOGGER = LogManager.getLogger(Test.class);
public static void main(String[] args) {
StructuredDataMessage msg = new StructuredDataMessage("1", "message", "event");
msg.put("testKey", "testValue");
LOGGER.info(msg);
EventLogger.logEvent(msg);
}
}
Output:
1 testValue event [1 testKey="testValue"] message
EVENT 1 testValue
event [1 testKey="testValue"] message
Note that the event appender properly dereferenced the sd keys from the StructuredDataMessage.
However, the following does not work in OSGi with pax-logging:
org.ops4j.pax.logging.cfg:
log4j2.appender.console.type = Console
log4j2.appender.console.name = STDOUT
log4j2.appender.console.layout.type = PatternLayout
log4j2.appender.console.layout.pattern = %m%n
log4j2.appender.event.type = Console
log4j2.appender.event.name = event
log4j2.appender.event.layout.type = PatternLayout
log4j2.appender.event.layout.pattern = %marker \$\\\{sd:id\} \$\\\{sd:testKey\} %n %m%n
log4j2.rootLogger.level = debug
log4j2.rootLogger.appenderRef.console.ref = STDOUT
log4j2.logger.event.name = EventLogger
log4j2.logger.event.level = debug
log4j2.logger.event.appenderRef.console.ref = event
log4j2.logger.event.additivity = false
Test.java:
public class Test implements BundleActivator {
private static final Logger LOGGER = LogManager.getLogger(Test.class);
#Override
public void start(BundleContext context) throws Exception {
StructuredDataMessage msg = new StructuredDataMessage("1", "message", "event");
msg.put("testKey", "testValue");
LOGGER.info(msg);
EventLogger.logEvent(msg, Level.INFO);
}
#Override
public void stop(BundleContext context) throws Exception {
}
}
Output:
event [1 testKey="testValue"] message
EVENT ${sd:id} ${sd:testKey}
event [1 testKey="testValue"] message
Is there a trick to getting this to work in pax-logging? I am able to access values from the MDC using \$\\\{ctx:key\} when applicable, so I'm assuming the syntax is similar. I've also tried using the lookups in patterns for RoutingAppender, FileAppender, etc. to no avail.
Thanks in advance!
Edit: I'm using the latest version of pax-logging-api and pax-logging-log4j2 (1.11.3)

OK, it's not yet a definitive answer - simply comment is too short to describe what happens.
The stack trace with your call is:
"pipe-restart 238#10666" prio=5 tid=0xc3 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at org.ops4j.pax.logging.log4j2.internal.PaxLoggerImpl.doLog0(PaxLoggerImpl.java:354)
at org.ops4j.pax.logging.log4j2.internal.PaxLoggerImpl.doLog(PaxLoggerImpl.java:337)
at org.ops4j.pax.logging.log4j2.internal.PaxLoggerImpl.inform(PaxLoggerImpl.java:233)
at org.ops4j.pax.logging.internal.TrackingLogger.inform(TrackingLogger.java:209)
at org.ops4j.pax.logging.log4jv2.Log4jv2Logger.logMessage(Log4jv2Logger.java:162)
at org.apache.logging.log4j.spi.AbstractLogger.log(AbstractLogger.java:2102)
at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2190)
at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2144)
at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2127)
at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1828)
at org.apache.logging.log4j.EventLogger.logEvent(EventLogger.java:56)
at grgr.test.ActivatorLogging.start(ActivatorLogging.java:39)
...
org.ops4j.pax.logging.log4jv2.Log4jv2Logger.logMessage() is bridge between logging facade and logging backend.
Remember - with pax-logging, you can, say, use Commons Logging facade with Log4J1 backend, or Log4j2 facade (and that's what you're doing) with e.g., Logback backend.
That's why org.ops4j.pax.logging.log4jv2.Log4jv2Logger.logMessage() does this:
} else if (level.intLevel() >= Level.INFO.intLevel()) {
m_delegate.inform(paxMarker, message.getFormattedMessage(), t, fqcn);
and your structured message is changed into String event [1 testKey="testValue"] message.
Only then configured appenders are called - and the appender with layout that extract structured data can't find it, because the structured message is already converted to plain String.
These 3 lines:
at org.ops4j.pax.logging.log4j2.internal.PaxLoggerImpl.inform(PaxLoggerImpl.java:233)
at org.ops4j.pax.logging.internal.TrackingLogger.inform(TrackingLogger.java:209)
at org.ops4j.pax.logging.log4jv2.Log4jv2Logger.logMessage(Log4jv2Logger.java:162)
do the crossing from pax-logging-api (facade) through TrackingLogger (bridge) to pax-logging-log4j2 (backend) losing structured information in between.
I've created https://ops4j1.jira.com/browse/PAXLOGGING-302 and hope to do something about it soon.
EDIT1
The key is that in org.apache.logging.log4j.core.lookup.StructuredDataLookup#lookup(), this condition is true:
if (event == null || !(event.getMessage() instanceof StructuredDataMessage)) {
return null;
}
EDIT2
I've just fixed https://ops4j1.jira.com/browse/PAXLOGGING-302 and this test proves it works:
Logger logger = LogManager.getLogger("my.logger");
logger.info(new StructuredDataMessage("1", "hello!", "typeX").with("key1", "sd1"));
logger.info(new StringMapMessage().with("key1", "map1"));
List<String> lines = readLines();
assertTrue(lines.contains("my.logger/org.ops4j.pax.logging.it.Log4J2MessagesIntegrationTest typeX/sd1 [INFO] typeX [1 key1=\"sd1\"] hello!"));
assertTrue(lines.contains("my.logger/org.ops4j.pax.logging.it.Log4J2MessagesIntegrationTest ${sd:type}/map1 [INFO] key1=\"map1\""));
the configuration is:
log4j2.appender.console.type = Console
log4j2.appender.console.name = console
log4j2.appender.console.layout.type = PatternLayout
log4j2.appender.console.layout.pattern = %c/%C ${sd:type}/${map:key1} [%p] %m%n
log4j2.rootLogger.level = info
log4j2.rootLogger.appenderRef.file.ref = console
(you have to use the escape sequences you've used, if configuring via etc/org.ops4j.pax.logging.cfg in Karaf).

GROOVY + Slf4j + rotation and backup log files

the following groovy script generate the C:\tmp\groovy.log log file by the Slf4j
groovy script works fine and log increased each time we use log.info/log.debug ... etc
but the problem is when the log file became with huge capacity and each day the log become more larger
I wonder how to add to my script the capability of the rotation mechanism
for example what I need is when the log file become 100K , then log will backup like zip file ( it will decrease the capacity for example to 10K )
is it possible ?
if yes please advice how to change the groovy script in order to give the backup capability
#Grab('org.slf4j:slf4j-api:1.6.1')
#Grab('ch.qos.logback:logback-classic:0.9.28')
import org.slf4j.*
import groovy.util.logging.Slf4j
import ch.qos.logback.core.*
import ch.qos.logback.classic.encoder.*
// Use annotation to inject log field into the class.
#Slf4j
class Family {
static {
new FileAppender().with {
name = 'file appender'
file = 'C:\\tmp\\groovy.log'
context = LoggerFactory.getILoggerFactory()
encoder = new PatternLayoutEncoder().with {
context = LoggerFactory.getILoggerFactory()
pattern = "%date{HH:mm:ss.SSS} [%thread] %-5level %logger{35} - %msg%n"
start()
it
}
start()
log.addAppender(it)
}
}
def father() {
log.debug 'car engine is hot'
log.error 'my car is stuck'
}
def mother() {
log.debug 'dont have a water in the kitchen'
log.error 'Cant make a cake'
}
}
def helloWorld = new Family()
helloWorld.father()
helloWorld.mother()

You can try with RollingFileAppender instead of FileAppender and set the RollingPolicy you need. i.e you can try with:
...
#Slf4j
class Family {
static {
new RollingFileAppender().with {
name = 'file appender'
file = 'C:\\tmp\\groovy.log'
// the policy to roll files
rollingPolicy = new TimeBasedRollingPolicy().with{
context = LoggerFactory.getILoggerFactory()
// file name pattern for the rolled files
fileNamePattern = 'C:\\tmp\\groovy.%date{yyyy-MM-dd}.%i.log'
// the maximum number of files to be keeped.
maxHistory = 10
timeBasedFileNamingAndTriggeringPolicy = new SizeAndTimeBasedFNATP().with{
context = LoggerFactory.getILoggerFactory()
// the max size of each rolled file
maxFileSize = '3MB'
}
}
context = LoggerFactory.getILoggerFactory()
encoder = new PatternLayoutEncoder().with {
context = LoggerFactory.getILoggerFactory()
pattern = "%date{HH:mm:ss.SSS} [%thread] %-5level %logger{35} - %msg%n"
start()
it
}
start()
log.addAppender(it)
}
}
...
Hope this helps,

Log4j file size

I am using log4j for logging of my Java project. I am creating three types of log files called "info","debug" and "error". I have set the option of max size of each file. But they are not limiting the size as we have set.
Code for debug file is -
public class DebugAppender extends RollingFileAppender {
/** Creates a new instance of DebugAppender */
public DebugAppender(String debuglogname, String path, String logfilesize) {
String Filename = null;
if (path == null || path.equalsIgnoreCase("")) {
Filename = "Logs" + File.separator + debuglogname + ".log";
} else {
Filename = path + File.separator + "Logs" + File.separator + debuglogname + ".log";
}
super.setFile(Filename);
setMaxFileSize(logfilesize);
setMaxBackupIndex(2);
setAppend(true);
setLayout(new PatternLayout("%d{DATE} %-5p %c{1} : %m%n"));
activateOptions();
}
public String getFName() {
return super.getFile();
}
}
Calling of Debug logger is -
Logger debugLogger = Logger.getLogger(LogHandler.class.getName() + "Debug");
DebugAppender objDebugAppender = new DebugAppender(debugFileName, sTempPath, "1MB");
debugLogger.addAppender(objDebugAppender);
We are setting its size to "1MB", but they all three files are not limiting to 1 MB, they get exceed to GBs. How can I limit their size. If any setting needed to set then please let me know.
Thanks

You have to set the file size of the log file through log4j properties file
log4j.appender.file.MaxFileSize=1MB
log4j.appender.file.MaxBackupIndex=1

In the xml format it would be
<param name="MaxFileSize" value="1MB" />
<param name="MaxBackupIndex" value="1" />

log4j to database

I used log4j to report error to database:
log4j.rootLogger=DEBUG, CUBRID
# CUBRID Database
log4j.appender.CUBRID = org.apache.log4j.jdbc.JDBCAppender
log4j.appender.CUBRID.driver = org.postgresql.Driver
log4j.appender.CUBRID.user = postgres
log4j.appender.CUBRID.password = postgres
log4j.appender.CUBRID.URL = jdbc:postgresql://localhost:5432/logs
log4j.appender.CUBRID.sql = INSERT INTO LOGS VALUES('%x','%d{yyyy-MM-dd HH:mm:ss.SSS}','%C','%p','%m')
log4j.appender.CUBRID.layout=org.apache.log4j.PatternLayout
I have a code:
public class LogTest extends BaseDAO<Object> {
public void show()
{
Logger log = Logger.getLogger(LogTest.class.getName());
log.info("Wystartowal");
log.warn("Trwanie aplikacji");
try {
if ( 1 / 0 == 0 ) {
System.out.println("Czekaj...");
}
} catch( Exception ex) {
log.error("Komunikat bledu", ex);
}
log.fatal("Koniec aplikacji");
}
}
All writes to the database but it saves me all the logs and I'd like to I would write only what I give myself using commands log.warn (), log.fatal (), etc.
How to do it?

This can be done by adjusting the logLevel. Try to change the value like this
log4j.rootLogger=WARN, CUBRID

You'll have to change the log level to something else than DEBUG.
log4j.rootLogger=WARN, CUBRID
This tutorial is very very comprehensive: Short introduction to log4j.
To only log your own info messages but allow other warnings that are >= WARN change to the following:
log4j.rootLogger=WARN, CUBRID
log4j.logger.LogTest = INFO, CUBRID
log4j.additivity.LogTest = false
# CUBRID Database
log4j.appender.CUBRID = org.apache.log4j.jdbc.JDBCAppender
log4j.appender.CUBRID.driver = org.postgresql.Driver
log4j.appender.CUBRID.user = postgres
log4j.appender.CUBRID.password = postgres
log4j.appender.CUBRID.URL = jdbc:postgresql://localhost:5432/logs
log4j.appender.CUBRID.sql = INSERT INTO LOGS VALUES('%x','%d{yyyy-MM-dd HH:mm:ss.SSS}','%C','%p','%m')
log4j.appender.CUBRID.layout=org.apache.log4j.PatternLayout
Here I am assuming that LogTest is not part of any package. If you have put it in a package then change the two lines to include the package (here I use my.package:
log4j.logger.my.package.LogTest = INFO, CUBRID
log4j.additivity.my.package.LogTest = false
Now if you want to apply the INFO level on all your loggers in your package then do this:
log4j.logger.my.package = INFO, CUBRID
log4j.additivity.my.package = false
Or even on top level package:
log4j.logger.my = INFO, CUBRID
log4j.additivity.my = false

Log4j : Creating/Modifying appenders at runtime, log file recreated and not appended

I want to create and enable an appender for a particular method call MyMethod(), whose log output is supposed to go to a file present at "logFilePath".
I do not want to include this appender in the xml configuration file, so I thought to create it at run time.
First, I tried to modify the logger properties at runtime and then calling activateOptions, eg. setting level to DEBUG before and setting it to Off in the finally block, so that output is logged only while the method is in use. That didn't work.
My problem here is that appender recreates a file everytime, and does not append to the same file. This is inspite of setAppend being true.
I am not very familiar with log4j, so please feel free to suggest an alternative approach.
The following is sample code to explain what I am trying.
private static FileAppender createNewAppender(String logFilePath) {
FileAppender appender = new FileAppender();
appender.setName("MyFileAppender");
appender.setLayout(new PatternLayout("%d %-5p [%c{1}] %m%n"));
appender.setFile(logFilePath);
appender.setAppend(true);
appender.setThreshold(Level.INFO);
appender.activateOptions();
Logger.getRootLogger().addAppender(appender);
return appender;
}
private static void removeAppender() {
Logger.getRootLogger().removeAppender(fileAppender) ; // ("MyFileAppender");
}
I call the above methods in the following way:
private static FileAppender fileAppender = null;
private static void myMethod(String logFilePath) {
try {
fileAppender = createNewAppender();
someOperation();
}
finally {
removeAppender();
fileAppender=null;
}
}

very easy just create a method and add this
String targetLog="where ever you want your log"
FileAppender apndr = new FileAppender(new PatternLayout("%d %-5p [%c{1}] %m%n"),targetLog,true);
logger.addAppender(apndr);
logger.setLevel((Level) Level.ALL);
then in any method you need to log just do this:
logger.error("your error here");

I do the following from scala (basically the same):
Set my root logging level to TRACE but set the threshold on my global appenders to info.
# Root logger option
log4j.rootLogger=TRACE, file, stdout
# log messages to a log file
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=./log.log
log4j.appender.file.MaxFileSize=100MB
log4j.appender.file.MaxBackupIndex=1
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{HH:mm:ss} %m%n
log4j.appender.file.Threshold=INFO
# log messages to stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{HH:mm:ss} %m%n
log4j.appender.stdout.Threshold=INFO
Then in the class I want to log:
private def set_debug_level(debug: String) {
import org.apache.log4j._
def create_appender(level: Level) {
val console_appender = new ConsoleAppender()
val pattern = "%d %p [%c,%C{1}] %m%n"
console_appender.setLayout(new PatternLayout(pattern))
console_appender.setThreshold(level)
console_appender.activateOptions()
Logger.getRootLogger().addAppender(console_appender)
}
debug match {
case "TRACE" => create_appender(Level.TRACE)
case "DEBUG" => create_appender(Level.DEBUG)
case _ => // just ignore other levels
}
}
So basically, since I set the threshold of my new appender to TRACE or DEBUG it will actually append. If I change the root to another level it will not log a lower level.