We Keep Coding

half-daily rollover in log4j2

Half-daily rollover for log4j2 not working for the given propertie file.
status = error
name = Proper
property.filepath = C\:/xi/doc/home/new/log
property.filename1 = ${filepath}/Common.log
filter.threshold.type = ThresholdFilter
filter.threshold.level = debug
appender.console.type = Console
appender.console.name = STDOUT
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss} %5p [%C].%M %m %n
appender.rolling1.type = RollingFile
appender.rolling1.name = RollingFile1
appender.rolling1.fileName = ${filename1}
appender.rolling1.filePattern = ${filepath}/Common.%d{yyyy-MM-dd-a}.%i.log.gz
appender.rolling1.layout.type = PatternLayout
appender.rolling1.layout.pattern = %d{yyyy-MM-dd HH:mm:ss} %5p [%C].%M %m %n
appender.rolling1.policies.type = Policies
appender.rolling1.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling1.policies.time.interval = 1
appender.rolling1.policies.time.modulate = true
appender.rolling1.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling1.policies.size.size = 100MB
appender.rolling1.strategy.type = DefaultRolloverStrategy
appender.rolling1.strategy.max = 20
appender.rolling1.strategy.delete.type = Delete
appender.rolling1.strategy.delete.basePath = ${basePath}
appender.rolling1.strategy.delete.maxDepth = 10
appender.rolling1.strategy.delete.ifLastModified.type = IfLastModified
# Delete all files older than 1 day
appender.rolling1.strategy.delete.ifLastModified.age = 1d
# loggers
logger.name = common
logger.level = debug
logger.additivity = false
logger.appenderRefs = rolling1
logger.appenderRefs.level = debug
logger.appenderRef.rolling1.ref = RollingFile1
The rollover works hourly and minutely if interval given and filepattern is given as ${filepath}/Common.%d{yyyy-MM-dd-HH} and ${filepath}/Common.%d{yyyy-MM-HH-minutely}
How to rollover half-daily? Is it supported in log4j2?

Log4j 1.x used to check at every message if the file pattern resolved to a different file name and rotated if it did.
However Log4j 2.x uses another approach. The date pattern is used twofold:
the date pattern %d{yyyy-MM-dd-a} is used to determine the rollover frequency (cf. source code), which in this case is days. This determines the time of the next rollover.
the date pattern is also used to determine the name of the file, which would contains AM in the morning and PM in the afternoon.
If you want two daily rotations you really need to add an hour specifier to the pattern: e.g. %d{yyyy-MM-dd-Ka} and set interval to 12.

log4j2 with pax-logging: can't use values from StructuredDataMessage

I'm using pax-logging-api along with pax-logging-log4j2 for logging from my OSGi bundles. I would like to utilize Log4J2's StructuredDataMessage (using EventLogger) to write some messages to a database. However, I'm unable to read the values I put into the StructuredDataMessage from the appenders when using Pax Logging.
The following works in a non-OSGi project using the Log4J2 libraries directly:
log4j2.properties:
appender.console.type = Console
appender.console.name = STDOUT
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %m%n
appender.event.type = Console
appender.event.name = event
appender.event.layout.type = PatternLayout
appender.event.layout.pattern = %marker ${sd:id} ${sd:testKey} %n %m%n
rootLogger.level = debug
rootLogger.appenderRef.console.ref = STDOUT
logger.event.name = EventLogger
logger.event.level = debug
logger.event.appenderRef.console.ref = event
logger.event.additivity = false
Test.java:
public class Test {
private static final Logger LOGGER = LogManager.getLogger(Test.class);
public static void main(String[] args) {
StructuredDataMessage msg = new StructuredDataMessage("1", "message", "event");
msg.put("testKey", "testValue");
LOGGER.info(msg);
EventLogger.logEvent(msg);
}
}
Output:
1 testValue event [1 testKey="testValue"] message
EVENT 1 testValue
event [1 testKey="testValue"] message
Note that the event appender properly dereferenced the sd keys from the StructuredDataMessage.
However, the following does not work in OSGi with pax-logging:
org.ops4j.pax.logging.cfg:
log4j2.appender.console.type = Console
log4j2.appender.console.name = STDOUT
log4j2.appender.console.layout.type = PatternLayout
log4j2.appender.console.layout.pattern = %m%n
log4j2.appender.event.type = Console
log4j2.appender.event.name = event
log4j2.appender.event.layout.type = PatternLayout
log4j2.appender.event.layout.pattern = %marker \$\\\{sd:id\} \$\\\{sd:testKey\} %n %m%n
log4j2.rootLogger.level = debug
log4j2.rootLogger.appenderRef.console.ref = STDOUT
log4j2.logger.event.name = EventLogger
log4j2.logger.event.level = debug
log4j2.logger.event.appenderRef.console.ref = event
log4j2.logger.event.additivity = false
Test.java:
public class Test implements BundleActivator {
private static final Logger LOGGER = LogManager.getLogger(Test.class);
#Override
public void start(BundleContext context) throws Exception {
StructuredDataMessage msg = new StructuredDataMessage("1", "message", "event");
msg.put("testKey", "testValue");
LOGGER.info(msg);
EventLogger.logEvent(msg, Level.INFO);
}
#Override
public void stop(BundleContext context) throws Exception {
}
}
Output:
event [1 testKey="testValue"] message
EVENT ${sd:id} ${sd:testKey}
event [1 testKey="testValue"] message
Is there a trick to getting this to work in pax-logging? I am able to access values from the MDC using \$\\\{ctx:key\} when applicable, so I'm assuming the syntax is similar. I've also tried using the lookups in patterns for RoutingAppender, FileAppender, etc. to no avail.
Thanks in advance!
Edit: I'm using the latest version of pax-logging-api and pax-logging-log4j2 (1.11.3)

OK, it's not yet a definitive answer - simply comment is too short to describe what happens.
The stack trace with your call is:
"pipe-restart 238#10666" prio=5 tid=0xc3 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at org.ops4j.pax.logging.log4j2.internal.PaxLoggerImpl.doLog0(PaxLoggerImpl.java:354)
at org.ops4j.pax.logging.log4j2.internal.PaxLoggerImpl.doLog(PaxLoggerImpl.java:337)
at org.ops4j.pax.logging.log4j2.internal.PaxLoggerImpl.inform(PaxLoggerImpl.java:233)
at org.ops4j.pax.logging.internal.TrackingLogger.inform(TrackingLogger.java:209)
at org.ops4j.pax.logging.log4jv2.Log4jv2Logger.logMessage(Log4jv2Logger.java:162)
at org.apache.logging.log4j.spi.AbstractLogger.log(AbstractLogger.java:2102)
at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2190)
at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2144)
at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2127)
at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1828)
at org.apache.logging.log4j.EventLogger.logEvent(EventLogger.java:56)
at grgr.test.ActivatorLogging.start(ActivatorLogging.java:39)
...
org.ops4j.pax.logging.log4jv2.Log4jv2Logger.logMessage() is bridge between logging facade and logging backend.
Remember - with pax-logging, you can, say, use Commons Logging facade with Log4J1 backend, or Log4j2 facade (and that's what you're doing) with e.g., Logback backend.
That's why org.ops4j.pax.logging.log4jv2.Log4jv2Logger.logMessage() does this:
} else if (level.intLevel() >= Level.INFO.intLevel()) {
m_delegate.inform(paxMarker, message.getFormattedMessage(), t, fqcn);
and your structured message is changed into String event [1 testKey="testValue"] message.
Only then configured appenders are called - and the appender with layout that extract structured data can't find it, because the structured message is already converted to plain String.
These 3 lines:
at org.ops4j.pax.logging.log4j2.internal.PaxLoggerImpl.inform(PaxLoggerImpl.java:233)
at org.ops4j.pax.logging.internal.TrackingLogger.inform(TrackingLogger.java:209)
at org.ops4j.pax.logging.log4jv2.Log4jv2Logger.logMessage(Log4jv2Logger.java:162)
do the crossing from pax-logging-api (facade) through TrackingLogger (bridge) to pax-logging-log4j2 (backend) losing structured information in between.
I've created https://ops4j1.jira.com/browse/PAXLOGGING-302 and hope to do something about it soon.
EDIT1
The key is that in org.apache.logging.log4j.core.lookup.StructuredDataLookup#lookup(), this condition is true:
if (event == null || !(event.getMessage() instanceof StructuredDataMessage)) {
return null;
}
EDIT2
I've just fixed https://ops4j1.jira.com/browse/PAXLOGGING-302 and this test proves it works:
Logger logger = LogManager.getLogger("my.logger");
logger.info(new StructuredDataMessage("1", "hello!", "typeX").with("key1", "sd1"));
logger.info(new StringMapMessage().with("key1", "map1"));
List<String> lines = readLines();
assertTrue(lines.contains("my.logger/org.ops4j.pax.logging.it.Log4J2MessagesIntegrationTest typeX/sd1 [INFO] typeX [1 key1=\"sd1\"] hello!"));
assertTrue(lines.contains("my.logger/org.ops4j.pax.logging.it.Log4J2MessagesIntegrationTest ${sd:type}/map1 [INFO] key1=\"map1\""));
the configuration is:
log4j2.appender.console.type = Console
log4j2.appender.console.name = console
log4j2.appender.console.layout.type = PatternLayout
log4j2.appender.console.layout.pattern = %c/%C ${sd:type}/${map:key1} [%p] %m%n
log4j2.rootLogger.level = info
log4j2.rootLogger.appenderRef.file.ref = console
(you have to use the escape sequences you've used, if configuring via etc/org.ops4j.pax.logging.cfg in Karaf).

error when start up tomcat with log4j: log4j:ERROR Could not find value for key log4j.appender

I have 2 issues with the log4j and maybe one causes the other.
when I tried to start up my tomcat I get this error:
log4j:ERROR Could not find value for key log4j.appender.rollingFile
I cannot insert the log line to DB.
log4j.properties file:
log4j.rootLogger=INFO, S, rollingFile, sql
log4j.appender.S =org.apache.log4j.ConsoleAppender
log4j.appender.S.layout =org.apache.log4j.PatternLayout
log4j.appender.S.layout.ConversionPattern = %d{dd MMM yyyy HH:mm:ss,SSS} %c{1} [%p] %m%n
log4j.appender.rollingFile = org.apache.log4j.DailyRollingFileAppender
log4j.appender.rollingFile.File =C:\\log.log
log4j.appender.rollingFile.Append = true
log4j.appender.rollingFile.MaxFileSize=2000KB
log4j.appender.rollingFile.MaxBackupIndex=19
log4j.appender.rollingFile.Threshold = ALL
log4j.appender.rollingFile.DatePattern = '.'yyy-MM-dd
log4j.appender.rollingFile.layout = org.apache.log4j.PatternLayout
log4j.appender.rollingFile.layout.ConversionPattern = %d{dd MMM yyyy HH:mm:ss,SSS} %c{1} [%p] %m%n
log4j.appender.sql=org.apache.log4j.jdbc.JDBCAppender
log4j.appender.sql.URL=jdbc:jtds:sqlserver://devsql/DBName
log4j.appender.sql.driver=net.sourceforge.jtds.jdbc.Driver
log4j.appender.sql.user=user_name
log4j.appender.sql.password=password
log4j.appender.sql.sql=INSERT INTO Log (logTypeId,logServiceId,hostId,logDateTime,logTitle,logModule,logDesc,logLineNumber,logStackTrace,userId) VALUES('%p','Project','172.0.12.123',now(),'%c{1}','%c{2}',N'%m','%L','%throwable{200}','%X{userId}')
log4j.appender.sql.layout=org.apache.log4j.PatternLayout
my class is:
import org.apache.log4j.Logger;
public class BasicApi {
private static final Logger LOG = Logger.getLogger(BasicApi.class.getName());
public ResponseEntity<String> runApi(InputBo input) {
LOG.info("!!!!!!!!!!!!!trying1!!!!!!!!!!!!!!");
LOG.debug("!!!!!!!!!!!!!trying2!!!!!!!!!!!!!!");
}
}
the error is:
log4j:ERROR Could not find value for key log4j.appender.rollingFile
log4j:ERROR Could not instantiate appender named "rollingFile".
important thing: this code is under maven jar project that another project adds it as a dependency.
can you please help me whats wrong in this code? Thanks

GROOVY + Slf4j + rotation and backup log files

the following groovy script generate the C:\tmp\groovy.log log file by the Slf4j
groovy script works fine and log increased each time we use log.info/log.debug ... etc
but the problem is when the log file became with huge capacity and each day the log become more larger
I wonder how to add to my script the capability of the rotation mechanism
for example what I need is when the log file become 100K , then log will backup like zip file ( it will decrease the capacity for example to 10K )
is it possible ?
if yes please advice how to change the groovy script in order to give the backup capability
#Grab('org.slf4j:slf4j-api:1.6.1')
#Grab('ch.qos.logback:logback-classic:0.9.28')
import org.slf4j.*
import groovy.util.logging.Slf4j
import ch.qos.logback.core.*
import ch.qos.logback.classic.encoder.*
// Use annotation to inject log field into the class.
#Slf4j
class Family {
static {
new FileAppender().with {
name = 'file appender'
file = 'C:\\tmp\\groovy.log'
context = LoggerFactory.getILoggerFactory()
encoder = new PatternLayoutEncoder().with {
context = LoggerFactory.getILoggerFactory()
pattern = "%date{HH:mm:ss.SSS} [%thread] %-5level %logger{35} - %msg%n"
start()
it
}
start()
log.addAppender(it)
}
}
def father() {
log.debug 'car engine is hot'
log.error 'my car is stuck'
}
def mother() {
log.debug 'dont have a water in the kitchen'
log.error 'Cant make a cake'
}
}
def helloWorld = new Family()
helloWorld.father()
helloWorld.mother()

You can try with RollingFileAppender instead of FileAppender and set the RollingPolicy you need. i.e you can try with:
...
#Slf4j
class Family {
static {
new RollingFileAppender().with {
name = 'file appender'
file = 'C:\\tmp\\groovy.log'
// the policy to roll files
rollingPolicy = new TimeBasedRollingPolicy().with{
context = LoggerFactory.getILoggerFactory()
// file name pattern for the rolled files
fileNamePattern = 'C:\\tmp\\groovy.%date{yyyy-MM-dd}.%i.log'
// the maximum number of files to be keeped.
maxHistory = 10
timeBasedFileNamingAndTriggeringPolicy = new SizeAndTimeBasedFNATP().with{
context = LoggerFactory.getILoggerFactory()
// the max size of each rolled file
maxFileSize = '3MB'
}
}
context = LoggerFactory.getILoggerFactory()
encoder = new PatternLayoutEncoder().with {
context = LoggerFactory.getILoggerFactory()
pattern = "%date{HH:mm:ss.SSS} [%thread] %-5level %logger{35} - %msg%n"
start()
it
}
start()
log.addAppender(it)
}
}
...
Hope this helps,

Log4j : Creating/Modifying appenders at runtime, log file recreated and not appended

I want to create and enable an appender for a particular method call MyMethod(), whose log output is supposed to go to a file present at "logFilePath".
I do not want to include this appender in the xml configuration file, so I thought to create it at run time.
First, I tried to modify the logger properties at runtime and then calling activateOptions, eg. setting level to DEBUG before and setting it to Off in the finally block, so that output is logged only while the method is in use. That didn't work.
My problem here is that appender recreates a file everytime, and does not append to the same file. This is inspite of setAppend being true.
I am not very familiar with log4j, so please feel free to suggest an alternative approach.
The following is sample code to explain what I am trying.
private static FileAppender createNewAppender(String logFilePath) {
FileAppender appender = new FileAppender();
appender.setName("MyFileAppender");
appender.setLayout(new PatternLayout("%d %-5p [%c{1}] %m%n"));
appender.setFile(logFilePath);
appender.setAppend(true);
appender.setThreshold(Level.INFO);
appender.activateOptions();
Logger.getRootLogger().addAppender(appender);
return appender;
}
private static void removeAppender() {
Logger.getRootLogger().removeAppender(fileAppender) ; // ("MyFileAppender");
}
I call the above methods in the following way:
private static FileAppender fileAppender = null;
private static void myMethod(String logFilePath) {
try {
fileAppender = createNewAppender();
someOperation();
}
finally {
removeAppender();
fileAppender=null;
}
}

very easy just create a method and add this
String targetLog="where ever you want your log"
FileAppender apndr = new FileAppender(new PatternLayout("%d %-5p [%c{1}] %m%n"),targetLog,true);
logger.addAppender(apndr);
logger.setLevel((Level) Level.ALL);
then in any method you need to log just do this:
logger.error("your error here");

I do the following from scala (basically the same):
Set my root logging level to TRACE but set the threshold on my global appenders to info.
# Root logger option
log4j.rootLogger=TRACE, file, stdout
# log messages to a log file
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=./log.log
log4j.appender.file.MaxFileSize=100MB
log4j.appender.file.MaxBackupIndex=1
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{HH:mm:ss} %m%n
log4j.appender.file.Threshold=INFO
# log messages to stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{HH:mm:ss} %m%n
log4j.appender.stdout.Threshold=INFO
Then in the class I want to log:
private def set_debug_level(debug: String) {
import org.apache.log4j._
def create_appender(level: Level) {
val console_appender = new ConsoleAppender()
val pattern = "%d %p [%c,%C{1}] %m%n"
console_appender.setLayout(new PatternLayout(pattern))
console_appender.setThreshold(level)
console_appender.activateOptions()
Logger.getRootLogger().addAppender(console_appender)
}
debug match {
case "TRACE" => create_appender(Level.TRACE)
case "DEBUG" => create_appender(Level.DEBUG)
case _ => // just ignore other levels
}
}
So basically, since I set the threshold of my new appender to TRACE or DEBUG it will actually append. If I change the root to another level it will not log a lower level.