Based on https://logging.apache.org/log4j/2.x/manual/async.html I want to use the Mixing Synchronous and Asynchronous Loggers approach in order to benefit from some performance improvement over the all synchronous loggers.
The benchmark code:
public static void main(String[] args) {
org.apache.logging.log4j.Logger log4j2Logger = org.apache.logging.log4j.LogManager
.getLogger("com.foo.Bar");
long start = System.currentTimeMillis();
int nbLogMessages = 1 * 1000 * 1000;
for (int i = 0; i < nbLogMessages; i++) {
log4j2Logger.info("Log Message");
}
org.apache.logging.log4j.core.Logger coreLogger = (org.apache.logging.log4j.core.Logger) log4j2Logger;
org.apache.logging.log4j.core.LoggerContext context = (org.apache.logging.log4j.core.LoggerContext) coreLogger
.getContext();
Map<String, org.apache.logging.log4j.core.Appender> appenders = context
.getConfiguration().getAppenders();
for (org.apache.logging.log4j.core.Appender appender : appenders
.values()) {
appender.stop();
}
long elapsed = System.currentTimeMillis() - start;
System.out.println("Elapsed " + elapsed + "ms "
+ (nbLogMessages * 1000 / elapsed) + "logs/sec.");
}
The Log4j2 configuration is exactly the one in the doc (https://logging.apache.org/log4j/2.x/manual/async.html):
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
<Appenders>
<RandomAccessFile name="RandomAccessFile" fileName="/tmp/asyncWithLocation.log" immediateFlush="false" append="false">
<PatternLayout>
<Pattern>%d %p %class{1.} [%t] %location %m %ex%n</Pattern>
</PatternLayout>
</RandomAccessFile>
</Appenders>
<Loggers>
<!-- pattern layout actually uses location, so we need to include it -->
<AsyncLogger name="com.foo.Bar" level="trace" includeLocation="true">
<AppenderRef ref="RandomAccessFile"/>
</AsyncLogger>
<Root level="info" includeLocation="true">
<AppenderRef ref="RandomAccessFile"/>
</Root>
</Loggers>
</Configuration>
With that mixed sync/async logger config I get around 33,400 logs/sec.
Now if I replace the AsyncLogger "com.foo.Bar" with a regular Logger I get around 35,300 logs/sec.
How come the synchronous logger strategy is faster, according to the graphs it should have a way higher throughput no?
I've tried various other things like "warming up" the JVM before doing the benchmark like they recommend but it didn't help.
Note that if I set the property Log4jContextSelector to "org.apache.logging.log4j.core.async.AsyncLoggerContextSelector" to activate the "all async" loggers, I get around 86,400 logs/seconds. Unfortunately I can't use that option for other reasons.
Using log4j2-2.3, disruptor-3.3.2.
OS is Ubuntu, 8 cores.
Before you start measuring, I would recommend that you log a smaller number of messages, say 100,000 or so, then sleep for one second before starting the measured loop. (You don't want to measure log4j's initialization or the JVM optimization here.)
What are your performance results if you measure until after the logging loop finishes (before trying to stop the appenders)?
Also try setting includeLocation="false" in your config. This should make a big difference.
I don't think you should manually stop the appenders (see my answer to your other question).
Related
I want to print the event severity in sentence case, in the logs instead of default Uppercase. I have modified the log4j2 xml like below,
<Console name="STDOUT" target="SYSTEM_OUT" direct=true>
<PatternLayout pattern="%level{WARN=Warning, DEBUG=Debug, ERROR=Error, TRACE=Trace, INFO=Info}"/>
</Console>
<Loggers>
<AsyncRoot level="INFO" includeLocation="false">
<AppenderRef ref="STDOUT">
</AsyncRoot>
</Loggers>
Current Event Severity Printed in Logs :
INFO / WARNING / DEBUG / ERROR / TRACE
Expected Event Severity Printed in Logs :
Info/ Warning / Debug / Error / Trace
I still see the event is getting printed in Uppercase in logs. Something else need to be changed ?
According to my reading of the org.apache.logging.log4j.core.pattern.LevelPatternConverter source code, Log4j2 should output the level replacement strings exactly as you gave them in the pattern.
If that's not happening, check that Log4j2 is actually using that config.
If that doesn't resolve the problem, you may need to use a debugger to figure out what is going on.
This is a Java question.
I have around 100 static functions that use a slf4j logger.
I want to add some metadata to standardise the logs - assume it's some kind of preamble that is a function of what processing is currently going on.
How can I get the logger to print that metadata without going in to each of the static functions and changing them to explicitly add in the metadata.
e.g.
static Logger logger; ...
void mainProcessing() {
String file = "current_file.txt";
int line = 3;
...
func1();
func2();
...
}
void func1() {
...
logger.warn("some warning");
}
I'd like to see "WARN: File current_file.txt, line 3, msg: some warning" in the logs.
Any ideas?
(Prefer not to have to change each of the func1() functions obviously, if possible)
Thanks in advance.
You need to specify print format. However beware, that obtaining line number and/or file will greatly decrease your application performance. This is not C++, Logback will probably create a Throwable object for each line, to retrieve the stacktrace and extract the line number and file name.
Regarding line:
L / line Outputs the line number from where the logging request was
issued.
Generating the line number information is not particularly fast. Thus,
its use should be avoided unless execution speed is not an issue.
Regarding file:
F / file Outputs the file name of the Java source file where the
logging request was issued.
Generating the file information is not particularly fast. Thus, its
use should be avoided unless execution speed is not an issue.
http://logback.qos.ch/manual/layouts.html#file
Sample logback.xml configuration would be:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="console" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{"yy-MM-dd HH:mm:ss,SSS"}: %-5p %F:%L [%t] %c{0}: %M - %m%n</pattern>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="console" />
</root>
</configuration>
Better to redesign your code, that messages are more verbose and not unique, containing the context and required data to debug it by a person that does not know the code very well (your future co-worker)
I am new to Log4j2. I have written a simple program that actually logs data into RollingRandomAccessFile using log4j2. Below is the program:
public class Log4j2Example {
/**
* #param args the command line arguments
*/
public static Logger mlogger = null;
public static void main(String[] args) throws InterruptedException {
mlogger = LogManager.getLogger("Messages-log");
int i = 0;
while (true) {
String str = "Hello" + i;
System.out.println(str);
mlogger.info(str);
i++;
Thread.sleep(20);
}
}
}
My log4j2.xml file
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn">
<Appenders>
<RollingRandomAccessFile name="Messages-log" fileName="Log4J/Messages-${date:yyyy-MM-dd}.log"
immediateflush="true" filePattern="Log4J/Messages-%d{MM-dd-yyyy-HH}-%i.log.gz">
<PatternLayout>
<Pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} %p %m%n</Pattern>
</PatternLayout>
<Policies>
<TimeBasedTriggeringPolicy />
<SizeBasedTriggeringPolicy size="500 MB"/>
</Policies>
<DefaultRolloverStrategy max="50"/>
</RollingRandomAccessFile>
</Appenders>
<Loggers>
<Logger name="Messages-log" level="info" additivity="false">
<appender-ref ref="Messages-log" level="info"/>
</Logger>
<root level="info">
<appender-ref ref="Messages-log"/>
</root>
</Loggers>
</Configuration>
I am writing a simple statement into the log file sleeping for 20 milli-seconds for every record. Now the time stamp in the file should be for example:
If the first statement is logged at 17:20:32:354 then the next statement should be logged at 17:20:32:374 but it is logging at 17:20:32:384. An extra of 11 milli-seconds constantly is added for every record. Below is my log file output
2017-12-04 17:40:42.205 INFO Hello11
2017-12-04 17:40:42.236 INFO Hello12
2017-12-04 17:40:42.268 INFO Hello13
2017-12-04 17:40:42.299 INFO Hello14
2017-12-04 17:40:42.330 INFO Hello15
2017-12-04 17:40:42.361 INFO Hello16
2017-12-04 17:40:42.393 INFO Hello17
2017-12-04 17:40:42.424 INFO Hello18
You can see that first statement is logged at .205 milli second and the second statement is logged at .236 milli second. Infact I am sleeping the thread for 20 milliseconds so the correct timestamp should be .226 milli second. What am I doing wrong here? I need the exact time stamp to be written as it is very important in production. I have also tried this with log4j 1 but the same result. I have synchronized my System time with Internet time also. And one thing I found it worked perfectly with 5 and 15 milli-seconds of sleep but from 20 milli seconds this is causing a big issue.
What am I doing wrong here?
Mostly flawed assumptions and expectations:
Incorrect micro-benchmark.
Flawed assumption that System.currentTimeMillis has sufficient granularity.
Assuming Thread.sleep is accurate.
Assuming adjustments to the system clock won't change wake up time.
Trying to solve a realtime problem without using realtime software.
Not accounting for code logging framework.
Not accounting for appenders that perform I/O.
I don’t think that this is caused by logging. You can verify this by changing your test code to the below (to exclude the logging) and see if you are getting similar results:
public static void main(String[] args) throws InterruptedException {
SimpleDateFormat format = new SimpleDateFormat("HH:mm:ss.SSS");
int i = 0;
while (true) {
System.out.println(format.format(new Date()) + " Hello " + i);
i++;
Thread.sleep(20);
}
}
As Ralph pointed out in Log4j2-2141, it is likely that you are running into the granularity of the Windows scheduler. See WinAPI Sleep() function call sleeps for longer than expected and https://forum.sysinternals.com/bug-in-waitable-timers_topic16229.html.
Do any one knows how to add a empty line in slf4j logs without its formatting.
To get empty log I have added empty string as log.
log.error("");
I need to get it with out formatting, Just empty line, Please help.
In your logback.xml, declare a "minimal" appender that doesn't embellish the log message with timestamps or log levels or anything like so:
<appender name="minimal" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%m%n</pattern>
</encoder>
</appender>
Then declare a logger that uses that appender like so:
<logger name="blank.line.logger" level="ALL" additivity="false">
<appender-ref ref="minimal"/>
</logger>
In your Java code where you'd like to produce empty log lines, you'll need a reference to that logger separate from whatever other logger you're using for regular log messages, something like
private static final Logger blankLineLogger = LoggerFactory.getLogger("blank.line.logger");
Then you can get your desired blank line in the log output with
blankLineLogger.info("");
The accepted answser is a bit tricky, for me I just use a simple log.error("\n") , or append a \n in the previous log line.
Is it possible some how to use MDC to name the log file at run time.
I have a single web application which is being called by different names at the same time using tomcat docbase. So i need to have separate log files for each of them.
This can be accomplished in Logback, the successor to Log4J.
Logback is intended as a successor to the popular log4j project, picking up where log4j leaves off.
See the documentation for Sifting Appender
The SiftingAppender is unique in its capacity to reference and configure nested appenders. In the above example, within the SiftingAppender there will be nested FileAppender instances, each instance identified by the value associated with the "userid" MDC key. Whenever the "userid" MDC key is assigned a new value, a new FileAppender instance will be built from scratch. The SiftingAppender keeps track of the appenders it creates. Appenders unused for 30 minutes will be automatically closed and discarded.
In the example, they generate a separate log file for each user based on an MDC value.
Other MDC values could be used depending on your needs.
This is also possible with log4j. You can do this by implementing your own appender. I guess the easiest way is to
subclass AppenderSkeleton.
All logging events end up in the append(LoggingEvent event) method you have to implement.
In that method you can access the MDC by event.getMDC("nameOfTheKeyToLookFor");
Then you could use this information to open the file to write to.
It may be helpful to have a look at the implementation of the standard appenders like RollingFileAppender to figure out the rest.
I used this approach myself in an application to separate the logs of different threads into different log files and it worked very well.
I struggled for a while to find SiftingAppender-like functionality in log4j (we couldn't switch to logback because of some dependencies), and ended up with a programmatic solution that works pretty well, using an MDC and appending loggers at runtime:
// this can be any thread-specific string
String processID = request.getProcessID();
Logger logger = Logger.getRootLogger();
// append a new file logger if no logger exists for this tag
if(logger.getAppender(processID) == null){
try{
String pattern = "%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n";
String logfile = "log/"+processID+".log";
FileAppender fileAppender = new FileAppender(
new PatternLayout(pattern), logfile, true);
fileAppender.setName(processID);
// add a filter so we can ignore any logs from other threads
fileAppender.addFilter(new ProcessIDFilter(processID));
logger.addAppender(fileAppender);
}catch(Exception e){
throw new RuntimeException(e);
}
}
// tag all child threads with this process-id so we can separate out log output
MDC.put("process-id", processID);
//whatever you want to do in the thread
LOG.info("This message will only end up in "+processID+".log!");
MDC.remove("process-id");
The filter appended above just checks for a specific process id:
public class RunIdFilter extends Filter {
private final String runId;
public RunIdFilter(String runId) {
this.runId = runId;
}
#Override
public int decide(LoggingEvent event) {
Object mdc = event.getMDC("run-id");
if (runId.equals(mdc)) {
return Filter.ACCEPT;
}
return Filter.DENY;
}
}
Hope this helps a bit.
As of 20-01-2020, this is now a default functionality of Log4j.
To achieve that you just need to use a RoutingAppender with MDC.
Example:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN" monitorInterval="30">
<Appenders>
<Routing name="Analytics" ignoreExceptions="false">
<Routes>
<Script name="RoutingInit" language="JavaScript"><![CDATA[
// This script must return a route name
//
// Example from https://logging.apache.org/log4j/2.x/manual/appenders.html#RoutingAppender
// on how to get a MDC value
// logEvent.getContextMap().get("event_type");
//
// but as we use only one route with dynamic name, we return 1
1
]]>
</Script>
<Route>
<RollingFile
name="analytics-${ctx:event_type}"
fileName="logs/analytics/${ctx:event_type}.jsonl"
filePattern="logs/analytics/$${date:yyyy-MM}/analytics-${ctx:event_type}-%d{yyyy-dd-MM-}-%i.jsonl.gz">
<PatternLayout>
<pattern>%m%n</pattern>
</PatternLayout>
<Policies>
<TimeBasedTriggeringPolicy/>
<SizeBasedTriggeringPolicy size="250 MB"/>
</Policies>
</RollingFile>
</Route>
</Routes>
<!-- Created appender TTL -->
<IdlePurgePolicy timeToLive="15" timeUnit="minutes"/>
</Routing>
</Appenders>
<Loggers>
<Logger name="net.bytle.api.http.AnalyticsLogger" level="debug" additivity="false">
<AppenderRef ref="Analytics"/>
</Logger>
</Loggers>
</Configuration>
To known more, see Log4j - How to route message to log file created dynamically.