We Keep Coding

Temporarily increase log4j2 logger level in multi-threaded service

Here is a long question for you Log4j2 gurus.
I have a service that:
has very strict performance requirements
is instrumented with a lot of logging calls using log4j2.
A typical call is gated, like:
if ( LOG.isInfoEnabled() ) {
LOG.info("everything's fine");
}
Because of the number of log messages and the performance needs, the service will generally run with logging set to WARN (i.e., not many messages).
However, I have been asked to build in a parameter to the service call that, if given, will cause it to:
Temporarily increase the logging level to whatever was requested in the parameter (e.g., INFO or TRACE)
Add a WriterAppender to capture the logging in a PrintWriter.
Append the PrintWriter log data to the request response.
It seems clear, due to the gating I put around each logging call, that I need to actually increase the logging level temporarily, like this:
LoggerContext ctx = (LoggerContext) LogManager.getContext(false);
Configuration cfg = ctx.getConfiguration();
LoggerConfig loggerCfg = cfg.getLoggerConfig("com.mycompany.scm");
loggerCfg.setLevel(logLevel);
.. other code to add `WriterAppender` ...
ctx.updateLoggers();
But I have an immediate problem with that, in that it causes the logging to ALSO go the log file of the service. That might not be the end of the world, but I'd like to avoid that, if possible.
I did that by having the default logging go through appenders that filter by level, so that even if logging is turned on, it won't write any messages more detailed than are wanted in the default log file. (Like this, from my .properties file):
appenders=scm_warn, scm_info
appender.scm_warn.type = Console
appender.scm_warn.name = SCM_WARN
appender.scm_warn.layout.type = PatternLayout
appender.scm_warn.layout.pattern = %d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
appender.scm_warn.filter.threshold.type = ThresholdFilter
appender.scm_warn.filter.threshold.level = warn
appender.scm_info.type = Console
appender.scm_info.name = SCM_INFO
appender.scm_info.layout.type = PatternLayout
appender.scm_info.layout.pattern = %d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
appender.scm_info.filter.threshold.type = ThresholdFilter
appender.scm_info.filter.threshold.level = info
loggers = coreConfigurator
logger.coreConfigurator.name = com.mycompany.scm
logger.coreConfigurator.level = warn
logger.coreConfigurator.additivity = false # do not let configurator log messages get processed by client application's parent or root logger.
logger.coreConfigurator.appenderRefs = core
logger.coreConfigurator.appenderRef.core.ref = SCM_WARN
... that way, even if the logging level gets increased, the extra messages will not go to the main log file (I only want them to go to my PrintWriter).
And now, the question!
How can I temporarily increase the log level (like I try to do in the code above) for the current thread only?
If there are three (3) simultaneous calls to the service, I...
... want each added Appender to only write log messages generated by the thread that created the Appender.
... want each added Appender to be removed after the request that added it finishes.
... want the logging level to get reset back to what it was, as long as there are no other requests with this logging parameter turned on still in process.
Ideally, I think it sounds like I want each thread to have a completely separate logging context. Is that possible? Any thoughts on how to do all this?

You could potentially use a custom Context Selector to have a different context per thread, but that's probably cause issues when multiple threads want to write to the same log file, so likely not a viable option.
The alternative is to write a custom Appender, that uses a ThreadLocal to store the StringWriter. If a StringWriter has not been established for the thread, the appended will skip logging. This custom Appender should be added in the Log4J config file, so it's always there and receiving log entries.
That way you enable logging for a particular thread by creating and assigning a StringWriter to the ThreadLocal, run the code, then clear the ThreadLocal and get the logged information from the StringWriter. Since there initially is no StringWriter for any thread, the appender will do nothing, so shouldn't affect performance in any noticeable way.
You'd still have to do the level-escalation you're already doing, with filters on the other appenders.

You could:
access to class logger and change the log level (as you suggested):
LogManager.getLogger(Class.forName("your.class.package")).setLevel(Level.FATAL);
use a different logger; just configure two different loggers.

Poor performance of log4j2 in combination with Kafka

I'm having performance issues using log4j2 (2.5) in combination with Kafka (0.10.1.0). When I enable Kafka in the log4j2.xml file, my application slows down to a crawl, whilst only outputting around 200KB/s of events to the Kafka broker. This is orders of magnitude lower than that what Kafka is supposed to achieve (https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines).
Here is the relevant part of my log4j2.xml config file:
<Kafka name="KafkaAll" topic="all">
<PatternLayout pattern="%date %message" />
<Property name="bootstrap.servers">localhost:9092</Property>
<Property name="buffer.memory">67108864</Property>
<Property name="batch.size">8196</Property>
<Property name="acks">1</Property>
</Kafka>
After running some tests I've been able to pin down the problem and found out that the ProducerPerformance test shipped with Kafka does achieve decent performance. Its performance is around 5MB/s, with similar sized messages of 100 bytes. After extensive testing, I found out that the difference lies not in the configuration, but in the way the calls are implemented. The log4j2 KafkaAppender uses the KafkaManager class to write log to Kafka:
public void send(final byte[] msg) throws ExecutionException, InterruptedException, TimeoutException {
if (producer != null) {
producer.send(new ProducerRecord<byte[], byte[]>(topic, msg)).get(timeoutMillis, TimeUnit.MILLISECONDS);
}
}
The performance problem is caused by the call to the "get" method, which blocks until the send has completed. Funny enough, there is a log4j appender included with Kafka that does take this issue into account:
Future<RecordMetadata> response = producer.send(new ProducerRecord<byte[], byte[]>(topic, message.getBytes()));
if (syncSend) {
try {
response.get();
} catch (InterruptedException ex) {
throw new RuntimeException(ex);
} catch (ExecutionException ex) {
throw new RuntimeException(ex);
}
}
In other words, when syncSend is set to false, the call to send returns immediately. This "syncSend" property is however nowhere to be found in the log4j2 implementation of the KafkaAppender.
I've tried making calls asynchronous through other means, such as using the AsyncAppender shipped with log4j2 and setting the acks property to 0 instead of 1. However, none of the settings achieve the performance gain by not waiting for the send to complete. I've also tried to use the log4j appender shipped with Kafka, but I did not manage to get it to work with log4j2 (and I want to stick with log4j2).
So, finally, I have decided to fork the KafkaAppender shipped with log4j2 and remove the blocking part of the call. This works, but of course I'd rather be using of-the-shelf packages.
Is there anyone who also ran into this problem? How did you solve the issue? Is there an easier way without changing code?

The KafkaAppender now has a new attribute "syncSend" that can be set to support asynchronous sending, significantly improving throughput to Kafka. Thank you for your support!

How to customize logging for Google App Engine Java?

Google App Engine Java uses java.util.logging to create logging messages. I want to modify the log messages, that are displayed in Developers Console - Monitoring - Logs. The idea is to add some additional output like username without putting it in each log message manually:
log.info("user action");
should result in an logging output like
user "testuser": user action
Therefore I created an own Formatter:
public class TestFormatter extends Formatter {
#Override
public String format(LogRecord record) {
// find out username..
return "user " + username + ": " + record.getMessage();
}
}
Setting this as formatter for the ConsoleHandler in the logging.properties has not effekt:
java.util.logging.ConsoleHandler.formatter = com.example.guestbook.TestFormatter
When deploying it in on the local machine, and trying to add it programmatically like this:
Logger rootLogger = Logger.getLogger("");
Handler[] handlers = rootLogger.getHandlers();
log.info("Handler[] size: " + handlers.length);
for(Handler h : handlers) {
log.info(h.toString());
h.setFormatter(new TestFormatter());
}
I get 2 handler, one ConsoleHandler and one DevLogHandler. But setting the formatter results in the fact that no further logs are displayed. On GAE instead I get 0 handler.
When trying to acces Logger.getGlobal() instead of Logger.getLogger(""), I get 0 Handler on the local instance and a SecurityException: No permission to modify global on GAE. This exception already arises when trying to get the list of Handlers.
Now my question: Is there a way to modify the logs of developer console in such a way? If yes, how?

As I reply I got in the past from a Google ticket I opened for a similar question
I would discourage tampering with the Loggers/Handlers used
internally by GAE.
Besides that, the Global Logger cannot be customized, you can try to it with a Logger with a custom name

Log4j2 custom email subject from Map

I've some applications installed to my customers and I configured smtp appender to receive errors email.
Unfortunally I need a way to understand from which customer is arriving the email.
I'm trying to set a parameter in the map in order to show it as the subject of the email. I can set this parameter only after my app is started and the db is up:
String[] parametri = {username};
MapLookup.setMainArguments(parametri);
and my log4j2.xml is:
<SMTP name="Mailer" subject="${sys:logPath} - ${map:0}" to="${receipients}"
from="${from}" smtpHost="${smtpHost}" smtpPort="${smtpPort}"
smtpProtocol="${smtpProtocol}" smtpUsername="${smtpUser}"
smtpPassword="${smtpPassword}" smtpDebug="false" bufferSize="200"
ignoreExceptions="false">
</SMTP>
the subject is the relevant part. Unfortunally the subject is not replaced from log4j and remains as it is.
What I'm doing wrong?
Thanks

Currently, the SmtpAppender class (actually its helper SmtpManager) creates a MimeMessage object once and reuses it for all messages to be sent. The message subject is initialized only once. The lookup is done only once when your configuration is read.
I suggest you raise a feature request on the Log4j2 Jira issue tracker for your use case.

Note: log4j 2.6+ supports this natively; you need Java7+ for this.
I created a free useable solution for log4j2 and also Java6 with an ExtendedSmtpAppender supporting PatternLayout in subject.
If you still use log4j 1.x (original question), simply replace your log4j-1.x.jar with log4j-1.2-api-2.x.jar - and log4j-core-2.x.jar + log4j-api-2.x.jar of course.
You get it from Maven Central as de.it-tw:log4j2-extras (This requires Java 7+ and log4j 2.8+).
If you are restricted to Java 6 (and thus log4j 2.3) then use de.it-tw:log4j2-Java6-extras
See also the GitLab project: https://gitlab.com/thiesw/log4j2-extras (or https://gitlab.com/thiesw/log4j2-Java6-extras)
Additionally, it supports burst summarizing, so you will not get 1000 error emails within a few seconds or minutes. Use case: Send all ERROR-logs via Email to support/developer. On a broken network or database this can cause hundreds of the same error email.
This appender does the following:
the first occurrence is emailed immediately
all following similar ERROR logs are buffered for a certain time (similarity and time is configurable)
after the time passed, a summary email with summary info (number of events, time) and the first and last event is send
Example configuration (inside <Appenders>):
<SMTPx name="ErrorMail" smtpHost="mailer.xxxx.de" smtpPort="25"
from="your name <noReply#xxx.de>" to="${errorEmailAddresses}"
subject="[PROJECT-ID, ${hostName}, ${web:contextPath}] %p: %c{1} - %m%notEmpty{ =>%ex{short})}"
subjectWithLayout="true" bufferSize="5"
burstSummarizingSeconds="300" bsCountInSubject="S" bsMessageMaskDigits="true"
bsExceptionOrigin="true" >
<PatternLayout pattern="-- %d %p %c [%.20t,%x] %m%n" charset="UTF-8" /> <!-- SMTP uses fixed charset for message -->
</SMTPx>
<Async name="AsyncErrorMail" blocking="false" errorRef="Console">
<AppenderRef ref="ErrorMail"/>
</Async>
See also https://issues.apache.org/jira/browse/LOG4J2-1192.

Log4j: Events appear in the wrong logfile

To be able to log and trace some events I've added a LoggingHandler class to my java project. Inside this class I'm using two different log4j logger instances - one for logging an event and one for tracing an event into different files. The initialization block of the class looks like this:
public void initialize()
{
System.out.print("starting logging server ...");
// create logger instances
logLogger = Logger.getLogger("log");
traceLogger = Logger.getLogger("trace");
// create pattern layout
String conversionPattern = "%c{2} %d{ABSOLUTE} %r %p %m%n";
try
{
patternLayout = new PatternLayout();
patternLayout.setConversionPattern(conversionPattern);
}
catch (Exception e)
{
System.out.println("error: could not create logger layout pattern");
System.out.println(e);
System.exit(1);
}
// add pattern to file appender
try
{
logFileAppender = new FileAppender(patternLayout, logFilename, false);
traceFileAppender = new FileAppender(patternLayout, traceFilename, false);
}
catch (IOException e)
{
System.out.println("error: could not add logger layout pattern to corresponding appender");
System.out.println(e);
System.exit(1);
}
// add appenders to loggers
logLogger.addAppender(logFileAppender);
traceLogger.addAppender(traceFileAppender);
// set logger level
logLogger.setLevel(Level.INFO);
traceLogger.setLevel(Level.INFO);
// start logging server
loggingServer = new LoggingServer(logLogger, traceLogger, serverPort, this);
loggingServer.start();
System.out.println(" done");
}
To make sure that only only thread is using the functionality of a logger instance at the same time each logging / tracing method calls the logging method .info() inside a synchronized-block. One example looks like this:
public void logMessage(String message)
{
synchronized (logLogger)
{
if (logLogger.isInfoEnabled() && logFileAppender != null)
{
logLogger.info(instanceName + ": " + message);
}
}
}
If I look at the log files, I see that sometimes a event appears in the wrong file. One example:
trace 10:41:30,773 11080 INFO masterControl(192.168.2.21): string broadcast message was pushed from 1267093 to vehicle 1055293 (slaveControl 1)
trace 10:41:30,784 11091 INFO masterControl(192.168.2.21): string broadcast message was pushed from 1156513 to vehicle 1105792 (slaveControl 1)
trace 10:41:30,796 11103 INFO masterControl(192.168.2.21): string broadcast message was pushed from 1104306 to vehicle 1055293 (slaveControl 1)
trace 10:41:30,808 11115 INFO masterControl(192.168.2.21): vehicle 1327879 was pushed to slave control 1
10:41:30,808 11115 INFO masterControl(192.168.2.21): string broadcast message was pushed from 1101572 to vehicle 106741 (slaveControl 1)
trace 10:41:30,820 11127 INFO masterControl(192.168.2.21): string broadcast message was pushed from 1055293 to vehicle 1104306 (slaveControl 1)
I think that the problem occures everytime two event happen at the same time (here: 10:41:30,808). Does anybody has an idea how to solve my problem? I already tried to add a sleep() after the method call, but that doesn't helped ...
BR,
Markus
Edit:
logtrace 11:16:07,75511:16:07,755 1129711297 INFOINFO masterControl(192.168.2.21): string broadcast message was pushed from 1291400 to vehicle 1138272 (slaveControl 1)masterControl(192.168.2.21): vehicle 1333770 was added to slave control 1
or
log 11:16:08,562 12104 INFO 11:16:08,562 masterControl(192.168.2.21): string broadcast message was pushed from 117772 to vehicle 1217744 (slaveControl 1)
12104 INFO masterControl(192.168.2.21): vehicle 1169775 was pushed to slave control 1
Edit 2:
It seems like the problem only occurs if logging methods are called from inside a RMI thread (my client / server exchange information using RMI connections). ...
Edit 3:
I solved the problem by myself: It seems like log4j is NOT completely thread-save. After synchronizing all log / trace methods using a separate object everything is working fine. Maybe the lib is writing the messages to a thread-unsafe buffer before writing them to file?

You don't need to synchronize on your logger but on the output stream.
If you use log4j, the output should be correctly synchronized. The only way to get what you see is that two threads write to the same file at the same time.
Is it possible that you configured two appenders with the same output file? Don't do that; every appender must have it's own, distinct file name.
If you're 100% sure that every appender writes to a different file, the only option left is that you're sometimes using the wrong logger.

Could it be that somehow the initialize method is called more than once? Every call would add two new appenders that write into the same file as the existing appenders.

This question is 4 years old but I just came across this same problem in 2013 and I think I've fixed it by creating a new PatternLayout for each Appender. Hope it helps anyone else with the same issue in the future.