I need to filter Log4j output to remove "sensitive" information like passwords from the log messages. The goal is to do something like this:
Replace:
05-Jan-2018 INFO [org.my.application] Username=Bob Password=myWeakPassword
With:
05-Jan-2018 INFO [org.my.application] Username=Bob Password=*********
This is fairly easy to do in Log4j V1, by extending the PatternLayout class:
public class CustomPatternLayout extends org.apache.log4j.PatternLayout {
#Override
public String format(LoggingEvent event) {
String temp = super.format(event);
return doFilteringStuff(temp);
}
}
However, in Log4j V2 the PatternLayout class was made "final" and the whole architecture was changed. There no longer seems to be a simple way to intercept/override the calls to the PatternLayout object. I looked at the Apache documentation but there's not much information.
I checked this question and this question but neither one has much help to offer.
I realize this is a very "general" question but does anyone know a straightforward way to do this in Log4j V2, or have any advice on this?
I think what you’re looking for is the RewriteAppender. From the manual:
The RewriteAppender allows the LogEvent to manipulated before it is processed by another Appender. This can be used to mask sensitive information such as passwords or to inject information into each event.
Please refer to this answer for a full example of using the RewriteAppender to mask sensitive content.
Related
We have a huge application using our custom RollingFileAppender (extended from org.apache.log4j.RollingFileAppender in log4j 1.x). We have overridden few methods, but two major methods are rollOver() and subAppend() as follows:
public void rollOver()
{
// Custom Code to specify how rolled-over files should be numbered and which file should be
// overwritten when MaxBackupIndex is reached
}
public void subAppend(LoggingEvent event)
{
// Custom Code to encrypt message before writing it
String data = this.layout.format(event);
if (isEncryptionOn())
{
data = PlatformEncryptDecrypt.encrypt2Way(data);
data = toUnicodeHexString(data);
}
this.qw.write(data);
.
.
.
// Other code taken from parent class
.
.
.
}
While migrating this to log4j2, I want to leverage log4j2's RollingFileAppender as much as possible, while overriding only selected methods. Since RollingFileAppender is final class, I would prefer to use builder for creating custom appender. I went through few other similar posts, but they are talking about logging few extra attributes with RollingFileAppender.
What would be the best way to achieve this using current latest version of log4j2 i.e. 2.13? Any help is highly appreciated.
If you are just wanting to encrypt the data then don't override the RollingFileAppender. Implement your own Layout instead. In fact, I would be open to adding an EncryptingLayout that wraps another Layout if you would create a Jira issue with your requirements.
In the meantime, just create a Layout that accepts another Layout as a configuration attribute. Then the methods in your Layout call that Layout's corresponding method and then encrypt the result.
I am using log4j2(version - 2.5) and I am trying write a message converter plugin which will mask some of the know patterns of the log message.
#Plugin(name = "CustomeMasking",
category = "Converter")
#ConverterKeys({"m"})
public class MyCustomFilteringLayout extends LogEventPatternConverter {
}
When I run my web application with this plugin then I see this warn message
WARN Converter key 'm' is already mapped to 'class
org.apache.logging.log4j.core.pattern.MessagePatternConverter'. Sorry,
Dave, I can't let you do that! Ignoring plugin [class
MyCustomFilteringLayout].
After exploring log4j2 site I have found these references.
Reference
If multiple Converters specify the same ConverterKeys, then the load
order above determines which one will be used. For example, to
override the %date converter which is provided by the built-in
DatePatternConverter class, you would need to place your plugin in a
JAR file in the CLASSPATH ahead of log4j-core.jar. This is not
recommended; pattern ConverterKeys collisions will cause a warning to
be emitted. Try to use unique ConverterKeys for your custom pattern
converters.
I need help to understand how can I write my custom converters for m/msg. Is there any better way to do it?
Additional Details:
I have created shaded jar for MyCustomFilteringLayout. Reason why I am doing this way is that I want to keep masking logic separate from application.
Updated
I have created converter for my own key which looks like this,
#Plugin(name = "CustomeMasking",
category = "Converter")
#ConverterKeys({"cm"})
public class MyCustomFilteringLayout extends LogEventPatternConverter {
}
Here I can't write another converter for same ConverterKeys - cm?
Now my log4j2.xml has this pattern layout,
<PatternLayout>
<Pattern>%d %p %c{1.} [%t] %cm %ex%n</Pattern>
</PatternLayout>
Your update solves the problem and answers the question how to replace the built-in message converter with a custom one. It needs a unique key.
Sounds like you want to parameterize your pattern. Many patterns take an options parameter. You can use this to control behavior, so specifying %cm{key1} in your layout pattern will produce different results than %cm{key2}.
For an example of a converter that takes parameters, see the source code of the MdcPatternConverter.
I am in the process of migrating my application from log4j 1.2 to log4j 2.0
I have existing code:
Enumeration appenders = logger.getAllAppenders();
.
.
.
fileBackupIndex = rollingFileAppender.getMaxBackupIndex();
In log4j 2.0 I could not find way to replace above java code. How to get list of all appenders and how to get the max value defined for RollingFile appender programatically?
With log4j2, there is a separation between API and CORE. This allows the team to make changes to the implementation without breaking client code.
So, if your code depends on implementation details, be aware that in the future this may change and your code may break.
That said, you can get a map of the appenders like this:
Logger logger = LogManager.getLogger();
Map<String, Appender> appenderMap =
((org.apache.logging.log4j.core.Logger) logger).getAppenders();
You can loop over the map until you find a RollingFileAppender. From this point it gets really ugly... The information you want is all in private fields, so you would need to use reflection to do the following:
get the fileAppender's "manager" field and cast it to RollingFileManager
get the manager's "strategy" field and cast it to DefaultRolloverStrategy
get the defaultRolloverStrategy's "maxIndex" field
This would obviously be pretty fragile... If you really need this you can request this feature on the log4j-dev mailing list or create a JIRA ticket. The quickest way to get this feature is if you provide a patch with the feature request.
I've added accessors for
minIndex, maxIndex, and compressionLevel.
triggeringPolicy and rolloverStrategy.
See our SVN trunk.
I do have some trouble understanding the log4j2 wrapper usage.
If you follow this link you will find attached an example using the AbstractLoggerWrapper. I just copied the following peace of code.
public class Log4j2Logger extends AbstractLogger
{
private static final String FQCN = AbstractLogger.class.getName();
private AbstractLoggerWrapper logImpl;
public Log4j2Logger(String name, String prefix, String logId, String instanceId)
{
super(name, prefix, logId, instanceId);
final AbstractLogger logger = (AbstractLogger) LogManager.getLogger(name);
this.logImpl = new AbstractLoggerWrapper(logger, name);
}
....
#Override
public void log(String message, LogLevel level)
{
logImpl.log(null, FQCN, toImplLevel(level), new SimpleMessage(createMessage(message)), null);
}
....
}
I don't understand the reason for subclassing AbstractLogger and intern using the AbstractLoggerWrapper. I actually could just remove the extend from the Log4j2Logger and encapsulate the AbstractLoggerWrapper. Do you see any reason of doing it like in the code snipped above?
Is there any way to subclass the AbstractLogger (like preferred) and just use it without the wrapper? And create it like a strategy pattern? e.g.,
LogManager.getLogger( class.getName(), Log4j2Logger.class )
Maybe this is what they tried to explain in the extending section and I don't understand it, yet. Somebody any idea how to do it?
Sincerely
Christian
Update: I missed to say, the reason why I am using the wrapper is because of an existing projekt with log4j (1.2) with a wrapper.
If you look at the documentation for AbstractLoggerWrapper:
Wrapper class that exposes the protected AbstractLogger methods to
support wrapped loggers.
You will see a clear indication as to why it is done the way Apache does it. If you instead decide to ignore the contract of the interface and go your own way, you are essentially saying
"I don't care how the library does things, I know better"
As such, you are taking on a great deal of risk, instead of using the general solution that has been provided. I sincerely doubt that you have such an esoteric environment that the library could would be insufficient.
To recap, follow the contract laid out by the library documentation.
I'm configuring the logging for a Java application. What I'm aiming for is two logs: one for all messages and one for just messages above a certain level.
The app uses the java.util.logging.* classes: I'm using it as is, so I'm limited to configuration through a logging.properties file.
I don't see a way to configure two FileHandlers differently: the docs and examples I've seen set properties like:
java.util.logging.FileHandler.level = INFO
While I want two different Handlers logging at different levels to different files.
Any suggestions?
http://java.sun.com/j2se/1.4.2/docs/guide/util/logging/overview.html is helpful. You can only set one Level for any individual logger (as you can tell from the setLevel() method on the logger). However, you can take the lowest of the two common levels, and then filter programmatically.
Unfortunately, you can't do this just with the configuration file. To switch with just the configuration file you would have to switch to something like log4j, which you've said isn't an option.
So I would suggest altering the logging in code, with Filters, with something like this:
class LevelFilter implements Filter {
private Level Level;
public LevelFilter(Level level) {
this.level = level;
}
public boolean isLoggable(LogRecord record) {
return level.intValue() < record.getLevel().intValue();
}
}
And then on the second handler, do setFilter(new LevelFilter(Level.INFO)) or whatever. If you want it file configurable you could use a logging properties setting you've made up yourself, and use the normal Properties methods.
I think the configuration code for setting up the two file handlers ad the programmatic code is fairly simple once you have the design, but if you want more detail add a comment and I'll edit.
I think you should be able to just subclass a handler and then override the methods to allow output to go to multiple files depending on the level of the message. This would be done by overriding the publish() method.
Alternatively, if you have to use the system-provided FileHandler, you could do a setFilter() on it to inject your own filter into the mix and, in that filter code, send ALL messages to your other file and return true if the LogRecord level if INFO or higher, causing the FileHandler.publish() to write it to the real file.
I'm not sure this is the way you should be using filters but I can't see why it won't work.