We Keep Coding

Spring Boot Logging override Colours

I wish to use different colours to distinguish between INFO, DEBUG and TRACE in Spring Ansi Coloured Logs, as they are currently all set to Green (see table below)
From the docs here https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-logging-color-coded-output
Color coding is configured by using the %clr conversion word. In its simplest form, the converter colors the output according to the log level, as shown in the following example:
%clr(%5p)
The following table describes the mapping of log levels to colors:
Level
Color
FATAL
Red
ERROR
Red
WARN
Yellow
INFO
Green
DEBUG
Green
TRACE
Green
It would appear I need to override the %clr conversion word, but I cannot find anything in the docs about that.
If it makes a difference I am using log4j2, and wish to build this into the application.

I don't see any conventional/documented way to override colors at a level scope. For example : ColorConverter for log4j2 doesn't like to be opened for that kind of options.
You could try to define your Log42 color plugin implementation, that is a plugin implementation annotating with that log4j2 Plugin annotation :
#Plugin(name = "color", category = PatternConverter.CATEGORY)
But not sure that it works or works reliably since Spring defines already one for that.
For the record here is the ColorConverter source code for logback.
By the way, if it is enough, you could define a pattern by starting from the CONSOLE_LOG_PATTERN defined in the Spring Boot source code for log4j2:
<Property name="CONSOLE_LOG_PATTERN">%clr{%d{${sys:LOG_DATEFORMAT_PATTERN}}}{faint} %clr{${sys:LOG_LEVEL_PATTERN}} %clr{%pid}{magenta} %clr{---}{faint} %clr{[%15.15t]}{faint} %clr{%-40.40c{1.}}{cyan} %clr{:}{faint} %m%n${sys:LOG_EXCEPTION_CONVERSION_WORD}</Property>

You can use GrepConsole plugin (Idea) for it.
See more here

First, the %clr defined is a logback conversion rule and is the one with the color logic.
This conversion rule is defined in the spring class DefaultLogbackConfiguration.
There we can see that spring adds the ColorConverter with the clr conversionWord in the logback configurator.
From this exploration, my conclusion is that you have to create your own logback.xml file (it disable spring default configuration, according to the documentation). With this file in place, you can add your own conversion rule like this:
<conversionRule conversionWord="nanos"
converterClass="chapters.layouts.MySampleConverter" />
After that you should change the property logging.pattern.console using your conversionWord created above, instead of %clr.

Log4j V2 Customizing Layout

I need to filter Log4j output to remove "sensitive" information like passwords from the log messages. The goal is to do something like this:
Replace:
05-Jan-2018 INFO [org.my.application] Username=Bob Password=myWeakPassword
With:
05-Jan-2018 INFO [org.my.application] Username=Bob Password=*********
This is fairly easy to do in Log4j V1, by extending the PatternLayout class:
public class CustomPatternLayout extends org.apache.log4j.PatternLayout {
#Override
public String format(LoggingEvent event) {
String temp = super.format(event);
return doFilteringStuff(temp);
}
}
However, in Log4j V2 the PatternLayout class was made "final" and the whole architecture was changed. There no longer seems to be a simple way to intercept/override the calls to the PatternLayout object. I looked at the Apache documentation but there's not much information.
I checked this question and this question but neither one has much help to offer.
I realize this is a very "general" question but does anyone know a straightforward way to do this in Log4j V2, or have any advice on this?

I think what you’re looking for is the RewriteAppender. From the manual:
The RewriteAppender allows the LogEvent to manipulated before it is processed by another Appender. This can be used to mask sensitive information such as passwords or to inject information into each event.
Please refer to this answer for a full example of using the RewriteAppender to mask sensitive content.

how to add new log value to all logging statements

I am using slf4j with logback in my project.there is one request_id stored in a ThreadLoacal.
I want to add the value of this request id to all log statements.
Is there any way so that logger implicitly pick up the value of request_id and log it as well, without being pass it in existing log statements?

Slf4j and logback both supports the usage of a mapped diagnostic context (MDC). You can add named values to the MDC, which are passed to the logger. The logging pattern supports tokens for output.
Note that the MDC is stuck to your thread, i.e. with a different thread the context is lost. And with thread reusage, the context will reappear, so cleaning is important in such situations.

You should be able to do this using a custom Converter.
This Answer to another question shows how you can use a Converter to incorporate a thread identifier that has been saves in a thread-local variable. You should be able to adapt this to incorporate a request id.
This section of the logback manual describes how to hook a custom converter into your logback configs.

This can be tedious, But, I am sure it would do it
Firstly configure sl4j logging with the JDK logger.
Write your own formatter, or override the Java SimpleFormatter like so:
package com.mypackage
public class MyFormatter extends SimpleFormatter{
#Override
public String format(LogRecord record){
String simpleFormattedLog = super.format(record);
String simpleFormattedLogWithThreadID = "THREAD_ID(" + getThreadId() + ") _ " + simpleFormattedLog;
return simpleFormattedLogWithThreadID;
}
private String getThreadId(){
return "GET THREAD ID HERE";
}
}
Then specify the formatter in the properties file as
java.util.logging.ConsoleHandler.formatter = com.mypackage.MyFormatter
This should work, all things being equal.

Splitting log4j Output with Quartz Worker Threads

I'm working on an application that consists of an overall Quartz-based scheduler and "CycledJob" run using CronTriggers. The purpose of the application is to process inputs from different email inboxes based on the source country.
Based on the country that it comes in from (i.e. US, UK, FR, etc.) the application triggers one job thread to run each country's processing cycle, so there would be a UK Worker thread, one for US, France, etc. When formatting the output to log4j, I'm using the thread parameter, so it emits [ApplicationName_Worker-1], [ApplicationName_Worker-2] etc. Try as I might, I can't find a way to name the threads since they're pulled out of Quartz's Thread Pools. Although I could possibly go so far as to extend Quartz, I'd like to work out a different solution instead of messing with the standard library.
Here's the problem: When using log4j, I'd like to have all log items from the US thread output to a US only file, likewise for each of the country threads. I don't care if they stay in one unified ConsoleAppender, the FileAppender split is what I'm after here. I already know how to specify multiple file appenders and such, my issue is I can't differentiate based on country. There are 20+ classes within the application that can be on the execution chain, very few of which I want to burden with the knowledge of passing an extra "context" parameter through EVERY method... I've considered a Strategy pattern extending a log4j wrapper class, but unless I can let every class in the chain know which thread it's on to parameterize the logger call, that seems impossible. Without being able to name the thread also creates a challenge (or else this would be easy!).
So here's the question: What would be a suggested approach to allow many subordinate classes in an application that are each used for every different thread to process the input know that they are within the context of a particular country thread when they are logging?
Good luck understanding, and please ask clarifying questions! I hope someone is able to help me figure out a decent way to tackle this. All suggestions welcome.

At the top of each country's processing thread, put the country code into Log4j's mapped diagnostic context (MDC). This uses a ThreadLocal variable so that you don't have to pass the country up and down the call stack explicitly. Then create a custom filter that looks at the MDC, and filters out any events that don't contain the current appender's country code.
In your Job:
...
public static final String MDC_COUNTRY = "com.y.foo.Country";
public void execute(JobExecutionContext context)
/* Just guessing that you have the country in your JobContext. */
MDC.put(MDC_COUNTRY, context.get(MDC_COUNTRY));
try {
/* Perform your job here. */
...
} finally {
MDC.remove(MDC_COUNTRY);
}
}
...
Write a custom Filter:
package com.y.log4j;
import org.apache.log4j.spi.LoggingEvent;
/**
* This is a general purpose filter. If its "value" property is null,
* it requires only that the specified key be set in the MDC. If its
* value is not null, it further requires that the value in the MDC
* is equal.
*/
public final class ContextFilter extends org.apache.log4j.spi.Filter {
public int decide(LoggingEvent event) {
Object ctx = event.getMDC(key);
if (value == null)
return (ctx != null) ? NEUTRAL : DENY;
else
return value.equals(ctx) ? NEUTRAL : DENY;
}
private String key;
private String value;
public void setContextKey(String key) { this.key = key; }
public String getContextKey() { return key; }
public void setValue(String value) { this.value = value; }
public String getValue() { return value; }
}
In your log4j.xml:
<appender name="fr" class="org.apache.log4j.FileAppender">
<param name="file" value="france.log"/>
...
<filter class="com.y.log4j.ContextFilter">
<param name="key" value="com.y.foo.Country" />
<param name="value" value="fr" />
</filter>
</appender>

I wish I could be a bit more helpful than this, but you may want to investigate using some filters? Perhaps your logging could output the country code and you could match your filter based on that?
A StringMatchFilter should probably be able to match it for you.
Couldn't get the below address to work properly as a link, but if you look at it, it has some stuff on separate file logging using filters.
http://mail-archives.apache.org/mod_mbox/logging-log4j-user/200512.mbox/<1CC26C83B6E5AA49A9540FAC8D35158B01E2968E#pune.kaleconsultants.com > (just remove the space before the >)
http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/spi/Filter.html

Why not just call Thread.setName() when your job starts to set the name the Thread? If there is an access problem, configure quartz to use your own thread pool.

I may be completely off base on my understanding of what you are attempting to accomplish, but I will take a stab at the solution. It sounds like you want a separate log file for each country for which you are processing email. Based on that understanding, here is a possible solution:
Set up an appender in your log4j configuration for each country for which you wish to log separately (US example provided):
log4j.appender.usfile=org.apache.log4j.FileAppender
log4j.appender.usfile.File=us.log
log4j.appender.usfile.layout=org.apache.log4j.PatternLayout
log4j.appender.usfile.layout.ConversionPattern=%m%n
Create a logger for each country and direct each of them to the appropriate appender (US example provided):
log4j.logger.my-us-logger=debug,usfile
In your code, create your Logger based on the country for which the email is being processed:
Logger logger = Logger.getLogger("my-us-logger");
Determine how you will accomplish step 3 for the subsequent method calls. You could repeat step 3 in each class/method; or you could modify the method signatures to accept a Logger as input; or you could possibly use ThreadLocal to pass the Logger between methods.
Extra info: If you do not want the log statements going to parent loggers (e.g. the rootLogger), you can set their additivity flags to false (US example provided):
log4j.additivity.my-us-logger=false

Separate INFO and ERROR logs from java.util.logging

I'm configuring the logging for a Java application. What I'm aiming for is two logs: one for all messages and one for just messages above a certain level.
The app uses the java.util.logging.* classes: I'm using it as is, so I'm limited to configuration through a logging.properties file.
I don't see a way to configure two FileHandlers differently: the docs and examples I've seen set properties like:
java.util.logging.FileHandler.level = INFO
While I want two different Handlers logging at different levels to different files.
Any suggestions?

http://java.sun.com/j2se/1.4.2/docs/guide/util/logging/overview.html is helpful. You can only set one Level for any individual logger (as you can tell from the setLevel() method on the logger). However, you can take the lowest of the two common levels, and then filter programmatically.
Unfortunately, you can't do this just with the configuration file. To switch with just the configuration file you would have to switch to something like log4j, which you've said isn't an option.
So I would suggest altering the logging in code, with Filters, with something like this:
class LevelFilter implements Filter {
private Level Level;
public LevelFilter(Level level) {
this.level = level;
}
public boolean isLoggable(LogRecord record) {
return level.intValue() < record.getLevel().intValue();
}
}
And then on the second handler, do setFilter(new LevelFilter(Level.INFO)) or whatever. If you want it file configurable you could use a logging properties setting you've made up yourself, and use the normal Properties methods.
I think the configuration code for setting up the two file handlers ad the programmatic code is fairly simple once you have the design, but if you want more detail add a comment and I'll edit.

I think you should be able to just subclass a handler and then override the methods to allow output to go to multiple files depending on the level of the message. This would be done by overriding the publish() method.
Alternatively, if you have to use the system-provided FileHandler, you could do a setFilter() on it to inject your own filter into the mix and, in that filter code, send ALL messages to your other file and return true if the LogRecord level if INFO or higher, causing the FileHandler.publish() to write it to the real file.
I'm not sure this is the way you should be using filters but I can't see why it won't work.