I am using slf4j with logback in my project.there is one request_id stored in a ThreadLoacal.
I want to add the value of this request id to all log statements.
Is there any way so that logger implicitly pick up the value of request_id and log it as well, without being pass it in existing log statements?
Slf4j and logback both supports the usage of a mapped diagnostic context (MDC). You can add named values to the MDC, which are passed to the logger. The logging pattern supports tokens for output.
Note that the MDC is stuck to your thread, i.e. with a different thread the context is lost. And with thread reusage, the context will reappear, so cleaning is important in such situations.
You should be able to do this using a custom Converter.
This Answer to another question shows how you can use a Converter to incorporate a thread identifier that has been saves in a thread-local variable. You should be able to adapt this to incorporate a request id.
This section of the logback manual describes how to hook a custom converter into your logback configs.
This can be tedious, But, I am sure it would do it
Firstly configure sl4j logging with the JDK logger.
Write your own formatter, or override the Java SimpleFormatter like so:
package com.mypackage
public class MyFormatter extends SimpleFormatter{
#Override
public String format(LogRecord record){
String simpleFormattedLog = super.format(record);
String simpleFormattedLogWithThreadID = "THREAD_ID(" + getThreadId() + ") _ " + simpleFormattedLog;
return simpleFormattedLogWithThreadID;
}
private String getThreadId(){
return "GET THREAD ID HERE";
}
}
Then specify the formatter in the properties file as
java.util.logging.ConsoleHandler.formatter = com.mypackage.MyFormatter
This should work, all things being equal.
Related
I would like to add a unique identifier to log statements, so I am able to add documentation (externally, e.g. a wiki) to every log statement, so a user can quickly access the message related documentation using the id. The logging framework I would like to use is SLF4J/logback.
I was not able to find documentation about related approaches except for some bits regarding auditing frameworks.
There is the Marker concept which I thought could be usable for ID injection, or I could just add the ID to the message text itself.
How would I add IDs to the logging statements "the right way"? Are there possibilities I didn't think of?
EDIT
The term unique ID just states there should be an identifier per log statement. A developer e.g. adds such an ID to a table/enum/whatever manually, which could be done wrong.
Such ID has to be stable, so documentation can be based on it. So the ID itself is not what I am wondering about.
My question is: what would be the right way of pushing the ID to the logger together with the message text? Would Markers be suited for this kind of requirement, should I embed the ID into the message text or is there some other possibility?
So, basically, would I use
logger.info(IDMarkers.DB_CONNECTION_FAILED, "no connection to the database");
or instead just
logger.info("[{}] no connection to the database", LogIDs.DB_CONNECTION_FAILED);
First approach has the advantage that showing the IDs is up to the logging system/its configuration.
Slf4j has http://www.slf4j.org/apidocs/org/slf4j/Marker.html
Unfortunately Markers are advertised for a different purpose. Still you can use them to uniquely mark logging statements.
More cumbersome solution is MDC:
MDC.put("MsgId", "EV-1234");
log.info()
MDC.remove("MsgId");
or with structural logging (requires v2.0.0):
logger.atDebug()
.addKeyValue("MsgId", "EV-1234")
.log("Temperature changed.");
Unique is only unique within some scope. Eventually, even every int or long value will be used.
So think about what "uniqueness" means to you. Then use a wrapper that will ensure your logging is handled with that id inserted.
Note that with slf4j you are dealing with an interface which will make a number of logging APIs consistent. This means you probably won't have the option to sub-class or even inject your implementation of the interface to ensure your consistent logging. Therefore you will be constrained to techniques like wrapping your logging API (preferably through the "consistent" interface).
package mypackage.log;
public class LoggerWrapper implements org.log4j.Logger {
private org.log4j.Logger logger;
public LoggerWrapper(org.log4j.Logger logger) {
this.logger = logger;
}
public String getUniqueId() {
return ...;
}
public void info(String message, Object params...) {
logger.info(String.format("[%d] %s", getUniqueId(), message), params));
}
... implement all the other methods ...
}
And this means that you will have to make your own LoggerFactory interface too
public Logger getLogger(String name) {
return new LoggerWrapper(org.sql4j.LoggerFactory(name));
}
While the code above has a few warts (not actually testing it); hopefully, it will give you an idea.
I currently use Log4J for a library and wanted to move to using SLF4J to allow the user of the library to choose the logging framework.
Everything is fine apart from one class that I use to replace text in the log message before it gets to screen or log:
public class FilteringLayout extends PatternLayout {
private static final String REPLACEMENT = "[APIKEY]";
private static Pattern replacementPattern = Pattern.compile("DO_NOT_MATCH");
public static void addReplacementString(String replacementString) {
replacementPattern = Pattern.compile(replacementString);
}
#Override
public String format(LoggingEvent event) {
if (event.getMessage() instanceof String) {
String message = event.getRenderedMessage();
Matcher matcher = replacementPattern.matcher(message);
if (matcher.find()) {
String maskedMessage = matcher.replaceAll(REPLACEMENT);
Throwable throwable = event.getThrowableInformation() != null
? event.getThrowableInformation().getThrowable() : null;
LoggingEvent maskedEvent = new LoggingEvent(event.fqnOfCategoryClass,
LoggerFactory.getLogger(event.getLoggerName()), event.timeStamp,
event.getLevel(), maskedMessage, throwable);
return super.format(maskedEvent);
}
}
return super.format(event);
}
}
How would I do this with pure SLF4J and remove the reliance on Log4J?
EDIT: Just to make it clearer, I want to remove a set sequence of characters from the log message automatically and before it's sent to the downstream logger.
I think you can achieve the same thing using SLF4J's Mapped Diagnostic Context feature. It works with both log4j and logback as backends. See http://logback.qos.ch/manual/mdc.html
Quote from SLF4J's manual:
"Mapped Diagnostic Context" is essentially a map maintained by the logging framework where the application can provided key-value pairs, which can then be inserted by the logging framework in log messages.
Edit:
Sorry, I haven't read your code carefully enough. I think SLF4J (being just a logging API) doesn't support filters and functionality like that by default. You could however implement a wrapper that processes the log messages before forwarding them. Extend LoggerWrapper and create a corresponding LoggerFactory that you could use throughout your codebase. See the sources of XLoggerFactory and XLogger in slf4j-ext as an example.
Is there anyway , or any configuration in log4j that tell him to log a line whenever some "logging level" configurations occur ?
I mean someone changed in some package the debug level from INFO to DEBUG , I want that event being logged by log4j.
Thanks
I'm not sure if it's your case, but if you are reloading the configuration file using configureAndWatch you should be able to see the information you need setting the system property -Dlog4j.debug=true.
See
http://logging.apache.org/log4j/1.2/apidocs/index.html?org/apache/log4j/xml/DOMConfigurator.html
There is a LogLog, the self logging of Log4J. I have never used it but if the solution exists it is there. I'd suggest you to download the source code of log4j and try to investigate it where it "knows" that the log level is being changed. Once you found the code see whether it prints something to self log and configure it as you need.
I'd be appreciate if you can post here more concrete answer if you find it. Good luck.
At runtime, if you have a servlet, jsp or a screen on your application where you are able to change log levels, you will most likely be doing something like this
public void changeLogLevel(String className, String logLevel){
Logger logger = Logger.getLogger(className);
Level level = Level.toLevel(logLevel);
logger.setLevel(level);
}
in order to log this event, all you would have to do is add an extra logger statement for this event
private static Logger classLogger = Logger.getLogger(ThisClass.class);
public void changeLogLevel(String className, String logLevel){
Logger logger = Logger.getLogger(className);
Level level = Level.toLevel(logLevel);
logger.setLevel(level);
classLogger.debug("The Level of " + className + " has changed to " + logLevel);
}
Then each time a log level occurs, you can log it here. If you want to get fancy, just send this log to its own file. You may want to advance the method even further to include an IP/username of the user who changed the log level.
If you have control over your application, ensure this is your only point in the application where a user can change your logging levels.
This doesn't answer your question, but the built-in java.util.logging.LogManager implementation has an addPropertyChangeListener() method that does exactly what you want.
I couldn't find anything comparable in Log4J, but I didn't look that hard...
I'm working on an application that consists of an overall Quartz-based scheduler and "CycledJob" run using CronTriggers. The purpose of the application is to process inputs from different email inboxes based on the source country.
Based on the country that it comes in from (i.e. US, UK, FR, etc.) the application triggers one job thread to run each country's processing cycle, so there would be a UK Worker thread, one for US, France, etc. When formatting the output to log4j, I'm using the thread parameter, so it emits [ApplicationName_Worker-1], [ApplicationName_Worker-2] etc. Try as I might, I can't find a way to name the threads since they're pulled out of Quartz's Thread Pools. Although I could possibly go so far as to extend Quartz, I'd like to work out a different solution instead of messing with the standard library.
Here's the problem: When using log4j, I'd like to have all log items from the US thread output to a US only file, likewise for each of the country threads. I don't care if they stay in one unified ConsoleAppender, the FileAppender split is what I'm after here. I already know how to specify multiple file appenders and such, my issue is I can't differentiate based on country. There are 20+ classes within the application that can be on the execution chain, very few of which I want to burden with the knowledge of passing an extra "context" parameter through EVERY method... I've considered a Strategy pattern extending a log4j wrapper class, but unless I can let every class in the chain know which thread it's on to parameterize the logger call, that seems impossible. Without being able to name the thread also creates a challenge (or else this would be easy!).
So here's the question: What would be a suggested approach to allow many subordinate classes in an application that are each used for every different thread to process the input know that they are within the context of a particular country thread when they are logging?
Good luck understanding, and please ask clarifying questions! I hope someone is able to help me figure out a decent way to tackle this. All suggestions welcome.
At the top of each country's processing thread, put the country code into Log4j's mapped diagnostic context (MDC). This uses a ThreadLocal variable so that you don't have to pass the country up and down the call stack explicitly. Then create a custom filter that looks at the MDC, and filters out any events that don't contain the current appender's country code.
In your Job:
...
public static final String MDC_COUNTRY = "com.y.foo.Country";
public void execute(JobExecutionContext context)
/* Just guessing that you have the country in your JobContext. */
MDC.put(MDC_COUNTRY, context.get(MDC_COUNTRY));
try {
/* Perform your job here. */
...
} finally {
MDC.remove(MDC_COUNTRY);
}
}
...
Write a custom Filter:
package com.y.log4j;
import org.apache.log4j.spi.LoggingEvent;
/**
* This is a general purpose filter. If its "value" property is null,
* it requires only that the specified key be set in the MDC. If its
* value is not null, it further requires that the value in the MDC
* is equal.
*/
public final class ContextFilter extends org.apache.log4j.spi.Filter {
public int decide(LoggingEvent event) {
Object ctx = event.getMDC(key);
if (value == null)
return (ctx != null) ? NEUTRAL : DENY;
else
return value.equals(ctx) ? NEUTRAL : DENY;
}
private String key;
private String value;
public void setContextKey(String key) { this.key = key; }
public String getContextKey() { return key; }
public void setValue(String value) { this.value = value; }
public String getValue() { return value; }
}
In your log4j.xml:
<appender name="fr" class="org.apache.log4j.FileAppender">
<param name="file" value="france.log"/>
...
<filter class="com.y.log4j.ContextFilter">
<param name="key" value="com.y.foo.Country" />
<param name="value" value="fr" />
</filter>
</appender>
I wish I could be a bit more helpful than this, but you may want to investigate using some filters? Perhaps your logging could output the country code and you could match your filter based on that?
A StringMatchFilter should probably be able to match it for you.
Couldn't get the below address to work properly as a link, but if you look at it, it has some stuff on separate file logging using filters.
http://mail-archives.apache.org/mod_mbox/logging-log4j-user/200512.mbox/<1CC26C83B6E5AA49A9540FAC8D35158B01E2968E#pune.kaleconsultants.com > (just remove the space before the >)
http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/spi/Filter.html
Why not just call Thread.setName() when your job starts to set the name the Thread? If there is an access problem, configure quartz to use your own thread pool.
I may be completely off base on my understanding of what you are attempting to accomplish, but I will take a stab at the solution. It sounds like you want a separate log file for each country for which you are processing email. Based on that understanding, here is a possible solution:
Set up an appender in your log4j configuration for each country for which you wish to log separately (US example provided):
log4j.appender.usfile=org.apache.log4j.FileAppender
log4j.appender.usfile.File=us.log
log4j.appender.usfile.layout=org.apache.log4j.PatternLayout
log4j.appender.usfile.layout.ConversionPattern=%m%n
Create a logger for each country and direct each of them to the appropriate appender (US example provided):
log4j.logger.my-us-logger=debug,usfile
In your code, create your Logger based on the country for which the email is being processed:
Logger logger = Logger.getLogger("my-us-logger");
Determine how you will accomplish step 3 for the subsequent method calls. You could repeat step 3 in each class/method; or you could modify the method signatures to accept a Logger as input; or you could possibly use ThreadLocal to pass the Logger between methods.
Extra info: If you do not want the log statements going to parent loggers (e.g. the rootLogger), you can set their additivity flags to false (US example provided):
log4j.additivity.my-us-logger=false
I'm configuring the logging for a Java application. What I'm aiming for is two logs: one for all messages and one for just messages above a certain level.
The app uses the java.util.logging.* classes: I'm using it as is, so I'm limited to configuration through a logging.properties file.
I don't see a way to configure two FileHandlers differently: the docs and examples I've seen set properties like:
java.util.logging.FileHandler.level = INFO
While I want two different Handlers logging at different levels to different files.
Any suggestions?
http://java.sun.com/j2se/1.4.2/docs/guide/util/logging/overview.html is helpful. You can only set one Level for any individual logger (as you can tell from the setLevel() method on the logger). However, you can take the lowest of the two common levels, and then filter programmatically.
Unfortunately, you can't do this just with the configuration file. To switch with just the configuration file you would have to switch to something like log4j, which you've said isn't an option.
So I would suggest altering the logging in code, with Filters, with something like this:
class LevelFilter implements Filter {
private Level Level;
public LevelFilter(Level level) {
this.level = level;
}
public boolean isLoggable(LogRecord record) {
return level.intValue() < record.getLevel().intValue();
}
}
And then on the second handler, do setFilter(new LevelFilter(Level.INFO)) or whatever. If you want it file configurable you could use a logging properties setting you've made up yourself, and use the normal Properties methods.
I think the configuration code for setting up the two file handlers ad the programmatic code is fairly simple once you have the design, but if you want more detail add a comment and I'll edit.
I think you should be able to just subclass a handler and then override the methods to allow output to go to multiple files depending on the level of the message. This would be done by overriding the publish() method.
Alternatively, if you have to use the system-provided FileHandler, you could do a setFilter() on it to inject your own filter into the mix and, in that filter code, send ALL messages to your other file and return true if the LogRecord level if INFO or higher, causing the FileHandler.publish() to write it to the real file.
I'm not sure this is the way you should be using filters but I can't see why it won't work.