We Keep Coding

how to add new log value to all logging statements

I am using slf4j with logback in my project.there is one request_id stored in a ThreadLoacal.
I want to add the value of this request id to all log statements.
Is there any way so that logger implicitly pick up the value of request_id and log it as well, without being pass it in existing log statements?

Slf4j and logback both supports the usage of a mapped diagnostic context (MDC). You can add named values to the MDC, which are passed to the logger. The logging pattern supports tokens for output.
Note that the MDC is stuck to your thread, i.e. with a different thread the context is lost. And with thread reusage, the context will reappear, so cleaning is important in such situations.

You should be able to do this using a custom Converter.
This Answer to another question shows how you can use a Converter to incorporate a thread identifier that has been saves in a thread-local variable. You should be able to adapt this to incorporate a request id.
This section of the logback manual describes how to hook a custom converter into your logback configs.

This can be tedious, But, I am sure it would do it
Firstly configure sl4j logging with the JDK logger.
Write your own formatter, or override the Java SimpleFormatter like so:
package com.mypackage
public class MyFormatter extends SimpleFormatter{
#Override
public String format(LogRecord record){
String simpleFormattedLog = super.format(record);
String simpleFormattedLogWithThreadID = "THREAD_ID(" + getThreadId() + ") _ " + simpleFormattedLog;
return simpleFormattedLogWithThreadID;
}
private String getThreadId(){
return "GET THREAD ID HERE";
}
}
Then specify the formatter in the properties file as
java.util.logging.ConsoleHandler.formatter = com.mypackage.MyFormatter
This should work, all things being equal.

IDs for log messages when using SLF4J

I would like to add a unique identifier to log statements, so I am able to add documentation (externally, e.g. a wiki) to every log statement, so a user can quickly access the message related documentation using the id. The logging framework I would like to use is SLF4J/logback.
I was not able to find documentation about related approaches except for some bits regarding auditing frameworks.
There is the Marker concept which I thought could be usable for ID injection, or I could just add the ID to the message text itself.
How would I add IDs to the logging statements "the right way"? Are there possibilities I didn't think of?
EDIT
The term unique ID just states there should be an identifier per log statement. A developer e.g. adds such an ID to a table/enum/whatever manually, which could be done wrong.
Such ID has to be stable, so documentation can be based on it. So the ID itself is not what I am wondering about.
My question is: what would be the right way of pushing the ID to the logger together with the message text? Would Markers be suited for this kind of requirement, should I embed the ID into the message text or is there some other possibility?
So, basically, would I use
logger.info(IDMarkers.DB_CONNECTION_FAILED, "no connection to the database");
or instead just
logger.info("[{}] no connection to the database", LogIDs.DB_CONNECTION_FAILED);
First approach has the advantage that showing the IDs is up to the logging system/its configuration.

Slf4j has http://www.slf4j.org/apidocs/org/slf4j/Marker.html
Unfortunately Markers are advertised for a different purpose. Still you can use them to uniquely mark logging statements.
More cumbersome solution is MDC:
MDC.put("MsgId", "EV-1234");
log.info()
MDC.remove("MsgId");
or with structural logging (requires v2.0.0):
logger.atDebug()
.addKeyValue("MsgId", "EV-1234")
.log("Temperature changed.");

Unique is only unique within some scope. Eventually, even every int or long value will be used.
So think about what "uniqueness" means to you. Then use a wrapper that will ensure your logging is handled with that id inserted.
Note that with slf4j you are dealing with an interface which will make a number of logging APIs consistent. This means you probably won't have the option to sub-class or even inject your implementation of the interface to ensure your consistent logging. Therefore you will be constrained to techniques like wrapping your logging API (preferably through the "consistent" interface).
package mypackage.log;
public class LoggerWrapper implements org.log4j.Logger {
private org.log4j.Logger logger;
public LoggerWrapper(org.log4j.Logger logger) {
this.logger = logger;
}
public String getUniqueId() {
return ...;
}
public void info(String message, Object params...) {
logger.info(String.format("[%d] %s", getUniqueId(), message), params));
}
... implement all the other methods ...
}
And this means that you will have to make your own LoggerFactory interface too
public Logger getLogger(String name) {
return new LoggerWrapper(org.sql4j.LoggerFactory(name));
}
While the code above has a few warts (not actually testing it); hopefully, it will give you an idea.

How to use the log4j2 wrapper API

I do have some trouble understanding the log4j2 wrapper usage.
If you follow this link you will find attached an example using the AbstractLoggerWrapper. I just copied the following peace of code.
public class Log4j2Logger extends AbstractLogger
{
private static final String FQCN = AbstractLogger.class.getName();
private AbstractLoggerWrapper logImpl;
public Log4j2Logger(String name, String prefix, String logId, String instanceId)
{
super(name, prefix, logId, instanceId);
final AbstractLogger logger = (AbstractLogger) LogManager.getLogger(name);
this.logImpl = new AbstractLoggerWrapper(logger, name);
}
....
#Override
public void log(String message, LogLevel level)
{
logImpl.log(null, FQCN, toImplLevel(level), new SimpleMessage(createMessage(message)), null);
}
....
}
I don't understand the reason for subclassing AbstractLogger and intern using the AbstractLoggerWrapper. I actually could just remove the extend from the Log4j2Logger and encapsulate the AbstractLoggerWrapper. Do you see any reason of doing it like in the code snipped above?
Is there any way to subclass the AbstractLogger (like preferred) and just use it without the wrapper? And create it like a strategy pattern? e.g.,
LogManager.getLogger( class.getName(), Log4j2Logger.class )
Maybe this is what they tried to explain in the extending section and I don't understand it, yet. Somebody any idea how to do it?
Sincerely
Christian
Update: I missed to say, the reason why I am using the wrapper is because of an existing projekt with log4j (1.2) with a wrapper.

If you look at the documentation for AbstractLoggerWrapper:
Wrapper class that exposes the protected AbstractLogger methods to
support wrapped loggers.
You will see a clear indication as to why it is done the way Apache does it. If you instead decide to ignore the contract of the interface and go your own way, you are essentially saying
"I don't care how the library does things, I know better"
As such, you are taking on a great deal of risk, instead of using the general solution that has been provided. I sincerely doubt that you have such an esoteric environment that the library could would be insufficient.
To recap, follow the contract laid out by the library documentation.

Making simple performance modifications to an already compiled jar?

Like many log4j users, we often have debug level logging that is expensive to evaluate. So we guard those cases with code like:
if( _logger.isDebugEnabled )
_logger.debug("Interesting, my foojes are goofed up: " + getFullDetails())
However, that is uglier than a plain _logger.debug call, and sometimes the programmer doesn't realize the evaluation could be expensive.
It seems like it should be fairly simple to write a program that takes a compiled jar and guards all the _logger.debug calls with the isDebugEnabled check. We would likely be willing to accept the extra overhead of checking isDebugEnabled in all cases.
Has anyone tried this approach, or done similar post-processing of a jar?

Rather than looking at modifying the jar, I'd search for a solution using Bytecode Instrumentation. The problem will be to identify those parts of the code you want to wrap inside a .isDebugEnabled() - you will have to identify objects that are only used for log4j invocations.

Have you looked at AspectJ ? This supports aspects using bytecode weaving, and can interceptions into a previously compiled .jar file.

I believe a good solution would be that the code would be efficient as is.
Consider that log4j is deprecated. Its author itself left it as is, to avoid breaking compatibility, but he created a new one, SLF4J (http://www.slf4j.org/ ). He provides both a facade and an implementation, according to the distinction commons-logging/log4j, but without the flaws of each...
I believe that, in this new logging facility, you can send Object parameters to the logging, and that the level is evaluated before converting the Objects (to String or otherwise). The idea is to use a format string, and parameters.
Our code doesn't use slf4j, but we have utility methods that do exactly that.
It is coded roughly as follow (from memory):
public enum LogLevel {
FATAL, ERROR, WARNING, INFO, DEBUG;
public void log(Logger logger, String format, Object... parameters) {
if (isEnabled(logger)) {
logImpl(logger, String.format(format, parameters));
}
}
public boolean isEnabled(Logger logger) {
switch(this) {
case WARNING : return logger.isWarningEnabled();
case INFO : return logger.isInfoEnabled();
case DEBUG : return logger.isDebugEnabled();
default: return true;
}
}
private void logImpl(Logger logger, String message) {
switch(this) {
case WARNING : logger.warn(message);
// other cases
}
}
}
It is used as:
public void myMethod(Object param) {
LogLevel.WARNING.log(LOGGER, "What is my message ....", "myMethod", param);
}
UPDATE : If you need to call a method in the log...
One possibility is to use toString method. This is appropriate if your logging is 'technical', and will be used also when debugging.
If your logging is more functional (not targeted to the developper), I suggest to define an interface (it is functionally sound in that case, so it is useful to provide meaning) :
public interface Detailable { // the name could also suggest logging?
String getFullDetails();
}
Implement that interface in any object that need to be passed as logging object, with a complex calculation to build the log.

Splitting log4j Output with Quartz Worker Threads

I'm working on an application that consists of an overall Quartz-based scheduler and "CycledJob" run using CronTriggers. The purpose of the application is to process inputs from different email inboxes based on the source country.
Based on the country that it comes in from (i.e. US, UK, FR, etc.) the application triggers one job thread to run each country's processing cycle, so there would be a UK Worker thread, one for US, France, etc. When formatting the output to log4j, I'm using the thread parameter, so it emits [ApplicationName_Worker-1], [ApplicationName_Worker-2] etc. Try as I might, I can't find a way to name the threads since they're pulled out of Quartz's Thread Pools. Although I could possibly go so far as to extend Quartz, I'd like to work out a different solution instead of messing with the standard library.
Here's the problem: When using log4j, I'd like to have all log items from the US thread output to a US only file, likewise for each of the country threads. I don't care if they stay in one unified ConsoleAppender, the FileAppender split is what I'm after here. I already know how to specify multiple file appenders and such, my issue is I can't differentiate based on country. There are 20+ classes within the application that can be on the execution chain, very few of which I want to burden with the knowledge of passing an extra "context" parameter through EVERY method... I've considered a Strategy pattern extending a log4j wrapper class, but unless I can let every class in the chain know which thread it's on to parameterize the logger call, that seems impossible. Without being able to name the thread also creates a challenge (or else this would be easy!).
So here's the question: What would be a suggested approach to allow many subordinate classes in an application that are each used for every different thread to process the input know that they are within the context of a particular country thread when they are logging?
Good luck understanding, and please ask clarifying questions! I hope someone is able to help me figure out a decent way to tackle this. All suggestions welcome.

At the top of each country's processing thread, put the country code into Log4j's mapped diagnostic context (MDC). This uses a ThreadLocal variable so that you don't have to pass the country up and down the call stack explicitly. Then create a custom filter that looks at the MDC, and filters out any events that don't contain the current appender's country code.
In your Job:
...
public static final String MDC_COUNTRY = "com.y.foo.Country";
public void execute(JobExecutionContext context)
/* Just guessing that you have the country in your JobContext. */
MDC.put(MDC_COUNTRY, context.get(MDC_COUNTRY));
try {
/* Perform your job here. */
...
} finally {
MDC.remove(MDC_COUNTRY);
}
}
...
Write a custom Filter:
package com.y.log4j;
import org.apache.log4j.spi.LoggingEvent;
/**
* This is a general purpose filter. If its "value" property is null,
* it requires only that the specified key be set in the MDC. If its
* value is not null, it further requires that the value in the MDC
* is equal.
*/
public final class ContextFilter extends org.apache.log4j.spi.Filter {
public int decide(LoggingEvent event) {
Object ctx = event.getMDC(key);
if (value == null)
return (ctx != null) ? NEUTRAL : DENY;
else
return value.equals(ctx) ? NEUTRAL : DENY;
}
private String key;
private String value;
public void setContextKey(String key) { this.key = key; }
public String getContextKey() { return key; }
public void setValue(String value) { this.value = value; }
public String getValue() { return value; }
}
In your log4j.xml:
<appender name="fr" class="org.apache.log4j.FileAppender">
<param name="file" value="france.log"/>
...
<filter class="com.y.log4j.ContextFilter">
<param name="key" value="com.y.foo.Country" />
<param name="value" value="fr" />
</filter>
</appender>

I wish I could be a bit more helpful than this, but you may want to investigate using some filters? Perhaps your logging could output the country code and you could match your filter based on that?
A StringMatchFilter should probably be able to match it for you.
Couldn't get the below address to work properly as a link, but if you look at it, it has some stuff on separate file logging using filters.
http://mail-archives.apache.org/mod_mbox/logging-log4j-user/200512.mbox/<1CC26C83B6E5AA49A9540FAC8D35158B01E2968E#pune.kaleconsultants.com > (just remove the space before the >)
http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/spi/Filter.html

Why not just call Thread.setName() when your job starts to set the name the Thread? If there is an access problem, configure quartz to use your own thread pool.

I may be completely off base on my understanding of what you are attempting to accomplish, but I will take a stab at the solution. It sounds like you want a separate log file for each country for which you are processing email. Based on that understanding, here is a possible solution:
Set up an appender in your log4j configuration for each country for which you wish to log separately (US example provided):
log4j.appender.usfile=org.apache.log4j.FileAppender
log4j.appender.usfile.File=us.log
log4j.appender.usfile.layout=org.apache.log4j.PatternLayout
log4j.appender.usfile.layout.ConversionPattern=%m%n
Create a logger for each country and direct each of them to the appropriate appender (US example provided):
log4j.logger.my-us-logger=debug,usfile
In your code, create your Logger based on the country for which the email is being processed:
Logger logger = Logger.getLogger("my-us-logger");
Determine how you will accomplish step 3 for the subsequent method calls. You could repeat step 3 in each class/method; or you could modify the method signatures to accept a Logger as input; or you could possibly use ThreadLocal to pass the Logger between methods.
Extra info: If you do not want the log statements going to parent loggers (e.g. the rootLogger), you can set their additivity flags to false (US example provided):
log4j.additivity.my-us-logger=false