We Keep Coding

Why are my logs being removed?

I have an app in java using log4j as my logging jar, but when I try to debug my app and open the log file I find it short and with only the last lines I logged and the rotation files don't show any of the logs that I have seen before.
Is there something wrong with my configuration file? I have multiple projects pointing to the same log file, Could this be the problem?
Here is my log4j.properties:
log4j.rootLogger=ERROR,logfile
log4j.appender.logfile=org.apache.log4j.RollingFileAppender
log4j.appender.logfile.File=/opt/tomcat7/logs/mylogs.log
log4j.appender.logfile.MaxFileSize=500KB
# Keep one backup file
log4j.appender.logfile.MaxBackupIndex=2
log4j.appender.logfile.layout=org.apache.log4j.PatternLayout
log4j.appender.logfile.layout.ConversionPattern=%d %5p (%F:%L) - %m%n
Thanks

You specified level of root logger to be ERROR, while in most cases there won't be a lot of ERROR messages - maybe one-two per many INFO level messages, while you suppressed them: ERROR level allows only ERROR and FATAL messages to be logged. And it filders out any INFO, WARN, DEBUG, TRACE level. Usually you don't use such setup in development. And in production you usually suppress only some verbose classes/packages, but not everything like in config you posted (meaning here ERROR level applies to every class).
So I suppose you should replace ERROR with INFO in your config.
UPDATE: Also likely log4j simply doesn't see your config file. I encountered this issues several times personally and each time I had a workaround like this:
public class Log4JProperties {
public static void setupLog4j(){
String log4jPathFile = Log4JProperties.class.getResource("/log4j.properties").getFile();
Properties props = new Properties();
try {
InputStream configStream = new FileInputStream(log4jPathFile);
props.load(configStream);
configStream.close();
} catch (IOException e) {
System.out.println("log4j configuration file not found");
}
LogManager.resetConfiguration();
PropertyConfigurator.configure(props);
}
}
This is not necessary when I run code from inside Intellij, but when I run using Maven, logging doesn't work without calling this function in the beginning of the execution.

When you specify MaxBackupIndex=2 it will only keep last 2 backups, if log file fills up quickly old backup files will be overridden.
You might have to increase the MaxFileSize or MaxBackupIndex based on your file logging behaviour…
https://logging.apache.org/log4php/docs/appenders/rolling-file.html

log4j Appender additivity for custom log level

I have a scenario where I have a separate custom level defined XYZLogLevel for logging and I have 2 rolling file appenders where 2nd appender is specifically reserved to log log messages from XYZLogLevel. However the logs are going in both the files which is undesirable.
Note:-
Logs are not package specific, so adding additivity for package wont
work.
Everything has to be done through log4j.properties file.
Adding LevelRangeFilter to first appender partially resolves it, but
when I add 3rd appender for another custom level, then I again see
duplication in 2nd and 3rd appender.
log4j.rootCategory=ERROR, F, XYZLOG, LMNLOG
log4j.appender.F=org.apache.log4j.RollingFileAppender
log4j.appender.F.File=f_log.log
log4j.appender.F.MaxFileSize=5MB
log4j.appender.F.MaxBackupIndex=10
log4j.appender.F.layout = org.apache.log4j.PatternLayout
log4j.appender.F.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss.SSS} %c{1} [%p] %m%n
log4j.appender.XYZLOG=org.apache.log4j.RollingFileAppender
log4j.appender.XYZLOG.File=xyz_reporting.log
log4j.appender.XYZLOG.threshold=XYZLOG#com.services.domain.xyzlogs.XYZLogLevel
log4j.appender.XYZLOG.filter.a=org.apache.log4j.varia.LevelMatchFilter
log4j.appender.XYZLOG.filter.a.LevelToMatch=XYZLOG#com.services.domain.xyzlogs.XYZLogLevel
log4j.appender.XYZLOG.filter.a.AcceptOnMatch=true
log4j.appender.XYZLOG.MaxFileSize=100KB
log4j.appender.XYZLOG.MaxBackupIndex=10
log4j.appender.XYZLOG.layout=org.apache.log4j.PatternLayout
log4j.appender.XYZLOG.layout.ConversionPattern=%m%n
log4j.appender.LMNLOG=org.apache.log4j.RollingFileAppender
log4j.appender.LMNLOG.File=lmn_reporting.log
log4j.appender.LMNLOG.threshold=LMNLOG#com.services.domain.lmnlogs.LMNLogLevel
log4j.appender.LMNLOG.filter.a=org.apache.log4j.varia.LevelMatchFilter
log4j.appender.LMNLOG.filter.a.LevelToMatch=LMNLOG#com.services.domain.lmnlogs.LMNLogLevel
log4j.appender.LMNLOG.filter.a.AcceptOnMatch=true
log4j.appender.LMNLOG.MaxFileSize=100KB
log4j.appender.LMNLOG.MaxBackupIndex=10
log4j.appender.LMNLOG.layout=org.apache.log4j.PatternLayout
log4j.appender.LMNLOG.layout.ConversionPattern=%m%n
Maybe this can be solved through org.apache.log4j.varia.DenyAllFilter but somehow I was not able to configure that for properties file. I believe that functionality is only for XML configuration.
Its a very tricky sitution as I dont have the liberty to switch to XML configuration, any help on this would be appreciated guys.
Link and examples would be great as I understand them quickly.

I am answering my own question.
First you need to create a logger and set additivity as FALSE.
log4j.logger.XZYLOG=XZYLOG#XYZLOG#com.services.domain.xyzlogs.XYZLogLevel, XYZAPPENDER
log4j.additivity.XZYLOG=false
log4j.logger.LMNLOG=LMNLOG#XYZLOG#com.services.domain.lmnlogs.LMNLogLevel, LMNAPPENDER
log4j.additivity.LMNLOG=false
Then configure the appenders in following way.
log4j.appender.XYZLOGAPPENDER=org.apache.log4j.RollingFileAppender
log4j.appender.XYZLOGAPPENDER.File=xyz_reporting.log
log4j.appender.XYZLOGAPPENDER.threshold=XYZLOG#com.services.domain.xyzlogs.XYZLogLevel
log4j.appender.XYZLOGAPPENDER.MaxFileSize=100KB
log4j.appender.XYZLOGAPPENDER.MaxBackupIndex=10
log4j.appender.XYZLOGAPPENDER.layout=org.apache.log4j.PatternLayout
log4j.appender.XYZLOGAPPENDER.layout.ConversionPattern=%m%n
log4j.appender.LMNLOGAPPENDER=org.apache.log4j.RollingFileAppender
log4j.appender.LMNLOGAPPENDER.File=lmn_reporting.log
log4j.appender.LMNLOGAPPENDER.threshold=LMNLOG#com.services.domain.lmnlogs.LMNLogLevel
log4j.appender.LMNLOGAPPENDER.MaxFileSize=100KB
log4j.appender.LMNLOGAPPENDER.MaxBackupIndex=10
log4j.appender.LMNLOGAPPENDER.layout=org.apache.log4j.PatternLayout
log4j.appender.LMNLOGAPPENDER.layout.ConversionPattern=%m%n
You may or may not need the filter in this case.
Note: I cannot post company code here thats why I modified the solution.
This is to give you general idea how I have solved this problem.

How to log to a specific file?

I have a log4j2 logger configuration that basically write all logging of the root logger to a basic logging file. I always use Logger.getRootLogger here.
For some specific events I'd like to log do a different file. How could I configure such a logger in the properties file (eg give it a name that I can then reference from code)?

You can do something like this.
log4j.appender.transaction=org.apache.log4j.DailyRollingFileAppender
log4j.appender.transaction.DatePattern='.'yyyy-MM-dd
log4j.appender.transaction.File=logs/transaction.log
log4j.appender.transaction.layout=org.apache.log4j.PatternLayout
log4j.appender.transaction.layout.ConversionPattern=%d{dd MMM yyyy HH\:mm\:ss} %-5p %c{2}\:%L - %m%n
log4j.appender.transaction.threshold=info
log4j.logger.transaction=INFO, transaction

I finally found out that Markers are the way to route messages:
What is markers in Java Logging frameworks and that is a reason to use them?

Log4j: Conditional additivity (or propagate event to root logger based on condition?)

I've very simple task: write all events (warn and above) to main log and for certain logger write debug only messages to second log file (warn and above must be logged to main). Sometimes I wish to see debug messages in main log as well (e.g. log4j.logger.com.test=debug)
Besides I must use log4j properties file syntax. I don't understand if additivity (see my last line) can be conditional in that case or should I use complete different approach for the task (which one?).
What I've for now:
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.EnhancedPatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d %-5p [%t] %c:%L - %m%n
log4j.rootLogger=warn, stdout
log4j.appender.file2=org.apache.log4j.RollingFileAppender
log4j.appender.file2.maxFileSize=10240KB
log4j.appender.file2.maxBackupIndex=30
log4j.appender.file2.File=${catalina.home}/logs/test.log
log4j.appender.file2.encoding=UTF-8
log4j.appender.file2.layout=org.apache.log4j.PatternLayout
log4j.appender.file2.layout.ConversionPattern=%d %-5p [%t] %c:%L - %m%n
log4j.logger.org.hibernate = debug, file2
log4j.additivity.org.hibernate = false
log4j.logger.com.test=debug

I would recommend you look at Log4J2 as log4j1.x is no longer actively maintained (and Log4J2 has several other advantages like performance improvements).
A common use case is writing debug and higher (info,warn, error, fatal) to a file for developers and warn and higher (warn, error, fatal) to another file eg for support.
From your description it sounds like you want to restrict output to only debug level, and exclude info, warn, error and fatal-level messages. This is possible but not as straight-forward as the use case above.
You probably need to use filters:
http://logging.apache.org/log4j/2.x/manual/filters.html
Take a look at the filter documentation, I'll see if I can dig up an example.

How to control logging by 3rd-party components in webapp in Tomcat 5.5?

We are using Tomcat 5.5 on Linux. Our webapp uses log4j for its logging (functionally-based and not many loggers), and intentionally sets the additivity of the loggers to false. Our loggers log to our own logfile. None of them log to the console.
A problem we are having is that when the level of our loggers are set to DEBUG we start getting lots of debug logging in catalina.out from 3rd party components, particularly oscache.
There is a minimal log4j.properties file in .../common/classes:
log4j.rootLogger=INFO, A1
log4j.appender.A1=org.apache.log4j.ConsoleAppender
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
# Print the date in ISO 8601 format
log4j.appender.A1.layout.ConversionPattern=%d [%t] %-5p %c - %m%n
There is no log4g.properties file in .../webapps/ourapp/WEB-INF/classes.
The first thing I tried was changing to log4j.rootLogger=ERROR, A1 but that made no difference.
The next thing i tried was creating a .../webapps/ourapp/WEB-INF/classes/log4j.properties file containing the single line
log4j.logger.com.opensymphony.oscache=ERROR
to see if that would stop the oscache logging. It did, but surprisingly (to me) it stopped all the unwanted logging, not just the oscache logging. So I commented the line out and tried again. Still no unwanted logging from anything. I moved the file aside and now the unwanted logging came back. I created a zero-length log4j.properties and all the unwanted logging went away again (!) While that's short-term what I want, it makes me wonder about what other logging is being thrown away (and why!). So I'm uncomfortable just relying on that.
The "Logging" chapter in the Tomcat 5.5 docs merely says that you can do per-app configuration by putting a properties file in WEB-INF/classes but doesn't (at least not that I could find) talk about how that interacts with the configuration specified in common/classes.
So:
How are the 3rd-party components logging to catalina.out in the first place? I guess their logging could bubble up to the root logger but then why do they still log even when the root logger level is turned up to ERROR?
Why does setting the logger level on our loggers to DEBUG make this logging start? We have our own names for our loggers so there is no way the oscache and other stuff could be ancestors of our loggers.
Why does even a zero-length WEB-INF/classes/log4j.properties file stop a whole bunch of logging?
How can I do this the "right" way and limit the logging in a meaningful way rather than relying on some weird (to me) side effect to turn it off?
Curiouser and curiouser. I tried matt's suggestion of turning on debugging. I also made a more extensive log4j.properties file for the webapp:
log4j.rootLogger=INFO, SSOA1
log4j.appender.SSOA1=org.apache.log4j.ConsoleAppender
log4j.appender.SSOA1.layout=org.apache.log4j.PatternLayout
# Print the date in ISO 8601 format
log4j.appender.SSOA1.layout.ConversionPattern=%d [%t] %-5p %c - %m%n
log4j.logger.com.opensymphony.oscache=ERROR
#log4j.additivity.com.opensymphony.oscache=false
When tomcat starts up I see:
log4j: Using URL [file:/srv/www/tomcat5/base/webapps/myapp/WEB-INF/classes/log4j.properties] for automatic log4j configuration.
log4j: Reading configuration from URL file:/srv/www/tomcat5/base/webapps/myapp/WEB-INF/classes/log4j.properties
log4j: Parsing for [root] with value=[INFO, SSOA1].
log4j: Level token is [INFO].
log4j: Category root set to INFO
log4j: Parsing appender named "SSOA1".
log4j: Parsing layout options for "SSOA1".
log4j: Setting property [conversionPattern] to [%d [%t] %-5p %c - %m%n].
log4j: End of parsing for "SSOA1".
log4j: Parsed "SSOA1" options.
log4j: Parsing for [com.opensymphony.oscache] with value=[ERROR].
log4j: Level token is [ERROR].
log4j: Category com.opensymphony.oscache set to ERROR
log4j: Handling log4j.additivity.com.opensymphony.oscache=[null]
log4j: Finished configuring.
But despite the fact the oscache logger is having its level set to error, I still see stuff like this in the log:
2011-03-30 14:53:22,076 [main] DEBUG com.opensymphony.oscache.base.algorithm.AbstractConcurrentReadCache - get called (key=AUDIT_KEY_OLDEST_TIMSTAMP)
If I'm forcing the oscache logger level to ERROR (which the log4j debug output says I am), then why is this DEBUG message being sent at all? The child logger overriding the level in code?
What I did notice is that if I uncomment out the "additivity" line for the oscache logger in the webapp's log4j.properties file then the logging really does go away. So it appears the oscache logging is relying on ancestor appenders and not its own. But then it seems all the weirder that setting the oscache logger level to ERROR isn't stopping these things.

You could configure log4j.properties to get only your code logs as below and put it inot different file than catalina.out too.
handlers = org.apache.juli.FileHandler
org.apache.juli.FileHandler.level=ALL
org.apache.juli.FileHandler.directory=${catalina.base}/logs
org.apache.juli.FileHandler.prefix=yourapp-name.
com.yourproject.module.package.level=ALL

I've figured out what the heck is going on. The problem is that buried down in the bowels of another part of the webapp is the following code that is run when that component is put into debug mode:
public static synchronized void setDebugOn(boolean debugOn) {
if (isAllDebugOn() ^ debugOn) {
setAllDebugOn(debugOn);
Enumeration en = LogManager.getCurrentLoggers();
while (en.hasMoreElements()) {
setDebugOn((Logger) en.nextElement(), debugOn);
}
setDebugOn(LogManager.getRootLogger(), debugOn);
}
}
public static void setDebugOn(String name, boolean debugOn) {
setDebugOn(getLogger(name), debugOn);
}
private static void setDebugOn(Logger logger, boolean debugOn) {
logger.setLevel(debugOn ? Level.DEBUG : Level.INFO);
}
In other words, when this component is put in debug mode, it also puts EVERY SINGLE LOG4J LOGGER IN THE WEBAPP into debug mode (which I've verified by changing the code to print out the name of every logger inside of the last method of those three. So blammo -- all the 3rd-party stuff that happens to use log4j starts getting its debug output logged regardless of what log4j.properties says. Sigh.
When I change the method with that loop to only mess with the levels of specific loggers related to that component then my log4j.properties configuration starts working as expected.