We Keep Coding

How to reduce logs size [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 8 years ago.
Improve this question
In every project I've been working on there's always been the issue of log files becoming too large.
A quick off the shelf solution was to use the Log4j RollingFileAppender and set the maximum size allowed.
However there are situations when the same exception happens repeatedly reaching the maximum size very quickly, before somebody manually intervenes. In that scenario because of the rolling policy you end up losing information of important events that happened just before the exception.
Can anybody suggest a fix for this issue?
P.S. Something I can think of is to hold a cache of the Exceptions happened so far, so that when the same Exception re-occurs I don't log tons of stacktrace lines. Still I think this must be a well-known issue and I don't want to reinvent the wheel.

There are two directions to approach this from: The System side and the development side. There are several answers already around dealing with this from the system side (i.e. after the application is deployed and running). However, I'd like to address the development side.
A very common pattern I see is to log exceptions at every level. I see UI components, EJB's, connectors, threads, helper classes, pojos, etc, etc, logging any and all exceptions that occur. In many cases, without bothering to check for the log level. This has the exact result you are encountering as well as making debugging and troubleshooting take more time than necessary as one has to sift through all of the duplication of errors.
My advice is to do the following in the code:
THINK. Not every exception is fatal, and in many cases actually irrelevant (e.g. IOException from a close() operation on a stream.) I don't want to say, "Don't log an exception," because you certainly don't want to miss any issues, so at worst, put the log statement within a conditional check for the debug level
if(logger.isDebugEnabled()){
// log exception
}
Log only at the top level. I'm sure this will meet with some negativity, but my feeling is that unless the class is a top-level interface into an application or component, or the exception ceases to be passed up, then the exception should not be logged. Said another way, if an exception is rethrown, wrapped and thrown or declared to be thrown from the method, do not log it at that level.
For example, the first case is contributing to the issue of too many log statements because it's likely the caller and whatever was called will also log the exception or something statement about the error.
public void something() throws IllegalStateException{
try{
// stuff that throws some exception
}catch(SomeException e){
logger.error(e); // <- NO because we're throwing one
throw new IllegalStateException("Can't do stuff.",e);
}
}
Since we are throwing it, don't log it.
public void something() throws IllegalStateException{
try{
// stuff that throws some exception
}catch(SomeException e){
// Whoever called Something should make the decision to log
throw new IllegalStateException("Can't do stuff.",e);
}
}
However, if something halts the propagation of the exception, it should log it.
public void something(){
try{
// stuff that throws some exception
}catch(SomeException e){
if(logger.isLogLevelEnabled(Log.INFO)){
logger.error(e); // DEFINITELY LOG!
}
}
}

Use Log4J feature to zip the log file after a specified size is reached using "Rolling File Appender". Zips are around 85KB for a 1MB file.
For this specify the trigger policy to zip based on size and specify the zip file in the rolling policy.
Let me know if you need for info.

In my experience, logging is used as a substitute for proper testing and debugging of code. Programmers say to themselves, "I can't be sure this code works, so I'll sprinkle logging messages in it, so when it fails I can use the log messages to figure out what went wrong."
Instead of just sprinkling logging messages around without thought, consider each log message as part of the user interface of your software. The user interface for the DBA, webmaster or system administrator, but part of the user interface nonetheless. Every message should do something useful. The message should be a spur for action, or provide information that they can use. If a message could not be useful, do not log it.
Give an appropriate logging level for each message. If the message is not describing an actual problem, and is not providing status information that is often useful, the message is probably useful only for debugging, so mark it as being a DEBUG or TRACING message. Your usual Log4J configuration should not write those messages at all. Change the configuration to write them only when you are debugging a problem.
You mention that the messages are due to an exception that happens often. Not all exceptions indicate a bug in the program, or even a problem in the operation of the program. You should log all exceptions that indicate a bug in your program, and log the stack trace for them. In many cases that is almost all you need to work out the cause of the bug. If the exception you are worried about is due to a bug, you are focusing on the wrong problem: you should fix the bug. If an exception does not indicate a bug in your program, you should not log a stacktrace for it. A stacktrace is useful only to programmers trying to debug a problem. If the exception does not indicate a problem at all, you need not log it at all.

Buy bigger hard drives and set up a batch process to automatically zip up older logs on a regular basis.
(Zip will detect the repeated exception pattern and compress it very effectively).

use the strategy if reach maximum size, append to the new log file. and run scheduler like everyday to wipe the old log file

What level to use for exception stack trace logging in Java?

I'm looking for best practices document (or your opinions) on how to effectively log exceptions and their stack traces. Of course, assuming one of popular logging frameworkks such as Log4J, SLF4J, java.util.logging, etc.
I'm particularly interested in your opinion about on what level stack traces should be logged.
I heard few contradicting each other opinions such as:
stack traces should be logged only on DEBUG level while ERROR level should contain only "human readable" error message
stack traces should be logged on ERROR level in order to give to the operator maximum amount of information required to find root cause of an exception
I have found couple of interesting articles however none of them touches this particular subject:
http://today.java.net/pub/a/today/2006/04/06/exception-handling-antipatterns.html
http://today.java.net/pub/a/today/2003/12/04/exceptions.html
which probably means that authors of these articles had same concerns as I do :-)
I'd be really interested in your view on this subject.

Stack trace is the most valuable piece of information you get when troubleshooting. I would never risk logging it on DEBUG level since it might be disabled. And I almost never want to suppress stack traces.
Also note that:
log.error("Houston, we have a problem", ex);
will print the human readable message in line marked as ERROR, while the stack trace is following that line. If you want your errors to be only human readable, just do grep ERROR.

I'm not sure about best-practice advice for this, but in the end, for me it boils down to this:
Exceptions should only be visible in exceptional circumstances. The concept of exception was invented to give developers a chance to handle errors internally.
In reality, most code I see doesn't even try to handle them, instead dumping them to the log, sysout, (or worst case of all) into dialog boxes. I know, that for a developer it is important in some cases to get the full stacetrace. But not nearly in all of them. Creating your own exception framework (which is definitely a best practice) might already be enough to figure out the context of an exception simply by classname.
So I would advise to do the following:
Create your own exception framework
Include specific error codes in the message, for your reference
Log the exception message on ERROR
Log the stacktrace on DEBUG
NEVER EVER display the user either of these. Instead show a useful message. Maybe include a way to report the error (with stacktrace) with minimal fuzz.
Note: If you are writing an internal "enterprise" software, forget everything I wrote. :-)

I think the stack trace should be logged at the appropriate place based on priority best practices. Depending on the nature of the exception and its place within your application, this may be one of many levels. Please see this related question:
Commons Logging priority best practices

Should you report the message text of exceptions?

Consider some code that can throw a checked exception (an exception of type Exception). Your code catches the exception, of course. You don't just swallow the exception either, your code reports it in some way to the user through your user interface. In a log file, perhaps, or using a GUI pop-up.
Should the text you report to the user include the message text of the exception. That is, the text provided by Throwable.getMessage() or Throwable.getLocalizedMessage()?
I think not, but it seems many disagree with me. So what have I got wrong? My argument is as follows.
The message was created when the exception was thrown. It therefore at best can provide only very low level information, which can be inappropriate for reporting to a user.
Philosophically, using the message seems to me against the whole point of exceptions, which is to separate the detection and initiation of error handling (the throw part) from completion of handling and reporting (the catch part). Using the message means the message must be good for reporting, which moves responsibility for reporting to the location that should be responsible for only detection and initiation. That is, I'd argue that the getMessage() part of the design of Throwable was a mistake.
The message is not localised. Despite its name, getLocalizedMessage() is not much good because you might not know what locale you want to use until you catch the exception (is the report to go to a system log read by your English system administrators, or is it to pop up in a window for the French user of the GUI?).
I hear that Java 7 has a much improved exception hierarchy for IOException, enabling you to handle different kinds of I/O erors in diffferent catch clauses, making the getMessage() text less important. That implies even the Java designers are somewhat uncomfortable with getMessage().
I'm not asking whether reporting the stack-trace is useful. A stack-trace is only ever going to be useful for an exception that suggests a bug. That is, for an unchecked exception. I think in such circumstances providing the low-level detail of the exception message is not just useful but mandatory. But my question deals with checked exceptions, such as file-not-found.
And I am not asking about the "best practice" for the message text.

If you are presenting an error condition to the user, it should probably be a user-friendly message. Exceptions contain technical details that the user should not/does not need to know.
In some situations it could be a security concern to present stacktrace information, so the user should never be shown stack traces.
If you are displaying error messages to the user, there is some point where you consciously make the decision to show a popup, or add a message to a log window. At that point you can translate any exception into a more user friendly message. Note that you might need more information than the default Exception types provide, so you can/should probably create you own Exception types that contain all the information you need to present all the data you need to the user.

No, exceptions shouldn't be shown directly in error messages directly to the user, they're low level technical details and the user almost always wants something more understandable, even if it doesn't provide as much information as a stack trace would!
I say almost always because there are cases (such as in IDEs) where you can consider your users technically competent enough to look at stack traces; indeed in this case they will probably prefer it to a "dumbed down" error message.
However, personally I think stack traces should always be logged somewhere that the user can access so that if they complain that "the program isn't working" you can see exactly what went on if they send you that file.

In some projects, I make a special kind of exception (e.g. UserFriendlyException). This exception type must have a user friendly error message. If I catch such an exception, I can display it to the user.
This makes it possible to use exceptions for user-friendly errors, and prevents that you show very technical messages to the user.

I would argue that you should never show the exception message itself to the user, it should only ever appear in a log. Even if you have purposefully made it user friendly, it should still not be displayed because you cannot internationalize those messages easily. You should come up with some mechanism that your UI layer can understand and resolve that to a code that you can look up an internationalized message to display to your user. I have found that an exception with a "code" attribute/enum works pretty well. Very specific exceptions work too, but that can be messy to maintain.

Jruby Stack Traces

So this might be a dumb question but I don't know what to ask for since 'jruby print stack trace' googled doesn't come up with much thats worthwhile but,
How does one print the stack trace when writing Jruby code that accesses Java classes that throw exceptions? ie right now I took code from someone else that looks like
rescue JavaSql::SQLException
puts "SQLException"
end
which works, but I'd like to see which particular SQLException is happening like I would in Java (ie. PrintStackTrace(Exception e))? How does one accomplish this?

I've never worked with Java exceptions in Ruby, but AFAIK, Java exceptions get presented to you as Ruby exceptions, which means you can get the exception message by sending the :message message to the exception object:
begin
# stuff
rescue JavaSql::SQLException => e
puts "#{e.class}: #{e.message}"
end
Is that what you mean? I'm not quite sure what you would need the stacktrace for in this particular situation, so I probably misunderstood.

One would imagine it relies on the underlying implementation. If the underlying implementation just swallows the exception you will not be able to see a stack trace. Also why don't you just allow the exception to bubble up in your ruby code so that the trace gets thrown out to the screen?

Most common cause of "java.lang.NullPointerException" when dealing with XMLs?

My strongest lead is that the code who deals with the incoming XMLs is actually receiving an invalid/incomplete file hence failing the DOM parsing. Any suggestions?

Incomplete file is definitely the place to start looking. I'd print out the file right before the point you parse it to see what's getting sent to the parser. If it's incomplete it will be obvious. If it's invalid, you'll have a little searching to do.

My first guess would be that the DOM-using code is treating elements that are marked as optional in the DTD as compulsory.
Edited to add:
What I mean is that unless you validate against a DTD, you cannot expect something like the following (example using dom4j) to return anything but null.
doc.selectSingleNode("//some/element/in/a/structure");
The same is of course true if you're stringing element navigation calls together, or generally don't check return values before using them.

You should have a stack trace pointing to where you NPE is thrown. That should narrow down the number of variables that can be null. Rather than getting the debugger or printf out, I suggest adding appropriate checks and throwing an exception where as soon as the error can be detected. It's a good habit to get into to avoid mysterious problems later.

Ideally you should be running your java application inside a debugger, thus when an uncaught exception is thrown you can examine the callstack, variables, etc and see exactly what line caused the crash, and perhaps which data is null that got used.
If you can't use a debugger for whatever reason, then compile your application with debugging support, and add an exception handler for this particular error, and print out the stack trace. Again, this will show exactly what line in what file caused the crash.