We Keep Coding

log4j/logback pass logger level as a parameter

I want to do something which seems really straightforward: just pass a lot of logging commands (maybe all, but particularly WARN and ERROR levels) through a method in a simple utility class. I want to do this in particular so that during testing I can suppress the actual output to logging by mocking the method which does this call.
But I can't find out how, with ch.qos.logback.classic.Logger, to call a single method with the Level as a parameter ... obviously I could use a switch command based on this value, but in some logging frameworks there's a method or two which lets you pass the logging Level as a parameter. Seems a bit "primitive" not to provide this.
The method might look a bit like this:
Logger.log( Level level, String msg )
Later
Having now looked up the "XY problem" I understand the scepticism about this question. Dynamic logging is considered bad, at least in Java (possibly less so in Python)... now I know and understand that the preferred route is to configure the logging configuration appropriately for testing.
One minor point, though, if I may: although I haven't implemented this yet with this particular project, I generally find "just" tracing the stacktrace back to the beginning of the particular Thread insufficient, and this is what logback does (with Exceptions passed at WARN or ERROR levels). I plan to implement a system for recording "snapshots" of Threads when they run new Threads... which can then be listed (right back to the start of the app's first Thread) if an error occurs. This is, if you like, another justification for using something to "handle" outgoing log calls. I suppose that if I want to implement something like this I will instead have to try to extend some elements of logback in some way.

Capture all thrown exceptions in java?

I doubt such a thing is possible, but without attaching a debugger to a java application, is it possible to have some collection populated with information about every exception that is generated in a java application, regardless of if it is caught or not? I know that in .NET, messages get generated by the application about exceptions which at that point are called "First Chance Exceptions", which may or may not subsequently be handled by the application. I'm wondering if there might be a similar mechanism in java I can exploit to view information about all the exceptions generated at runtime.
Just to clarify. This has nothing to do with the context in which an exception occurs. This question is not about what I do in a catch block, or unhandled exceptions. Its about knowing if the JVM provides a mechanism to see every exception generated at runtime, regardless of what generated it, or the context.

Why not, it's of course possible! But firstly.. Logging all exceptions encountered by the JVM is a waste of life. It's meaningless in every sense, there could be several excetion's thrown without any significance.
But if indeed if you have no choice, you could tweak your Java to do that.
Breaking every rule of good programming, what we live for, I give you this idea:
Copy the source code of java.lang.Exception from JDK sources to your project.
Create a method in your Exception.java like below:
private void logException() {
// Your logging routine here.
}
Edit java.lang.Exception to call this method logException() at the end of every constructor.
Add the new java.lang.Exception to bootstrap classpath.
Set your logging levels etc and run.
Put your heads up, present this to your weird client, use your diplomatic skills and scare them in few words 'we can do it.. but its your own risk'. Likely you will convince him not to use this.

How to reduce logs size [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 8 years ago.
Improve this question
In every project I've been working on there's always been the issue of log files becoming too large.
A quick off the shelf solution was to use the Log4j RollingFileAppender and set the maximum size allowed.
However there are situations when the same exception happens repeatedly reaching the maximum size very quickly, before somebody manually intervenes. In that scenario because of the rolling policy you end up losing information of important events that happened just before the exception.
Can anybody suggest a fix for this issue?
P.S. Something I can think of is to hold a cache of the Exceptions happened so far, so that when the same Exception re-occurs I don't log tons of stacktrace lines. Still I think this must be a well-known issue and I don't want to reinvent the wheel.

There are two directions to approach this from: The System side and the development side. There are several answers already around dealing with this from the system side (i.e. after the application is deployed and running). However, I'd like to address the development side.
A very common pattern I see is to log exceptions at every level. I see UI components, EJB's, connectors, threads, helper classes, pojos, etc, etc, logging any and all exceptions that occur. In many cases, without bothering to check for the log level. This has the exact result you are encountering as well as making debugging and troubleshooting take more time than necessary as one has to sift through all of the duplication of errors.
My advice is to do the following in the code:
THINK. Not every exception is fatal, and in many cases actually irrelevant (e.g. IOException from a close() operation on a stream.) I don't want to say, "Don't log an exception," because you certainly don't want to miss any issues, so at worst, put the log statement within a conditional check for the debug level
if(logger.isDebugEnabled()){
// log exception
}
Log only at the top level. I'm sure this will meet with some negativity, but my feeling is that unless the class is a top-level interface into an application or component, or the exception ceases to be passed up, then the exception should not be logged. Said another way, if an exception is rethrown, wrapped and thrown or declared to be thrown from the method, do not log it at that level.
For example, the first case is contributing to the issue of too many log statements because it's likely the caller and whatever was called will also log the exception or something statement about the error.
public void something() throws IllegalStateException{
try{
// stuff that throws some exception
}catch(SomeException e){
logger.error(e); // <- NO because we're throwing one
throw new IllegalStateException("Can't do stuff.",e);
}
}
Since we are throwing it, don't log it.
public void something() throws IllegalStateException{
try{
// stuff that throws some exception
}catch(SomeException e){
// Whoever called Something should make the decision to log
throw new IllegalStateException("Can't do stuff.",e);
}
}
However, if something halts the propagation of the exception, it should log it.
public void something(){
try{
// stuff that throws some exception
}catch(SomeException e){
if(logger.isLogLevelEnabled(Log.INFO)){
logger.error(e); // DEFINITELY LOG!
}
}
}

Use Log4J feature to zip the log file after a specified size is reached using "Rolling File Appender". Zips are around 85KB for a 1MB file.
For this specify the trigger policy to zip based on size and specify the zip file in the rolling policy.
Let me know if you need for info.

In my experience, logging is used as a substitute for proper testing and debugging of code. Programmers say to themselves, "I can't be sure this code works, so I'll sprinkle logging messages in it, so when it fails I can use the log messages to figure out what went wrong."
Instead of just sprinkling logging messages around without thought, consider each log message as part of the user interface of your software. The user interface for the DBA, webmaster or system administrator, but part of the user interface nonetheless. Every message should do something useful. The message should be a spur for action, or provide information that they can use. If a message could not be useful, do not log it.
Give an appropriate logging level for each message. If the message is not describing an actual problem, and is not providing status information that is often useful, the message is probably useful only for debugging, so mark it as being a DEBUG or TRACING message. Your usual Log4J configuration should not write those messages at all. Change the configuration to write them only when you are debugging a problem.
You mention that the messages are due to an exception that happens often. Not all exceptions indicate a bug in the program, or even a problem in the operation of the program. You should log all exceptions that indicate a bug in your program, and log the stack trace for them. In many cases that is almost all you need to work out the cause of the bug. If the exception you are worried about is due to a bug, you are focusing on the wrong problem: you should fix the bug. If an exception does not indicate a bug in your program, you should not log a stacktrace for it. A stacktrace is useful only to programmers trying to debug a problem. If the exception does not indicate a problem at all, you need not log it at all.

Buy bigger hard drives and set up a batch process to automatically zip up older logs on a regular basis.
(Zip will detect the repeated exception pattern and compress it very effectively).

use the strategy if reach maximum size, append to the new log file. and run scheduler like everyday to wipe the old log file

java - is there a standard way of collecting exceptions?

I have a Java program that parses several different input files. Even when an error is found in this input files, parsing could still continue and collect several other errors too. So what I want to do is instead of throwing and exception and stopping parsing process, I'd like to register the exception to somewhere, and then continue parsing and collect several other errors in similar way, then in the end I want to check if any errors are reported, and fail or continue depending on this.
Of course I can always do this manually by writing ExceptionRegister or whatever classes but I want to know two things:
1) Is there a problem with this approach? Do you know any alternative designs for what I want to do?
2) Is there a standard way of doing this? (e.g. instead of rolling my own classes I'd like to use built-in functionality if possible)
Thanks
EDIT: I don't know why but someone just removed his answer just before I accepted his answer. Anyway, I think simple data structures should work. Basically, I'll write an exception class that collects several error messages. Then I'll call it's throw method which throws itself if it has at least one error message registered.
EDIT2: Here are more clarifications. My problem has nothing to do with parsing. Parsing was just an example because my program does some parsing. Think this: I'm running an algorithm and in case of an error, I can continue the algorithm to collect more errors so that instead of printing one error and when it's fixed, printing second error, I can print this two errors together.

Exceptions should really be used when you can't handle the input anymore. They are the special case where your code says "I give up, I'm missing some information or I wasn't meant for this". This is a grey area on how to define such cases, but the usual philosophy, as put by Bill Venners in this (old!) article is:
Avoid using exceptions to indicate conditions that can reasonably be
expected as part of the typical functioning of the method.
In your case, it sounds like the content you have to parse might be incorrect, but this is expected by your program and doesn't break enough the contract to stop the parsing. On the other hand, an acceptable exception would be valid to use if an error in the syntax of the input causes the rest of the interpretation to fail, for example.
But people still uses exception because they are quite convenient for stopping execution and going up the stack without going in the tedious details of flowing through returns of results. But on its counterpart, they can have tricky results as you leave some unattended state in some objects.
Your requirements sounds more like a validation pattern is required than one single exception that could cause the processing to stop. One exception to stop all processing: if you throw one, the rest will be ignored. But you suggested that you would collect them instead of throwing those. So I'd say, in that case, why use exceptions at all? It seems you do want to return proper results and not stop the program's execution.
Because if you still go down this path, you could have a collection of exceptions to throw at the end. Which one do you throw? Which one takes precedence, in the Exception collector you created?
Take the example of Eclipse, which has this gigantic platform to handle with a massive collection plug-ins contribution. They use a proper communication channel to log any warning and errors, either in problems pane or through the execution of background task. The latter's execution will usually return an IStatus object or a variant. Based on this IStatus object, the code that receives the status decides to act upon it.
Hence personally, I'd develop a similar object that would collect all necessary user's errors (and not program's errors), that does not break the program's execution and an acceptable part of the contract. This object can contain the severity of the error, its source, a hint on how to fix it (this can be a string, or even a method that contains a pinpointing logic for showing the error or possibly a partial automated fix), etc... Once the execution is done, the parsing's result will get these status objects and act on it. If there are errors, inform the user through the UI and log it as well.
So it's basically the same approach as you initially suggested, minus the exceptions and minus the commodity of jumping through the stack that could lead to nasty side-effects and very difficult to debug errors.

I think I understand now. What you are actually trying to do is to collect the parse errors (which you are representing as exceptions) and continue parsing the current input file.
There is no standard way to do this:
The "exception register" is really nothing more than a list of parse error descriptors ... presumably some parser exception. If you can catch the exception at the appropriate point, it is trivial to add it to the "register".
The difficult part is the functionality you are not talking about:
How to capture the location of the error
How to get the parser to continue parsing when it gets a parser error.
The solutions to these depend on how you have implemented your parser.
If you are using a parser generator, there is a good chance that the PGS documentation explains how to implement this.
If you are implementing the parser by hand, you will need to roll your own code to track error locations and do syntax error recovery.
Basically, I'll write an exception class that collects several error messages. Then I'll call it's throw method which throws itself if it has at least one error message registered.
That sounds like an abuse of exceptions. A better idea is to accumulate the errors in a list, and then create / throw the exception if the list is non-empty. Use a static method in a helper class if you want to avoid duplicating code.
An exception that conditionally throws itself is (IMO) bizarre! And creating lots of exceptions that you are unlikely to throw is likely to be inefficient. (Depending on the JVM, the expensive part of exceptions is often creating the exception and capturing the stack frames. The expense can be significant.)

How to combine logging with an exception handling chain?

Suppose I have the following code:
void foo() {
/* ... */
try {
bar(param1);
} catch (MyException e) {
/* ??? */
}
}
void bar(Object param1) throws MyException {
/* ... */
try {
baz(param2);
} catch (MyException e) {
/* ??? */
}
}
void baz(Object param2) throws MyException {
/* ... */
if (itsAllATerribleMistakeOhNo) {
/* ??? */
throw new MyException("oops, error.");
}
}
I'm wondering where and how I should be logging the error.
Where the error occurs, below, in baz(), I know exactly what operation went awry and can log that fact.
At the top I have the most general context (e.g. what's the IP of the connection during whose handling we encountered the error.)
Along the way I might have some context which isn't known either at the top or at the bottom.
Another complication is that the error at the bottom might not really be considered an error when you look at it from the top (e.g. looking up something in a database fails; maybe you weren't sure ) - so I might choose to logger.WARN() instead of logger.ERROR().
So, above I described 3 locations (bottom, top, and along the way) - but it's not just a question of where to log, but also what to throw up. At every level in the middle, you have 2x2 options:
Log/Don't log a message
Throw the original exception / wrap the exception in a new exception with the added message.
What are the best practices, or some common wisdom, regarding these complex choices?
Note: I'm not asking about error handling/exception use in general, just about the dilemmae described above.

When it comes to logging, I prefer to keep all my logging at the top at the application boundary. Usually I use an interceptor or filter to handle all logging in a general way. By this concept, I can guarantee that everything is logged once and only once.
In this case, you would log inside your foo() method or whatever the entry point to your application is (you mentioned the IP address, I suppose we are talking about a servlet container or application server).
Than, catch your own exception in the filter/interceptor and log it depending on your needs. Add a catch throwable to catch all other exceptions that you did not handle in your code and log them as an error, since obviously you missed something further down in the stack trace.
This concept requires some planning ahead. You will probably use your own ApplicationException that stores the Error Message (String) along with some severity level (probably in an Enum). You need this to choose the correct log level when you do the actual logging.
This works well for all cases and has the advantage that all logging is happening exactly once. However, there is one case where you still need logging in your code: if you can fully deal with an error somewhere in your code (that is, an exception happens and you can do something that allows you to continue working without (re)throwing an exception). Since your are not throwing an exception, nothing would be logged otherwise.
To sum it up:
Log at the topmost position in a general way, preferably using an interceptor or filter.
Wrap exceptions inside your own ApplicationExceptions and add severity plus other things of interest for logging in your application.

Some suggestions that I tend to follow:
Link for some best practices
1) Trace the exception where it occurs. As the point where the exception occurs if the class or API knows the context in which the exception occurs then tracing and providing a proper log is better. But if the API cannot handle or comment on the exact context then API should not log the event and leave it on the caller.
2) Wrapping the exceptions : When there are lot of exceptions that can be thrown and all exceptions form a similar group (SQLException) which provides single exception and lets you to extract information if needed. Otherwise there would have been an explosion of exceptions that the caller needs to handle.
3) Re-Throwing the exceptions: If the API logs the exception and user can take some actions on that then the Exception MUST be rethrown to tell the user that some error condition occured.
4) Proper cause of exception : The exception message should not be too techy for the caller to understand, the message itself should guide the user to understand the underlying reason for the exception.
UPDATE:
Exception Management in Java

When I throw Exceptions in my code, I do not usually log anything. The exception is information enough.
The only exception to this is, when I am at the border of my system, that is, when the exception will leave the boundary of my system, then I log as I am not sure what the other system will do with the error.
When I handle exceptions, I log them when I actively handle them, that means when I am in a catch clause which does something more then just rethrowing the exception.
Usually this is rather at the top, but this depends on the situation.

When throwing an exception at the testing stage, you should remember:
Keep the exception message as clear as possible. Stack traces can be confusing at the best of times so ensure that what you are reading, at least, makes sense to you.
Ensure that the exception is relevant to the event. If the user types in the wrong value and you throw a NullPointerException, your code is illogical and loses it's value.
Ensure that it has as much information ABOUT THE EVENT as possible. That is, keep the message relevant. If a database call has gone wrong, print the connection string to the database, and the SQL query attempted. The state of every variable currently being used isn't necessary.
Don't waffle. It's tempting to type in technical jargon to make it look like you're hacking into the matrix. It doesn't help you in a stressful situation, and it certainly doesn't help anyone else using your code. Simple english words are always preferable.
Finally, NEVER IGNORE AN EXCEPTION. Always ensure you handle the exception, and you're outputting details in some way, following the rules I've stated above.