We Keep Coding

What are ways to change log level at runtime depending on occurrence rate

I would like to know whether there are conventional ways (probably some third-party libraries) to help me with the following problem. I would like to have an ability to change log level depending on occurrence rate. For example, if I catch some exception and it happens say once an hour I would like to log it with the WARN level. But if the rate becomes higher then I would like to switch to the ERROR level. The purpose of this behavior is to not get distracted by rare sporadic exceptions that are inevitable in general but shouldn't be brought to developers' attention unless there is a sufficient amount of them. When I open Sentry summary page I would like to see only what's really important and requires attention, not some rare request failures that will inevitably happen from time to time. But at the same time I can't just use WARN level because if there are many such failures then there's clearly something bad going on and it requires attention.
To better illustrate what I mean here's how in my opinion that could be done with an imaginary third-party library that would imitate (or be a wrapper over) slf4j/commons-logging:
LOG.warn("Failed to send request to ...", e).onRate(Rates.hourlyMoreThan(5)).switchTo(LogLevel.ERROR);
An alternative would be an option in Sentry not to show events (matching some criteria like a class of the exception) with a low occurrence rate but as far as I know that's not possible.

Assuming you want to do this with Log4j (other logging frameworks should have similar structures), this can probably be done with a custom appender. If that appender sees one message too many times, it could try to change the category's (or the logger's) level.
Here is how you could create your own Appender:
https://logging.apache.org/log4j/2.x/manual/extending.html
Here is what the appender would have to do to reconfigure the logging:
https://logging.apache.org/log4j/2.x/manual/customconfig.html

Not sure if it is possible to filter in Sentry UI, but you an definitely create a custom Sentry appender based on SentryAppender which will measure the occurrence and emit event to Sentry only when it reaches certain threshold.

performance improvement: toString()

We are developing quite big server app. Each action is tracked by log. There are many logs with calling of toString method. Unfortunatelly we need most of them on other hand we cannot track what happens on prod.
Is it make sense trying to improve toString method? Forexample put result of toString in memory and update if some field was updated.
example of my toString
public classs InMessage{
//declared 20+ fields
#Override
public String toString() {
StringBuilder builder = new StringBuilder(this.getClass().getSimpleName());
builder.append(": [");
builder.append(super.toString());
builder.append("; updateTime: ");
builder.append(updateTime);
//forexample 20 fields here
builder.append(";]");
return builder.toString();
}}
then we process InMessage in some way and log each action
log.debug("We received inMessage: {}", inMessage);
after this discussion we decided to decrease number of logs where it's possible and nothing more.

Unless you are actively experiencing performance or memory issues, don't worry about it. Useful logs can help enormously in prod diagnostics, you need to weigh that against hypothetical performance worries that might just never happen.
(Not saying you're definitely going to be problem-free here, just don't optimise for something that's not really a problem...)
Edit: trust your own judgement on whether you're logging "too much"; if a single UI or service operation generates 20 pages of incidental / debug level information, even at INFO level logging, you might want to refine the content to make it genuinely useful

That nano-optimization usually ends up in useless effort that add no overall value with the exception of some ms substracted from a time handled in seconds or, sometimes, minutes. Depends on the case but yours could be one of those where every little bit counts.
Identify an optimization need based on data, don't pre-optimize and in particular with such methods. I bet there are other approaches that could prove more valuable than questioning the use of toString like, for example, improving the order of a commonly used algorith or reducing the load of an operation (less calls, but significant ones).
Just compare the usage of toString with the solution you have in mind, you're implementing a kind of observer pattern that updates a central string repository if a field is changed? That's executing code on every setter (if you use that approach).

Do you provide DEBUG logging level for private methods?

I add logging for Java classes.In particularly, logger.debug level (for development mode).
As far private method called within public ones, is there sense to provide logging within private methods?Is it efficiently?
Thank you.

It's not necessarily a matter of efficiency or not. You have to ask yourself: "Does that private method do something worthy of logging?"
In order to avoid performance issues, you can always check the logger whether it will log debug messages or not, before actually logging those messages. In log4j, you'd write
if (logger.isDebugEnabled()) {
logger.debug("complicated" + "innefficient" + "string concatenation");
}
I imagine, other loggers provide similar concepts

yeah, if you want to "debug" your private methods in maintance mode or development, why not?
in production you'll set your logging level either warn or error! that's upto you...

Of course. Private methods usually hide implementation detail, and that's where most bugs lurk. Put your logging statements where you feel the need, and disable them in production.

Logging might seem wasteful when you don't need it, but it is indispensable when you do. Always provide meaningful log messages in your classes and functions. The log levels are there for you to be able to filter out the noise, and increase verbosity when you need it. Good loggers provide capabilities to change log levels at runtime.
Still, if you are worried about performance for whatever reason, you can use some simple guards to reduce the overhead of logging, especially debug logging, which is rarely on. With log4j/slf4j for example, you can wrap log debug statements in:
if(logger.isDebugEnabled()) {
logger.debug("something");
}
slf4j additionally has a printf-like syntax which only does string formatting if the log level is correct.
logger.debug("Object {} is not valid!", obj);
Like darioo commented, this second form removed the need to check the log level before the log statement.

Commenting out System.out.println

Do System.out.println(...) calls pose any effect if left in BlackBerry code or any other programming language?
When removed, the compilation time may be reduced, but is there any particular other reason to remove them?

There are a couple of things you need to know before using System.out.println() on Blackberry:
Once you print out something to the standard output any person that has your application installed on the device will be able to see them. All they need to do is to attach the device to the simulator and run in debug mode. So make sure you do not print out anything sensitive such as passwords, class names etc. in the released application.
The performance overhead that the System.out.println() itself makes is minimal, especially when the output stream is not attached to anything (i.e. Device is not connected and not in debug mode).
I myself rather use Blackberry preprocessor to be able to disable all logs before making a release. For this reason I define a logging directive LOGGING and then in my code:
//#ifdef LOGGING
System.out.println("LOGGING is enabled");
//#endif
For more on how to use preprocessors in Blackberry Eclipse plugin see this.

I prefer to use a flag to disable sysouts. Sysouts are really slow if you use them a lot, eg. in loops.

If you don't intend to use the output for anything like debugging ect. then it's best to take it out. Your program will only run as fast as the line can be output so in theory the less system.out line you have the faster the process will be.
Hope this helps.

Runtime might be also reduced, as the statements are actually executed - even if the user doesn't see the output on the screen. If you're using a lot of these (e.g. in tight loops) or you're passing to them Objects with expensive toString() methods, the useless output may be slowing you down.
Also, if you're passing String as an argument, those will take some space in bytecode and in memory. You on your souped-up machine with 173 PB of RAM may not care, but there are resource-constrained systems (such as mobile devices).

You should be able to use Ant to preprocess these lines out of your source code. (Make sure that none of them have side-effects!)

I don't know specifically about Blackberry, but if your program is writing to an unknown device (i.e. you are not sure where standard out is going), there may be a potential for your app to occasionally/sporadically/inexplicably block momentarily in the attempt to write.

Create your own method, i.e. :
public static void consoleMessage(String msg){
if(DEBUG_FLAG){
System.out.println(msg);
}
}
Then use only this throughout your code. It will save you the time for changing all the lines.

Use something like Log4J instead of system out print statements, it gives you much more flexibility

Keeping System.out statements isn't that bad a thing to do usually. Users might be able to see them so it doesnt always look good in a production environment. A better idea is to use a logging framework such as java.util.logging or log4j. These can be configured to dump output to the console, to a file, a DB, a webservice ...
Keep in mind that just becuase you can't see the output it doesn't mean that no work is being done at runtime. The JVM still has to create a String to pass to system.out (or a log statement) which can take a fair bit of memory/CPU for large/complex objects like collections.

Sysout statements access a synchronized, shared resource, which causes synchronization between threads using it. That can prevent memory consistency bugs in multithreaded programs if there is no other code which enforces synchronization. When the sysout statements are removed, any existing memory consistency bugs in the code may surface for the first time.
For an example of this effect, see: Loop doesn't see changed value without a print statement.

It's not an object and it doesn't have any memory attached to it so there shouldn't be any effect besides the time to run it and compile it. And of course readability maybe lol

How to know when there's too much logging messages?

I came across one very good library for parsing CUE files. But when I started to read its source code, I realized that it is almost unreadable:
public void setParent(final CueSheet parent) {
FileData.logger.entering(FileData.class.getCanonicalName(), "setParent(CueSheet)", parent);
this.parent = parent;
FileData.logger.exiting(FileData.class.getCanonicalName(), "setParent(CueSheet)");
}
every method has logger.entering() and logger.exiting() messages. Isn't that too much?
There's another java library for parsing audio tags. It also had like 15 log messages for each file it read. It was annoying so I commented out every call to logger. And the library became twice as fast, because they used a lot of string concatenation for log messages.
So the question is: should I really log everything, even if it is not large enterprise application? Because these libraries obviously don't need any logging, except for error messages. And my experience shows that loggers are terrible tool for debugging. Why should I use it?

How to know when is too much logging? When you know that the logged information isn't important in the long term, such as for straightforward debug actions or bug correction, or for when the application doesn't deal with too much important information.
Sometimes you need to log almost everything. Is performance or full possibility of analysis the most important part of an application? It really depends.
I've worked in the past with some integration with a lot of different webservices, like 10 in a same app. We logged all xml requests and responses. Is this an overhead? In the long term, I don't think so because we worked with a lot of credit card operations and should have every process made with the server logged. How to know what happened when there was a bug?
You wouldn't believe what I've seen in some of the xml responses. I've even received a xml without closing tags, from a BIG airplane company. Were the "excessive logs" a bad practice? Say that to your clients when you have to prove that the error came from the other vendor.

Ideally, you use a logger that allows logging levels; log4j has fatal/error/warn/debug/info, for example. That way, if you set the level to "only show errors", you don't lose speed to the software building log messages you didn't need.
That said, it's only too much logging until you wind up needing something that would have been logged. It sounds like most of the logging that's slowing you down should be "trace" level, though; it's showing you what a profiler would have.

Most logging libraries incorporate a means to confirm that logging is enabled before processing an instruction:
For example:
public void foo(ComplicatedObject bar) {
Logger.getInstance(Foo.class).trace("Entering foo(" + bar + ")");
}
Could be quite costly depending on the efficiency of the bar.toString() method. However, if you instead wrap that in a check for the logging level before doing the string concatenation:
static {
Logger log = Logger.getInstance(Foo.class);
public void foo(ComplicatedObject bar) {
if (log.isTraceEnabled()) {
log.trace("Entering foo(" + bar + ")");
}
}
Then the string concatenation only occurs if at least one appender for the class is set to Trace. Any complicated log message should do this to avoid unnecessary String creation.

This level of logging is canonically bad - in fact, I saw code exactly like this in the Daily WTF a few days ago.
But logging is in general a Very Good Thing.

It depends, it this code for an application, or a library? For an application, logger are useful once the code is in production. It should not be used to debug, but to help you replicate a bug. When a user tells you that your application crashed, you always want the maximum logging information.
I agree that it makes the code less readable. It even make the application slower!
It's a total different game for a library. You should have consistent logging with adequate level. The library SHOULD inform the development team when an error occurs.

Logging should provide you with information that a stack trace can't in order to track down a problem. This usually means that the info is some kind of historical trace of what the program did, as opposed to what state it's in at the time of failure.
Too much historical data will be ignored. If you can safely deduce that a method was called without having to actually log its entry and exit, then it's safe to remove those logging entries.
Another bad sign is if your logging files start to use up a huge amounts of disk space. You're not only sacrificing space, you're probably slowing down the app too much.

To answer the question, why should I use loggers?
Have you ever encountered a piece of software where the only error indicated presented to the end user is Fatal error occured. Would it not be nice to find out what have caused it?
Logging is a tool that can really help you narrow these kind of problems in the field.
Remember, end-user systems don't have nice IDE's to debug and the end-users usually are not knowledgeable enough to run these tools. However end-users, in most cases, are capable of copying log configuration files ( written by us, clever programmers ) into predefined location and fetch log files and email them back to us ( poor soles for having to parse megabytes of log output ) when they encounter problems.
Having said this, logging should be highly configurable and under normal conditions produce minimal output. Also, guards should protect finer level logging from consuming too many resources.
I think in the example that you have provided all logging should have been done on a TRACE level. Also, because nothing bad can really happen between function entry point and exit, it probably make sense to have only one log statement there.

Over the years I've swayed backwards and forwards between promoting logging everything at the appropriate levels (trace, info, etc...) and thinking that any is a complete waste of time. In reality it depends on what is going to be useful to track down or required (logging can be a cheap way of maintaining an audit trail).
Personally, I tend to log entry/exit at a component or service level and then log significant points in the processing such as a business logic decision or a call on another service/component. Of course errors are always logged, but once only and at the place they were handled (the stack trace and exception message should have sufficient info to diagnose the problem) and any service/component interface should always handle an errors (even if it is just converting it into another more appropriate to the caller).
The problem with logging stuff on the off chance something goes wrong is that you end up with too much information that it is impossible to identify the issue, especially if it is running under a server as you end up with loads of intertwined log entries. Obviously you can get around that by incorporating a request id in the entry and using some software to filter on that. Of course you also have the case where your application is distributed and/or cluster and you have multiple logs.
Nowadays I would never actually write trace entering/exiting entries code, the code just gets in the way and it is so much easier to use something like aspectj if it is really needed. Using aspectj also would guarantee to be consistent (you can change the log format in one place rather than having to change every operation) and accurate (in case some refactoring adds a new paramater and teh developer forgets to add it to the logging).
One thing I have thought about doing or looking to see if someone already has is a logger that will hold the entries in memory, then if an error is encountered they are written, if the operation succeeds the entries are just discarded. If anyone knows of one (ideally for log4j) please let me know, alternatively I have a few ideas on how to implement this if anyone is interested in doing one.

This is where log levels are helpful. In general, log levels in the order of verbosity and priority are TRACE, DEBUG, INFO, WARN, ERROR, FATAL.
The developer has to take a conscious call to use the correct log level while logging in the code.
While creating an instance of Logger we have to pass the correct log level by choosing it from a config (always prefer config). This decides which levels to be logged. For example, while creating the logger, if the config for log level is set to "INFO", anything below "INFO" (TRACE, DEBUG) won't be logged.
For instance, in the example you mentioned above, a TRACE OR DEBUG level would make more sense.
In runtime in production, the config for log level should always be set to INFO.
When an issue occurs in production and if the developer wants to find out the root cause, they can request for changing the log level to TRACE or DEBUG (mostly inside a QA environment where they can replicate the scenario), to see what exactly is happening (The app sometimes has to be restarted to have the log level changed, but it is helpful).
Log levels is a great practice, as most of the times, we won't be able to launch a debugger in the landscapes. As we are skipping the unnecessary file writes by choosing a higher log level, the performance won't take a hit