I am using slf4j, implementation of log4j for logging in my java project. Currently I am having 2 appenders, FILE and CONSOLE.
I want to know following 2 things:
Does using multiple appenders (in this case CONSOLE and FILE) causes performance issue in logging?
When somebody would want to use CONSOLE and FILE appenders both?
When writing to CONSOLE and FILE, you are writing to 2 different streams. In a multithreaded system, the performance hit will not be much, but with big volumes it is still apparent.
From the log4J manual
The typical cost of actually logging is about 100 to 300 microseconds.
This includes building the statement and writing it, but the time taken for writing will still be apparent if you are logging heavily.
But you need to ask a more basic question - Why are you logging?
to keep track of what is going on
to find out errors
The CONSOLE is not useful for the first part as the logs are not saved anywhere. If the logging is heavy, and all the logs are sent to the CONSOLE, the amount of logs will make the output on the console unreadable, so purpose 2 is also defeated.
IMO it makes much more sense reading logs from a file using something like less. As a general practice, you log to file and if you must, log only the ERROR messages to console, as a few ERROR messages would be an indicator of something going wrong, whereas hundreds of log lines on console is just junk, as you cannot make any sense of it when the console is refreshing so rapidly.
TL-DR
The cost might not be much, but why incur an added cost when you are getting no added advantage?
Read these links on log 4j performance.
log4j-performance
log4j-decreased application performance
log4j appenders
I challenge you to notice any performance change.
For instance you might want a daemon application to log both in the console and in a file. It does not seem to be such an uncommon behavior.
Related
I have a server which generates about 5MB/s of logs and much more during peaks. My initial test setup was to just use systemd to forward the output to rsyslog and then sending the data to a file. As I expected, this is very expensive with systemd-journal eating up a large chunk of my CPU resources and prevents the server from being able to handle traffic peaks.
I'm looking for a logging setup that would be both high-performance as well as support log rotation. I ran into the following article
https://medium.com/javarevisited/high-performance-logging-java-59ba374b2166
which mentions the memory mapped file appender for log4j. Log4j also has a rotating file logger, but these two do not compose, so I would need to use something external to rotate the logs. However, normally logrotate on the machine sends a HUP signal to tell syslog that a file has been rotated, so that it can close file handles and reopen the new file. This won't really work for a Java server using log4j.
Any recommendations for what to use? My code currently uses slf4j to actual emit stuff to logs, so a big plus if it can interoperate with it.
Encountered a frustrating problem in our application today which came down to an ArrayIndexOutOfBounds exception being thrown. The exception's type was just about all that was logged which is fairly useless (but, oh dear legacy app, we still love you, mostly). I've redeployed the application with a change which logs the stack trace on exception handling (and immediately found the root cause of the problem) and wondered why no one else did this before. Do you generally log the stack trace and is there any reason you wouldn't do this?
Bonus points if you can explain (why, not how) the rationale behind having to jump hoops in java to get a string representation of a stack trace!
Some logs might contain sensitive data, log facilities are not necessarily secure enough to track that data in production.
Logging to much can result in too much information, i.e. no information at all for the sysadmins. If their logs are filled up with debug messages, they won't be able to recognize suspicious patterns. (Years ago I saw a system logging all system calls for security reasons. There were so many logs, that nobody saw it when some unprivileged users started to become root.)
Best thing to do to log everything with appropriate log levels, and be able to set log levels in production (at least in Java not that a big issue).
Please see these questions also
Logging in Java and in general: Best Practices?
Best practices for Java logging from multiple threads?
Important things here to consider
Handling Sensitive data
Compact exception messages and mailing those to appropriate fixers
Logging what is required. Because its logging expensive in terms space
and time
I generally do log the stack trace, because it has information for troubleshooting/debugging the problem. It's the best think next to a minidump and often leads to a solution simply by code inspection and identifying the problem.
BTW, I agree with sibidiba about the potential information disclosure about your app internals a full stack exposes: the function names, along with the stack call sequence, can tell a lot to an educated reader. This is the reason why some products only log the symbol address on the stack, and rely on the devs to resolve the address to the name from internal pdbs.
But in I reckon that logging text into files containing 1 line of error and 14 lines of stack makes it very difficult to navigate the error logs. It also causes problem on high concurency apps because the lock on the log file is held longer (or worse, the log files get interleaved). Having encountered these problems my self many times, along with other issues in supporting and troubleshooting deployments of my own apps, led me to actually create a service for logging errors at bugcollect.com. When designing the error collection policies I chose to collect the stack dumps every time, and to use the stacks as part of the bucket keys (to group errors that happen on the same stack into same bucket).
Restrictions on logging are often pushed through when developers log too liberally and sysadmins discover that the app, once put under a production load, thrashes and fills the HD with huge log files. It can then be hard to convince them that you've seen the error of your ways and have reduced logging (or adjusted log levels) sufficiently but really need those remaining log entries.
For us it is very simple: If there is an unexpected exception thrown, we log the stack trace along with as telling a message as possible.
My guess is that the developer who wrote the original code in the question, simply wasn't experienced enough to know that it is not enough with just the message. I thought so too, once.
The reason why it is convoluted to get a stack trace as a string is because there is no StringPrintWriter in the JRE - I believe the line of thinking has been that they provide a lot of orthogonal building blocks which you then combine as needed. You have to assemble the needed PrintWriter yourself.
Bonus points if you can explain (why,
not how) the rationale behind having
to jump hoops in java to get a string
representation of a stack trace!
Shouldn't you just log the throwable instead of going through hoops to print the stacktrace? Like this: log.error("Failed to deploy!", ex). Given a throwable, Log4J will print both the error message obtained via getMessage() and the stack trace.
What I've seen a lot is code logging an exception like this:
LOG.error(ex);
Because log4j accepts an Object as the first argument, it will log the String representation of the Exception, which is often only the name of the class. This is usually just an oversight on the developer's part. It's better to log and error like this:
LOG.error("foo happened", ex);
..so that if configured properly, the logging framework will log the stack trace.
Are there any known bugs with the Log4J rolling file appender. I have been using log4j happily for a number of years but was not aware of this. A colleague of mine is suggesting that there are known issues ( and i found one a Bugzilla entry on this) where under heavy load,the rolling file appender (we use the time-based one) might not perform correctly when the rollover occurs # midnight.
Bugzilla entry - https://issues.apache.org/bugzilla/show_bug.cgi?id=44932
Appreciate inputs and pointers on how others overcome this.
Thanks,
Manglu
I have not encountered this issue myself, and from the bug report, I would suspect that it is very uncommon. Th Log4j RollingFileAppender has always worked in a predictable and reliable fashion for the apps I have developed and maintained.
This particular bug, If I understand correctly, would only happen if there are multiple instances of Log4j, like if you had multiple instances of the same app running simultaneously, writing to the same log file. Then, when it is rollover time, one instance cannot get a lock on the file in order to delete it and archive its contents, resulting in the loss of the data that was to be archived.
I cannot speak to any of the other known bugs your colleague mentioned unless you would like to cite them specifically. In general, I believe Log4j is reliable for production apps.
#kg, this happens to me too. This exact situation. 2 instances of the same program.
I updated it to the newer rolling.RollingFileAppender instead of using DailyFileRoller( whatever it was called ).
I run two instances simultenously via crontab. The instances output as many messages as they can until 5 seconds is reached. They measure the time for 1 second by using System.currentTimeMillis, and append to a counter to estimate a 5 second timeperiod for the loop. So there's minimum overhead in this test. Each output log message contains an incrementing number, and the message contains identifiers set from commandline to be able to separate them.
From putting the log message order together, one of the processes succeeds in writing from the start to end of the sequence, the other one loses the first entries of its output (from 0 onward).
This really ought to be fixed...
I have an application that runs slow. This is because of a huge amount of loggings at DEBUG and INFO levels inside the code. I have made some modifications in the code and changed the log level to WARN and it works well now.
But there is only one log file (currently at 1.6GB). I want to use a RollingFileAppender to have more, smaller, files. What is the best (maximum) size that I should use for the appender’s MaxFileSize property so that performance won’t degrade?
That really depends on many factors so to answer the question, you'd have to run a profiler with various file sizes. But since log4j only writes to the log file, you can simply create files of various sizes on your system and time how long it takes.
To be able to find errors in the file, I suggest to use a DailyRollingFileAppender, though. This will make it much more simple to look for something "yesterday" or "two weeks ago".
Having smaller chunks will make your log files more manageable, but it will not improve performance.
It seems that your limitation is the hdd. One solution to the performance issue would be to have WARN level logged in one file, and DEBUG and INFO in another. Ideally, you would have this larger file kept on a dedicated fast hdd.
Another solution to the performance issue would be to tweak logging to different package. You rarely need INFO on all packages, because parsing 2 GB of data would be hard to do, especially in real time.
Answer to the smallest size question:
It should be as large as your tools can handle without trouble. Let's say that you would use a log viewer to watch the log file. Some log viewers will perform badly on files, let's say, bigger than 10 MB. But, again, on 1 GB of log data generated in ... 1 hour maybe, you won't be able to watch it in real time.
The server.xml which controls the startup of Apache Tomcat's servlet container contains a debug attribute for nearly every major component. The debug attribute is more or less verbose depending upon the number you give it, zero being least and 99 being most verbose. How does the debug level affect Tomcat's speed when servicing large numbers of users? I assume zero is fast and 99 is relatively slower, but is this true. If there are no errors being thrown, does it matter?
Extensive logging takes a significant amount of time. This is why it is so important to put
if (log.isDebugEnabled())
log.debug(bla_bla_bla);
so I would say that seting your production server to being verbose would seriously affect performance. I assume it's a production server you're talking about since you say it must service a large number of users.
Logging is not only responsible for giving you errors, but also for tracking of what's going on. In some cases, code cannot run inside a debugger, then logging is your only option.
This is why logging output can be extremely verbose. And I really mean that. I remember setting Catalina's loglevel to TRACE once and ended up with a several megabyte logfile. That was before the server received any hits at all. It was a huge performance hog. Countable in several seconds.
If you don't need logging for Tomcat itself, don't activate it on any of its components. You will typically only want to tinker with Tomcat's loglevel if you suspect a bug in either your setup or Tomcat itself.
For your own applications, measure the logging cost using a profiler or just some stress testing. Whatever your results, I would recommend against running an application with a high loglevel setting in a production environment. My current project dumps about a megabyte per request at TRACE setting, only about three to four lines on INFO and nothing on WARNING (iff everything goes well :-). I recommend not more than the most necessary logging. Your app should really just report startup, shutdown and failure, and - at most - one line per request.