Are logging tools like log4j useful only during development phase? [closed] - java

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 years ago.
Improve this question
I need to log the activity of the users connected to my server, should I use log4j? Or is log4j beneficial only during development phase?

They are actually not particularly useful during development, System.out.println is pretty good for most dev debug logging, but once you deploy the following abilities become really useful:
roll logfiles so they don't get too big allowing for continuous maintenance-free operation
add times/dates so you can look at the logs for a certain time period
Change verbosity on the fly (You don't always want trace or debug info, but being able to flip it on when he system isn't running well can be a lifesaver)
Re-route logfiles to a more accessible place... Log4j can send your logs to various databases or other locations for when you can't actually reach your server directly.
Some of our code has trace statements on every significant line. If we run into problems when we are developing we leave the debugging/trace statements in and are able to turn them on when we need to in production--almost equivalent to single-stepping through your deployed code. In addition most methods have trace or debug statements at the top showing the parameters being passed in and program flow--again only really useful for a deployed system where a debugger is unavailable.
So in short, yes it's useful after development.
edit (in response to comment question)--
Just as an example. The app I'm working on now has 20ish logs. One is "Performance", it logs data coming in including timings--sometimes more than one line a second. This logfile "Rolls" at 10mb (about hourly) but we use it to find lags in our data delivery. We even use other software to analyze this log sometimes to look for patterns in data timing.
We have a separate "Error" log that logs all error-level activity. This log doesn't roll so fast that we lose data when we are getting a bunch of other log information.
There is another log to put problems related to Hibernate/SQL and one for problems related to our message queue and one for our inter-app cache....
These are all also combined into a "main" log through the log4j config file. We can reconfigure any one log to a different level (for instance, we were having authentication problems with a data source so we turned up it's debugging level on that source to find out what had changed in our server's environment to cause that)
Even though some of the logs scroll 10mb in an hour (our max file size). Log4j will roll them into .1 and .2 files so we can keep 10-50 of them depending on need.
All of this is done through config files and can be changed after deployment without rebuilding the system.
Edit 2--another thought
Another useful point about using log4j and the java logging interface is that libraries like hibernate using it can be configured through xml files without rebuilding.
Without Log4j/java's logging APIs you would either A) have a custom API to control the logs, B) only have default logging or C) have no logging from that subsystem. Since Hibernate uses java's APIs however, you can set the log level for "Hibernate" log info in a standard, documented xml config file and even re-route it's logs to a database or other logging device.

Logging is especially useful to locate errors that occur in productive code. During development you can use more powerful tools like debuggers.
Of course you have to be aware that logging potentially affects performance and can create huge files. That's why tools like log4j can be configured to turn on and off logging or to control its verbosity.
It's ok to use log4j because it is the most common library for Java. However I personally find its configuration a bit unitntuitive.

Another reason for using loggers such as Log4j is to allow for different logging levels for different components. Logs can get pretty large and messy if you turn DEBUG on for everything. If you know what area of code you want to "magnify" in your logs you can ratchet up the logging for that area alone.

yes (or any other logging framework / SLF4J)
no

Related

log4j2 multiple log files performance vs one big log file

Working on a huge Java/Spring monolith app. We use log4j2 for logging. We have very few log files because our chief architect does not accept creating more log files for separate things, because a) he "knows" that every new file will impact performance negatively (writing to many files in a monolith will impact performance much more than writing to single one or the few we have) b) we use a log aggregator service in production and CI that can be used to filter out/sort all logs, and he considers that this is the way to go all time.
Problem is, though, that using so few log files makes a mess in the actual logs and without using any tools to sort and filter, developers usually look into those files only for errors and not understanding the flow or anything else, thus more time spent actually debugging app than adding logs or going through them. We do not have log aggregators for development machines obviously as it is a paid service for production/CI only.
Personally I do not like the idea of having a mess in the log files the way we have and consider it a mistake as it is hard to understand something in there if you open the logs with Notepad++ only, for example.
I have found the answer logging big file small files which somehow answers part of my question from one point of view, but still, does anyone have any idea if there is any issue with multiple files from performance perspective, or shall I spend time actually trying to prove one idea or the other?

How to avoid Log4J exploit? [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 1 year ago.
Improve this question
There is a serious security flaw in Log4J that has apparently been patched. But I have not found an intelligible explanation as to how there could even be a security flaw in a logging framework.
It seems to have something to do with "Lookups".
https://logging.apache.org/log4j/2.x/manual/lookups.html
But that is in a config file, not a log file that could be injected into?
Question. How do I kill all clever stuff once and for all to prevent any future exploits? I just want logging.
It smells like I need to remove any "$" signs but that is just a guess. (One should also remove any \ns to avoid log file spoofing. I always write a very simple wrapper for the logging to be able to do this sort of thing.)
(I have a very simple shim that all log messages go through, so it is easy for me to remove all "$"s that are not in format strings.)
The only way to prevent vulnaribilites caused by lookups is to disable them completely.
According to log4j2 team, the way to do that is by appending Java parameter
-Dlog4j2.formatMsgNoLookups=true
This definitely fixes the ability to call lookups by putting ${...} sequence s in the user input (like in URLs that are logged).
Unfortunately, the lookups are applied to the logged sequence, even if you use parameters or log exception stack traces. The lookups sequeces in your code are not dangerous, because they are controllable. But those that come from user are not predictable. If you log 'wrong username: xxx', you have no idea, what the user can type to exploit another vulnaribility.
However, you can't be sure there are no other grave errors in the framework, so switching to other logging framework is reasonable.
A good candidate is Logback, which is started by the original author of the Log4j. The developer states clearly, his framework has nothing to do with the security issues introduced in Log4j2:
Unless specified otherwise, when we say log4j we mean log4j 1.x. We
should also like to emphasize that logback is unrelated to log4j 2.x.
It does not share code nor vulnerabilities with log4j 2.x.
The bug is apparently fixed in the version 2.16.0, however it addresses only that particular vulnaribility, and the dangerous lookup feature was not removed.
According to the Sophos Report,
The updated version of Log4j still supports the potentially dangerous
what-you-see-is-not-what-you-get system of string “lookups”, but
network-based JNDI connections, whether on the same machine or
reaching out to somewhere else, are no longer enabled by default.
In other words, log4j2 still remains unsecure, but not as ridiculously insecure as previously.
For security issues like this one it makes sense to keep the up to date information in a single place.
Please refer to the official Log4j information about this vulnerability here: https://logging.apache.org/log4j/2.x/security.html
The document states several mitigation measures applicable for different versions of Log4j.
I realize that in general the StackOverflow convention is to have explicit steps and examples in the answer here on StackOverflow, rather than link to external documentation, but I believe that sensitive security flaws should be an exception.

Logging from Java app to ELK without need for parsing logs

I want to send logs from a Java app to ElasticSearch, and the conventional approach seems to be to set up Logstash on the server running the app, and have logstash parse the log files (with regex...!) and load them into ElasticSearch.
Is there a reason it's done this way, rather than just setting up log4J (or logback) to log things in the desired format directly into a log collector that can then be shipped to ElasticSearch asynchronously? It seems crazy to me to have to fiddle with grok filters to deal with multiline stack traces (and burn CPU cycles on log parsing) when the app itself could just log it the desired format in the first place?
On a tangentially related note, for apps running in a Docker container, is best practice to log directly to ElasticSearch, given the need to run only one process?
If you really want to go down that path, the idea would be to use something like an Elasticsearch appender (or this one or this other one) which would ship your logs directly to your ES cluster.
However, I'd advise against it for the same reasons mentioned by #Vineeth Mohan. You'd also need to ask yourself a couple questions, but mainly what would happen if your ES cluster goes down for any reason (OOM, network down, ES upgrade, etc)?
There are many reasons why asynchronicity exists, one of which is robustness of your architecture and most of the time that's much more important than burning a few more CPU cycles on log parsing.
Also note that there is an ongoing discussion about this very subject going on in the official ES discussion forum.
I think it's usually ill-advised to log directly to Elasticsearch from a Log4j/Logback/whatever appender, but I agree that writing Logstash filters to parse a "normal" human-readable Java log is a bad idea too. I use https://github.com/logstash/log4j-jsonevent-layout everywhere I can to have Log4j's regular file appenders produce JSON logs that don't require any further parsing by Logstash.
There is also https://github.com/elastic/java-ecs-logging which provides a layout for log4j, log4j2 and Logback. It's quite efficient and the Filebeat configuration is very minimal.
Disclaimer: I'm the author of this library.
If you need a quick solution I've written this appender here Log4J2 Elastic REST Appender if you want to use it. It has the ability to buffer log events based on time and/or number of events before sending it to Elastic (using the _bulk API so that it sends it all in one go). It has been published to Maven Central so it's pretty straight forward.
As the other folks have already mentioned the best way to do it would be to save it to file, and then ship it to ES separately. However I think that there is value if you need to get something running quickly until you have time/resources implement the optimal way.

Advantage of log4j

What's the advantage of log4j over set System.out and System.err to output to a log file?
At a high level, the win from Log4j over manual logging is that you can decouple your logging code from what you actually want to log and where and how you want to log it. Details about logging verbosity/filtering, formatting, log location, and even log type (files, network, etc.) are handled declaratively using configuration and extensibly via custom appenders, rather you having to code that flexibility yourself.
This is critically important because it's often hard for developers to predict how logging needs will change once their software is in production. Operations teams managing that software may need less verbose logs, may need mulitple logs, may need to ship those logs to multiple servers, may need to sometimes get really verbose data for troubleshooting, etc. And it's usually impossible for operations teams, if they need to change how logging works, to convince the developer to make big code changes. This often leads to production downtime, friction between operations and development, and wasted time all around.
From the developer's point of view, Log4j insulates you from having to make code changes to support logging, and insulates you from being pestered by people who want logging changes. It enables people managing your code to scratch their own itch rather than bugging you!
Also, since Log4j is the de-facto standard for Java logging, there are lots of tools available which can do cool things with Log4j-- furthermore preventing you and your operations teams from re-inventing the wheel.
My favorite feature is the ability to easily write appenders send data to non-file sources, like SYSLOG, Splunk, etc. which makes it easy to your app's custom logging into operations management tools your IT department is already using.
Actually, you should look into the slf4j facade these days, as it allows you to use {}-placeholders for the most concise statements. You can then use the appropriate logging framework behind slf4j to handle the actual treatment of your log statements. This could be log4j or the slf4j-simple which just prints out all of INFO, WARN and ERROR, and discards the rest.
The crucial observation you need to make is that the WRITING of log statements is done when the code is written, and the DECISION of what is needed is done when the code is deployed, which may be years after the code was written and tested. System.out.println requires you to physically change your code to get rid of them, which is unacceptable in a rigid write-test-deploy cycle. IF the code changes, it must be retested. With slf4j you just enable those you want to see.
We have full logging in the test phase, and rather verbose logging in the initial period of a production deployment, after which we go down to information only. This gives us full information in a scenario where debugging a case is very rarely possible.
You might find this article I wrote interesting. The target audience is beginning Java programmers, with my intention of giving them good habits from the start. http://runjva.appspot.com/logging101/index.html
my favorites (not all)
Ability to set parameters of logging in config, without recompiling
Ability to set the way log is written (from text file to SMTP sender)
Ability to filter by severity
Levels, formatting, logging to multiple files... A logging framework (even if it's java.util.logging) is really beneficial if there's a chance anything may go wrong while your code is running.
log4j allows you to log to various resources e.g. event log, email, file system etc while allowing your application to remain decoupled from all of these resources. Furthermore, you get to use a common interface to log to all of the various resources without having to learn or integrate thier corresponding APIs.
Log4j offers the ability to rotate your log files based on size and delete them based on quantity (logrotate), so your servers don't fill up their disks. Personally I think that is one of the more valuable features in Log4j.
Also Log4j is popular and understood by many developers. The last three companies I've worked at have all used Log4j in most projects.
Take a look and you will understand the power of log4j :
log4j.properties I used once for a project :
# ALL < DEBUG < INFO < WARN < ERROR < FATAL < OFF
# No appenders for rootLogger
log4j.rootLogger=OFF
folder=..
prefix=
fileExtension=.log
htmlExtension=${fileExtension}.html
datestamp=yyyy-MM-dd/HH:mm:ss.SSS/zzz
layout=%d{${datestamp}} ms=%-4r [%t] %-5p %l %n%m %n%n
# myLogger logger
log4j.logger.myLogger=ALL, stdout, infoFile, infoHtml, errorFile
# stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=${layout}
# infoFile
log4j.appender.infoFile=org.apache.log4j.FileAppender
log4j.appender.infoFile.File=${folder}/${prefix}_info${fileExtension}
log4j.appender.infoFile.layout=org.apache.log4j.PatternLayout
log4j.appender.infoFile.layout.ConversionPattern=${layout}
# infoHtml
log4j.appender.infoHtml=org.apache.log4j.FileAppender
log4j.appender.infoHtml.File=${folder}/${prefix}_info${htmlExtension}
log4j.appender.infoHtml.layout=org.apache.log4j.HTMLLayout
log4j.appender.infoHtml.layout.Title=Logs
log4j.appender.infoHtml.layout.LocationInfo=true
# errorFile
log4j.appender.errorFile=org.apache.log4j.FileAppender
log4j.appender.errorFile.File=${folder}/${prefix}_error${fileExtension}
log4j.appender.errorFile.layout=org.apache.log4j.PatternLayout
log4j.appender.errorFile.layout.ConversionPattern=${layout}
# APPENDERS SETTINGS
log4j.appender.stdout.Threshold = ALL
log4j.appender.infoFile.Threshold = INFO
log4j.appender.infoHtml.Threshold = INFO
log4j.appender.errorFile.Threshold = WARN.
To change the variables in your java code you can do :
Loading Configuration
Log4j will automatically load the configuration if it is stored in a
file called "log4j.properties" and is present on the classpath under
"" (e.g. WEB-INF/classes/log4j.properties).
I don't like that approach and prefer to load the configuration
explicitly by calling:
PropertyConfigurator.configure( Config.ETC + "/log4j.properties" );
This way I can reload the configuration at any time as long as my
application is still running. I like to add a button to an
administrative jsp, "Reload Log4J".
Dynamic Log File Location
Many people complain that Log4j forces you to hard-code the location
where your logs will be kept. Actually, it is possible to dynamically
choose the log-file location, especially if you use the ${log.dir}
property substitution technique above. Here's how:
String dynamicLog = // log directory somehow chosen...
Properties p = new Properties( Config.ETC + "/log4j.properties" );
p.put( "log.dir", dynamicLog ); // overwrite "log.dir"
PropertyConfigurator.configure( p );
logging (Document historical business events that occur, you can check old logs)
track the application (project flow)
debugging the application (Detailed information what occurs in a method at granular level //data, value and all inside methods)
error handling (information about specific error that occur)

Java Logging vs Log4J [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
Is it still worth to add the log4j library to a Java 5 project just to log
let's say some exceptions to a file with some nice rollover settings.
Or will the standard util.logging facility do the job as well?
What do you think?
I'd say you're probably fine with util.logging for the needs you describe.
For a good decision tree, have a look at Log4j vs java.util.logging
Question One :
Do you anticipate a need for any of the clever handlers that Log4j has that JUL does not have, such as the SMTPHandler, NTEventLogHandler, or any of the very convenient FileHandlers?
Question Two :
Do you see yourself wanting to frequently switch the format of your logging output? Will you need an easy, flexible way to do so? In other words, do you need Log4j's PatternLayout?
Question Three :
Do you anticipate a definite need for the ability to change complex logging configurations in your applications, after they are compiled and deployed in a production environment? Does your configuration sound something like, "Severe messages from this class get sent via e-mail to the support guy; severe messages from a subset of classes get logged to a syslog deamon on our server; warning messages from another subset of classes get logged to a file on network drive A; and then all messages from everywhere get logged to a file on network drive B"? And do you see yourself tweaking it every couple of days?
If you can answer yes to any of the above questions, go with Log4j. If you answer a definite no to all of them, JUL will be more than adequate and it's conveniently already included in the SDK.
That said, pretty much every project these days seems to wind up including log4j, if only because some other library uses it.
I recommend that you use the Simple Logging Facade for Java (SLF4J). It supports different providers that include Log4J and can be used as a replacement for Apache Commons Logging.
Log4j has been around for a long time, and it works very well. I have no scientific study to back it, but based on what I've seen at a large number of clients, it is easily the logging framework that I see used more than any other. It has been around for a long time, and not been replaced by the Next Big Logging Framework, which says something.
It is dead simple to set up, and easy to learn the basic appenders (outputs). There are a whole host appenders that are available, including:
ConsoleAppender
DailyRollingFileAppender
ExternallyRolledFileAppender
FileAppender
JDBCAppender
JMSAppender
NTEventLogAppender
RollingFileAppender
SMTPAppender
SocketAppender
SyslogAppender
TelnetAppender
WriterAppender
Plus others. It isn't difficult to write your own appender either. Additionally there is a great deal of flexibility in each of the appenders that allow you to control specifically what is output in your log.
One note, I had a series of classloader problems when I used apache commons logging in addition to log4j. It was only for one specific application, but I found it simpler to use log4j alone, rather than to have the flexibility offered when using an abstraction layer like commons logging.
See this article for
more details:
Good luck!
java.util.logging offers a comprehensive logging package without the excess baggage some of the others provide..
log4j is a much nicer package overall, and doesn't have some of the hiccups that java.util.logging contains. I'd second that using log4j directly is easier than using the commons logging.
I recommend using Apache Commmons Logging as your logging interface. That way you have the flexibility to switch logging implementations anytime you want without requiring any code changes on your end.
I would go with log4j. The possibilites with log4j is not obsolete at all!

Categories