I see that SLF4J is used for logging entries during stormcrawler components execution. However I cannot find the file containing the configuration used for SLF4J logging. Do I need a separate configuration file to enable debug level logging?
Related
We understand that Jetty 11 has basically changed logging from version 10 (no internal Jetty classes, moreover Jetty 11 is commited to use SLF4 as a base logging).
The problem
We have a rudimentary knowledge of SLF4J (used it before and we've even read the Jetty 11 SLF4J sources ,too), but currently we don't see any way to "teach" Jetty 11 a new logging (aka there are no "setLogging()" methods in the Jetty 11 source code as there were before).
Global (Jetty) parameters, alas, can't be our solution just yet.
The state (aka our requirements)
We have already solved the "RequestLog" outputs of Jetty, no problems there, we need the "normal" Jetty-log outputs.
We need to control (many) modules/jars etc. via a unified logging.
Our logging is simple but requires that no output happens on the console (stdout / stderr etc). In best case the logging gets an instance of an Exception/Runtime, too.
Therefore, we need to route the Jetty output from the "Jetty server" through our internal logging. Using SLF4? If there is no other way (and we see no other way up to now), gladly.
Switching back to Jetty 10, sadly, is not an option.
Could this be solved in any way we are not aware of (yet)? Any idea would be very appreciated, thank you!
The switch from Jetty logging to Slf4j was actually done in Jetty 10.0.0.
slf4j was designed for unified logging, it can capture into a single logging location implementation all of the logging events generated from libraries that use ...
slf4j API
java.util.logging API
log4j1 API
log4j2 API
commons-logging API
logback API
org.apache.juli.logging API
and if you use slf4j version 2.x series, there's even rudimentary support for capturing java.lang.System.Logger API.
With slf4j, you have 2 categories of jar files to think about.
Bridge API JARs
These are slf4j based JARs that merely capture the above logging events and route them to slf4j. You can choose 0..n of these JARs to use.
There's dozens of options here.
Here's some common ones
jcl-overs-slf4j - captures Jakarta Commons Logging events and sends to slf4j
jul-to-slf4j - captures Java Util Logging events and sends them to slf4j
log4j-over-slf4j - captures Log4j 1.x events and sends them to slf4j
log4j2-overs-slf4j - captures Log4j 2.x events and sends them to slf4j
osgi-over-slf4j - captures osgi logging bundle events and sends them to slf4j
See http://www.slf4j.org/legacy.html
Implementation Binding JAR
These are the implementation of slf4j-api, and is the final binding of all logging events, it is the thing that decides what to do with the logging event (eg: write it to disk, ignore it, send it to a logging database, etc)
You have many choices here as well, here's some common jars to pick from (pick only 1!)
logback-classic - slf4j to Logback (Eclipse Jetty group's favorite logging implementation)
slf4j-jdk14 - slf4j to Java Util Logging
slf4j-log4j12 - slf4j to Log4j 1.2.x
log4j-slf4j-impl - slf4j to Log4j 2.x (see https://logging.apache.org/log4j/2.x/log4j-slf4j-impl/)
slfj-jcl - slf4j to Jakarta Commons Logging
jetty-slf4j-impl - Jetty 10+ implementation of the slf4j api
See: http://www.slf4j.org/manual.html#swapping
Since Jetty 10.0.x, the jetty-slf4j-impl exists, which provides an out of the box implementation that simply writes to System.err (aka STDERR) with some decent logging filtering by level in the usual jetty-logging.properties.
See https://search.maven.org/artifact/org.eclipse.jetty/jetty-slf4j-impl
Important advice
Don't use multiple binding implementations. Narrow it down to 1 binding implementation and purge all other logging implementation jars.
Don't accidentally create a loop with introducing a Bridge API Jar and a Binding Implementation JAR with the same logging technology. (eg: using binding log4j-over-slf4j and slf4j-log4j12 at the same time)
There is no "configuration" to wire up these binding or bridge jars, their mere existence in the classloader is enough to make them work. See the slf4j manual on how that works.
We have already solved the "RequestLog" outputs of Jetty, no problems there, we need the "normal" Jetty-log outputs.
Interesting, this is "solved" by actually using slf4j, as that's the only non-deprecated implementation of RequestLog.Writer in Jetty 10 and Jetty 11.
The way this works, is the Slf4jRequestLogWriter will emit events to a single named logger (the name of which you can configure in Slf4jRequestLogWriter.setLoggerName(String)) using the slf4j-api. Then it reaches the logging implementation and is routed wherever that logging configuration decides based on that logger name (file, with rolling, syslog, sent to a different system for aggregation, logstash, etc)
Did you really implement your own RequestLog.Writer instead of just using your preferred logging logging library? (libraries like logback, log4j2, log4j1, and even java.util.logging can easily create separate log files for RequestLog events).
⚠️ Note: do not use logback-access for RequestLog at this time (It does not fully support jakarta.servlets yet, and has many bugs that result in bad request log data. See open PR at https://github.com/qos-ch/logback/pull/532)
I have the following simple code
AmazonEC2 ec2 = AmazonEC2ClientBuilder.standard().withRegion(Regions.US_EAST_1).build();
DescribeInstancesRequest request = new DescribeInstancesRequest();
DescribeInstancesResult response = ec2.describeInstances(request);
While running this code I see the multiline debug output. I show only the beginning below. How to remove this output?
14:03:29.877 [main] DEBUG com.amazonaws.AmazonWebServiceClient - Internal logging successfully configured to commons logger: true
14:03:38.129 [main] DEBUG com.amazonaws.metrics.AwsSdkMetrics - Admin mbean registered under com.amazonaws.management:type=AwsSdkMetrics
14:03:38.480 [main] DEBUG com.amazonaws.auth.AWSCredentialsProviderChain - Unable to load credentials from EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
14:03:38.482 [main] DEBUG com.amazonaws.auth.AWSCredentialsProviderChain - Unable to load credentials from SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey)
14:03:38.580 [main] DEBUG com.amazonaws.auth.AWSCredentialsProviderChain - Loading credentials from com.amazonaws.auth.profile.ProfileCredentialsProvider#7fb9f71f
14:03:38.786 [main] DEBUG com.amazonaws.request - Sending Request: POST https://ec2.us-east-1.amazonaws.com / Parameters: ({"Action":["DescribeInstances"],"Version":["2016-11-15"]}Headers: (User-Agent: aws-sdk-java/1.11.135 Mac_OS_X/10.12.6 Java_HotSpot(TM)_64-Bit_Server_VM/25.144-b01/1.8.0_144, amz-sdk-invocation-id: 0a1303b3-16c0-b140-8d9b-b2e22dc685b1, )
14:03:40.160 [main] DEBUG com.amazonaws.auth.AWS4Signer - AWS4 Canonical Request: '"POST /
amz-sdk-invocation-id:0a1303b3-16c0-b140-8d9b-b2e22dc685b1
amz-sdk-retry:0/0/500
host:ec2.us-east-1.amazonaws.com
user-agent:aws-sdk-java/1.11.135 Mac_OS_X/10.12.6 Java_HotSpot(TM)_64-Bit_Server_VM/25.144-b01/1.8.0_144
x-amz-date:20171113T190338Z
Log4j Assumption
Under the assumption your application's logging system is configured via a property file like log4j.properties present at the base of your Java classpath, you can reduce verbosity of the loggers by including the line
log4j.logger.com.amazonaws=WARN
WARN or ERROR will guarantee you get the least amount of logs for the loggers inside the AWS SDK.
Should you wish to keep a more granular level of logging for a specific component of the AWS SDK, you can always override the level on a restricted scope (i.e. a subpackage of the com.amazonaws), e.g.
log4j.logger.com.amazonaws.request=INFO
Logging frameworks supported by AWS Java SDK
The AWS SDK documentation states "Supported logging systems include the Java Logging Framework and Apache Log4j, among others".
I refer you to Logging AWS SDK for Java Calls for further reference.
Checking which logging framework is used by the test
The Apache Commons logging library referenced from Gradle is an abstraction layer on top of logging libraries which provide actual implementations of loggers. It is not, in itself, the logging implementation used by your test. In the statement you provided, you are using the Java logging framework ( java.util.Logger a.k.a JUL). But is it really the logging implementation your test uses?
A quick check you could do to verify if log4j is on your test's classpath is to add the VM argument -Dlog4j.debug=true and launch your test. If log4j is on your classpath you will see some output from the library printed on the console.
You could also check the runtime classpath of your test, because there may be transitive dependencies (as opposed to direct) to logging libraries, which may not be visible from the list of compile-time dependencies.
One last resort possibility could be to put a breakpoint there and debug to see which logger is used from the SDK itself (this code is called in your test because it is shown in the logs you provided).
I have been using spring boot and use aws client to get some secrets before bootstrap spring boot environment, but all secrets go to debug messages then I tried a lot of methods but this is the one works for me:
LoggerContext loggerContext = (LoggerContext)
LoggerFactory.getILoggerFactory();
loggerContext.getLogger("org.apache.http").setLevel(Level.INFO);
loggerContext.getLogger("com.amazonaws").setLevel(Level.INFO);
I am using Jetty http client in my application. I am also using SLF4J with LogBack.
How to set Jetty's overall logging level to INFO?
I have followed the instructions on Jetty's documentation to provide jetty-logging.properties file with contents like this:
# Configure Jetty for SLf4j Logging
org.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.Slf4jLog
# Overall Logging Level is INFO
org.eclipse.jetty.LEVEL=INFO
Jetty is logging fine via SLF4J, but the LEVEL setting doesn't seem to work.
When I set the level from Logback configuration then it works, but I want my code to be logged in DEBUG level and Jetty in INFO level.
Jetty is very noisy in DEBUG level.
Any help is appreciated!
You need to set the Properties....
// change Jetty Logger
Properties p = new Properties();
p.setProperty("org.eclipse.jetty.LEVEL", "OFF");
org.eclipse.jetty.util.log.StdErrLog.setProperties(p);
I wasn't able to make Jetty to log at custom level. Because if Jetty finds SLF4J in my application then it will start using it with the logging level I have specified in logback.xml configuration file.
So I made a little hack. I specified Jetty to use Java Utils Logging and set level to INFO.
// Setting Jetty logger implementation and level (DEBUG | INFO | WARN | IGNORE)
System.setProperty("org.eclipse.jetty.util.log.class",
"org.eclipse.jetty.util.log.JavaUtilLog");
System.setProperty("org.eclipse.jetty.util.log.class.LEVEL", "INFO");
And for redirecting Java Utils Logging messages to SLF4J I used jul-to-slf4j-1.7.12.jar
That's how I got it working. If there is a better way I sure want to know.
I declare loggers in my code like this:
Logger logger = LoggerFactory.getLogger(MyClass.class);
I would like to be able to change the log level of my loggers programmatically at runtime. I have tried iterating over the loggers in the repository via the LogManager, but it only applies the new log level to loggers that are instantiated in the code. If a new logger instance is created, it does not use the new log level. My requirement is such that the log level in my web application needs to be configurable via an Administrative GUI, and this new log level needs to apply to all loggers in my code (exclude third party library logging, e.g. JSF, Spring, Hibernate etc).
This is what I'm doing now which does not meet my requirements as it does not apply the log level to newly instantiated loggers:
Enumeration<Logger> currentLoggers = LogManager.getCurrentLoggers();
while(currentLoggers.hasMoreElements()) {
Logger logger = currentLoggers.nextElement();
if (StringUtils.startsWith(logger.getName(), "com.my.company")) {
logger.setLevel(logLevel);
}
}
I am using Log4j 1.2.17.
Didn't find the resource anymore, but as to my faded memory LogManager.getLogger("com.my.company").setLevel(whateverloglevel) should do the job.
All loggers that are created with LogManager.getLogger(MyClass.class) where MyClass is in com.my.company or in a subtree of that will be affected.
whateverloglevel is one of Level:
ALL The ALL has the lowest possible rank and is intended to turn on all logging.
DEBUG The DEBUG Level designates fine-grained informational events that are most useful to debug an application.
ERROR The ERROR level designates error events that might still allow the application to continue running.
FATAL The FATAL level designates very severe error events that will presumably lead the application to abort.
INFO The INFO level designates informational messages that highlight the progress of the application at coarse-grained level.
OFF The OFF has the highest possible rank and is intended to turn off logging.
TRACE The TRACE Level designates finer-grained informational events than the DEBUG
WARN The WARN level designates potentially harmful situations.
Be aware that this probably does not work in log4j 2.
For (I think) version >= 2.4 see:
log4j FAQ
Programmatically change log level in Log4j2
Suppose I have a web project (war) which is using logback logging but this project has a dependency of a jar which internally uses log4j logging. What will happen in this case? Will dependency logs appear in my logback log file or they just disappear (means I wont be able to see them anywhere) or some exception?
The logs coming from your dependency won't appear in your log file unless you use a bridge to redirect calls made to log4j by calls made to your logger.
If you use logback with slf4j, you can refer to this link to have more explanation on how to do that : http://www.slf4j.org/legacy.html