I'm using log4j2 and splunk within java to send logs into my Splunk Enterprise HEC (HTTP Event Collector) Splunk Enterprise is running in my local machine.
I'm doing all log4j2 configuration programmatically. (I know this is not the correct way to do this but I'm still doing this for learning purpose).
I tried to send the logs into Splunk Enterprise directly from postman with the same URL and token and it works fine, but when I tried to send the logs from java using log4j2 I don't get anything in splunk.
My code is =>
import org.apache.logging.log4j.Level;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.core.config.Configurator;
import org.apache.logging.log4j.core.config.builder.api.ConfigurationBuilder;
import org.apache.logging.log4j.core.config.builder.api.ConfigurationBuilderFactory;
import org.apache.logging.log4j.core.config.builder.impl.BuiltConfiguration;
import org.apache.logging.log4j.core.layout.PatternLayout;
import com.splunk.logging.*;
public class Main {
private static final Logger log;
static {
configureLog4J();
log = LogManager.getLogger(Main.class);
}
public static void configureLog4J() {
ConfigurationBuilder<BuiltConfiguration> builder =
ConfigurationBuilderFactory.newConfigurationBuilder();
// configure a splunk appender
builder.add(
builder.newAppender("splunkH", "SplunkHttp")
.add(
builder.newLayout(PatternLayout.class.getSimpleName())
.addAttribute(
"pattern",
"%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"
)
)
.addAttribute("sourcetype", "log4j2")
.addAttribute("index", "main")
.addAttribute("url", "http://localhost:8088/services/collector") //I tried this url in postman and its working fine there
.addAttribute("token", "xxx")
.addAttribute("disableCertificateValidation", "true")
);
// configure the root logger
builder.add(
builder.newRootLogger(Level.INFO)
.add(builder.newAppenderRef("splunkH"))
);
// apply the configuration
Configurator.initialize(builder.build());
}//end of configureLog4J
public static void main(String ar[]) {
log.log(Level.INFO, "Hello from log4j2");
log.log(Level.ERROR, "Error from log4j2");
}//end of main method
}//end of class
my POM file
<dependencies>
<dependency>
<groupId>com.splunk.logging</groupId>
<artifactId>splunk-library-javalogging</artifactId>
<version>1.11.4</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.11.2</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.11.2</version>
</dependency>
<dependency>
<groupId>com.splunk</groupId>
<artifactId>splunk</artifactId>
<version>1.6.5.0</version>
</dependency>
</dependencies>
<repositories>
<repository>
<id>splunk-artifactory</id>
<name>Splunk Releases</name>
<url>https://splunk.jfrog.io/splunk/ext-releases-local</url>
</repository>
</repositories>
I cannot see any logs in splunk. Did I miss something ?
Add .addAttribute("batch_size_count", "1") or make a loop producing 10 log messages, becasue that's the default value of batch_size_count. This has been explained in splunk docs "Configure Log4j 2" section.
By the way, I reckon the services/collector endpoint should be used with JSON messages (e.g. .add(builder.newLayout("JSONLayout"))). Also, you are using a log4j2 version that has the Log4Shell (CVE-2021-44228) vulnerability. It has been fixed in 2.15.0, switch to anything between that and the newest version 2.17.2.
Finally, I share the sentiment of the answers to the question How to configure log4j 2.x purely programmatically? that log4j2 is troublesome to use when configured programmatically. I had issues with it in a cluster env and switching to file configuration solved all my problems.
Related
I want to bean or inject logger so that I don't end up creating an object of logger in each and every class.
So I am trying to integrate Lombok which would help me to resolve lot of things including logger.
Here is my code:
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import lombok.extern.slf4j.Slf4j;
#Slf4j
#SpringBootApplication
public class MyApplication {
private static final Logger logger = LoggerFactory.getLogger(MyApplication.class);
public static void main(String[] args) {
SpringApplication.run(MyApplication.class, args);
log.debug("Hello");
logger.info("Info log", MyApplication.class.getSimpleName());
}
}
But this is giving me error when I try to use log directly
IDE error -
log cannot be resolved
Console runtime error -
Exception in thread "main" java.lang.NullPointerException: Cannot
invoke "org.slf4j.Logger.info(String, Object)" because
"com.MyApplication .log" is null at
com.MyApplication.main(MyApplication .java:27)
Pom.xml
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.12</version>
<scope>provided</scope>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-logging</artifactId>
<version>2.4.5</version>
</dependency>
I have also installed lomback to my Eclipse IDE
Looks like you didn't install the Lombok plugin in your IDE.
Take a look at https://projectlombok.org/setup/overview under IDEs. There you can find instructions on how to install the plugin on most common used IDEs.
Also try to remove the logger from your code. This line:
private static final Logger logger = LoggerFactory.getLogger(MyApplication.class);
Because this is what lombok is adding at compile time. And you don't need the logging dependency because it's already part of spring-boot
I have existing project with very standart Logback implementation.
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
class SomeClass{
Logger log = LoggerFactory.getLogger(getClass());
void someMethod(){
log.info("some log message");
}
}
This is used in many places and in code I dont have sources. ch.qos.logback.classic.Logger is used as implementation.
I want to use my own implementation of Logger (I wanted to extend ch.qos.logback.classic.Logger but it's final) and use it applicatio wide without modifying the code (which I don't have anyway).
I would expect that configuration of LoggerFactory in logback.xml would be possible, but it doesn't seem so.
I'm using maven artifacts
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.30</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
</dependency>
I could not find straight way of doing it. I came over XLogger and logback-ext but it seems I would need to modify all factory usages.
I am trying to write to my webapp logs to a .log file in the /logs directory in tomcat but the file is not generated nor is any logging output going to the console other than the spring logs and tomcat logs. When I run spring boot as a jar file with the embedded tomcat it writes to the log file just fine, but as soon as I deploy to tomcat via the webapps folder the application logs are no where to be found.
SpringBoot 2.1.2
Java 1.8
Tomcat 8.5
I have tried:
configuring LOGGING_CONFIG in setenv.sh
Multiple loggers.. logback, java utils etc..
application.properties:
logging.file=../logs/my-app.log
logging.level.org.springframework=INFO
logging.level.com.bose=DEBUG
log4j.properties for log4j:
log4j.rootLogger=${marge.log.level}, stdout, file
# Direct log messages to a log file
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=marge.log
#when stdout is also specified it will not write to the file
log4j.appender.file.MaxFileSize=1MB
# Keep one backup file
log4j.appender.file.MaxBackupIndex=1
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{dd MMM yyyy HH:mm:ss,SSS} [%c] [%-5p] %n%m%n
# Direct log messages to stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
# stdout uses PatternLayout
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{HH:mm:ss,SSS} [%c] [%-5p] %n%m%n
# Print only messages of level DEBUG or above in the package com.bose
log4j.logger.com.app=${log.level}
Expected: when I deploy my webapp in /webapps the application logs (generated by log4j) should be in my-app.log in the /logs directory
Actual: No file is generated and no logs are even in stdout/console
By default spring boot uses logback as a logging binder, so the key concept here to exclude logback first then include log4j
Example:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-logging</artifactId>
</exclusion>
</exclusions>
</dependency>
After that add log4j 2 dependency:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-log4j2</artifactId>
</dependency>
Add your log4j2.properties file to src/main/resources to be on classpath
Finally pay attention to what logging interface you are using this is important, with the above configuration you should use apache logging like:
package com.example;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.web.servlet.support.SpringBootServletInitializer;
import org.springframework.context.ApplicationContext;
#SpringBootApplication
public class Application extends SpringBootServletInitializer {
private static final Logger LOGGER = LogManager.getLogger(Application.class);
public static void main(String[] args){
ApplicationContext ctx = SpringApplication.run(Application.class, args);
LOGGER.info("Info level log message");
LOGGER.debug("Debug level log message");
LOGGER.error("Error level log message");
}
}
I am newbie to implement log4j2 with my aws java lambda cloud watch. I need custom log instead of cloud watch logs. I am uploading a csv of large size record using step function.So the built in cloud watch logs the same thing repeatedly. So I am planning to add log4j2 with my java lambda. For this I added below dependency in my pom.xml
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-log4j2</artifactId>
<version>1.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.8.2</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.8.2</version>
</dependency>
Then added the log4j2.xml under src/main/resources. The log4j2.xml is like below
<?xml version="1.0" encoding="UTF-8"?>
<Configuration packages="com.amazonaws.services.lambda.runtime.log4j2.LambdaAppender">
<Appenders>
<Lambda name="Lambda">
<PatternLayout>
<pattern>%d{yyyy-MM-dd HH:mm:ss} %X{AWSRequestId} %-5p %c{1}:%L - %m%n</pattern>
</PatternLayout>
</Lambda>
</Appenders>
<Loggers>
<Root level="debug">
<AppenderRef ref="Lambda" />
</Root>
</Loggers>
</Configuration>
After that to check the log i have created one aws java lambda project and my code looks like below.
package com.amazonaws.lambda.demo;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.S3Event;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.GetObjectRequest;
import com.amazonaws.services.s3.model.S3Object;
import com.amazonaws.services.stepfunctions.AWSStepFunctions;
import com.amazonaws.services.stepfunctions.AWSStepFunctionsClientBuilder;
import com.amazonaws.services.stepfunctions.model.StartExecutionRequest;
import com.amazonaws.services.lambda.runtime.Context;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
public class log4jTest implements RequestHandler<S3Event, String> {
static final Logger logger = LogManager.getLogger(log4jTest.class);
private AmazonS3 s3 = AmazonS3ClientBuilder.standard().build();
public log4jTest() {}
log4jTest(AmazonS3 s3) {
this.s3 = s3;
}
#Override
public String handleRequest(S3Event event, Context context) {
String bucket = event.getRecords().get(0).getS3().getBucket().getName();
String key = event.getRecords().get(0).getS3().getObject().getKey();
try {
for(int i=0;i<10;i++)
{
if(i==10)
{
logger.error("log data Error");
}
}
} catch (Exception e) {
context.getLogger().log(String.format(
"Error getting object %s from bucket %s. Make sure they exist and"
+ " your bucket is in the same region as this function.", key, bucket));
}
return null;
}
}
According to the aws document Logging (Java).
Everything I did as the doc says. But when I am run the lambda I am getting an error like below
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'org.apache.logging.log4j.simplelog.StatusLogger.level' to TRACE to show Log4j2 internal initialization logging.
I am checking the log in lambda -> Monitoring -> jump to logs [Screen shot given below]
I googled and went through below sites.
No log4j2 configuration file found. Using default configuration: logging only errors to the console
No log4j2 configuration file found. Using default configuration: logging only errors to the console
But I dont know I can't fix this error anyway. Please anyone can help me on this. It would be really grateful if u can do this.
Thanks in advance
I found I did not have any luck placing my log4j2.xml file at either src/main/resources or at src/main/resources/path/to/my/Lambda/log4j2.xml.
So I did a little digging. In my case, Lambda deploys my handler with a classpath whose first entry is a folder /var/task. The class files are rooted directly at /var/task - but the resource files are placed in /var/task/resources - so Log4J2 cannot find log4j2.xml in the classpath. The solution for me was to use the log4j.configurationFile property to specify the location of the file within the resources folder:
static {
// System.setProperty("org.apache.logging.log4j.simplelog.StatusLogger.level","TRACE");
System.setProperty("java.util.logging.manager", "org.apache.logging.log4j.jul.LogManager");
System.setProperty("log4j.configurationFile", "resources/lambda-log4j2.xml");
}
I also renamed the file "lambda-log4j2.xml" - to make sure it was reading my file, during debugging this...
This static block is at the top of my handler class.
Can anyone give an example of pom.xml dependencies configuration that will make OpenEJB to use slf4j logging, instead of JCL (this is what it uses now, as I understand).
see also How to configure OpenEJB logging?
We have support for:
Log4j
java.uti.logging
New implementations of LogStreamFactory can be plugged in by setting the class name as the value of openejb.log.factory system property.
Feel free to copy one of the existing impls and update for either JCL or SLF4J. The project is always accepting new committers if you get the urge to hack on other things as well!
Thanks to David's pointers, this is actually quite easy. You only need to implement two clases:
- Slf4jLogStreamFactory
- Slf4jLogStream
then set your factory :
System.setProperty("openejb.log.factory", "de.glauche.Slf4jLogStreamFactory");
package de.glauche;
import org.apache.openejb.util.LogCategory;
import org.apache.openejb.util.LogStream;
import org.apache.openejb.util.LogStreamFactory;
public class Slf4jLogStreamFactory implements LogStreamFactory {
#Override
public LogStream createLogStream(LogCategory logCategory) {
return new Slf4jLogStream(logCategory);
}
}
and the Slf4jLogStream:
package de.glauche;
import org.apache.openejb.util.LogCategory;
import org.apache.openejb.util.LogStream;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class Slf4jLogStream implements LogStream {
private Logger log;
public Slf4jLogStream(LogCategory logCategory) {
log = LoggerFactory.getLogger(logCategory.getName());
}
#Override
public boolean isFatalEnabled() {
return log.isErrorEnabled();
}
#Override
public void fatal(String message) {
log.error(message);
}
#Override
public void fatal(String message, Throwable t) {
log.error(message,t);
}
... (overwrite the remaining methods like this)
With this i'm getting all output nicely formated in my logback logger via slf4j :)
This is how I made OpenEJB to use external logging:
[...]
#Before
public void before() throws Exception {
System.setProperty("openejb.logger.external", "true");
InitialContext ctx = new InitialContext();
}
[...]
Maybe it's possible to move this system property to some global resource, like pom.xml?
If you add the API and JCL dependencies then your logging using the SLF4J API will get directed to the default JCL logs.
Is that what you want? Or do you want to use some other back end for the logging?
<dependencies>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.6.1</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-jcl</artifactId>
<version>1.6.1</version>
</dependency>
</dependencies>
Not a Maven expert, but from my understanding, you want this:
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>jul-to-slf4j</artifactId>
<version>1.6.1</version>
</dependency>
If OpenEJB doesn't support JCL, it would be pointless to add that dependency, so go with JUL or Log4j.
You can find more details at Maven repository.
Just in case you really want to use JCL bridge, use this:
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>jcl-over-slf4j</artifactId>
<version>1.6.1</version>
</dependency>
If you are executing via Maven's exec plugin, try this:
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.2.1</version>
<configuration>
<mainClass>your Mainclass here</mainClass>
<systemProperties>
<systemProperty>
<key>openejb.log.factory</key>
<value>slf4j</value>
</systemProperty>
</systemProperties>
</configuration>
</plugin>