I am using AWS Step Functions Local for testing Step Functions on my local machine, but the level of logging has significantly increased the time for my test suite to run. I cannot find any configuration options on the AWS website to control the log level so I have been trying to roll my own with no success.
The logging I am trying to hide looks like this:
2022-12-15 12:11:36.562: CreateStateMachine => {"requestClientOptions":{"readLimit":131073,"skipAppendUriPath":false},"requestMetricCollector":null,"customRequestHeaders":null,"customQueryParameters":null,"cloneSource":null,"sdkRequestTimeout":null,"sdkClientExecutionTimeout":null,"name":"<MY STEP FUNCTION NAME","definition":"{MY STEP FUNCTION DEFINITION}"...}
2022-12-15 12:11:36.782: [200] CreateStateMachine <= {"sdkResponseMetadata":null,"sdkHttpMetadata":null,"stateMachineArn":"arn:aws:states:eu-west-1:123456789012:stateMachine:MY_STEP_FUNCTION_NAME","creationDate":1671106296763}
I downloaded the Zip file containing the Step Functions Local JAR file, extracted the zip file, extracted StepFunctionsLocal.jar, and found there was a log4j.properties file. "Excellent!", I thought, I can provide my own configuration file and it will be great. So I created a Docker image with the following Dockerfile:
FROM amazoncorretto:11
WORKDIR /jar
COPY . /jar
RUN yum -y install unzip zip && yum -y clean all && rm -rf /var/cache
RUN mkdir -p /jar/extracted
RUN unzip StepFunctionsLocal.zip -d /jar/extracted
ENV SFN_MOCK_CONFIG=""
ENV LAMBDA_ENDPOINT=""
ENV AWS_DEFAULT_REGION=""
EXPOSE 8083
CMD ["java", "-Dlog4j2.configurationFile=file:/jar/log4j.properties", "-jar", "/jar/extracted/StepFunctionsLocal.jar"]
With the following log4.properties file which is in the same directory as the Dockerfile:
# console is set to be a ConsoleAppender.
log4j.appender.console=org.apache.log4j.ConsoleAppender
# console uses PatternLayout.
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n
log4j.appender.APPLOG=amazon.platform.logging.PeriodicConcurrentFileAppender
log4j.appender.APPLOG.layout=amazon.platform.logging.SanitizingPatternLayout
log4j.appender.APPLOG.File=../logs/application.log
log4j.appender.APPLOG.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.APPLOG.layout.ConversionPattern=%p %d{D/HH:mm:ss:SSS} %t %c - %m%n
# Set root logger level to INFO and its only appender to console.
log4j.rootLogger=ERROR, APPLOG
#log4j.logger.com.amazon.coral.reflect=WARN
#log4j.logger.httpclient.wire=OFF
# Enable this to see everything that httpclient does.
#log4j.logger.org.apache.commons.httpclient=WARN
#log4j.logger.org.springframework=WARN
#log4j.logger.org.apache=WARN
#log4j.logger.com.amazonaws.swf.console=ERROR
#log4j.logger.com.amazonaws.console.auth=ERROR
This is the same file I found in the extracted StepFunctionsLocal.jar except all of the lines which configure the log level have been commented out and the root logger level set to ERROR.
I built the image and ran it, but the same logging still appeared. At this point I'm stumped. Am I going about this the right way? Have I missed something obvious?
Related
I'm currently writing a bash script to copy log files from one host server to another host server. I have an application installed and running on a Websphere application server. The script would copy both the Server logs as well as application log to a remote staging directory. I can retrieve the server log files' path using wsadmin. I'm facing a challenge to retrieve the application log file path from log4j.properties under WEB-INF/classes directory.
Currently, I'm hard coding the log file path in a csv and from the csv, I'm retrieving it and using the command :
find <LOG_FILE_PATH> -type f -name $filename -newermt "$user_date_from" ! -newermt "$user_date_to"
Sample csv file content:
user#server_IP,AppServerName,LogFilePath,IdentifierToDifferentiateWhetherFileisServerLogOrAppLog
I want to do away with this csv file hard coding option because:
I know that log4j.properties already stores the log file paths.
I want to retrieve the log file path from log4j.properties instead at runtime, just to make sure that I do not have to update the csv hard coded value in case there is any change in log4j.properties.
Sample log4j.properties file content:
logFileLoc=<AbsoluteLogFilePath>
log4j.rootLogger=INFO,RollingAppender,stdout
log4j.appender.RollingAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.RollingAppender.File=${logFileLoc}/<logFileName>.log
log4j.appender.RollingAppender.append=true
log4j.appender.RollingAppender.DatePattern='.'yyyy-MM-dd
log4j.appender.RollingAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.RollingAppender.layout.ConversionPattern=[%p] %d %c %M - %m%n
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{ABSOLUTE} %5p %c{2}:%L - %m%n
So, I wonder if there exists any wsadmin like admin service available with log4j that can serve my purpose? Any help will be much appreciated.
This is my log4j.properties file:
# LOG4J configuration
log4j.rootLogger=INFO, Appender1,Appender2
log4j.logger.org.hibernate=info
log4j.logger.org.quartz=info
log4j.appender.Appender1=org.apache.log4j.ConsoleAppender
log4j.appender.Appender1.layout=org.apache.log4j.PatternLayout
log4j.appender.Appender1.layout.ConversionPattern=%-7p %d [%t] %c %x - %m%n
log4j.appender.Appender2=org.apache.log4j.FileAppender
log4j.appender.Appender2.File=/home/diego/proyect/log/general.log
log4j.appender.Appender2.layout=org.apache.log4j.PatternLayout
log4j.appender.Appender2.layout.ConversionPattern=%-7p %d [%t] %c %x - %m%n
Locally works on my windows machine, but on the server (ubuntu 16.04) side the web application cannot write on the .log file, already modified permissions for the file:
chmod 777 /home/diego/proyect/log/general.log
this is the resuls of ls -lhs on the log directory:
-rwxrwxrwx 1 diego diego 0 Nov 16 14:23 general.log
Also not sure which is the tomcat user on the server.
I solved, it seems I need to add the log4j.properties files into the classpath, my project has the following structure:
main
src/main/java
src/main/resources/log4j.properties
webapp
WEB-INF/log4j.properties
So I put the file under src/main/resources directory and it works.
The fact that it is working on local environment mean that your configuration is correct. I think there is an issue with your environment. Here are a few things to check:
Does the tomcat process has permissions to write in file . i.e. are administrator rights required to write?
Check that there aren't any temporary files near tomcat's directories which may be preventing your changes.
Check that you don't have any other log4j config files on your classpath, as these could override your log4j file and preventing it . To double check this, you could try temporarily removing your log4j.properties file .
I have a kafka connect jar which needs to be run as a docker container. I need to capture all my connect logs on a log file in the container (preferably at a directory/file - /etc/kafka/kafka-connect-logs) which can later be pushed to localhost (on which docker engine is running) using volumes in docker. When I change my connect-log4j.properties to append into a log file, I see that no log file is created. If I try the same without docker and run the kafka connect on a local linux VM by changing connect-log4j.properties to write logs to a log file, it works perfectly but not from docker. Any suggestions will be very helpful.
Docker File
FROM confluent/platform
COPY Test.jar /usr/local/bin/
COPY kafka-connect-docker.sh /usr/local/bin/
COPY connect-distributed.properties /usr/local/bin/
COPY connect-log4j.properties /etc/kafka/connect-log4j.properties
RUN ["apt-get", "update"]
RUN ["apt-get", "install", "-yq", "curl"]
RUN ["chown", "-R", "confluent:confluent", "/usr/local/bin/kafka-connect-docker.sh", "/usr/local/bin/connect-distributed.properties", "/usr/local/bin/Test.jar"]
RUN ["chmod", "+x", "/usr/local/bin/kafka-connect-docker.sh", "/usr/local/bin/connect-distributed.properties", "/usr/local/bin/Test.jar"]
RUN ["chown", "-R", "confluent:confluent", "/etc/kafka/connect-log4j.properties"]
RUN ["chmod", "777", "/usr/local/bin/kafka-connect-docker.sh", "/etc/kafka/connect-log4j.properties"]
EXPOSE 8083
CMD [ "/usr/local/bin/kafka-connect-docker.sh" ]
connect-log4j.properties
# Root logger option
log4j.rootLogger = INFO, FILE
# Direct log messages to stdout
log4j.appender.FILE=org.apache.log4j.FileAppender
log4j.appender.FILE.File=/etc/kafka/log.out
# Define the layout for file appender
log4j.appender.FILE.layout=org.apache.log4j.PatternLayout
log4j.appender.FILE.layout.conversionPattern=%m%
log4j.logger.org.apache.zookeeper=ERROR
log4j.logger.org.I0Itec.zkclient=ERROR
kafka-connect-docker.sh
#!/bin/bash
export CLASSPATH=/usr/local/bin/Test.jar
exec /usr/bin/connect-distributed /usr/local/bin/connect-distributed.properties
It works fine when I am using the default connect-log4j.properties (appends logs to console), but am unable to create a log file in docker. Also, same process without docker works fine (creates log file ) in local VM.
Result of the discussion:
Declare the volume right away in the dockerfile and configure your logging configuration to place the log output directly to the volume. On docker run (or however you start your container) mount the volume and you should be fine.
I have maven application with log4j.properties in it with setting to write the logs to a specified file instead of console. When I run the EAR on one of the websphere servers it is creating the file as expected and writing logs to it. But, when I am running the same EAR on other webspehere server it is writing to console instead of writing the logs to the specified file. I have checked the permissions and everything seems to be fine. Please help me in identifying what the issue is. Thanks in advance.
# CONSOLE APPENDER (stdout)
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Threshold=DEBUG
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] [%t] %-5p %20c - %m%n
# ROLLING FILE APPENDER (on the file system) for memberpolicyattributesservice code
log4j.appender.xxxxService=org.apache.log4j.RollingFileAppender
log4j.appender.xxxxService.Threshold=DEBUG
log4j.appender.xxxxService.File=/var/logs/xxxServer1/xxxServiceLog.log
log4j.appender.xxxxService.layout=org.apache.log4j.PatternLayout
log4j.appender.xxxxService.layout.ConversionPattern=%d{MM-dd#HH:mm:ss} %-5p (%13F:%L) %3x - %m%n
log4j.appender.xxxxService.MaxFileSize=10000KB
log4j.appender.xxxxService.MaxBackupIndex=30
log4j.appender.xxxxService.layout=org.apache.log4j.PatternLayout
log4j.appender.xxxxService.layout.ConversionPattern=[%d] [%t] %-5p %20c - %m%n
# ROLLING FILE APPENDER (on the file system) for hiberate, open source code log files
log4j.appender.open_source_code=org.apache.log4j.RollingFileAppender
log4j.appender.open_source_code.layout=org.apache.log4j.PatternLayout
log4j.appender.open_source_code.Threshold=DEBUG
#message format:YYYY-MM-DD HH:mm:ss,ms [ThreadId] <PRIORITY> classname.message
log4j.appender.open_source_code.layout.ConversionPattern=%d [%t]<%-5p> %c.%m \r\n
#file that will be logged to
log4j.appender.open_source_code.File=/var/logs/xxxServer1/open_source_code.log
log4j.appender.open_source_code.Append=true
log4j.appender.open_source_code.MaxFileSize=1024KB
log4j.appender.open_source_code.MaxBackupIndex=5
#turn on log4j verbose mode
log4j.debug = true
# Set root logger level to INFO and its appender to DSInstrumentor,stdout.
log4j.rootLogger=DEBUG,stdout,xxxxService
# YOUR CATEGORIES (to customize logging per class/pkg/project/etc)
log4j.category.fatal=FATAL,xxxxService
log4j.category.error=ERROR,xxxxService
#This will also enable the logging for all the children (packages and classes) of this package
log4j.logger.com.xxxxx=ALL,xxxxService
# Print only messages of level INFO in the open source code
log4j.logger.org=INFO,open_source_code
You have multiple loggers defined in your root logger (DEBUG, stdout, xxxxService) but the xxxxService loggers look like they're bound to the file system on one of the systems:
log4j.appender.open_source_code.File=/var/logs/xxxServer1/open_source_code.log
Make sure that path is valid for each server in your WAS cluster.
As a side note, you should probably avoid using debug and stdout on a remote server. This is fine for local development in your workstation, but not on a remote box. Instead, provide different log4j properties on your various deployment tiers. This lets you customize the log location or appender (say c:\temp or CONSOLE on your desktop, but /var/logs ... on all remote machines) as well as your log levels (DEBUG for desktop, maybe INFO for QA or Staging, and WARN or ERROR for Production).
I am trying to run a spark job in EMR cluster.
I my spark-submit I have added configs to read from log4j.properties
--files log4j.properties --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/log4j.properties"
Also I have added
log4j.rootLogger=INFO, file
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=/log/test.log
log4j.appender.file.MaxFileSize=10MB
log4j.appender.file.MaxBackupIndex=10
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %5p %c{7} - %m%n
in my log4j configurations.
Anyhow I see the logs in the console, though I don't see the log file generated. What am I doing wrong here ?
Quoting spark-submit --help:
--files FILES Comma-separated list of files to be placed in the working directory of each executor. File paths of these files in executors can be accessed via SparkFiles.get(fileName).
That doesn't much say what to do with the FILES if you cannot use SparkFiles.get(fileName) (which you cannot for log4j).
Quoting SparkFiles.get's scaladoc:
Get the absolute path of a file added through SparkContext.addFile().
That does not give you much either, but suggest to have a look at the source code of SparkFiles.get:
def get(filename: String): String =
new File(getRootDirectory(), filename).getAbsolutePath()
The nice thing about it is that getRootDirectory() uses an optional property or just the current working directory:
def getRootDirectory(): String =
SparkEnv.get.driverTmpDir.getOrElse(".")
That gives as something to work on, doesn't it?
On the driver the so-called driverTmpDir directory should be easy to find in Environment tab of web UI (under Spark Properties for spark.files property or Classpath Entries marked as "Added By User" in Source column).
On executors, I'd assume a local directory so rather than using file:/log4j.properties I'd use
-Dlog4j.configuration=file://./log4j.properties
or
-Dlog4j.configuration=file:log4j.properties
Note the dot to specify the local working directory (in the first option) or no leading / (in the latter).
Don't forget about spark.driver.extraJavaOptions to set the Java options for the driver if that's something you haven't thought about yet. You've been focusing on executors only so far.
You may want to add -Dlog4j.debug=true to spark.executor.extraJavaOptions that is supposed to print what locations log4j uses to find log4j.properties.
I have not checked that answer on a EMR or YARN cluster myself but believe that may have given you some hints where to find the answer. Fingers crossed!
With Spark 2.2.0 standalone cluster, executor JVM is started first and only then Spark distributes application jar and --files
Which means passing
spark.executor.extraJavaOptions=-Dlog4j.configuration=file:log4j-spark.xml
does not makes sense as this file does not exist yet (is not downloaded) at the executor JVM launch time and log4j initialization
If you pass
spark.executor.extraJavaOptions=-Dlog4j.debug -Dlog4j.configuration=file:log4j-spark.xml
you will find at the beginning of the executor's stderr failed attempt to load log4j config file
log4j:ERROR Could not parse url [file:log4j-spark.xml].
java.io.FileNotFoundException: log4j-spark.xml (No such file or directory)
...
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
And bit later is logged download of --files from driver
18/07/18 17:24:12 INFO Utils: Fetching spark://123.123.123.123:35171/files/log4j-spark.xml to /ca/tmp-spark/spark-49815375-3f02-456a-94cd-8099a0add073/executor-7df1c819-ffb7-4ef9-b473-4a2f7747237a/spark-0b50a7b9-ca68-4abc-a05f-59df471f2d16/fetchFileTemp5898748096473125894.tmp
18/07/18 17:24:12 INFO Utils: Copying /ca/tmp-spark/spark-49815375-3f02-456a-94cd-8099a0add073/executor-7df1c819-ffb7-4ef9-b473-4a2f7747237a/spark-0b50a7b9-ca68-4abc-a05f-59df471f2d16/-18631083971531927447443_cache to /opt/spark-2.2.0-bin-hadoop2.7/work/app-20180718172407-0225/2/./log4j-spark.xml
It may work differently with yarn or another cluster manager but with standalone cluster, it seems there is no way you can specify your own logging configuration for executors on spark-submit.
You can dynamically reconfigure log4j in your job code (override log4j configuration programmatically: file location for FileAppender), but you would need to do it carefully in some mapPartition lambda that is executed in executor's JVM. Or maybe you can dedicate first stage of your job to it. All that sucks though...
Here is the complete command I used to run my uber-jar in EMR and I see log files generated in driver and executor nodes.
spark-submit --class com.myapp.cloud.app.UPApp --master yarn --deploy-mode client --driver-memory 4g --executor-memory 2g --executor-cores 8 --files log4j.properties --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:log4j.properties" --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=file:log4j.properties -Dlog4j.debug=true" --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:log4j.properties" --conf "spark.eventLog.dir=/mnt/var/log/" uber-up-0.0.1.jar
where log4j.properties is in my local filesystem.