I have a kafka connect jar which needs to be run as a docker container. I need to capture all my connect logs on a log file in the container (preferably at a directory/file - /etc/kafka/kafka-connect-logs) which can later be pushed to localhost (on which docker engine is running) using volumes in docker. When I change my connect-log4j.properties to append into a log file, I see that no log file is created. If I try the same without docker and run the kafka connect on a local linux VM by changing connect-log4j.properties to write logs to a log file, it works perfectly but not from docker. Any suggestions will be very helpful.
Docker File
FROM confluent/platform
COPY Test.jar /usr/local/bin/
COPY kafka-connect-docker.sh /usr/local/bin/
COPY connect-distributed.properties /usr/local/bin/
COPY connect-log4j.properties /etc/kafka/connect-log4j.properties
RUN ["apt-get", "update"]
RUN ["apt-get", "install", "-yq", "curl"]
RUN ["chown", "-R", "confluent:confluent", "/usr/local/bin/kafka-connect-docker.sh", "/usr/local/bin/connect-distributed.properties", "/usr/local/bin/Test.jar"]
RUN ["chmod", "+x", "/usr/local/bin/kafka-connect-docker.sh", "/usr/local/bin/connect-distributed.properties", "/usr/local/bin/Test.jar"]
RUN ["chown", "-R", "confluent:confluent", "/etc/kafka/connect-log4j.properties"]
RUN ["chmod", "777", "/usr/local/bin/kafka-connect-docker.sh", "/etc/kafka/connect-log4j.properties"]
EXPOSE 8083
CMD [ "/usr/local/bin/kafka-connect-docker.sh" ]
connect-log4j.properties
# Root logger option
log4j.rootLogger = INFO, FILE
# Direct log messages to stdout
log4j.appender.FILE=org.apache.log4j.FileAppender
log4j.appender.FILE.File=/etc/kafka/log.out
# Define the layout for file appender
log4j.appender.FILE.layout=org.apache.log4j.PatternLayout
log4j.appender.FILE.layout.conversionPattern=%m%
log4j.logger.org.apache.zookeeper=ERROR
log4j.logger.org.I0Itec.zkclient=ERROR
kafka-connect-docker.sh
#!/bin/bash
export CLASSPATH=/usr/local/bin/Test.jar
exec /usr/bin/connect-distributed /usr/local/bin/connect-distributed.properties
It works fine when I am using the default connect-log4j.properties (appends logs to console), but am unable to create a log file in docker. Also, same process without docker works fine (creates log file ) in local VM.
Result of the discussion:
Declare the volume right away in the dockerfile and configure your logging configuration to place the log output directly to the volume. On docker run (or however you start your container) mount the volume and you should be fine.
Related
I am using AWS Step Functions Local for testing Step Functions on my local machine, but the level of logging has significantly increased the time for my test suite to run. I cannot find any configuration options on the AWS website to control the log level so I have been trying to roll my own with no success.
The logging I am trying to hide looks like this:
2022-12-15 12:11:36.562: CreateStateMachine => {"requestClientOptions":{"readLimit":131073,"skipAppendUriPath":false},"requestMetricCollector":null,"customRequestHeaders":null,"customQueryParameters":null,"cloneSource":null,"sdkRequestTimeout":null,"sdkClientExecutionTimeout":null,"name":"<MY STEP FUNCTION NAME","definition":"{MY STEP FUNCTION DEFINITION}"...}
2022-12-15 12:11:36.782: [200] CreateStateMachine <= {"sdkResponseMetadata":null,"sdkHttpMetadata":null,"stateMachineArn":"arn:aws:states:eu-west-1:123456789012:stateMachine:MY_STEP_FUNCTION_NAME","creationDate":1671106296763}
I downloaded the Zip file containing the Step Functions Local JAR file, extracted the zip file, extracted StepFunctionsLocal.jar, and found there was a log4j.properties file. "Excellent!", I thought, I can provide my own configuration file and it will be great. So I created a Docker image with the following Dockerfile:
FROM amazoncorretto:11
WORKDIR /jar
COPY . /jar
RUN yum -y install unzip zip && yum -y clean all && rm -rf /var/cache
RUN mkdir -p /jar/extracted
RUN unzip StepFunctionsLocal.zip -d /jar/extracted
ENV SFN_MOCK_CONFIG=""
ENV LAMBDA_ENDPOINT=""
ENV AWS_DEFAULT_REGION=""
EXPOSE 8083
CMD ["java", "-Dlog4j2.configurationFile=file:/jar/log4j.properties", "-jar", "/jar/extracted/StepFunctionsLocal.jar"]
With the following log4.properties file which is in the same directory as the Dockerfile:
# console is set to be a ConsoleAppender.
log4j.appender.console=org.apache.log4j.ConsoleAppender
# console uses PatternLayout.
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n
log4j.appender.APPLOG=amazon.platform.logging.PeriodicConcurrentFileAppender
log4j.appender.APPLOG.layout=amazon.platform.logging.SanitizingPatternLayout
log4j.appender.APPLOG.File=../logs/application.log
log4j.appender.APPLOG.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.APPLOG.layout.ConversionPattern=%p %d{D/HH:mm:ss:SSS} %t %c - %m%n
# Set root logger level to INFO and its only appender to console.
log4j.rootLogger=ERROR, APPLOG
#log4j.logger.com.amazon.coral.reflect=WARN
#log4j.logger.httpclient.wire=OFF
# Enable this to see everything that httpclient does.
#log4j.logger.org.apache.commons.httpclient=WARN
#log4j.logger.org.springframework=WARN
#log4j.logger.org.apache=WARN
#log4j.logger.com.amazonaws.swf.console=ERROR
#log4j.logger.com.amazonaws.console.auth=ERROR
This is the same file I found in the extracted StepFunctionsLocal.jar except all of the lines which configure the log level have been commented out and the root logger level set to ERROR.
I built the image and ran it, but the same logging still appeared. At this point I'm stumped. Am I going about this the right way? Have I missed something obvious?
Im using kubenetes and I have a pod contain ignite db I added to the pod another container - sscaling/jmx-prometheus-exporter:latest
I read in git I should run this
To run as a javaagent download the jar and run:
java -javaagent:./jmx_prometheus_javaagent-0.14.0.jar=8080:config.yaml -jar yourJar.jar
but I didnt understand, should I also download the file for the kubenrtes container as well?
can someone assist how can I continue from here
I have the following default configuration:
---
hostPort: localhost:5555
username:
password:
rules:
- pattern: ".*"
The situation is as follows:
The spring-boot service writes logs via logback to the output.log file
Logback is configured to rotate logs and it periodically empties output.log, dropping logs into archive files specified in the rotation policy. That is, the output.log itself is such a temporary buffer with the most recent log
That all works fine: https://www.youtube.com/watch?v=hrWmfC3y9zM
But if we mount the logs folder from the container outside to the host volume (and we need this for reading with a third-party utility), then everything breaks down: output.log stops being automatically cleared, grows indefinitely, and the archive files specified in the rotation policy are not created at all. This happens even when no one reads the file from this directory on the host machine.
For example
1) Directories inside container:
/app/
|-> service.jar
the buffer log is written here:
/app/logs/
|-> output.log
rotated archive log files are stored here
/app/
| -> out1.log
| -> out2.log
directory is mounted to the host in docker compose:
volumes:
- /LOGS:/app/logs:rw
2. The second container reading logs is mounted like this:
volumes:
- /LOGS:/var/log
But even when reading container is stopped, the file is locked. Just mounting folder from the source container is enough for blocking
Is this some kind of feature of docker volumes? Has it some workaround?
We were using the cf-uaa's gradle tasks to create a docker image but those have been removed in the latest version. I've loaded the war in a recent version, but the service does not seem to be starting correctly.
I've been building the war from the v74 tag, adding it to tomcat:8.5.45-jdk12-openjdk-oracle or tomcat:9.0.24-jdk12-openjdk-oracle, and setting the various env vars that we were passing in to the previous image. I'm not seeing any log entries after the initial tomcat output stating that my war has been deployed and the server startup time.
The Dockerfile is basically just an adaptation of what was being passed in the previous image:
FROM tomcat:8.5.45-jdk12-openjdk-oracle
#FROM tomcat:9.0.24-jdk12-openjdk-oracle
ENV LOGIN_CONFIG_URL WEB-INF/classes/required_configuration.yml
ENV UAA_CONFIG_PATH /uaa
RUN bash -c "rm -r /usr/local/tomcat/webapps/ROOT"
RUN bash -c "rm -r /usr/local/tomcat/webapps/host-manager"
RUN bash -c "rm -r /usr/local/tomcat/webapps/manager"
RUN bash -c "rm -r /usr/local/tomcat/webapps/examples"
RUN bash -c "rm -r /usr/local/tomcat/webapps/docs"
ADD *.war /usr/local/tomcat/webapps/uaa.war
RUN bash -c "echo $LOGIN_CONFIG_URL"
EXPOSE 8080
I would expect to see the service responding to my requests, or some errors in the log indicating that the war failed to deploy. I am not currently getting any log output generated from the application code. When I send a request to the service, the response is a 500 with the an error header from the service.
X-Cf-Uaa-Error:Server failed to start. Possible configuration error.
update: I've located the uaa logs within .../tomcat/logs/uaa.log I'm not seeing anything indicating that the service failed to deploy, but I am also not seeing anything to indicate that it is picking up the env vars I have set in the container. I recreated the service using the war from the original setup which started successfully using the uaa.yml which I mounted as a volume. Comparing the logs, the original setup's first log entry is YamlProcessor which does not show up in the v75 logs at all. In fact, no debug entries show up at all, which suggests to me that my LOG_LEVEL env var is not propagating either.
Update 2: We reverted the image base to FROM tomcat:8.5-jre8 and started seeing flyway errors in the uaa.log. Our previous datasource url format was url: jdbc:postgresql://${POSTGRES_NAME}:5432/${DB}?currentSchema=uaa which caused a flyway exception. After removing the schema reference, it created the tables in the public schema. By creating the uaa schema manually before starting the service, it was able to run with the original format. The flyway version has updated, so perhaps there something new that needs to be set.
The application seems to be running, but when I try to get a token at /uaa/oauth/token I get a 500 with this error in the logs: Caused by: java.lang.NoSuchMethodError: java.nio.CharBuffer.limit(I)Ljava/nio/CharBuffer;
Since Jan 2021, UAA server docker images is now be available on cloudfoundry/uaa dockerhub repository.
docker pull cloudfoundry/uaa:75.0.0
See its Dockerfile for more details.
Can you try following ?
https://github.com/hortonworks/docker-cloudbreak-uaa
This works very well.
I am trying to run a spark job in EMR cluster.
I my spark-submit I have added configs to read from log4j.properties
--files log4j.properties --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/log4j.properties"
Also I have added
log4j.rootLogger=INFO, file
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=/log/test.log
log4j.appender.file.MaxFileSize=10MB
log4j.appender.file.MaxBackupIndex=10
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %5p %c{7} - %m%n
in my log4j configurations.
Anyhow I see the logs in the console, though I don't see the log file generated. What am I doing wrong here ?
Quoting spark-submit --help:
--files FILES Comma-separated list of files to be placed in the working directory of each executor. File paths of these files in executors can be accessed via SparkFiles.get(fileName).
That doesn't much say what to do with the FILES if you cannot use SparkFiles.get(fileName) (which you cannot for log4j).
Quoting SparkFiles.get's scaladoc:
Get the absolute path of a file added through SparkContext.addFile().
That does not give you much either, but suggest to have a look at the source code of SparkFiles.get:
def get(filename: String): String =
new File(getRootDirectory(), filename).getAbsolutePath()
The nice thing about it is that getRootDirectory() uses an optional property or just the current working directory:
def getRootDirectory(): String =
SparkEnv.get.driverTmpDir.getOrElse(".")
That gives as something to work on, doesn't it?
On the driver the so-called driverTmpDir directory should be easy to find in Environment tab of web UI (under Spark Properties for spark.files property or Classpath Entries marked as "Added By User" in Source column).
On executors, I'd assume a local directory so rather than using file:/log4j.properties I'd use
-Dlog4j.configuration=file://./log4j.properties
or
-Dlog4j.configuration=file:log4j.properties
Note the dot to specify the local working directory (in the first option) or no leading / (in the latter).
Don't forget about spark.driver.extraJavaOptions to set the Java options for the driver if that's something you haven't thought about yet. You've been focusing on executors only so far.
You may want to add -Dlog4j.debug=true to spark.executor.extraJavaOptions that is supposed to print what locations log4j uses to find log4j.properties.
I have not checked that answer on a EMR or YARN cluster myself but believe that may have given you some hints where to find the answer. Fingers crossed!
With Spark 2.2.0 standalone cluster, executor JVM is started first and only then Spark distributes application jar and --files
Which means passing
spark.executor.extraJavaOptions=-Dlog4j.configuration=file:log4j-spark.xml
does not makes sense as this file does not exist yet (is not downloaded) at the executor JVM launch time and log4j initialization
If you pass
spark.executor.extraJavaOptions=-Dlog4j.debug -Dlog4j.configuration=file:log4j-spark.xml
you will find at the beginning of the executor's stderr failed attempt to load log4j config file
log4j:ERROR Could not parse url [file:log4j-spark.xml].
java.io.FileNotFoundException: log4j-spark.xml (No such file or directory)
...
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
And bit later is logged download of --files from driver
18/07/18 17:24:12 INFO Utils: Fetching spark://123.123.123.123:35171/files/log4j-spark.xml to /ca/tmp-spark/spark-49815375-3f02-456a-94cd-8099a0add073/executor-7df1c819-ffb7-4ef9-b473-4a2f7747237a/spark-0b50a7b9-ca68-4abc-a05f-59df471f2d16/fetchFileTemp5898748096473125894.tmp
18/07/18 17:24:12 INFO Utils: Copying /ca/tmp-spark/spark-49815375-3f02-456a-94cd-8099a0add073/executor-7df1c819-ffb7-4ef9-b473-4a2f7747237a/spark-0b50a7b9-ca68-4abc-a05f-59df471f2d16/-18631083971531927447443_cache to /opt/spark-2.2.0-bin-hadoop2.7/work/app-20180718172407-0225/2/./log4j-spark.xml
It may work differently with yarn or another cluster manager but with standalone cluster, it seems there is no way you can specify your own logging configuration for executors on spark-submit.
You can dynamically reconfigure log4j in your job code (override log4j configuration programmatically: file location for FileAppender), but you would need to do it carefully in some mapPartition lambda that is executed in executor's JVM. Or maybe you can dedicate first stage of your job to it. All that sucks though...
Here is the complete command I used to run my uber-jar in EMR and I see log files generated in driver and executor nodes.
spark-submit --class com.myapp.cloud.app.UPApp --master yarn --deploy-mode client --driver-memory 4g --executor-memory 2g --executor-cores 8 --files log4j.properties --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:log4j.properties" --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=file:log4j.properties -Dlog4j.debug=true" --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:log4j.properties" --conf "spark.eventLog.dir=/mnt/var/log/" uber-up-0.0.1.jar
where log4j.properties is in my local filesystem.