My sonarqube doesn't start up and I keep on getting a 502 or 400 nginx error. I have JDK 11 and everything installed and I can't seem to find the problem in the logs. I don't know if the issue is with Java or nginx or sonarqube itself.
es.log
2023.01.27 10:58:39 INFO es[][o.e.n.Node] version[7.9.3], pid[47598], build[oss/tar/c4138e51121ef06a6404866cddc601906fe5c868/2020-10-16T10:36:16.141335Z], OS[Linux/6.>
2023.01.27 10:58:39 INFO es[][o.e.n.Node] JVM home [/usr/lib/jvm/java-11-openjdk-amd64]
2023.01.27 10:58:39 INFO es[][o.e.n.Node] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io>
2023.01.27 10:58:40 INFO es[][o.e.p.PluginsService] loaded module [analysis-common]
2023.01.27 10:58:40 INFO es[][o.e.p.PluginsService] loaded module [lang-painless]
2023.01.27 10:58:40 INFO es[][o.e.p.PluginsService] loaded module [parent-join]
2023.01.27 10:58:40 INFO es[][o.e.p.PluginsService] loaded module [percolator]
2023.01.27 10:58:40 INFO es[][o.e.p.PluginsService] loaded module [transport-netty4]
2023.01.27 10:58:41 INFO es[][o.e.p.PluginsService] no plugins loaded
2023.01.27 10:58:41 INFO es[][o.e.e.NodeEnvironment] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [42.9gb], net total_space [78.2gb], types [ext4]
2023.01.27 10:58:41 INFO es[][o.e.e.NodeEnvironment] heap size [495.3mb], compressed ordinary object pointers [true]
2023.01.27 10:58:41 WARN es[][o.e.d.c.s.Settings] [node.master] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking chan>
2023.01.27 10:58:41 WARN es[][o.e.d.c.s.Settings] [node.data] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking change>
2023.01.27 10:58:41 INFO es[][o.e.n.Node] node name [sonarqube], node ID [AMSlGyGQTgWwvRq78aVdog], cluster name [sonarqube]
2023.01.27 10:58:47 WARN es[][o.e.d.c.r.OperationRouting] searches will not be routed based on awareness attributes starting in version 8.0.0; to opt into this behavi>
2023.01.27 10:58:49 INFO es[][o.e.t.NettyAllocator] creating NettyAllocator with the following configs: [name=unpooled, factors={es.unsafe.use_unpooled_allocator=fals>
2023.01.27 10:58:50 INFO es[][o.e.d.DiscoveryModule] using discovery type [zen] and seed hosts providers [settings]
2023.01.27 10:58:51 WARN es[][o.e.g.DanglingIndicesState] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imp>
2023.01.27 10:58:51 INFO es[][o.e.n.Node] initialized
2023.01.27 10:58:51 INFO es[][o.e.n.Node] starting ...
2023.01.27 10:58:52 INFO es[][o.e.t.TransportService] publish_address {127.0.0.1:39879}, bound_addresses {127.0.0.1:39879}
2023.01.27 10:58:52 INFO es[][o.e.b.BootstrapChecks] explicitly enforcing bootstrap checks
2023.01.27 10:58:52 INFO es[][o.e.c.c.Coordinator] cluster UUID [VZtnXYVBTdqpaiXh8hM6jA]
2023.01.27 10:58:53 INFO es[][o.e.c.s.MasterService] elected-as-master ([1] nodes joined)[{sonarqube}{AMSlGyGQTgWwvRq78aVdog}{fpJWgQCCTya62AbPbg6XEQ}{127.0.0.1}{127.0>
2023.01.27 10:58:53 INFO es[][o.e.c.s.ClusterApplierService] master node changed {previous [], current [{sonarqube}{AMSlGyGQTgWwvRq78aVdog}{fpJWgQCCTya62AbPbg6XEQ}{12>
2023.01.27 10:58:53 INFO es[][o.e.h.AbstractHttpServerTransport] publish_address {127.0.0.1:9001}, bound_addresses {127.0.0.1:9001}
2023.01.27 10:58:53 INFO es[][o.e.n.Node] started
2023.01.27 10:58:53 INFO es[][o.e.g.GatewayService] recovered [0] indices into cluster_state
2023.01.27 10:58:56 INFO es[][o.e.n.Node] stopping ...
2023.01.27 10:58:56 INFO es[][o.e.n.Node] stopped
2023.01.27 10:58:56 INFO es[][o.e.n.Node] closing ...
2023.01.27 10:58:56 INFO es[][o.e.n.Node] closed
sonar log:
2023.01.27 10:51:07 INFO app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube/temp
2023.01.27 10:51:07 INFO app[][o.s.a.es.EsSettings] Elasticsearch listening on [HTTP: 127.0.0.1:9001, TCP: 127.0.0.1:34703]
2023.01.27 10:51:07 INFO app[][o.s.a.ProcessLauncherImpl] Launch process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/opt/sonarqube/elasticsearch]: /opt/sonar>
2023.01.27 10:51:08 INFO app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
2023.01.27 10:51:23 INFO app[][o.s.a.SchedulerImpl] Process[es] is up
2023.01.27 10:51:23 INFO app[][o.s.a.ProcessLauncherImpl] Launch process[[key='web', ipcIndex=2, logFilenamePrefix=web]] from [/opt/sonarqube]: /usr/lib/jvm/java-11-o>
2023.01.27 10:51:26 INFO app[][o.s.a.SchedulerImpl] Process[web] is stopped
2023.01.27 10:51:26 WARN app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [es]: 143
2023.01.27 10:51:26 INFO app[][o.s.a.SchedulerImpl] Process[es] is stopped
2023.01.27 10:51:26 INFO app[][o.s.a.SchedulerImpl] SonarQube is stopped
<-- Wrapper Stopped
--> Wrapper Started as Daemon
Launching a JVM...
Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
Copyright 1999-2006 Tanuki Software, Inc. All Rights Reserved.
web.log:
2023.01.27 11:38:16 WARN web[][o.s.p.ProcessEntryPoint] Fail to start web
java.lang.RuntimeException: org.apache.catalina.LifecycleException: Failed to initialize connector [Connector[HTTP/1.1-9000]]
at com.google.common.base.Throwables.propagate(Throwables.java:241)
at org.sonar.server.app.EmbeddedTomcat.start(EmbeddedTomcat.java:70)
at org.sonar.server.app.WebServer.start(WebServer.java:52)
at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:97)
at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:81)
at org.sonar.server.app.WebServer.main(WebServer.java:99)
Caused by: org.apache.catalina.LifecycleException: Failed to initialize connector [Connector[HTTP/1.1-9000]]
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:559)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:848)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:173)
at org.apache.catalina.startup.Tomcat.start(Tomcat.java:440)
at org.sonar.server.app.EmbeddedTomcat.start(EmbeddedTomcat.java:66)
... 4 common frames omitted
2023.01.27 11:38:16 INFO web[][o.s.p.ProcessEntryPoint] Hard stopping process
Please help thank you
When deploying my Keycloak to our Docker swarm, we get the following warning and error in the stack trace:
We are trying to deploy the keycloak SPI into the given volume. This works but keycloak can't write/ read into the volume.
10:13:05,350 INFO [org.jboss.modules] (main) JBoss Modules version 1.10.2.Final
10:13:06,057 INFO [org.jboss.msc] (main) JBoss MSC version 1.4.12.Final
10:13:06,069 INFO [org.jboss.threads] (main) JBoss Threads version 2.4.0.Final
10:13:06,220 INFO [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 12.0.1 (WildFly Core 13.0.3.Final) starting
10:13:06,392 INFO [org.jboss.vfs] (MSC service thread 1-4) VFS000002: Failed to clean existing content for temp file provider of type temp. Enable DEBUG level log to find what caused this
10:13:07,466 INFO [org.wildfly.security] (ServerService Thread Pool -- 22) ELY00001: WildFly Elytron version 1.13.1.Final
10:13:08,587 INFO [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
10:13:08,653 WARN [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) WFLYDS0039: /opt/jboss/keycloak/standalone/deployments is not writable
10:13:08,675 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) WFLYDS0042: Boot-time scan failed due to inaccessible deployment directory: /opt/jboss/keycloak/standalone/deployments
10:13:08,690 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 3) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
10:13:08,861 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)
The errors WFLYDS0039 and WFLYDS0042 are the issue.
We mounted the deployments folder to a docker volume on the server on the path: /srv/keycloak_providers
When deploying the SPI I can see that the .JAR file can be found on the server in the path above.
Here is the keycloak docker-compose file:
version: "3.7"
services:
db:
image: mariadb
volumes:
- keycloak_db:/var/lib/mysql
environment:
MYSQL_DATABASE: keycloak
networks:
db:
aliases:
- mariadb
keycloak:
image: jboss/keycloak:12.0.1
environment:
- DB_ADDR=mariadb
- DB_PORT=3306
- DB_DATABASE=keycloak
- DB_VENDOR=mariadb
- PROXY_ADDRESS_FORWARDING=true
networks:
- public
- db
volumes:
- keycloak_themes:/opt/jboss/keycloak/themes
- keycloak_providers:/opt/jboss/keycloak/standalone/deployments/
networks:
db:
driver: overlay
public:
external: true
volumes:
keycloak_db:
driver: local-persist
driver_opts:
mountpoint: /srv/keycloak_db
keycloak_themes:
driver: local-persist
driver_opts:
mountpoint: /srv/keycloak_themes
keycloak_providers:
driver: local-persist
driver_opts:
mountpoint: /srv/keycloak_providers
Keycloak server is run as the jboss user which has the uid/gid set to 1000. Make sure that user id 1000 has read/write permissions to that source folder /srv/keycloak_providers. The easiest solution is chmod -R 777 /srv/keycloak_providers from the host OS CLI. But you may need more safe permission configuration - just keep in mind uid 1000 must have read/write access + eventually also some user(s) from the host OS.
Of course it can be more complicated if Docker daemon/Keycloak container has configured uid remapping - there will be another uid on the scene.
I start Keycloak 7.0.1 on new MySQL 5.7 database
This is the Kubernetes deployment
spec:
containers:
- env:
- name: KEYCLOAK_USER
value: admin
- name: KEYCLOAK_PASSWORD
value: password
- name: PROXY_ADDRESS_FORWARDING
value: "true"
- name: KEYCLOAK_LOGLEVEL
value: INFO
- name: DB_VENDOR
value: mysql
- name: DB_ADDR
value: db.rds.amazonaws.com
- name: DB_DATABASE
value: keycloak
- name: DB_SCHEMA
value: keycloak
- name: DB_PORT
value: "3306"
- name: DB_USER
value: keycloak
- name: DB_PASSWORD
value: 789
- name: JDBC_PARAMS
value: character_set_server=utf8mb4&useUnicode=true&verifyServerCertificate=false&useSSL=true&requireSSL=true&allowPublicKeyRetrieval=true&serverTimezone=Europe/Paris
image: jboss/keycloak:7.0.1
And the stackstrace
Added 'admin' to '/opt/jboss/keycloak/standalone/configuration/keycloak-add-user.json', restart server to load user
-b 0.0.0.0
=========================================================================
Using MySQL database
=========================================================================
19:29:43,163 INFO [org.jboss.modules] (CLI command executor) JBoss Modules version 1.9.1.Final
19:29:43,233 INFO [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.8.Final
19:29:43,241 INFO [org.jboss.threads] (CLI command executor) JBoss Threads version 2.3.3.Final
19:29:43,407 INFO [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 7.0.1 (WildFly Core 9.0.2.Final) starting
19:29:43,483 INFO [org.jboss.vfs] (MSC service thread 1-2) VFS000002: Failed to clean existing content for temp file provider of type temp. Enable DEBUG level log to find what caused this
19:29:44,245 INFO [org.wildfly.security] (ServerService Thread Pool -- 19) ELY00001: WildFly Elytron version 1.9.1.Final
19:29:44,799 INFO [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
19:29:44,902 INFO [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
19:29:45,059 INFO [org.jboss.as.patching] (MSC service thread 1-2) WFLYPAT0050: Keycloak cumulative patch ID is: base, one-off patches include: none
19:29:45,078 WARN [org.jboss.as.domain.management.security] (MSC service thread 1-2) WFLYDM0111: Keystore /opt/jboss/keycloak/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self signed certificate for host localhost
19:29:45,169 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
19:29:45,170 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 7.0.1 (WildFly Core 9.0.2.Final) started in 1998ms - Started 64 of 78 services (29 services are lazy, passive or on-demand)
The batch executed successfully
19:29:45,343 INFO [org.jboss.as] (MSC service thread 1-2) WFLYSRV0050: Keycloak 7.0.1 (WildFly Core 9.0.2.Final) stopped in 18ms
19:29:46,892 INFO [org.jboss.modules] (CLI command executor) JBoss Modules version 1.9.1.Final
19:29:46,965 INFO [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.8.Final
19:29:46,984 INFO [org.jboss.threads] (CLI command executor) JBoss Threads version 2.3.3.Final
19:29:47,148 INFO [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 7.0.1 (WildFly Core 9.0.2.Final) starting
19:29:47,232 INFO [org.jboss.vfs] (MSC service thread 1-1) VFS000002: Failed to clean existing content for temp file provider of type temp. Enable DEBUG level log to find what caused this
19:29:48,101 INFO [org.wildfly.security] (ServerService Thread Pool -- 22) ELY00001: WildFly Elytron version 1.9.1.Final
19:29:49,023 INFO [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
19:29:49,111 INFO [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
19:29:49,254 INFO [org.jboss.as.patching] (MSC service thread 1-2) WFLYPAT0050: Keycloak cumulative patch ID is: base, one-off patches include: none
19:29:49,264 WARN [org.jboss.as.domain.management.security] (MSC service thread 1-2) WFLYDM0111: Keystore /opt/jboss/keycloak/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self signed certificate for host localhost
19:29:49,365 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
19:29:49,370 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 7.0.1 (WildFly Core 9.0.2.Final) started in 2467ms - Started 64 of 85 services (36 services are lazy, passive or on-demand)
The batch executed successfully
19:29:49,569 INFO [org.jboss.as] (MSC service thread 1-2) WFLYSRV0050: Keycloak 7.0.1 (WildFly Core 9.0.2.Final) stopped in 20ms
=========================================================================
JBoss Bootstrap Environment
JBOSS_HOME: /opt/jboss/keycloak
JAVA: java
JAVA_OPTS: -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED
=========================================================================
19:29:50,369 INFO [org.jboss.modules] (main) JBoss Modules version 1.9.1.Final
19:29:50,895 INFO [org.jboss.msc] (main) JBoss MSC version 1.4.8.Final
19:29:50,906 INFO [org.jboss.threads] (main) JBoss Threads version 2.3.3.Final
19:29:51,047 INFO [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 7.0.1 (WildFly Core 9.0.2.Final) starting
19:29:51,162 INFO [org.jboss.vfs] (MSC service thread 1-1) VFS000002: Failed to clean existing content for temp file provider of type temp. Enable DEBUG level log to find what caused this
19:29:51,978 INFO [org.wildfly.security] (ServerService Thread Pool -- 21) ELY00001: WildFly Elytron version 1.9.1.Final
19:29:52,813 INFO [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
19:29:52,885 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 29) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
19:29:53,069 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)
19:29:53,093 INFO [org.xnio] (MSC service thread 1-2) XNIO version 3.7.2.Final
19:29:53,106 INFO [org.xnio.nio] (MSC service thread 1-2) XNIO NIO Implementation Version 3.7.2.Final
19:29:53,153 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 39) WFLYCLINF0001: Activating Infinispan subsystem.
19:29:53,159 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 52) WFLYNAM0001: Activating Naming Subsystem
19:29:53,169 INFO [org.wildfly.extension.microprofile.config.smallrye._private] (ServerService Thread Pool -- 48) WFLYCONF0001: Activating WildFly MicroProfile Config Subsystem
19:29:53,170 INFO [org.jboss.remoting] (MSC service thread 1-1) JBoss Remoting version 5.0.12.Final
19:29:53,175 WARN [org.jboss.as.txn] (ServerService Thread Pool -- 57) WFLYTX0013: The node-identifier attribute on the /subsystem=transactions is set to the default value. This is a danger for environments running multiple servers. Please make sure the attribute value is unique.
19:29:53,170 INFO [org.jboss.as.jaxrs] (ServerService Thread Pool -- 41) WFLYRS0016: RESTEasy version 3.7.0.Final
19:29:53,224 INFO [org.wildfly.extension.microprofile.metrics.smallrye] (ServerService Thread Pool -- 50) WFLYMETRICS0001: Activating Eclipse MicroProfile Metrics Subsystem
19:29:53,250 INFO [org.jboss.as.security] (ServerService Thread Pool -- 55) WFLYSEC0002: Activating Security Subsystem
19:29:53,259 INFO [org.wildfly.extension.microprofile.health.smallrye] (ServerService Thread Pool -- 49) WFLYHEALTH0001: Activating Eclipse MicroProfile Health Subsystem
19:29:53,247 INFO [org.wildfly.extension.io] (ServerService Thread Pool -- 40) WFLYIO001: Worker 'default' has auto-configured to 2 core threads with 16 task threads based on your 1 available processors
19:29:53,267 INFO [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 43) WFLYCLJG0001: Activating JGroups subsystem. JGroups version 4.0.19
19:29:53,282 INFO [org.jboss.as.connector] (MSC service thread 1-2) WFLYJCA0009: Starting JCA Subsystem (WildFly/IronJacamar 1.4.16.Final)
19:29:53,328 INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 34) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.4)
19:29:53,450 INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 34) WFLYJCA0005: Deploying non-JDBC-compliant driver class com.mysql.jdbc.Driver (version 5.1)
19:29:53,482 INFO [org.jboss.as.security] (MSC service thread 1-1) WFLYSEC0001: Current PicketBox version=5.0.3.Final
19:29:53,521 WARN [org.wildfly.clustering.web.undertow] (ServerService Thread Pool -- 58) WFLYCLWEBUT0007: No routing provider found for default-server; using legacy provider based on static configuration
19:29:53,576 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0018: Started Driver service with driver-name = h2
19:29:53,598 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0003: Undertow 2.0.21.Final starting
19:29:53,598 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0018: Started Driver service with driver-name = mysql
19:29:53,648 INFO [io.smallrye.metrics] (MSC service thread 1-2) Converted [2] config entries and added [4] replacements
19:29:53,651 INFO [io.smallrye.metrics] (MSC service thread 1-2) Converted [3] config entries and added [18] replacements
19:29:53,640 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 58) WFLYUT0014: Creating file handler for path '/opt/jboss/keycloak/welcome-content' with options [directory-listing: 'false', follow-symlink: 'false', case-sensitive: 'true', safe-symlink-paths: '[]']
19:29:53,685 INFO [org.jboss.as.ejb3] (MSC service thread 1-1) WFLYEJB0482: Strict pool mdb-strict-max-pool is using a max instance size of 4 (per class), which is derived from the number of CPUs on this host.
19:29:53,696 INFO [org.jboss.as.ejb3] (MSC service thread 1-2) WFLYEJB0481: Strict pool slsb-strict-max-pool is using a max instance size of 16 (per class), which is derived from thread worker pool sizing.
19:29:53,697 INFO [org.jboss.as.naming] (MSC service thread 1-2) WFLYNAM0003: Starting Naming Service
19:29:53,803 INFO [org.jboss.as.mail.extension] (MSC service thread 1-2) WFLYMAIL0001: Bound mail session [java:jboss/mail/Default]
19:29:54,048 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0012: Started server default-server.
19:29:54,108 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0006: Undertow HTTP listener default listening on 0.0.0.0:8080
19:29:54,108 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0006: Undertow AJP listener ajp listening on 0.0.0.0:8009
19:29:54,112 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0018: Host default-host starting
19:29:54,154 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 60) MODCLUSTER000001: Initializing mod_cluster version 1.4.1.Final
19:29:54,167 INFO [org.jboss.as.ejb3] (MSC service thread 1-1) WFLYEJB0493: EJB subsystem suspension complete
19:29:54,170 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 60) MODCLUSTER000032: Listening to proxy advertisements on /224.0.1.105:23364
19:29:54,270 INFO [org.jboss.as.patching] (MSC service thread 1-2) WFLYPAT0050: Keycloak cumulative patch ID is: base, one-off patches include: none
19:29:54,272 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS]
19:29:54,273 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) WFLYJCA0001: Bound data source [java:jboss/datasources/KeycloakDS]
19:29:54,287 WARN [org.jboss.as.domain.management.security] (MSC service thread 1-1) WFLYDM0111:
...
...
WFLYCLINF0002: Started work cache from keycloak container
19:29:59,495 WARN [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0273: Excluded subsystem webservices via jboss-deployment-structure.xml does not exist.
19:30:00,077 INFO [org.keycloak.services] (ServerService Thread Pool -- 66) KC-SERVICES0001: Loading config from standalone.xml or domain.xml
19:30:00,086 DEBUG [org.keycloak.provider.ProviderManager] (ServerService Thread Pool -- 66) Provider loaders [org.keycloak.provider.DefaultProviderLoaderFactory#33f94f36, org.keycloak.provider.FileSystemProviderLoaderFactory#381df96, org.keycloak.provider.wildfly.ModuleProviderLoaderFactory#15544a2]
19:30:00,087 DEBUG [org.keycloak.provider.FileSystemProviderLoaderFactory] (ServerService Thread Pool [org.keycloak.connections.jpa.updater.liquibase.lock.LiquibaseDBLockProviderFactory] (ServerService Thread Pool -- 66) Liquibase lock provider configured with lockWaitTime: 900 seconds
19:30:00,395 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanStickySessionEncoderProviderFactory] (ServerService Thread Pool -- 66) Should attach route to the sticky session cookie: true
19:30:00,409 DEBUG [org.keycloak.services.DefaultKeycloakSessionFactory] (ServerService Thread Pool -- 66) SPI client-storage provider openshift-oauth-client disabled
19:30:00,411 DEBUG [org.keycloak.keys.infinispan.InfinispanPublicKeyStorageProviderFactory] (ServerService Thread Pool -- 66) minTimeBetweenRequests is 10
19:30:00,422 DEBUG [org.keycloak.services.DefaultKeycloakSessionFactory] (ServerService Thread Pool -- 66) Loaded SPI timer (provider = basic)
19:30:00,428 DEBUG [org.keycloak.services.DefaultKeycloakSessionFactory] (ServerService Thread Pool -- 66) Loaded SPI hostname (provider = request)
19:30:00,481 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 66) WFLYCLINF0002: Started realmRevisions cache from keycloak container
19:30:00,487 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 66) WFLYCLINF0002: Started userRevisions cache from keycloak container
19:30:00,499 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 66) WFLYCLINF0002: Started authorizationRevisions cache from keycloak container
19:30:00,501 DEBUG [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (ServerService Thread Pool -- 66) Using container managed Infinispan cache container, lookup=java:jboss/infinispan/container/keycloak
19:30:00,502 INFO [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (ServerService Thread Pool -- 66) Node name: keycloak-7569d49d7d-sjpwf, Site name: null
19:30:00,520 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (ServerService Thread Pool -- 66) new JtaTransactionWrapper
19:30:00,520 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (ServerService Thread Pool -- 66) was existing? false
19:30:00,541 DEBUG [org.keycloak.connections.jpa.updater.liquibase.conn.DefaultLiquibaseConnectionProvider] (ServerService Thread Pool -- 66) Added package org.keycloak.connections.jpa.updater.liquibase.lock to liquibase
19:30:01,129 WARN [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ServerService Thread Pool -- 66) IJ000604: Throwable while attempting to get a new connection: null: javax.resource.ResourceException: IJ031084: Unable to create connection
at org.jboss.ironjacamar.jdbcadapters#1.4.16.Final//org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createLocalManagedConnection(LocalManagedConnectionFactory.java:345)
at org.jboss.ironjacamar.jdbcadapters#1.4.16.Final//org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:352)
at org.jboss.ironjacamar.jdbcadapters#1.4.16.Final//org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:287)
at org.jboss.ironjacamar.impl#1.4.16.Final//org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.createConnectionEventListener(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:1325)
at org.jboss.ironjacamar.impl#1.4.16.Final//org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.getConnection(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:499)
at org.jboss.ironjacamar.impl#1.4.16.Final//org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:632)
at org.jboss.ironjacamar.impl#1.4.16.Final//org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:604)
at org.jboss.ironjacamar.impl#1.4.16.Final//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:624)
at org.jboss.ironjacamar.impl#1.4.16.Final//org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:440)
at org.jboss.ironjacamar.impl#1.4.16.Final//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:789)
at org.jboss.ironjacamar.jdbcadapters#1.4.16.Final//org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151)
at org.jboss.as.connector#17.0.1.Final//org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:64)
at org.keycloak.keycloak-model-jpa#7.0.1//org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.getConnection(DefaultJpaConnectionProviderFactory.java:366)
at org.keycloak.keycloak-model-jpa#7.0.1//org.keycloak.connections.jpa.updater.liquibase.lock.LiquibaseDBLockProvider.lazyInit(LiquibaseDBLockProvider.java:65)
at org.keycloak.keycloak-model-jpa#7.0.1//org.keycloak.connections.jpa.updater.liquibase.lock.LiquibaseDBLockProvider.lambda$waitForLock$2(LiquibaseDBLockProvider.java:96)
at org.keycloak.keycloak-server-spi-private#7.0.1//org.keycloak.models.utils.KeycloakModelUtils.suspendJtaTransaction(KeycloakModelUtils.java:682)
at org.keycloak.keycloak-model-jpa#7.0.1//org.keycloak.connections.jpa.updater.liquibase.lock.LiquibaseDBLockProvider.waitForLock(LiquibaseDBLockProvider.java:94)
at org.keycloak.keycloak-services#7.0.1//org.keycloak.services.resources.KeycloakApplication$1.run(KeycloakApplication.java:144)
at org.keycloak.keycloak-server-spi-private#7.0.1//org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:227)
at org.keycloak.keycloak-services#7.0.1//org.keycloak.services.resources.KeycloakApplication.<init>(KeycloakApplication.java:137)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at org.jboss.resteasy.resteasy-jaxrs#3.7.0.Final//org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:152)
at org.jboss.resteasy.resteasy-jaxrs#3.7.0.Final//org.jboss.resteasy.spi.ResteasyProviderFactory.createProviderInstance(ResteasyProviderFactory.java:2784)
at org.jboss.resteasy.resteasy-jaxrs#3.7.0.Final//org.jboss.resteasy.spi.ResteasyDeployment.createApplication(ResteasyDeployment.java:364)
at org.jboss.resteasy.resteasy-jaxrs#3.7.0.Final//org.jboss.resteasy.spi.ResteasyDeployment.startInternal(ResteasyDeployment.java:277)
at org.jboss.resteasy.resteasy-jaxrs#3.7.0.Final//org.jboss.resteasy.spi.ResteasyDeployment.start(ResteasyDeployment.java:89)
at org.jboss.resteasy.resteasy-jaxrs#3.7.0.Final//org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.init(ServletContainerDispatcher.java:119)
at org.jboss.resteasy.resteasy-jaxrs#3.7.0.Final//org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.init(HttpServletDispatcher.java:36)
at io.undertow.servlet#2.0.21.Final//io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:117)
at org.wildfly.extension.undertow#17.0.1.Final//org.wildfly.extension.undertow.security.RunAsLifecycleInterceptor.init(RunAsLifecycleInterceptor.java:78)
at io.undertow.servlet#2.0.21.Final//io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:103)
at io.undertow.servlet#2.0.21.Final//io.undertow.servlet.core.ManagedServlet$DefaultInstanceStrategy.start(ManagedServlet.java:303)
at io.undertow.servlet#2.0.21.Final//io.undertow.servlet.core.ManagedServlet.createServlet(ManagedServlet.java:143)
at io.undertow.servlet#2.0.21.Final//io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:583)
at io.undertow.servlet#2.0.21.Final//io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:554)
at io.undertow.servlet#2.0.21.Final//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
at io.undertow.servlet#2.0.21.Final//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at org.wildfly.extension.undertow#17.0.1.Final//org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at org.wildfly.extension.undertow#17.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at org.wildfly.extension.undertow#17.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at org.wildfly.extension.undertow#17.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at org.wildfly.extension.undertow#17.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at io.undertow.servlet#2.0.21.Final//io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:596)
at org.wildfly.extension.undertow#17.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:97)
at org.wildfly.extension.undertow#17.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:78)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.jboss.threads#2.3.3.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads#2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
at org.jboss.threads#2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
at org.jboss.threads#2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
at java.base/java.lang.Thread.run(Thread.java:834)
at org.jboss.threads#2.3.3.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 117 milliseconds ago. The last packet sent successfully to the server was 110 milliseconds ago.
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:990)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.ExportControlled.transformSocketToSSLSocket(ExportControlled.java:201)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.MysqlIO.negotiateSSLConnection(MysqlIO.java:4912)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1663)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1224)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2190)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2221)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2016)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:776)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:386)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
at org.jboss.ironjacamar.jdbcadapters#1.4.16.Final//org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createLocalManagedConnection(LocalManagedConnectionFactory.java:321)
... 55 more
Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
at java.base/sun.security.ssl.HandshakeContext.<init>(HandshakeContext.java:169)
at java.base/sun.security.ssl.ClientHandshakeContext.<init>(ClientHandshakeContext.java:98)
at java.base/sun.security.ssl.TransportContext.kickstart(TransportContext.java:216)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:395)
at com.mysql.jdbc#5.1.46//com.mysql.jdbc.ExportControlled.transformSocketToSSLSocket(ExportControlled.java:186)
... 71 more
19:30:01,145 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (ServerService Thread Pool -- 66) JtaTransactionWrapper rollback
19:30:01,150 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (ServerService Thread Pool -- 66) JtaTransactionWrapper end
19:30:01,158 INFO [org.jboss.as.server] (Thread-1) WFLYSRV0220: Server shutdown has been requested via an OS signal
Do you think there is a configuration I do not see ?
Since 7.0.1, Keycloak has switched to Redhat UBI base images (registry.access.redhat.com/ubi8-minimal). In the Dockerfile it install's a version of OpenJDK-11 (11.0.5 currently) which contains modified version of java.security file under /etc/alternatives/jre/conf/security/.
Switching to the standard OpenJDK-11 as the base image in https://github.com/keycloak/keycloak-containers/blob/7.0.1/server/Dockerfile fixes the issue:
FROM openjdk:11.0.5-jdk
ENV KEYCLOAK_VERSION 7.0.1
ENV JDBC_POSTGRES_VERSION 42.2.5
ENV JDBC_MYSQL_VERSION 5.1.46
ENV JDBC_MARIADB_VERSION 2.2.3
ENV JDBC_MSSQL_VERSION 7.4.1.jre8
ENV LAUNCH_JBOSS_IN_BACKGROUND 1
ENV PROXY_ADDRESS_FORWARDING false
ENV JBOSS_HOME /opt/jboss/keycloak
ENV LANG en_US.UTF-8
ARG GIT_REPO
ARG GIT_BRANCH
ARG KEYCLOAK_DIST=https://downloads.jboss.org/keycloak/$KEYCLOAK_VERSION/keycloak-$KEYCLOAK_VERSION.tar.gz
USER root
ADD tools /opt/jboss/tools
RUN /opt/jboss/tools/build-keycloak.sh
USER 1000
EXPOSE 8080
EXPOSE 8443
ENTRYPOINT [ "/opt/jboss/tools/docker-entrypoint.sh" ]
CMD ["-b", "0.0.0.0"]
However, the OpenJDK image is larger and image scanning (at least in Google Cloud registry) reveals several vulnerabilities.
An alternative is to copy java.security from the standard OpenJDK image to Keycloak image or to a new image based on Keycloak:
FROM openjdk:11.0.5-jdk as openjdk
FROM jboss/keycloak:7.0.1
COPY --from=openjdk /usr/local/openjdk-11/conf/security/java.security /etc/alternatives/jre/conf/security/
This is a quick workaround. A more security-aware approach would be to compare the 2 versions of the file and introduce only minimal changes.
For keycloak 12.0.1 you still experience this problem when the database uses anything weaker than TLSv1.2. I think the problem occurs, because the RedHat UBI 8 image does not accept anything less than TLSv1.2. For more information check https://access.redhat.com/articles/3642912.
Setting crypto-policy to LEGACY, as the article suggests, did not work for me. Unchanged, the image uses DEFAULT crypto-policy, you need to override the file /etc/crypto-policies/config with LEGACY. To check the contents execute:
docker run --rm -ti --entrypoint bash jboss/keycloak:12.0.1 \
-c 'cat /etc/crypto-policies/config'
What fixed it for me was the idea from John Georgladis's answer here. I am posting here a complete working version, as the code above did not work for me right away. Overriding java.security did not work as well, particularly because of the line:
security.useSystemPropertiesFile=true
To check the contents of java.security file you could use:
docker run --rm -ti --entrypoint bash jboss/keycloak:12.0.1 \
-c 'cat /etc/java/java-11-openjdk/java-11-openjdk-*/conf/security/java.security | grep -v ^# | grep -v ^$'
Using opnejdk as base image lowers the cryptographic expectations and allows TLSv1.0 or TLSv1.1. So, if you cannot upgrade your DB to use TLSv1.2 or above, then you could install and ship keycloak like below (for Keycloak 12.0.1).
Base for this script is taken from current Keycloak 12.0.1 Dockerfile
FROM jboss/keycloak:12.0.1 as keycloak
FROM /openjdk:11.0.9-jdk
ENV KEYCLOAK_VERSION 12.0.1
ENV JDBC_POSTGRES_VERSION 42.2.5
ENV JDBC_MYSQL_VERSION 8.0.22
ENV JDBC_MARIADB_VERSION 2.5.4
ENV JDBC_MSSQL_VERSION 8.2.2.jre11
ENV LAUNCH_JBOSS_IN_BACKGROUND 1
ENV PROXY_ADDRESS_FORWARDING false
ENV JBOSS_HOME /opt/jboss/keycloak
ENV LANG en_US.UTF-8
ARG GIT_REPO
ARG GIT_BRANCH
ARG KEYCLOAK_DIST=https://github.com/keycloak/keycloak/releases/download/$KEYCLOAK_VERSION/keycloak-$KEYCLOAK_VERSION.tar.gz
USER root
COPY --from=keycloak /opt/jboss/tools /opt/jboss/tools
RUN /opt/jboss/tools/build-keycloak.sh
USER 1000
EXPOSE 8080
EXPOSE 8443
ENTRYPOINT [ "/opt/jboss/tools/docker-entrypoint.sh" ]
CMD ["-b", "0.0.0.0"]
You will need to tweak java.security file. See similar issue for RDS Postgres keycloak - SSL error: Certificates do not conform to algorithm constraints
Wildfly are suspending intermittently my java web application. This occur often one time in the week.
I'm running my service with the following configuration:
Ubuntu 16 8GB
Intel(R) Xeon(R) CPU E5-2640 v3 # 2.60GHz
Web Application:
Wildfly 14
Java 11
JavaEE 8
Postgresql
2019-08-08 07:35:14,619 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) WFLYDS0012: Scan of /opt/wildfly-14.0.1.Final/standalone/deployments threw Exception: java.lang.RuntimeException: WFLYDS0032: Failed to list files in directory /opt/wildfly-14.0.1.Final/standalone/deployments. Check that the contents of the directory are readable.
at org.jboss.as.deployment-scanner#6.0.2.Final//org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.listDirectoryChildren(FileSystemDeploymentService.java:1365)
at org.jboss.as.deployment-scanner#6.0.2.Final//org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scanDirectory(FileSystemDeploymentService.java:846)
at org.jboss.as.deployment-scanner#6.0.2.Final//org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:598)
at org.jboss.as.deployment-scanner#6.0.2.Final//org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:493)
at org.jboss.as.deployment-scanner#6.0.2.Final//org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$DeploymentScanRunnable.run(FileSystemDeploymentService.java:255)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
at org.jboss.threads#2.3.2.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
Caused by: java.nio.file.FileSystemException: /opt/wildfly-14.0.1.Final/standalone/deployments: Too many open files
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:428)
at java.base/java.nio.file.Files.newDirectoryStream(Files.java:603)
at org.jboss.as.deployment-scanner#6.0.2.Final//org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.listDirectoryChildren(FileSystemDeploymentService.java:1358)
... 11 more
2019-08-08 07:55:11,261 INFO [org.jboss.as.server] (Thread-1) WFLYSRV0236: Suspending server with no timeout.
2019-08-08 07:55:11,272 INFO [org.jboss.as.ejb3] (Thread-1) WFLYEJB0493: EJB subsystem suspension complete
2019-08-08 07:55:11,285 INFO [org.jboss.as.server] (Thread-1) WFLYSRV0220: Server shutdown has been requested via an OS signal
2019-08-08 07:55:11,324 INFO [org.jboss.as.mail.extension] (MSC service thread 1-8) WFLYMAIL0002: Unbound mail session [java:jboss/mail/Default]
2019-08-08 07:55:11,327 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 72) WFLYUT0022: Unregistered web context: '/service' from server 'default-server'
Looks like some process may be your web server process is keeping files open and it is reaching beyond ulimit.
To find root cause, you can simply use lsof linux command to see all the open files and their respective process ids.
1) if you find you could control this count of open files, then it's good.
2) if it's unavoidable, then try increasing ulimit
I'm running a single node application with Spark on a machine with 32 GB RAM.
More than 12GB of the memory is available at the time I'm running the applicaton.
But From the spark UI and logs, I see that it using 3.8GB of RAM (which is gradually decreased as the jobs run).
At this time this is logged, 5GB more memory is avilable. Where as Spark is using 3.8GB
UPDATE
I set these parameters in conf/spark-env.sh but still each time I run the application It is using exactly 3.8 GB
export SPARK_WORKER_MEMORY=6g
export SPARK_MEM=6g
export SPARK_DAEMON_MEMORY=6g
Log
2015-11-19 13:05:41,701 INFO org.apache.spark.SparkEnv.logInfo:59 - Registering MapOutputTracker
2015-11-19 13:05:41,716 INFO org.apache.spark.SparkEnv.logInfo:59 - Registering BlockManagerMaster
2015-11-19 13:05:41,735 INFO org.apache.spark.storage.DiskBlockManager.logInfo:59 - Created local directory at /usr/local/TC_SPARCDC_COM/temp/blockmgr-8513cd3b-ac03-4c0a-b291-65aba4cbc395
2015-11-19 13:05:41,746 INFO org.apache.spark.storage.MemoryStore.logInfo:59 - MemoryStore started with capacity 3.8 GB
2015-11-19 13:05:41,777 INFO org.apache.spark.HttpFileServer.logInfo:59 - HTTP File server directory is /usr/local/TC_SPARCDC_COM/temp/spark-b86380c2-4cbd-43d6-a3b7-aa03d9a05a84/httpd-ceaffbd0-eac4-447e-9d3f-c452627a28cb
2015-11-19 13:05:41,781 INFO org.apache.spark.HttpServer.logInfo:59 - Starting HTTP Server
2015-11-19 13:05:41,842 INFO org.spark-project.jetty.server.Server.doStart:272 - jetty-8.y.z-SNAPSHOT
2015-11-19 13:05:41,854 INFO org.spark-project.jetty.server.AbstractConnector.doStart:338 - Started SocketConnector#0.0.0.0:5279
2015-11-19 13:05:41,855 INFO org.apache.spark.util.Utils.logInfo:59 - Successfully started service 'HTTP file server' on port 5279.
2015-11-19 13:05:41,867 INFO org.apache.spark.SparkEnv.logInfo:59 - Registering OutputCommitCoordinator
2015-11-19 13:05:42,013 INFO org.spark-project.jetty.server.Server.doStart:272 - jetty-8.y.z-SNAPSHOT
2015-11-19 13:05:42,039 INFO org.spark-project.jetty.server.AbstractConnector.doStart:338 - Started SelectChannelConnector#0.0.0.0:4040
2015-11-19 13:05:42,039 INFO org.apache.spark.util.Utils.logInfo:59 - Successfully started service 'SparkUI' on port 4040.
2015-11-19 13:05:42,041 INFO org.apache.spark.ui.SparkUI.logInfo:59 - Started SparkUI at http://103.252.184.181:4040
2015-11-19 13:05:42,114 WARN org.apache.spark.metrics.MetricsSystem.logWarning:71 - Using default name DAGScheduler for source because spark.app.id is not set.
2015-11-19 13:05:42,117 INFO org.apache.spark.executor.Executor.logInfo:59 - Starting executor ID driver on host localhost
2015-11-19 13:05:42,307 INFO org.apache.spark.util.Utils.logInfo:59 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 31334.
2015-11-19 13:05:42,308 INFO org.apache.spark.network.netty.NettyBlockTransferService.logInfo:59 - Server created on 31334
2015-11-19 13:05:42,309 INFO org.apache.spark.storage.BlockManagerMaster.logInfo:59 - Trying to register BlockManager
2015-11-19 13:05:42,312 INFO org.apache.spark.storage.BlockManagerMasterEndpoint.logInfo:59 - Registering block manager localhost:31334 with 3.8 GB RAM, BlockManagerId(driver, localhost, 31334)
2015-11-19 13:05:42,313 INFO org.apache.spark.storage.BlockManagerMaster.logInfo:59 - Registered BlockManager
If you are using SparkSubmit you can use the --executor-memory and --driver-memory flags. Otherwise, change these configurations spark.executor.memory and spark.driver.memory either directly in your program or in spark-defaults.
Note that you should not set memory too high. As a rule of thumb, aim for ~75% of available memory. That will leave enough memory for other processes (like your OS) running on your machines.
It is correctly stated by #Glennie Helles Sindholt but setting driver flags while submitting jobs on a standalone machine won't affect the usage as the JVM has been already been initialized. Checkout this link of discussion:
How to set Apache Spark Executor memory
If you are using Spark submit command to submit a job following is an example for how to set parameters while submitting the job:
spark-submit --master spark://127.0.0.1:7077 \
--num-executors 2 \
--executor-cores 8 \
--executor-memory 3g \
--class <Class name> \
$JAR_FILE_NAME or path \
/path-to-input \
/path-to-output \
By varying the number of parameters in this you can see and understand how the usage of RAM is changing. Also, there is a utility named htop on Linux. It is useful to instantaneous usage of memory, CPU cores and Swap space to have an understanding of what is happening. To install htop, use the following:
sudo apt-get install htop
It will look something like this:
htop utility
For more information you can check out the following links:
https://spark.apache.org/docs/latest/configuration.html