We Keep Coding

Failed to setup gcp repository using elasticsearch operator

I'm launching 3 elastic nodes using elastic operator and i tried to set up automated snapshots for these instances.
I followed this doc
I minified the json of the service account key and created a file called gcs.client.default.credentials_file with no file extension and added this file to kubernetes secrets.
And added the secureSettings.secretName field to the spec of the elastic cluster and added the secret name to it which was gcs-credentials
But i get this error on the logs
{"#timestamp":"2022-12-26T18:45:40.037Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"elasticsearch-cluster-es-node-1","elasticsearch.cluster.name":"elasticsearch-cluster","error.type":"java.lang.IllegalStateException","error.message":"failed to load plugin class [org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin]","error.stack_trace":"java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin]\n\tat org.elasticsearch.server#8.5.0/org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607)\n\tat org.elasticsearch.server#8.5.0/org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:482)\n\tat org.elasticsearch.server#8.5.0/org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:290)\n\tat org.elasticsearch.server#8.5.0/org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:159)\n\tat org.elasticsearch.server#8.5.0/org.elasticsearch.plugins.PluginsService.lambda$getPluginsServiceCtor$14(PluginsService.java:634)\n\tat org.elasticsearch.server#8.5.0/org.elasticsearch.node.Node.<init>(Node.java:406)\n\tat org.elasticsearch.server#8.5.0/org.elasticsearch.node.Node.<init>(Node.java:316)\n\tat org.elasticsearch.server#8.5.0/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:214)\n\tat org.elasticsearch.server#8.5.0/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:214)\n\tat org.elasticsearch.server#8.5.0/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)\nCaused by: java.lang.reflect.InvocationTargetException\n\tat java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:79)\n\tat java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)\n\tat java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:484)\n\tat org.elasticsearch.server#8.5.0/org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:600)\n\t... 9 more\nCaused by: java.lang.IllegalArgumentException: failed to load GCS client credentials from [gcs.client.default.credentials_file]\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.loadCredential(GoogleCloudStorageClientSettings.java:265)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.getClientSettings(GoogleCloudStorageClientSettings.java:221)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.load(GoogleCloudStorageClientSettings.java:209)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin.reload(GoogleCloudStoragePlugin.java:88)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin.<init>(GoogleCloudStoragePlugin.java:36)\n\tat java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:67)\n\t... 12 more\nCaused by: java.io.IOException: Invalid PKCS#8 data.\n\tat com.google.auth.oauth2.ServiceAccountCredentials.privateKeyFromPkcs8(ServiceAccountCredentials.java:496)\n\tat com.google.auth.oauth2.ServiceAccountCredentials.fromPkcs8(ServiceAccountCredentials.java:474)\n\tat com.google.auth.oauth2.ServiceAccountCredentials.fromJson(ServiceAccountCredentials.java:212)\n\tat com.google.auth.oauth2.ServiceAccountCredentials.fromStream(ServiceAccountCredentials.java:548)\n\tat com.google.auth.oauth2.ServiceAccountCredentials.fromStream(ServiceAccountCredentials.java:520)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.lambda$loadCredential$13(GoogleCloudStorageClientSettings.java:257)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:569)\n\tat org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedIOException(SocketAccess.java:33)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.loadCredential(GoogleCloudStorageClientSettings.java:256)\n\t... 17 more\n"}
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/elasticsearch-cluster.log

Try adding the following lines to your configuration (on each Elasticsearch):
elasticsearch01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.6.2
...
ulimits:
memlock:
soft: -1
hard: -1
Also check this link on Elasticsearch for more detailed information.

How can I add a user that can connect to ActiveMQ Artemis in WildFly via Quarkus JMS?

I have a WildFly application with an embedded ActiveMQ Artemis and I can't seem to figure how to debug it. It seems to connect fine since I'm getting no errors in the log, but I'm unable to read any queues. There is no error but nothing is happening.
I'm running the Quarkus JMS client 1.03 which is using org.apache.activemq » artemis-jms-client 2.19. The server is running WildFly 26.1 with the embedded ActiveMQ Artemis server 2.19.1 so the client and server should be compatible. However, I'm not sure if I've configured it correctly. I'm using the standalone-full.xml.
For testing I've added a user test1:
bash-4.4$ ./add-user.sh
What type of user do you wish to add?
a) Management User (mgmt-users.properties)
b) Application User (application-users.properties)
(a): b
Enter the details of the new user to add.
Using realm 'ApplicationRealm' as discovered from the existing property files.
Username : test1
Password recommendations are listed below. To modify these restrictions edit the add-user.properties configuration file.
- The password should be different from the username
- The password should not be one of the following restricted values {root, admin, administrator}
- The password should contain at least 8 characters, 1 alphabetic character(s), 1 digit(s), 1 non-alphanumeric symbol(s)
Password :
WFLYDM0098: The password should be different from the username
Are you sure you want to use the password entered yes/no? yes
Re-enter Password :
What groups do you want this user to belong to? (Please enter a comma separated list, or leave blank for none)[ ]: guest
About to add user 'test1' for realm 'ApplicationRealm'
Is this correct yes/no? yes
Is this new user going to be used for one AS process to connect to another AS process?
e.g. for a slave host controller connecting to the master or for a Remoting connection for server to server Jakarta Enterprise Beans calls.
yes/no? yes
To represent the user add the following to the server-identities definition <secret value="dGVzdDE=" />
This is the log for creating that user. In my standalone-full.xml I have the role guest for ActiveMQ Artemis and the queue prices set up:
<subsystem xmlns="urn:jboss:domain:messaging-activemq:13.1">
<server name="default">
...
<security-setting name="#">
<role name="guest" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>
</security-setting>
...
<address-setting name="jms.queue.prices" expiry-address="jms.queue.ExpiryQueue" redelivery-delay="1000" max-delivery-attempts="0"/>
...
<jms-queue name="prices" entries="java:/jms/queue/prices" durable="true"/>
...
</server>
</subsystem>
My Quarkus client tries to connect with the following:
quarkus.artemis.url=tcp://localhost:8080
quarkus.artemis.username=test1
quarkus.artemis.password=test1
It has a consumer:
#Override
public void run() {
try (JMSContext context = connectionFactory.createContext(JMSContext.AUTO_ACKNOWLEDGE)) {
JMSConsumer consumer = context.createConsumer(context.createQueue("prices"));
while (true) {
Message message = consumer.receive();
if (message == null) {
// receive returns `null` if the JMSConsumer is closed
return;
}
lastPrice = message.getBody(String.class);
}
} catch (JMSException e) {
throw new RuntimeException(e);
}
}
and a producer:
#Override
public void run() {
try (JMSContext context = connectionFactory.createContext(JMSContext.AUTO_ACKNOWLEDGE)) {
context.createProducer().send(context.createQueue("prices"), Integer.toString(random.nextInt(100)));
}
}
It seem to connect just fine. I can see in the Quarkus log:
2022-12-08 14:29:59,596 DEBUG [org.apa.act.art.cor.cli.imp.ClientSessionFactoryImpl] (pool-10-thread-1) Trying reconnection attempt 0/1
2022-12-08 14:29:59,596 DEBUG [org.apa.act.art.cor.cli.imp.ClientSessionFactoryImpl] (pool-10-thread-1) Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory#6096907d and currentConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=8080&host=localhost
2022-12-08 14:29:59,596 DEBUG [org.apa.act.art.cor.rem.imp.net.NettyConnector] (pool-10-thread-1) Connector NettyConnector [host=localhost, port=8080, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using native epoll
2022-12-08 14:29:59,596 DEBUG [org.apa.act.art.cor.client] (pool-10-thread-1) AMQ211002: Started EPOLL Netty Connector version 4.1.82.Final to localhost:8080
2022-12-08 14:29:59,597 DEBUG [org.apa.act.art.cor.rem.imp.net.NettyConnector] (pool-10-thread-1) Remote destination: localhost/127.0.0.1:8080
2022-12-08 14:29:59,598 DEBUG [org.apa.act.art.cor.rem.imp.net.NettyConnector] (Thread-1 (ActiveMQ-client-netty-threads)) Added ActiveMQClientChannelHandler to Channel with id = 6339db8b
2022-12-08 14:29:59,598 DEBUG [org.apa.act.art.cor.cli.imp.ClientSessionFactoryImpl] (pool-10-thread-1) Connected with the currentConnectorConfig=TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=8080&host=localhost
2022-12-08 14:29:59,599 DEBUG [org.apa.act.art.cor.cli.imp.ClientSessionFactoryImpl] (pool-10-thread-1) Reconnection successful
So it seems to be working fine, but I can't seem to send or consume from the queue.
Answers to Justin questions in the comments
Have you tried adding ?httpUpgradeEnabled=true
yes no change
quarkus.artemis.url=tcp://localhost:8080?httpUpgradeEnabled=true
Can you check the consumer-count
Actually there does not even seem to be a session active. I tried to close the connection for my test user but it shown me a dialog "no connection for that user" So maybe it's not even connecting as I thought
Why do you suspect the problem is with the user
I thought it might have something to do with some permission error. I was not sure if I was suppost to add the user to the "guest" group or not
But since I have in my config
<security-setting name="#">
<role name="guest" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>
</security-setting>
EDIT2. Actually I see now in the log that quarkus seems to be missing permission for creating a consumer. So i guess "guest" role is not the correct one.
I assume that was corrected

using ApacheMina SSHD to transfer files, then I got : java.lang.IllegalStateException: Channel id=2 not registered because session is being closed

I'm using ApacheMina SSHD to transfer files to a server, then I got errors like this: java.lang.IllegalStateException: Channel id=2 not registered because session is being closed. and this is happening in specific environment, when i'm transfering files with another linux environment, it doesn't occur any errors. here is the error stack, can somebody help me explain why this happened?
Exception in thread "Thread-5210" java.lang.IllegalStateException: Channel id=2 not registered because session is being closed: SftpChannelSubsystem[id=2, recipient=-1]-ClientSessionImpl[xxx#xxx/xxx:22][sftp]
at org.apache.sshd.common.channel.AbstractChannel.handleChannelRegistrationResult(AbstractChannel.java:446)
at org.apache.sshd.common.session.helpers.AbstractConnectionService.registerChannel(AbstractConnectionService.java:423)
at org.apache.sshd.sftp.client.impl.DefaultSftpClient.<init>(DefaultSftpClient.java:100)
at org.apache.sshd.sftp.client.impl.DefaultSftpClientFactory.createDefaultSftpClient(DefaultSftpClientFactory.java:66)
at org.apache.sshd.sftp.client.impl.DefaultSftpClientFactory.createSftpClient(DefaultSftpClientFactory.java:50)
at org.apache.sshd.sftp.client.fs.SftpFileSystem.getClient(SftpFileSystem.java:183)
at org.apache.sshd.sftp.client.fs.SftpFileSystemProvider.resolveRemoteFileAttributes(SftpFileSystemProvider.java:978)
at org.apache.sshd.sftp.client.fs.SftpFileSystemProvider.lambda$readRemoteAttributes$0(SftpFileSystemProvider.java:969)
at org.apache.sshd.sftp.client.impl.SftpPathImpl.withAttributeCache(SftpPathImpl.java:120)
at org.apache.sshd.sftp.client.impl.SftpPathImpl.withAttributeCache(SftpPathImpl.java:140)
at org.apache.sshd.sftp.client.fs.SftpFileSystemProvider.readRemoteAttributes(SftpFileSystemProvider.java:969)
at org.apache.sshd.sftp.client.impl.AbstractSftpFileAttributeView.readRemoteAttributes(AbstractSftpFileAttributeView.java:71)
at org.apache.sshd.sftp.client.fs.SftpPosixFileAttributeView.readAttributes(SftpPosixFileAttributeView.java:52)
at org.apache.sshd.sftp.client.fs.SftpPosixFileAttributeView.readAttributes(SftpPosixFileAttributeView.java:40)
at org.apache.sshd.sftp.client.fs.SftpFileSystemProvider.checkAccess(SftpFileSystemProvider.java:826)
at java.nio.file.Files.exists(Files.java:2385)
at com.sftpmgmt.utils.ChannelSftp.isReadableAndWritable(ChannelSftp.java:62)

IBM MQ - Java api - PCFMessageAgent - connection fails

I changed the MQIVP sample in MQ with a server connection channel of my own local.server.con and it is working fine. But I tried connecting to the same channel with PCFMessageAgent and the connection is failing with errors in MQ log. What is the relation between my channel and SYSTEM.DEFAULT.MODEL.QUEUE which gives the error.
C:\Program Files\IBM\WebSphere MQ\Tools\wmqjava\samples>java -Djava.library.path="C:\Program Files\IBM\WebSphere MQ\java\lib" MQIVPMod
Websphere MQ for Java Installation Verification Program
5724-B4 (C) Copyright IBM Corp. 2002, 2014. All Rights Reserved.
================================================================
Please enter the IP address of the MQ server :10.40.1.16
Please enter the port to connect to : (1414)1415
Please enter the server connection channel name :local.server.con
Please enter the user name (or RETURN for none) :test
Please enter the password for the user :test123
Please enter the queue manager name :local
Success: Connected to queue manager.
Success: Opened SYSTEM.DEFAULT.LOCAL.QUEUE
Success: Put a message to SYSTEM.DEFAULT.LOCAL.QUEUE
Success: Got a message from SYSTEM.DEFAULT.LOCAL.QUEUE
Success: Closed SYSTEM.DEFAULT.LOCAL.QUEUE
Success: Disconnected from queue manager
Tests complete -
SUCCESS: This MQ Transport is functioning correctly.
Press Enter to continue ...
My PCFMessageAgent code and error:
new PCFMessageAgent(host, Integer.parseInt(port), channelName); // connect
com.ibm.mq.MQException: MQJE001: Completion Code '2', Reason '2035'.
at com.ibm.mq.MQDestination.open(MQDestination.java:323)
at com.ibm.mq.MQQueue.<init>(MQQueue.java:236)
at com.ibm.mq.MQQueueManager.accessQueue(MQQueueManager.java:2674)
at com.ibm.mq.pcf.PCFAgent.open(PCFAgent.java:448)
at com.ibm.mq.pcf.PCFAgent.open(PCFAgent.java:394)
at com.ibm.mq.pcf.PCFAgent.connect(PCFAgent.java:287)
at com.ibm.mq.pcf.PCFAgent.<init>(PCFAgent.java:190)
at com.ibm.mq.pcf.PCFMessageAgent.<init>(PCFMessageAgent.java:157)
at test.wmq.PCFTest.main(PCFTest.java:49)
And the MQ log :
5/2/2017 14:01:31 - Process(6048.60) User(MUSR_MQADMIN) Program(amqzlaa0.exe)
Host(BLR_SWG_N09505) Installation(Installation1)
VRMF(8.0.0.4) QMgr(local)
AMQ8077: Entity 'test#blr_swg_n09505' has insufficient authority to access
object 'SYSTEM.DEFAULT.MODEL.QUEUE'.
EXPLANATION:
The specified entity is not authorized to access the required object. The
following requested permissions are unauthorized: get
ACTION:
Ensure that the correct level of authority has been set for this entity against
the required object, or ensure that the entity is a member of a privileged
group.
----- amqzfubn.c : 518 --------------------------------------------------------
5/2/2017 14:01:32 - Process(8004.41) User(MUSR_MQADMIN) Program(amqrmppa.exe)
Host(BLR_SWG_N09505) Installation(Installation1)
VRMF(8.0.0.4) QMgr(local)
AMQ9208: Error on receive from host BLR_SWG_N09505 (10.40.1.16).
EXPLANATION:
An error occurred receiving data from BLR_SWG_N09505 (10.40.1.16) over TCP/IP.
This may be due to a communications failure.
ACTION:
The return code from the TCP/IP recv() call was 10054 (X'2746'). Record these
values and tell the systems administrator.
----- amqccita.c : 4076 -------------------------------------------------------
5/2/2017 14:01:32 - Process(8004.41) User(MUSR_MQADMIN) Program(amqrmppa.exe)
Host(BLR_SWG_N09505) Installation(Installation1)
VRMF(8.0.0.4) QMgr(local)
AMQ9999: Channel 'local.server.con' to host '10.40.1.16' ended abnormally.
EXPLANATION:
The channel program running under process ID 8004(7988) for channel
'local.server.con' ended abnormally. The host name is '10.40.1.16'; in some
cases the host name cannot be determined and so is shown as '????'.
ACTION:
Look at previous error messages for the channel program in the error logs to
determine the cause of the failure. Note that this message can be excluded
completely or suppressed by tuning the "ExcludeMessage" or "SuppressMessage"
attributes under the "QMErrorLog" stanza in qm.ini. Further information can be
found in the System Administration Guide.
----- amqrmrsa.c : 930 --------------------------------------------------------

You need to go and read up on MQ permissions (i.e. authorizations). It is best to do permissions on the group rather than principle (UserId).
setmqaut -m {QM_NAME} -n SYSTEM.ADMIN.COMMAND.QUEUE -t queue -g {GROUP} +put +inq +dsp
setmqaut -m {QM_NAME} -n SYSTEM.DEFAULT.MODEL.QUEUE -t queue -g {GROUP} +get +inq +dsp

There is no relation between channels and model queues.
But I think, that the PCFMessageAgent is trying to create a dynamic queue to use as a ReplyToQ to receive responses, and it seems it tries to create the dynamic queue by opening the SYSTEM.DEFAULT.MODEL.QUEUE.

Sonar stops working every few days: jdbc connection error

I have a Sonar server which is used once a day from maven/jenkins and every few days, say every 4 or 5 days, it crashes and and shows the message "We're sorry, but something went wrong".
In the log, the error is always about a jdbc connection problem. I thought that it was a problem with the database but then if I restart the sonar server everything goes fine again.
So, it looks like a memory leak or something in the sonar server that makes it crash every few days until someone restarts it. Does that makes sense?. This is the configuration I have:
sonar.jdbc.username: xxxx
sonar.jdbc.password: xxxx
sonar.jdbc.url: jdbc:mysql://x.x.x.x:3306/sonar?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true
#----- Connection pool settings
sonar.jdbc.maxActive: 20
sonar.jdbc.maxIdle: 5
sonar.jdbc.minIdle: 2
sonar.jdbc.maxWait: 5000
sonar.jdbc.minEvictableIdleTimeMillis: 600000
sonar.jdbc.timeBetweenEvictionRunsMillis: 30000
sonar.updatecenter.activate=true
http.proxyHost=xxxx
http.proxyPort=3128
sonar.notifications.delay=60
That's it. And this is the error log:
INFO o.s.s.p.ServerImpl SonarQube Server / 3.7.3 /
INFO o.s.c.p.Database Create JDBC datasource for jdbc:mysql://x.x.x.x:3306/sonar?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true
ERROR o.s.c.p.Database Can not connect to database.
Please check connectivity and settings (see the properties prefixed by 'sonar.jdbc.').
org.apache.commons.dbcp.SQLNestedException:
Cannot create PoolableConnectionFactory (Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.)
.
.
.
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
.
.
.
Caused by: java.net.ConnectException: Connection refused
.
.
.
INFO jruby.rack An exception happened during JRuby-Rack startup
no connection available
--- System
jruby 1.6.8 (ruby-1.8.7-p357) (2012-09-18 1772b40) (Java HotSpot(TM) 64-Bit Server VM 1.6.0_43) [linux-amd64-java]
Time: Thu Jan 02 08:04:08 -0500 2014
Server: jetty/7.6.11.v20130520
jruby.home: file:/opt/sonar/war/sonar-server/WEB-INF/lib/jruby-complete-1.6.8.jar!/META-INF/jruby.home
--- Context Init Parameters:
jruby.compat.version = 1.8
jruby.max.runtimes = 1
jruby.min.runtimes = 1
jruby.rack.logging = slf4j
public.root = /
rails.env = production
--- Backtrace
ActiveRecord::ConnectionNotEstablished: no connection available
set_native_database_types at arjdbc/jdbc/RubyJdbcConnection.java:517
initialize at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-jdbc-adapter-1.1.3/lib/arjdbc/jdbc/connection.rb:61
initialize at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-jdbc-adapter-1.1.3/lib/arjdbc/jdbc/adapter.rb:31
jdbc_connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-jdbc-adapter-1.1.3/lib/arjdbc/jdbc/connection_methods.rb:6
send at org/jruby/RubyKernel.java:2109
new_connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:223
checkout_new_connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:245
checkout at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:188
loop at org/jruby/RubyKernel.java:1439
checkout at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:184
mon_synchronize at file:/opt/sonar/war/sonar-server/WEB-INF/lib/jruby-complete-1.6.8.jar!/META-INF/jruby.home/lib/ruby/1.8/monitor.rb:191
checkout at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:183
connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:98
retrieve_connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:326
retrieve_connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_specification.rb:123
connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_specification.rb:115
initialize at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/migration.rb:440
up at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/migration.rb:401
migrate at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/migration.rb:383
upgrade_and_start at /opt/sonar/war/sonar-server/WEB-INF/lib/database_version.rb:62
automatic_setup at /opt/sonar/war/sonar-server/WEB-INF/lib/database_version.rb:74
(root) at /opt/sonar/war/sonar-server/WEB-INF/config/environment.rb:213
load at org/jruby/RubyKernel.java:1087
load_environment at /opt/sonar/war/sonar-server/WEB-INF/config/environment.rb:23
load_environment at file:/opt/sonar/war/sonar-server/WEB-INF/lib/jruby-rack-1.1.10.jar!/jruby/rack/rails_booter.rb:65
(root) at <script>:1
--- RubyGems
Gem.dir: /opt/sonar/war/sonar-server/WEB-INF/gems
Gem.path:
/opt/sonar/war/sonar-server/WEB-INF/gems
Activated gems:
rake-0.9.2.2
activesupport-2.3.15
activerecord-2.3.15
rack-1.1.3
actionpack-2.3.15
actionmailer-2.3.15
activeresource-2.3.15
rails-2.3.15
color-tools-1.3.0
i18n-0.4.2
json-jruby-1.2.3-universal-java-1.6
activerecord-jdbc-adapter-1.1.3
fastercsv-1.4.0
--- Bundler
undefined method `bundle_path' for Bundler:Module
--- JRuby-Rack Config
compat_version = RUBY1_8
default_logger = org.jruby.rack.logging.StandardOutLogger#4fbbe4e1
err = java.io.PrintStream#d2284af
filter_adds_html = true
filter_verifies_resource = false
ignore_environment = false
initial_memory_buffer_size =
initial_runtimes = 1
jms_connection_factory =
jms_jndi_properties =
logger = org.jruby.rack.logging.Slf4jLogger#566dc8f0
logger_class_name = slf4j
logger_name = jruby.rack
maximum_memory_buffer_size =
maximum_runtimes = 1
num_initializer_threads =
out = java.io.PrintStream#6aeeefcf
rackup =
rackup_path =
rewindable = true
runtime_arguments =
runtime_timeout_seconds =
serial_initialization = false
servlet_context = ServletContext#o.e.j.w.WebAppContext{/,file:/opt/sonar/war/sonar-server/},file:/opt/sonar/war/sonar-server
ERROR jruby.rack unable to create shared application instance
org.jruby.rack.RackInitializationException: no connection available
.
.
.
org.jruby.exceptions.RaiseException:
(ConnectionNotEstablished) no connection available
.
.
.
ERROR jruby.rack Error: application initialization failed
org.jruby.rack.RackInitializationException: no connection available
.
.
.
org.jruby.exceptions.RaiseException:
(ConnectionNotEstablished) no connection available
Any help will be appreciated :)

This sounds familiar to me :P Try asking your operators if they have some automated 'cleansing' operation that just kills periodically open database connections automatically in order to prevent leaked connections to the database.

It happened to me, in a Windows 2012 Server and SQL Server 2012 database in a different server. It seems that sonar service maintains a connection open, created at start up time, so any disconnection (networking, database restart, etc.) causes this unrecoverable connection problem. Restarting the Sonar (SonarQube) Windows service solved the problem to me. But, if this problem is frequent, as in your case, it would be a good idea to schedule a service restart task, or find out what is taking the connection down.