I have a Sonar server which is used once a day from maven/jenkins and every few days, say every 4 or 5 days, it crashes and and shows the message "We're sorry, but something went wrong".
In the log, the error is always about a jdbc connection problem. I thought that it was a problem with the database but then if I restart the sonar server everything goes fine again.
So, it looks like a memory leak or something in the sonar server that makes it crash every few days until someone restarts it. Does that makes sense?. This is the configuration I have:
sonar.jdbc.username: xxxx
sonar.jdbc.password: xxxx
sonar.jdbc.url: jdbc:mysql://x.x.x.x:3306/sonar?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true
#----- Connection pool settings
sonar.jdbc.maxActive: 20
sonar.jdbc.maxIdle: 5
sonar.jdbc.minIdle: 2
sonar.jdbc.maxWait: 5000
sonar.jdbc.minEvictableIdleTimeMillis: 600000
sonar.jdbc.timeBetweenEvictionRunsMillis: 30000
sonar.updatecenter.activate=true
http.proxyHost=xxxx
http.proxyPort=3128
sonar.notifications.delay=60
That's it. And this is the error log:
INFO o.s.s.p.ServerImpl SonarQube Server / 3.7.3 /
INFO o.s.c.p.Database Create JDBC datasource for jdbc:mysql://x.x.x.x:3306/sonar?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true
ERROR o.s.c.p.Database Can not connect to database.
Please check connectivity and settings (see the properties prefixed by 'sonar.jdbc.').
org.apache.commons.dbcp.SQLNestedException:
Cannot create PoolableConnectionFactory (Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.)
.
.
.
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
.
.
.
Caused by: java.net.ConnectException: Connection refused
.
.
.
INFO jruby.rack An exception happened during JRuby-Rack startup
no connection available
--- System
jruby 1.6.8 (ruby-1.8.7-p357) (2012-09-18 1772b40) (Java HotSpot(TM) 64-Bit Server VM 1.6.0_43) [linux-amd64-java]
Time: Thu Jan 02 08:04:08 -0500 2014
Server: jetty/7.6.11.v20130520
jruby.home: file:/opt/sonar/war/sonar-server/WEB-INF/lib/jruby-complete-1.6.8.jar!/META-INF/jruby.home
--- Context Init Parameters:
jruby.compat.version = 1.8
jruby.max.runtimes = 1
jruby.min.runtimes = 1
jruby.rack.logging = slf4j
public.root = /
rails.env = production
--- Backtrace
ActiveRecord::ConnectionNotEstablished: no connection available
set_native_database_types at arjdbc/jdbc/RubyJdbcConnection.java:517
initialize at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-jdbc-adapter-1.1.3/lib/arjdbc/jdbc/connection.rb:61
initialize at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-jdbc-adapter-1.1.3/lib/arjdbc/jdbc/adapter.rb:31
jdbc_connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-jdbc-adapter-1.1.3/lib/arjdbc/jdbc/connection_methods.rb:6
send at org/jruby/RubyKernel.java:2109
new_connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:223
checkout_new_connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:245
checkout at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:188
loop at org/jruby/RubyKernel.java:1439
checkout at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:184
mon_synchronize at file:/opt/sonar/war/sonar-server/WEB-INF/lib/jruby-complete-1.6.8.jar!/META-INF/jruby.home/lib/ruby/1.8/monitor.rb:191
checkout at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:183
connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:98
retrieve_connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:326
retrieve_connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_specification.rb:123
connection at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_specification.rb:115
initialize at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/migration.rb:440
up at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/migration.rb:401
migrate at /opt/sonar/war/sonar-server/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/migration.rb:383
upgrade_and_start at /opt/sonar/war/sonar-server/WEB-INF/lib/database_version.rb:62
automatic_setup at /opt/sonar/war/sonar-server/WEB-INF/lib/database_version.rb:74
(root) at /opt/sonar/war/sonar-server/WEB-INF/config/environment.rb:213
load at org/jruby/RubyKernel.java:1087
load_environment at /opt/sonar/war/sonar-server/WEB-INF/config/environment.rb:23
load_environment at file:/opt/sonar/war/sonar-server/WEB-INF/lib/jruby-rack-1.1.10.jar!/jruby/rack/rails_booter.rb:65
(root) at <script>:1
--- RubyGems
Gem.dir: /opt/sonar/war/sonar-server/WEB-INF/gems
Gem.path:
/opt/sonar/war/sonar-server/WEB-INF/gems
Activated gems:
rake-0.9.2.2
activesupport-2.3.15
activerecord-2.3.15
rack-1.1.3
actionpack-2.3.15
actionmailer-2.3.15
activeresource-2.3.15
rails-2.3.15
color-tools-1.3.0
i18n-0.4.2
json-jruby-1.2.3-universal-java-1.6
activerecord-jdbc-adapter-1.1.3
fastercsv-1.4.0
--- Bundler
undefined method `bundle_path' for Bundler:Module
--- JRuby-Rack Config
compat_version = RUBY1_8
default_logger = org.jruby.rack.logging.StandardOutLogger#4fbbe4e1
err = java.io.PrintStream#d2284af
filter_adds_html = true
filter_verifies_resource = false
ignore_environment = false
initial_memory_buffer_size =
initial_runtimes = 1
jms_connection_factory =
jms_jndi_properties =
logger = org.jruby.rack.logging.Slf4jLogger#566dc8f0
logger_class_name = slf4j
logger_name = jruby.rack
maximum_memory_buffer_size =
maximum_runtimes = 1
num_initializer_threads =
out = java.io.PrintStream#6aeeefcf
rackup =
rackup_path =
rewindable = true
runtime_arguments =
runtime_timeout_seconds =
serial_initialization = false
servlet_context = ServletContext#o.e.j.w.WebAppContext{/,file:/opt/sonar/war/sonar-server/},file:/opt/sonar/war/sonar-server
ERROR jruby.rack unable to create shared application instance
org.jruby.rack.RackInitializationException: no connection available
.
.
.
org.jruby.exceptions.RaiseException:
(ConnectionNotEstablished) no connection available
.
.
.
ERROR jruby.rack Error: application initialization failed
org.jruby.rack.RackInitializationException: no connection available
.
.
.
org.jruby.exceptions.RaiseException:
(ConnectionNotEstablished) no connection available
Any help will be appreciated :)
This sounds familiar to me :P Try asking your operators if they have some automated 'cleansing' operation that just kills periodically open database connections automatically in order to prevent leaked connections to the database.
It happened to me, in a Windows 2012 Server and SQL Server 2012 database in a different server. It seems that sonar service maintains a connection open, created at start up time, so any disconnection (networking, database restart, etc.) causes this unrecoverable connection problem. Restarting the Sonar (SonarQube) Windows service solved the problem to me. But, if this problem is frequent, as in your case, it would be a good idea to schedule a service restart task, or find out what is taking the connection down.
Related
I am connecting to remote Oracle db from AWS server using Java Hibernate connection, while the fetch is on if their is any network fluctuation, query hangs.
Below is the log of query:
23 Sep 2018 10:46:01 LiveSync INFO {Query----->Select count(rec_no)
as count_rec from TBL_NAME where REC_NO='XXXX'Booking Id--->XXXXX} 23
Sep 2018 11:02:09 LiveSync INFO {SQL Error...could not execute
query}
In the first line, Query was fired at 10:46:01 to provide count of record, however due to network packet loss during that time, the response came to Java at 11:02:09, almost after 15 min.
Can anyone help me to immediately respond to Java on the network breakage?
After running my application, i am getting this error after around 5 mins.
Even though i am returning the resource after use, i keep getting this.
I have built jedis-2.2.2-SNAPSHOT.jar from the jedis code base, since its not released yet
I had set the minIdle = 100, maxIdle=200 & maxActive=200. At the time of this exception, the connection count to redis was 122 from my application
redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:42)
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:442)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:360)
at redis.clients.util.Pool.getResource(Pool.java:40)
... 6 more
Did you check that redis is still up & running ?
If not, investigate why it died.
try a redis-cli in a terminal if you can. "info" would give you more details.
I have the following problem:
I try to connect to an ActiveMQ broker (which is now down) using the following piece of code
connectionFactory = new ActiveMQConnectionFactory(this.url + "?timeout=2000");
connection = connectionFactory.createConnection();
connection.start();
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
LOGGER.info("Connected to " + this.url);
The problem is that the timeout does not have any effect
connection.start()
is blocked forever.
I inspected ActiveMQ log and found the following info:
2013-12-20 01:49:03,149 DEBUG [ActiveMQ Task-1] (FailoverTransport.java:786) - urlList connectionList:[tcp://localhost:61616?timeout=2000], from: [tcp://localhost:61616?timeout=2000]
2013-12-20 01:49:03,149 DEBUG [ActiveMQ Task-1] (FailoverTransport.java:1040) - Connect fail to: tcp://localhost:61616?timeout=2000, reason: java.lang.IllegalArgumentException: Invalid connect parameters: {timeout=2000}
The timeout parameter is specified here http://activemq.apache.org/cms/configuring.html
Has anybody any idea how to pass timeout argument to ActiveMQConnectionFactory?
Or how to set a timeout for connection.start() ?
Thank you!
Update: I found this on Stackoverflow: ActiveMQ - CreateSession failover timeout after a connection is resumed . I tried it but the following exception is thrown:
javax.jms.JMSException: Could not create Transport. Reason: java.lang.IllegalArgumentException: Invalid connect parameters: {transport.timeout=5000}
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:35)
I use ActiveMQ 5.8.0 from maven repo
It appears that your url is invalid still in both cases when attempting to set the timeout property.
If you're trying to have a failover URL, which it looks like you are since it is getting in to the Failover code then you're probably looking for initialReconnectDelay (and possibly maxReconnectAttempts which would throw an exception if the server is still down after the number of attempts is reached).
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory("failover://(tcp://localhost:61616)?initialReconnectDelay=2000&maxReconnectAttempts=2");
I'm getting an error while attempting to get a connection to a local(same computer) database "oracle.net.ns.NetException: Got minus one from a read call".
here's the code
OracleDataSource ods = new OracleDataSource();
String jdbcURL = "jdbc:oracle:thin:username/userpass#mylinux:1522:sid";
ods.setURL(jdbcURL);
Connection conn = ods.getConnection();
Caused by: oracle.net.ns.NetException: Got minus one from a read call
at oracle.net.ns.Packet.receive(Packet.java:286)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:287)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1054)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:308)
... 8 more
What am I missing here ? My brief search on the net didn't turn up anything helpful. I'm linking ojdbc6.jar with the test app.
I can connect just fine from sqlplus using the above credentials.
UPDATE 1 : Am getting the following exception stack trace
Exception in thread "main" java.sql.SQLRecoverableException: IO Error: Got minus one from a read call
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:419)
at oracle.jdbc.driver.PhysicalConnection.(PhysicalConnection.java:536)
at oracle.jdbc.driver.T4CConnection.(T4CConnection.java:228)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
at oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:280)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:207)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:157)
at TestOraConn.main(TestOraConn.java:17)
Caused by: oracle.net.ns.NetException: Got minus one from a read call
at oracle.net.ns.Packet.receive(Packet.java:286)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:287)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1054)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:308)
... 8 more
UPDATE 2:
lsnrctl status
LSNRCTL for Linux: Version 11.2.0.2.0 - Production on 28-MAY-2013 22:34:16
Copyright (c) 1991, 2010, Oracle. All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
TNS-12541: TNS:no listener
TNS-12560: TNS:protocol adapter error
TNS-00511: No listener
Linux Error: 111: Connection refused
I use glassfish 3.0.1 on windows 2003 with JDK 6u24.
When I start the server and try to login to the admin console (4848) I will receive this error message:
[#|2011-02-24T12:59:37.578+0330|INFO|glassfish3.0.1|javax.enterprise.system.std.
com.sun.enterprise.v3.services.impl|_ThreadID=25;_ThreadName=http-thread-pool-48
48-(2);|Cannot refresh Catalog : Connection timed out: connect|#]
Update
[#|2011-02-24T12:58:51.343+0330|INFO|glassfish3.0.1|null|_ThreadID=24;_ThreadNam
e=Thread-23;|The Admin Console is already installed, but not yet loaded.|#]
[#|2011-02-24T12:58:51.343+0330|INFO|glassfish3.0.1|null|_ThreadID=24;_ThreadNam
e=Thread-23;|The Admin Console is loading...|#]
[#|2011-02-24T12:58:52.296+0330|INFO|glassfish3.0.1|javax.enterprise.system.cont
ainer.web.com.sun.enterprise.web|_ThreadID=24;_ThreadName=Thread-23;|Created HTT
P listener http-listener-1 on port 80|#]
[#|2011-02-24T12:58:52.296+0330|INFO|glassfish3.0.1|javax.enterprise.system.cont
ainer.web.com.sun.enterprise.web|_ThreadID=24;_ThreadName=Thread-23;|Created HTT
P listener http-listener-2 on port 8181|#]
[#|2011-02-24T12:58:52.296+0330|INFO|glassfish3.0.1|javax.enterprise.system.cont
ainer.web.com.sun.enterprise.web|_ThreadID=24;_ThreadName=Thread-23;|Created HTT
P listener admin-listener on port 4848|#]
[#|2011-02-24T12:58:52.343+0330|INFO|glassfish3.0.1|javax.enterprise.system.cont
ainer.web.com.sun.enterprise.web|_ThreadID=24;_ThreadName=Thread-23;|Created vir
tual server server|#]
[#|2011-02-24T12:58:52.343+0330|INFO|glassfish3.0.1|javax.enterprise.system.cont
ainer.web.com.sun.enterprise.web|_ThreadID=24;_ThreadName=Thread-23;|Created vir
tual server __asadmin|#]
[#|2011-02-24T12:58:53.531+0330|INFO|glassfish3.0.1|javax.enterprise.system.cont
ainer.web.com.sun.enterprise.web|_ThreadID=24;_ThreadName=Thread-23;|Virtual ser
ver server loaded system default web module|#]
[#|2011-02-24T12:58:54.671+0330|INFO|glassfish3.0.1|javax.enterprise.system.core
.security.com.sun.enterprise.security|_ThreadID=24;_ThreadName=Thread-23;|SEC100
2: Security Manager is OFF.|#]
[#|2011-02-24T12:58:54.937+0330|INFO|glassfish3.0.1|javax.enterprise.system.core
.security.com.sun.enterprise.security|_ThreadID=24;_ThreadName=Thread-23;|Securi
ty startup service called|#]
[#|2011-02-24T12:58:54.953+0330|INFO|glassfish3.0.1|javax.enterprise.system.core
.security.com.sun.enterprise.security|_ThreadID=24;_ThreadName=Thread-23;|SEC114
3: Loading policy provider com.sun.enterprise.security.provider.PolicyWrapper.|#
]
[#|2011-02-24T12:58:55.062+0330|INFO|glassfish3.0.1|javax.enterprise.system.core
.security.com.sun.enterprise.security.auth.realm|_ThreadID=24;_ThreadName=Thread
-23;|Realm admin-realm of classtype com.sun.enterprise.security.auth.realm.file.
FileRealm successfully created.|#]
[#|2011-02-24T12:58:55.062+0330|INFO|glassfish3.0.1|javax.enterprise.system.core
.security.com.sun.enterprise.security.auth.realm|_ThreadID=24;_ThreadName=Thread
-23;|Realm file of classtype com.sun.enterprise.security.auth.realm.file.FileRea
lm successfully created.|#]
[#|2011-02-24T12:58:55.062+0330|INFO|glassfish3.0.1|javax.enterprise.system.core
.security.com.sun.enterprise.security.auth.realm|_ThreadID=24;_ThreadName=Thread
-23;|Realm certificate of classtype com.sun.enterprise.security.auth.realm.certi
ficate.CertificateRealm successfully created.|#]
[#|2011-02-24T12:58:55.078+0330|INFO|glassfish3.0.1|javax.enterprise.system.core
.security.com.sun.enterprise.security|_ThreadID=24;_ThreadName=Thread-23;|Securi
ty service(s) started successfully....|#]
[#|2011-02-24T12:58:56.687+0330|INFO|glassfish3.0.1|javax.enterprise.resource.we
bcontainer.jsf.config|_ThreadID=24;_ThreadName=Thread-23;|Initializing Mojarra 2
.0.2 (FCS b10) for context ''|#]
[#|2011-02-24T12:58:58.546+0330|INFO|glassfish3.0.1|javax.enterprise.system.cont
ainer.web.com.sun.enterprise.web|_ThreadID=24;_ThreadName=Thread-23;|Loading app
lication __admingui at /|#]
[#|2011-02-24T12:58:58.546+0330|INFO|glassfish3.0.1|null|_ThreadID=24;_ThreadNam
e=Thread-23;|Loading __admingui Application done is 7203 ms|#]
[#|2011-02-24T12:58:58.546+0330|INFO|glassfish3.0.1|null|_ThreadID=24;_ThreadNam
e=Thread-23;|The Admin Console application is loaded.|#]
[#|2011-02-24T12:59:02.515+0330|INFO|glassfish3.0.1|javax.enterprise.system.tool
s.admin.org.glassfish.server|_ThreadID=25;_ThreadName=http-thread-pool-4848-(2);
|AMX ComplianceMonitor: ValidationLevel = full, UnregisterNonCompliant = false,
LogInaccessibleAttributes = true|#]
[#|2011-02-24T12:59:02.828+0330|INFO|glassfish3.0.1|javax.enterprise.system.tool
s.admin.org.glassfish.server|_ThreadID=26;_ThreadName=Thread-24;|AMX config read
, domain config registered as amx:pp=/,type=domain|#]
[#|2011-02-24T12:59:03.046+0330|INFO|glassfish3.0.1|javax.enterprise.system.tool
s.admin.org.glassfish.server|_ThreadID=27;_ThreadName=Thread-27;|J2EEDomain regi
stered at amx:pp=/,type=J2EEDomain,j2eeType=J2EEDomain,name=amx|#]
[#|2011-02-24T12:59:03.046+0330|INFO|glassfish3.0.1|javax.enterprise.system.tool
s.admin.org.glassfish.server|_ThreadID=25;_ThreadName=http-thread-pool-4848-(2);
|AMXStartupServiceNew: AMX ready for use, DomainRoot = amx:pp=,type=domain-root|
#]
[#|2011-02-24T12:59:10.843+0330|WARNING|glassfish3.0.1|javax.enterprise.system.c
ontainer.web.com.sun.web.security|_ThreadID=25;_ThreadName=http-thread-pool-4848
-(2);|Web login failed: Login failed: javax.security.auth.login.LoginException:
Failed file login for admin.|#]
[#|2011-02-24T12:59:10.843+0330|WARNING|glassfish3.0.1|org.apache.catalina.conne
ctor.Request|_ThreadID=25;_ThreadName=http-thread-pool-4848-(2);|PWC4011: Unable
to set request character encoding to UTF-8 from context , because request param
eters have already been read, or ServletRequest.getReader() has already been cal
led|#]
[#|2011-02-24T12:59:16.093+0330|INFO|glassfish3.0.1|org.glassfish.admingui|_Thre
adID=25;_ThreadName=http-thread-pool-4848-(2);|admin console: initSessionAttribu
tes()|#]
[#|2011-02-24T12:59:37.578+0330|INFO|glassfish3.0.1|javax.enterprise.system.std.
com.sun.enterprise.v3.services.impl|_ThreadID=25;_ThreadName=http-thread-pool-48
48-(2);|Cannot refresh Catalog : Connection timed out: connect|#]
[#|2011-02-24T13:08:09.015+0330|INFO|glassfish3.0.1|org.glassfish.admingui|_Thre
adID=28;_ThreadName=http-thread-pool-4848-(1);|admin console: initSessionAttribu
tes()|#]
[#|2011-02-24T13:08:30.125+0330|INFO|glassfish3.0.1|javax.enterprise.system.std.
com.sun.enterprise.v3.services.impl|_ThreadID=28;_ThreadName=http-thread-pool-48
48-(1);|Cannot refresh Catalog : Connection timed out: connect|#]
[#|2011-02-24T13:14:16.546+0330|WARNING|glassfish3.0.1|com.sun.grizzly.config.Gr
izzlyServiceListener|_ThreadID=17;_ThreadName=Thread-13;|Interrupting idle Threa
d: http-thread-pool-4848-(2)|#]
It seems that Glassfish has a lot of trouble recently caused by the new version 3.1. The admin console tries to look for new updates at start-up. If the update server will not respond in a certain time, the update count fails or take a long time.
You could first try to choose a different update server with glassfishs update tool. You will find it in glassfish/bin folder.
If this doesn't work, you could disable the update module of the glassfish admin console by renaming the file console-updatecenter-plugin.jar in the folder glassfish/glassfish/modules.
You could also disable network access from the admin console by adding the following to the JVM-Options of your domain.xml:
-Dcom.sun.enterprise.tools.admingui.NO_NETWORK=true