I would like to find out how I can set a timeout for a Statement.executeBatch() in Java. The code works well when the database is up, but when I block the db IP in iptables, the JVM hangs at executeBatch(). I tried using .setQueryTimeout(10) but an exception is thrown:
SQLException cannot be cast to SybSQLException
On database side I use Sybase and jdbc2 for connection
Thank you.
PS: I can't post the code because it is company copyrighted.
Related
I am trying to connect to a Postgresql database, I am getting the following Error:
Error:org.postgresql.util.PSQLException: FATAL: sorry, too many clients already
What does the error mean and how do I fix it?
My server.properties file is following:
serverPortData=9042
serverPortCommand=9078
trackConnectionURL=jdbc:postgresql://127.0.0.1:5432/vTrack?user=postgres password=postgres
dst=1
DatabaseName=vTrack
ServerName=127.0.0.1
User=postgres
Password=admin
MaxConnections=90
InitialConnections=80
PoolSize=100
MaxPoolSize=100
KeepAliveTime=100
TrackPoolSize=120
TrackMaxPoolSize=120
TrackKeepAliveTime=100
PortNumber=5432
Logging=1
An explanation of the following error:
org.postgresql.util.PSQLException: FATAL: sorry, too many clients already.
Summary:
You opened up more than the allowed limit of connections to the database. You ran something like this: Connection conn = myconn.Open(); inside of a loop, and forgot to run conn.close();. Just because your class is destroyed and garbage collected does not release the connection to the database. The quickest fix to this is to make sure you have the following code with whatever class that creates a connection:
protected void finalize() throws Throwable
{
try { your_connection.close(); }
catch (SQLException e) {
e.printStackTrace();
}
super.finalize();
}
Place that code in any class where you create a Connection. Then when your class is garbage collected, your connection will be released.
Run this SQL to see postgresql max connections allowed:
show max_connections;
The default is 100. PostgreSQL on good hardware can support a few hundred connections at a time. If you want to have thousands, you should consider using connection pooling software to reduce the connection overhead.
Take a look at exactly who/what/when/where is holding open your connections:
SELECT * FROM pg_stat_activity;
The number of connections currently used is:
SELECT COUNT(*) from pg_stat_activity;
Debugging strategy
You could give different usernames/passwords to the programs that might not be releasing the connections to find out which one it is, and then look in pg_stat_activity to find out which one is not cleaning up after itself.
Do a full exception stack trace when the connections could not be created and follow the code back up to where you create a new Connection, make sure every code line where you create a connection ends with a connection.close();
How to set the max_connections higher:
max_connections in the postgresql.conf sets the maximum number of concurrent connections to the database server.
First find your postgresql.conf file
If you don't know where it is, query the database with the sql: SHOW config_file;
Mine is in: /var/lib/pgsql/data/postgresql.conf
Login as root and edit that file.
Search for the string: "max_connections".
You'll see a line that says max_connections=100.
Set that number bigger, restart postgresql database.
What's the maximum max_connections?
Use this query:
select min_val, max_val from pg_settings where name='max_connections';
I get the value 8388607 so in theory that's the most you are allowed to have, but then a runaway process can eat up thousands of connections, and surprise, your database is unresponsive until reboot. If you had a sensible max_connections like 100. The offending program would be denied a new connection and the database is safu.
We don't know what server.properties file is that, we neither know what SimocoPoolSize means (do you?)
Let's guess you are using some custom pool of database connections. Then, I guess the problem is that your pool is configured to open 100 or 120 connections, but you Postgresql server is configured to accept MaxConnections=90 . These seem conflictive settings. Try increasing MaxConnections=120.
But you should first understand your db layer infrastructure, know what pool are you using, if you really need so many open connections in the pool. And, specially, if you are gracefully returning the opened connections to the pool
No need to increase the MaxConnections & InitialConnections. Just close your connections after after doing your work. For example if you are creating connection:
try {
connection = DriverManager.getConnection(
"jdbc:postgresql://127.0.0.1/"+dbname,user,pass);
} catch (SQLException e) {
e.printStackTrace();
return;
}
After doing your work close connection:
try {
connection.commit();
connection.close();
} catch (SQLException e) {
e.printStackTrace();
}
The offending lines are the following:
MaxConnections=90
InitialConnections=80
You can increase the values to allow more connections.
You need to close all your connexions for example:
If you make an INSERT INTO statement you need to close the statement and your connexion in this way:
statement.close();
Connexion.close():
And if you make a SELECT statement you need to close the statement, the connexion and the resultset in this way:
resultset.close();
statement.close();
Connexion.close();
I did this and it worked
I had postgres and other apps up in docker. I was facing this problem when more than ten apps connected to postgres database. The solution was to increase postgres max_connection count. This is 100 by default. To increase this value, either find max_connection in the /var/lib/pgsql/data/postgresql.conf file and edit it. Another way is to add and run the docker-compose.yml document as follows.
version: '3'
services:
taxi-postgresql:
container_name: my-postgresql
image: postgres:13.3
volumes:
- ./postgres-volume:/var/lib/postgresql/data
environment:
- POSTGRES_DB=taxi-postgresql
- POSTGRES_USER=user
- POSTGRES_PASSWORD=user
- POSTGRES_HOST_AUTH_METHOD=trust
command: postgres -c 'max_connections=1000'
ports:
- 127.0.0.1:5432:5432
The same error appears in our microservices deployment, and it is solved by increasing the below value in the Postgresql container:
num_init_children
I am using JDBC driver to connect to mySql from my java code (read client).
Driver = com.mysql.jdbc.Driver
JdbcUrl = jdbc:mysql://<<IpOftheDb>>/<<DbSchema Name>>?autoReconnect=true&connectTimeout=5000&socketTimeout=10000
In case the database is down ( machine hosting the db is up but the mysqld process is not running) , it takes some time to get the exception , The exception is
"com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:Could not create connection to database server. Attempted reconnect 3 times. Giving up."
In the statement above socketTimeout is 10 sec . Now if I bring up the db with 10 sec as SocketTimeout I get the response correctly.
But If i reduce it to one sec and am executing the query I get the same exception.
"com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:Could not create connection to database server. Attempted reconnect 3 times. Giving up."
But connectTimeout doesnt change anything. Can someone explain me what socketTimeout and connectTimeout means.
Also , If we are setting up replication and specifying the 2nd database as failover i.e.
my connection string changes to
jdbc:mysql://<<PrimaryDbIP>>,<<SecondaryDbIp>>/<<DbSchema>>?useTimezone=true
&serverTimezone=UTC&useLegacyDatetimeCode=false
&failOverReadOnly=false&autoReconnect=true&maxReconnects=3
&initialTimeout=5000&connectTimeout=6000&socketTimeout=6000
&queriesBeforeRetryMaster=50&secondsBeforeRetryMaster=30
I see that if primary is down then I get the response from secondary (failover Db) .
Now when client executes a query , does it go to primary database and then wait for socketTimeout (or whatever) and then goes to Secondary or it goes to Seconday before timeout occurs.
Moreover, the second time when the same connection Object is used , does it go directly to the secondary or again the above process is repeated .
I tried find some documentation which explains this but couldnt get .
Hopefully , someone can help here explaining the various timeout parameters and their usefulness.
I'm writing a program that does some stuff on the database. Users are allowed to configure db processes, by passing db host port, type and credentials. It all works fine when values are correct. But when user passes invalid credentials I would like to show an error. So here is the part where I create my connection pool
ComboPooledDataSource cpds = new ComboPooledDataSource();
cpds.setJdbcUrl( connectionUrl );
cpds.setUser(username);
cpds.setPassword(password);
And later to verify if all is ok with the connection I do
cpds.getConnection()
I would expect to get some SQLException with vendor specific error saying that credentials are invalid (which happens when you use typical DriverManager way of getting the connection), but instead the process waits until a connection checkout exception is thrown
java.sql.SQLException: An attempt by a client to checkout a Connection has timed out.
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118)
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:77)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:690)
....
Caused by: com.mchange.v2.resourcepool.TimeoutException: A client timed out while waiting to acquire a resource from com.mchange.v2.resourcepool.BasicResourcePool#20014b8 -- timeout at awaitAvailable()
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1467)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:644)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:554)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:758)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:685)
... 66 more
How can I identify that there is a invalid credential issue with c3p0?
Your best way to validate provided credentials/JDBC params is to avoid connection pool at all.
Open dedicated connection just for this purpose and try to execute simplest SQL against new connection (eg SELECT 1 or similar).
After success, you can pass them to C3P0 otherwise propagate error back to user.
JDBC providers are free to create whatever error/exception messages they want. So you need to be ready to parse the error message of each provider in order to make sense of what is happening.
You can also try to get information from exception types if the JDBC provider segregates errors in separate types.
As a side note, giving too much information regarding why the connection failed may be considered a security breach. So one should not expect the JDBC driver to give you such information. For instance, why would any database collaborate with invasion attempts by saying "the username is correct, but the password is not."?
My server calls sometimes the MSSQL JDBC CallableStatement.execute() that works with a previously connected connection. Everything is ok while the connection is alive. But if the connection is disconnected somehow (e.g. somebody turn SQL server down) the execute() call throws SQLException. I need to differ between 'connection was dropped' error and any other JDBC error (like table I'm trying to use doesn't exist). Since if I hit the connection error - I need to reconnect. And in any other case I need just to give error message to the user.
SQLException.getSQLState() always returns null and getErrorCode() always returns 0.
Thanks
I am no Java expert but this seems to be mainly based on Java.
You'd need to catch the exception from the .execute() command and query the string to see what the error is, then make a decision on which method to execute next based on that information.
I am getting the following error:
java.sql.SQLException: Closed
Connection at
oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
at
oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
at
oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:208)
at
oracle.jdbc.driver.PhysicalConnection.getMetaData(PhysicalConnection.java:1508)
at
com.ibatis.sqlmap.engine.execution.SqlExecutor.moveToNextResultsSafely(SqlExecutor.java:348)
at
com.ibatis.sqlmap.engine.execution.SqlExecutor.handleMultipleResults(SqlExecutor.java:320)
at
com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQueryProcedure(SqlExecutor.java:277)
at
com.ibatis.sqlmap.engine.mapping.statement.ProcedureStatement.sqlExecuteQuery(ProcedureStatement.java:34)
at
com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithCallback(GeneralStatement.java:173)
at
com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryForList(GeneralStatement.java:123)
at
com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForList(SqlMapExecutorDelegate.java:614)
at
com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForList(SqlMapExecutorDelegate.java:588)
at
com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryForList(SqlMapSessionImpl.java:118)
at
org.springframework.orm.ibatis.SqlMapClientTemplate$3.doInSqlMapClient(SqlMapClientTemplate.java:268)
at
org.springframework.orm.ibatis.SqlMapClientTemplate.execute(SqlMapClientTemplate.java:193)
at
org.springframework.orm.ibatis.SqlMapClientTemplate.executeWithListResult(SqlMapClientTemplate.java:219)
at
org.springframework.orm.ibatis.SqlMapClientTemplate.queryForList(SqlMapClientTemplate.java:266)
at
gov.hud.pih.eiv.web.authentication.AuthenticationUserDAO.isPihUserDAO(AuthenticationUserDAO.java:24)
at
gov.hud.pih.eiv.web.authorization.AuthorizationProxy.isAuthorized(AuthorizationProxy.java:125)
at
gov.hud.pih.eiv.web.authorization.AuthorizationFilter.doFilter(AuthorizationFilter.java:224)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
at
I am really stumped and can't figure out what could be causing this error. I am not able to reproduce the error on my machine but on production it is coming a lot of times. I am using iBatis in the whole application so there are no chances of my code not closing connections.
We do have stored procedures that run for a long time before they return results (around 15 seconds).
does anyone have any ideas on what could be causing this? I dont think raising the # of connections on the application server will fix this issue buecause if connections were running out then we'd see "Error on allocating connections"
Sample code snippet:
this.setSqlMapClientTemplate(getSqlTempl());
getSqlMapClientTemplate().queryForList("authentication.isUserDAO", parmMap);
this.setSqlMapClientTemplate(getSqlTemplDW());
List results = (List) parmMap.get("Result0");
I am using validate in my connection pool.
Based on the stack trace, the likely cause is that you are continuing to use a ResultSet after close() was called on the Connection that generated the ResultSet.
What is your DataSource framework? Apache Commons DBCP?
do you use poolPrepareStatement property in data source configuration?
Check the following:
Make sure testOnBorrow and testOnReturn are true and place a simple validationQuery like select 0 from dual.
Do you use au
do you use autoCommit? Are you using START TRANSACTION, COMMIT in your stored procedures? After several days of debugging we found out that you can't mix transaction management both in Java and in SQL - you have to decide on one place to do it. Where are you doing yours?
Edit your question with answers to this, an we'll continue from there.
When a db server reboots, or there are some problems with a network, all the connections in the connection pool are broken and this usuall requires a reboot of application server
And if broken connection detected, you shold create a new one to replace it in connection pool. It's common problem called deadly connections.