i am running a web app where 3 databases are involved
the first database is the admin database and the two other databases are for two separate institutions,
meaning both institutions are using the same app but can access their separate database per a unique_code entered.
the databases are starter(admin database),company1 and company2.
when the web app is started, the admin database is initially connected to automatically. (starter database).
(first connection pool) code below: which works perfectly.
comboPooledDataSource.setDriverClass("com.mysql.cj.jdbc.Driver");
comboPooledDataSource.setJdbcUrl("jdbc:mysql://host.com/starter");
comboPooledDataSource.setUser("username");
comboPooledDataSource.setPassword("password");
comboPooledDataSource.setMinPoolSize(2);
comboPooledDataSource.setMaxPoolSize(3000);
comboPooledDataSource.setAcquireIncrement(1);
comboPooledDataSource.setMaxIdleTime(1800);
comboPooledDataSource.setMaxStatements(0);
comboPooledDataSource.setIdleConnectionTestPeriod(3);
comboPooledDataSource.setBreakAfterAcquireFailure(false);
comboPooledDataSource.setUnreturnedConnectionTimeout(5);
and the user must enter a code in a textfield on the homepage (like a login).
if the code exist in the starter database, the database related to the code is connected to and the user can view their contents from that database.
//code to fetch database name is written below: which also works successfully
String entry_code=request.getParameter("Ecode");
//where 'Ecode' is the name of the html textfield where the user types the code
try{
con=Main_C3Po_Connection.getInstance().getConnection();
String sql="select db from checker where code='"+entry_code+"'";
pst=con.prepareStatement(sql);
rs=pst.executeQuery();
if(rs.next()){
get_db=rs.getString("db");
}
}catch(SQLException e){
out.println(e);
}
eg: starter(admin database)
table name : checker
id | code | db |
11 | 44 | company1 |
12 | 35 | company2 |
so the second connection pool doesnt have a fixed database url but a variable database name.
eg:("jdbc:mysql://host.com/"+get_db+"?autoReconnect=true&useUnicode=yes");
where get_db is the variable name.
so when the user enters code 44, the value in the db column relating to the code entered is (company1), is then placed into the get_db variable and the database is connected to and can be accessed.
when the first code(44) is entered, the 'company1' value is placed into the 'get_db' variable and the connection is made successfully.
but the problem is after logging out and the second code (35) is entered, the 'company2' value is also placed into the 'get_db' variable BUT
the connection pool for some reason still keeps the previous database connection and cannot switch to the other database chosen.
below is the second connection pool which cannot switch to a different database, though the database variable is changed:
comboPooledDataSource.setDriverClass("com.mysql.cj.jdbc.Driver");
comboPooledDataSource.setJdbcUrl("jdbc:mysql://host.com/"+get_db+"?autoReconnect=true&useUnicode=yes");
comboPooledDataSource.setUser("username");
comboPooledDataSource.setPassword("password");
comboPooledDataSource.setMinPoolSize(2);
comboPooledDataSource.setMaxPoolSize(3000);
comboPooledDataSource.setAcquireIncrement(1);
comboPooledDataSource.setMaxIdleTime(1800);
comboPooledDataSource.setMaxStatements(0);
comboPooledDataSource.setIdleConnectionTestPeriod(5);
comboPooledDataSource.setBreakAfterAcquireFailure(false);
comboPooledDataSource.setUnreturnedConnectionTimeout(5);
please how do i configure the second connection pool to kill all connections after logging out so that it can **switch** and access any other database chosen. thank you.
This is an awkward configuration; I don't recommend it. But it should work. The act of calling
comboPooledDataSource.setJdbcUrl("jdbc:mysql://host.com/"+get_db+"?autoReconnect=true&useUnicode=yes");
should cause a "soft reset", so any new Connections you get from the pool will be to the new DB. Are you sure that you are not still using the old Connection? That is, have you been sure to close() all Connection objects from before the change?
A less awkward approach would just be to make multiple Connection pools, one for each database you need to access. When you are done with a Connection pool, free the threads and Connections associated with it by calling close() on the pool itself.
Related
I am trying to connect to a Postgresql database, I am getting the following Error:
Error:org.postgresql.util.PSQLException: FATAL: sorry, too many clients already
What does the error mean and how do I fix it?
My server.properties file is following:
serverPortData=9042
serverPortCommand=9078
trackConnectionURL=jdbc:postgresql://127.0.0.1:5432/vTrack?user=postgres password=postgres
dst=1
DatabaseName=vTrack
ServerName=127.0.0.1
User=postgres
Password=admin
MaxConnections=90
InitialConnections=80
PoolSize=100
MaxPoolSize=100
KeepAliveTime=100
TrackPoolSize=120
TrackMaxPoolSize=120
TrackKeepAliveTime=100
PortNumber=5432
Logging=1
An explanation of the following error:
org.postgresql.util.PSQLException: FATAL: sorry, too many clients already.
Summary:
You opened up more than the allowed limit of connections to the database. You ran something like this: Connection conn = myconn.Open(); inside of a loop, and forgot to run conn.close();. Just because your class is destroyed and garbage collected does not release the connection to the database. The quickest fix to this is to make sure you have the following code with whatever class that creates a connection:
protected void finalize() throws Throwable
{
try { your_connection.close(); }
catch (SQLException e) {
e.printStackTrace();
}
super.finalize();
}
Place that code in any class where you create a Connection. Then when your class is garbage collected, your connection will be released.
Run this SQL to see postgresql max connections allowed:
show max_connections;
The default is 100. PostgreSQL on good hardware can support a few hundred connections at a time. If you want to have thousands, you should consider using connection pooling software to reduce the connection overhead.
Take a look at exactly who/what/when/where is holding open your connections:
SELECT * FROM pg_stat_activity;
The number of connections currently used is:
SELECT COUNT(*) from pg_stat_activity;
Debugging strategy
You could give different usernames/passwords to the programs that might not be releasing the connections to find out which one it is, and then look in pg_stat_activity to find out which one is not cleaning up after itself.
Do a full exception stack trace when the connections could not be created and follow the code back up to where you create a new Connection, make sure every code line where you create a connection ends with a connection.close();
How to set the max_connections higher:
max_connections in the postgresql.conf sets the maximum number of concurrent connections to the database server.
First find your postgresql.conf file
If you don't know where it is, query the database with the sql: SHOW config_file;
Mine is in: /var/lib/pgsql/data/postgresql.conf
Login as root and edit that file.
Search for the string: "max_connections".
You'll see a line that says max_connections=100.
Set that number bigger, restart postgresql database.
What's the maximum max_connections?
Use this query:
select min_val, max_val from pg_settings where name='max_connections';
I get the value 8388607 so in theory that's the most you are allowed to have, but then a runaway process can eat up thousands of connections, and surprise, your database is unresponsive until reboot. If you had a sensible max_connections like 100. The offending program would be denied a new connection and the database is safu.
We don't know what server.properties file is that, we neither know what SimocoPoolSize means (do you?)
Let's guess you are using some custom pool of database connections. Then, I guess the problem is that your pool is configured to open 100 or 120 connections, but you Postgresql server is configured to accept MaxConnections=90 . These seem conflictive settings. Try increasing MaxConnections=120.
But you should first understand your db layer infrastructure, know what pool are you using, if you really need so many open connections in the pool. And, specially, if you are gracefully returning the opened connections to the pool
No need to increase the MaxConnections & InitialConnections. Just close your connections after after doing your work. For example if you are creating connection:
try {
connection = DriverManager.getConnection(
"jdbc:postgresql://127.0.0.1/"+dbname,user,pass);
} catch (SQLException e) {
e.printStackTrace();
return;
}
After doing your work close connection:
try {
connection.commit();
connection.close();
} catch (SQLException e) {
e.printStackTrace();
}
The offending lines are the following:
MaxConnections=90
InitialConnections=80
You can increase the values to allow more connections.
You need to close all your connexions for example:
If you make an INSERT INTO statement you need to close the statement and your connexion in this way:
statement.close();
Connexion.close():
And if you make a SELECT statement you need to close the statement, the connexion and the resultset in this way:
resultset.close();
statement.close();
Connexion.close();
I did this and it worked
I had postgres and other apps up in docker. I was facing this problem when more than ten apps connected to postgres database. The solution was to increase postgres max_connection count. This is 100 by default. To increase this value, either find max_connection in the /var/lib/pgsql/data/postgresql.conf file and edit it. Another way is to add and run the docker-compose.yml document as follows.
version: '3'
services:
taxi-postgresql:
container_name: my-postgresql
image: postgres:13.3
volumes:
- ./postgres-volume:/var/lib/postgresql/data
environment:
- POSTGRES_DB=taxi-postgresql
- POSTGRES_USER=user
- POSTGRES_PASSWORD=user
- POSTGRES_HOST_AUTH_METHOD=trust
command: postgres -c 'max_connections=1000'
ports:
- 127.0.0.1:5432:5432
The same error appears in our microservices deployment, and it is solved by increasing the below value in the Postgresql container:
num_init_children
I have to extend one of the queries, to join with the table in another database, and possibly another server (linked).
Here are some domain facts, I can't change:
- Weblogic 10 used. EJB2 on the Java side
- Database driver is Non-XA. And we can't change that
- Both of the data sources are set as "Emulate 2 phase commit" transaction
option
- Query that is used is not directly in the business method called by the client, but somewhere in a private method, that is called at some point in the execution.
I am getting the following
java.sql.SQLException: Connection has already been created in this tx
context
for pool named dataSourceA.datasource. Illegal attempt to create
connection from another pool: dataSourceB.datasource
I only need to get the Database name and database host from the data source.
Exceptions comes, when I try to make connection to the data source dataSourceB.
Is there any way to get its properties without making a connection? Just to get String properties, database name and host?
I have a code in Java for insert many sql lines in database from a text file.
I have the connection programmed with setAutoCommit(false), then at the end if none error happen (detected having all methods throws Throwable), I send the commit.
The task normally take 30 minutes.
It works very good in cable connection, but in a wifi connection it never reach the end because the connection is lost sometimes, for a short time.
For solve it. I programmed two things: The lines of the text file is converted to a serialized object that have all the lines in a ArrayList, and I created other serialized object that have a int index, that save the index of the last line inserted succesfully.
Then, in the program I do this:
charge in memory the object.lines, from the serialized object.
charge in memory the object.index, from the serialized object.
pseudo code:
loop:
index = sum 1 to the object.index
line = object.getLine(index)
insert line
if error continue (or goto loop)
send commit
if error continue (or goto loop)
object.index = index
serialize object
in this way I have a backup of the lines that are succesfully commited to the database, and I can continue the job in other time. If I have a connection problem in a line, I can try insert again the line.
If i have a connection problem, i wait 1 minute. The connection is recovered but is reseted automatically, not by me.
Then for example in a lines like this:
INSERT INTO my_table1 (id) VALUES (sq_mytable1_id.NEXTVAL);
//success
//connection lost
//connection reset
INSERT INTO my_table2 (id) VALUES (sq_mytable1_id.CURRVAL);
//error, sq_mytable1_id.CURRVAL is not in session.
I get a ORA-08002 exception because, the connection was reset, I can get the sq_mytable1_id.CURRVAL from the session.
Please, you can give me ideas of how programm a batch sql inserter tolerant to connection downs in wi-fi ?
I think serialize the connection, but I cannot: oracle.jdbc.driver.T4CConnection is not serializable.
I have Java application with Hibernate framework(no spring) connect to MySQL DB , manage connection pooling via c3p0
i try to configure my apllication to read from slave db and write to master db , i have following this link to some extend Master/Slave load balance
let's say if the application already got a session with connection in pool and it need to execute a read-only method , like this
public someReadOnlyMethod()
{
Session session = (get session from current Thread)
//set read-only so that it read from slave db
session.connection().setReadOnly(true);
(...connect to db to do something...)
//set it back in case of this method is followed by write method so that it go to master db
session.connection().setReadOnly(false);
}
Is the pooling create a new connection to connect to db 2 times for read-only and write operation(if so,this will heavily impact performance) or it smart enough to swap the operation to already existing read-only and writable connection pool ?
thx for your advice.
so this has nothing to do with the pool; it's all in the mysql driver. c3p0 will pass your call to setReadOnly (whether true or false) to the underlying Connection, and the Connection will route to the master or the slaves accordingly.
if you don't like how your Connections default (probably by default they are not read only), you can set the read-only property in the onAcquire method of a c3p0 ConnectionCustomizer, and the value use set (true or false) will become th default that c3p0 resets Connections to.
good luck!
tl;dr: It will re-use existing connections whenever you switch setReadOnly(true/false).
JDBC will connect to all servers listed in your connection URL when you do ReplicationDriver().connect(url). Those connections will remain open for re-use no matter how many times you switch setReadOnly().
Source: I just tested Connector/J version 5.1.38 with com.mysql.jdbc.ReplicationDriver.
The database connection is get like below
public Connection getDBConection(){
Context context = new InitialContext();
DataSource dataSource = (javax.sql.DataSource) context.lookup("java:myDataSource");
Connection conn = dataSource.getConnection();
}
For a userA, is each database request should call getDBConnection() once; but no need to control all request use the same connection?
That is, if userA has three database request, then userA should call getDBConnection() three times, and call Connection.closed() after used in each request?
If the userA call getDBConnection() three times (that is, call dataSource.getConnection() three times), is three connection created? Or it is unknown and controlled by weblogic?
I feel very chaos, is it true that there should be one new connection for one database request? or just call DataSource.getConnection() for each database request and the number of new connection created is controlled by web server, no need to think how many connection is actually created.
Every time you call DataSource.getConnection, the data source will retrieve a connection for you. It should be true that the returned connection is not being actively used by anyone else, but it is not necessarily a brand-new connection.
For example, if you use a connection pool, which is a very common practice, then when you call Connection.close, the connection is not actually closed, but instead returns to a pool of available connections. Then, when you call DataSource.getConnection, the connection pool will see if it has any spare connections lying around that it hasn't already handed out. If so, it will typically test that they haven't gone stale (usually by executing a very quick query against a dummy table). If not, it will return the existing connection to the caller. But if the connection is stale, then the connection pool will retrieve a truly new connection from the underlying database driver, and return that instead.
Typically, connection pools have a maximum number of real connections that they will keep at any one time (say, 50). If your application tries to request more than 50 simultaneous connections, DataSource.getConnection will throw an exception. Or in some implementations, it will block for a while until one becomes available, and then throw an exception after that time expires. For a sample implementation, have a look at Apache Commons DBCP.
Hopefully that answers your question!