I have a problem that thought I will break down to the simplest. There two applications on LAMP stack, one PHP and another Java that do the only and exactly the same thing: run a simple query:
SELECT * FROM test
PHP execution takes 30 ms in total
Java excution takes 230 ms in total
Query run on a local MySQL client takes 10-15 ms in total
Java takes ~200 ms roughly every time only to establish a connection to the db. I understand that PHP uses some kind of built in connection pooling, therefor it doesn't need to establish a new connection every time and only takes 30 ms as a result of it.
Is the same thing possible on Java? So far I failed to achieve that. I tried to use Apache Commons DBCP connection pooling, no change at all, still takes the same time to connect to the database.
UPDATE: This is a separate question where I'm trying to make connection pooling work on Java, for those who are asking for a code example: Java MySQL connetion pool is not working
You are misunderstanding the concept and purpose of connection pooling.
Connection pooling is meant to maintain a (set of) connections on a single (Java Virtual) machine (typically, an application server). The goal is to allow multiple threads on the same machine to essentially share their connection to the database server without the need to open one everytime they need to query the database.
Stand-alone applications cannot share connections as they run on different Java Virtual Machines, or perhaps even on different physical machines.
Connection pooling would help in your stand-alone application if you had several threads making concurrent access to the database.
However, you can still measure the benefit of connection pooling by wrapping your test (the code inside your try...catch) in a loop and iterating a few times. At the first iteration, the connection needs to be opened. Don't forget to release the connection (call Con.close()), then it would be reused at the next iteration.
Related
This question already has answers here:
Should a database connection stay open all the time or only be opened when needed?
(5 answers)
Closed 3 years ago.
I'm connecting to a small PostgreSQL database on the local LAN using Java and running some queries and I'm not sure what is best practice.
Should I close the connection after each query or keep the connection open for the next query?
you probably want to keep the connection open for as long a possible, but in practice it probably won't matter much unless you're making hundreds of little queries and reconnecting for every one
here's an overview of what's going on, assuming your network has ping times of around 1ms:
making the socket connection, a TCP handshake should take <5ms
your client with present authentication credentials, and the server will check them. this might take a while depending on how you've got things configured but by default it'll take <1ms
your client sends a query, the server will need to populate various internal data-structures and caches (this will take a few milliseconds), the server will then be able to plan and execute the query which will vary according to the query
more details here and here
if you reuse this connection then you save the cost of authentication and catalog loading. you're therefore recommended to reuse connections where possible so this work isn't performed every time. depending on your usage this may or may not actually make much difference. for example:
if you're running lots of queries that take a few milliseconds each then it'll certainly be helpful to reuse connections
if you're mostly running queries that take many seconds to execute then this cost will be negligible
My use case is a bunch of isolated calls that at some point interact with redis.
The problem is I'm seeing a super long wait time for acquiring a connection, having tried both predis and credis on my LAN environment. Over 1-3 client threads, the time it takes for my PHP scripts to connect to redis and select a database ranges from 18ms to 700ms!
Normally, I'd use a connection pool or cache a connection and use it across all my threads, but I don't think this can be done in PHP over different scripts.
Is there anything I can do to speed this up?
Apparently, Predis needs the persistent flag set: https://github.com/nrk/predis/wiki/Connection-Parameters) and also FPM, which was frustrating to set up on both Windows and Linux, not to mention testing before switching to FPM on our live setup.
I've switched to Phpredis (https://github.com/phpredis/phpredis), which is a PHP module/extension, and all is good now. The connection times have dropped dramatically using $redis->pconnect() and are consistent across multiple scripts/threads.
Caveat: it IS a little different from Predis in terms of error handling (it fails when instantiating the object, not when running the first call, it returns false instead of null for nonexistent values - ???), so watch out for that if switching from Predis.
I am in the process of building a client-server application and I would really like an advise on how to design the server-database connection part.
Let's say the basic idea is the following:
Client authenticates himself on the server.
Client sends a request to server.
Server stores client's request to the local database.
In terms of Java Objects we have
Client Object
Server Object
Database Object
So when a client connects to the server a session is created between them through which all the data is exchanged. Now what bothers me is whether i should create a database object/connection for each client session or whether I should create one database object that will handle all requests.
Thus the two concepts are
Create one database object that handles all client requests
For each client-server session create a database object that is used exclusively for the client.
Going with option 1, I guess that all methods should become synchronized in order to avoid one client thread not overwriting the variables of the other. However, making it synchronize it will be time consuming in the case of a lot of concurrent requests as each request will be placed in queue until the one running is completed.
Going with option 2, seems a more appropriate solution but creating a database object for every client-server session is a memory consuming task, plus creating a database connection for each client could lead to a problem again when the number of concurrent connected users is big.
These are just my thoughts, so please add any comments that it may help on the decision.
Thank you
Option 3: use a connection pool. Every time you want to connect to the database, you get a connection from the pool. When you're done with it, you close the connection to give it back to the pool.
That way, you can
have several clients accessing the database concurrently (your option 1 doesn't allow that)
have a reasonable number of connections opened and avoid bringing the database to its knees or run out of available connections (your option 2 doesn't allow that)
avoid opening new database connections all the time (your option 2 doesn't allow that). Opening a connection is a costly operation.
Basically all server apps use this strategy. All Java EE servers come with a connection pool. You can also use it in Java SE applications, by using a pool as a library (HikariCP, Tomcat connection pool, etc.)
I would suggested a third option, database connection pooling. This way you create a specified number of connections and give out the first available free connection as soon as it becomes available. This gives you the best of both worlds - there will almost always be free connections available quickly and you keep the number of connections the database at a reasonable level. There are plenty of the box java connection pooling solutions, so have a look online.
Just use connection pooling and go with option 2. There are quite a few - C3P0, BoneCP, DBCP. I prefer BoneCP.
Both are not good solutions.
Problem with Option 1:
You already stated the problems with synchronizing when there are multiple threads. But apart from that there are many other problems like transaction management (when are you going to commit your connection?), Security (all clients can see precommitted values).. just to state a few..
Problem with Option 2:
Two of the biggest problems with this are:
It takes a lot of time to create a new connection each and every time. So performance will become an issue.
Database connections are extremely expensive resources which should be used in limited numbers. If you start creating DB Connections for every client you will soon run out of them although most of the connections would not be actively used. You will also see your application performance drop.
The Connection Pooling Option
That is why almost all client-server applications go with the connection pooling solution. You have a set connections in the pool which are obtained and released appropriately. Almost all Java Frameworks have sophisticated connection pooling solutions.
If you are not using any JDBC framework (most use the Spring JDBC\Hibernate) read the following article:
http://docs.oracle.com/javase/jndi/tutorial/ldap/connect/pool.html
If you are using any of the popular Java Frameworks like Spring, I would suggest you use Connection Pooling provided by the framework.
I have a requirement to write a java application (web based) that will connect to an Oracle 11G databases (currently is connecting 10-12 different oracle databases), read some data from it (all are select queries).
After iterating this arraylist, I am connecting each database, fire select query (same query I am firing for all databases), get record , put it in one global collection list and close connection and continuing same process over this loop.
Currently I am "Executors" to connect multiple databases.
again using ExecutorService executor = Executors.newFixedThreadPool(20);
give one surprise to me. for creting first database connection, it shows immediate log, but for subsequent database connection, it prints logs after 60 seconds, so i am not getting why this is taking 60 secs or more for all connections?
logically, it should take time for all connections as like one.
Please suggest performance improvement for this application.
Opening a database connection is an expensive operation; if possible you should connect to each database once and reuse the connection for all queries made to that database (also known as connection pooling). It's not clear if this is what you're already doing.
Another thing that seems strange is that you have made the openConnection method synchronized. I see no need for doing that, and depending on how your code is structured it may mean that only one thread can be opening a connection at a given time. Since connecting takes a long time you'd want each thread to make its connection in parallel.
I have a java program running 24/7. It accesses mysql database every 3 seconds only from 9.am to 3.PM. In this case when should I should open and close the MySql connection?
Should I open and close every 3 sec?
Should I open at 9.am and close at 3.pm?
Should I open once when the program start and never close it. But reconnect when connection is closed automatically and exceptions is thrown?
Why don't you simply use a connection pool. If that is too tedious since the connection will be frequently used you can reuse the same one imho.
While it is true that setting up and tearing down a MySQL connection is relatively cheap (when compared to, for example, Oracle), doing it every 3 seconds is a waste of resources. I'd cache the connection and save the overhead of creating a new database connection every time.
This depends very much on the situation. Do you connect over a WAN, is the MySQL server shared with other applications or will you be the only user (or at least will your application create most of the load?) If the database is mostly yours and it is near enough, there is little benefit in setting up and tearing down the connection daily.
This is what most applications do and this is what I'd recommend you do by default.
If you do not want to leave connections open overnight, you might able to configure your connection pool to open connections on demand and close them when they have been idle for a certain period of time -- say, 15 minutes. This would give you the benefit of being able to query the database whenever you wish and not having too many idle connections.