I am dealing with high traffic in my Spring Boot project and my goal is serving clients as much fast as possible. In this case, I have more than 500 requests per second. In each rest endpoint call, I should connect my schema and gather multiple information from multiple tables. To be able to do that, should I create new connection for each eendpoint call or create & close before each db query?
I wrote a JDBC connection class but I am not sure that it is a good way. Maybe you can give me some opinion.
JDBC Connection Class
#PropertySource({"classpath:application.properties"})
#Configuration
public class FraudJDBConfiguration {
private final Logger LOGGER = LogManager.getLogger(FraudJDBConfiguration.class);
private final Environment env;
#Autowired
public FraudJDBConfiguration(Environment env) {
this.env = env;
}
#Bean
public Connection getFraudConnection() {
// Step 1: Loading or
// registering Oracle JDBC driver class
String connectionClass = env.getProperty("fraud.db.driver-class-name");
try {
Class.forName(connectionClass);
} catch (ClassNotFoundException cnfex) {
LOGGER.error(cnfex.getMessage());
throw new RuntimeException("JDBC driver class'ı bulunamadı");
}
// Step 2: Opening database connection
try {
String environmentType = env.getProperty("environment");
if (environmentType == null) {
LOGGER.error("environment Tip Hatası (TEST - UAT - LIVE)");
throw new RuntimeException("environment Tip Hatası (TEST - UAT - LIVE)");
} else {
String connectionString = null;
String username = null;
String password = null;
switch (environmentType.toLowerCase()) {
case "dev":
connectionString = env.getProperty(/*someurl*/);
username = env.getProperty(/*someusername*/);
password = env.getProperty(/*somepassword*/);
break;
case "tst":
connectionString = env.getProperty(/*someurl*/);
username = env.getProperty(/*someusername*/);
password = env.getProperty(/*somepassword*/);
break;
case "liv":
connectionString = env.getProperty(/*someurl*/);
username = env.getProperty(/*someusername*/);
password = env.getProperty(/*somepassword*/);
break;
case "uat":
connectionString = env.getProperty(/*someurl*/);
username = env.getProperty(/*someusername*/);
password = env.getProperty(/*somepassword*/);
break;
}
// Step 2.A: Create and
// get connection using DriverManager class
if (connectionString == null) {
LOGGER.error("fraud şeması için connection string bulunamadı");
throw new RuntimeException("fraud şeması için connection string bulunamadı");
}
return DriverManager.getConnection(connectionString, username, password);
}
} catch (SQLException e) {
LOGGER.error(e.getMessage());
}
return null;
}
}
DAO
#Component
public interface FraudCommTransactionsDao {
Long count();
}
DAO IMPL
#Service
public class FraudCommTransactionsDaoImpl implements FraudCommTransactionsDao {
private final FraudJDBConfiguration fraudJDBConfiguration;
#Autowired
public FraudCommTransactionsDaoImpl(FraudJDBConfiguration fraudJDBConfiguration) {
this.fraudJDBConfiguration = fraudJDBConfiguration;
}
#Override
public Long count() {
try(Connection connection = fraudJDBConfiguration.getFraudConnection()) {
Statement stmt = connection.createStatement();
ResultSet rs = stmt.executeQuery(/*some query*/);
if (rs.next()) {
return rs.getLong("transaction_id");
} else {
return 0L;
}
} catch (SQLException ex) {
ex.printStackTrace();
}
return null;
}
}
No, establishing a new physical connection to a database server is costly. It involves multiple steps: user authorization, establishing session defaults, allocating memory on both client and server, etc. This overhead should not be added to every single request.
It's a common practice to create a connection pool to share the physical connections between application threads. This introduces a concept of logical connections e.g. a Connection object created with DriverManager.getConnection() is a physical connection while DataSource.getConnection() returns a logical connection which is a proxy.
There are multiple database connection pooling libraries for Java that you can use e.g. HikariCP. Don't write your own, this is not simple.
Get fast data and deliver to client could be possible using the simplest way of using application.properties file. You may use this to get database connection to your datasource.
Related
I'm trying to implement Postgres Row Level Security on my app that uses R2DBC.
I found this AWS post that implements this but uses a non-reactive approach.
I'm having problems converting this to a reactive approach since I can't find a class equivalent to the AbstractRoutingDataSource:
public class TenantAwareDataSource extends AbstractRoutingDataSource {
private static final Logger LOGGER = LoggerFactory.getLogger(TenantAwareDataSource.class);
#Override
protected Object determineCurrentLookupKey() {
Object key = null;
// Pull the currently authenticated tenant from the security context
// of the HTTP request and use it as the key in the map that points
// to the connection pool (data source) for each tenant.
Authentication authentication = SecurityContextHolder.getContext().getAuthentication();
try {
if (!(authentication instanceof AnonymousAuthenticationToken)) {
Tenant currentTenant = (Tenant) authentication.getPrincipal();
key = currentTenant.getId();
}
} catch (Exception e) {
LOGGER.error("Failed to get current tenant for data source lookup", e);
throw new RuntimeException(e);
}
return key;
}
#Override
public Connection getConnection() throws SQLException {
// Every time the app asks the data source for a connection
// set the PostgreSQL session variable to the current tenant
// to enforce data isolation.
Connection connection = super.getConnection();
try (Statement sql = connection.createStatement()) {
LOGGER.info("Setting PostgreSQL session variable app.current_tenant = '{}' on {}", determineCurrentLookupKey().toString(), this);
sql.execute("SET SESSION app.current_tenant = '" + determineCurrentLookupKey().toString() + "'");
} catch (Exception e) {
LOGGER.error("Failed to execute: SET SESSION app.current_tenant = '{}'", determineCurrentLookupKey().toString(), e);
}
return connection;
}
#Override
public String toString() {
return determineTargetDataSource().toString();
}
}
What would be the equivalent on R2DBC to AbstractRoutingDataSource?
Thanks
Full source code here.
So I'm using Java and MySQL, I created a database with different roles (user/manager/admin etc....) By default, I'm connected as a simple user, but how can I switch my user to the super admin after he authenticates?
Because right now, my connection with MySQL is made in a private constructor with private methods, so at first I thought about putting setters, but I can't access them given that I can't instantiate an object from my class with the private constructor.
ConnexionMySQL.java:
private ResourceBundle bundle =
ResourceBundle.getBundle("domaine.properties.config");
private String url = bundle.getString("sgbd.url");
private String driver = bundle.getString("sgbd.driver");
private String mysqlUser = bundle.getString("sgbd.login");
private String mysqlPassword = bundle.getString("sgbd.password");
private ConnexionMysql(){
try {
session = doSshTunnel(this.sshUser, this.sshPassword, this.sshHost, this.sshPort, this.url, this.sshLocalPort,this.sshRemotePort);
System.out.println("Opened SSH on " + sshHost);
Class.forName(driver);
connect = DriverManager.getConnection(url, mysqlUser ,mysqlPassword);
System.out.println("connect� Mysql");
} catch (Exception e) {
System.out.println(e.getMessage());
}
}
public static Connection getInstance(){
if(instance == null){
instance = new ConnexionMysql();
System.out.println("INSTANCIATION DE LA CONNEXION SQL ! ");
}
else{
System.out.println("CONNEXION SQL EXISTANTE ! ");
}
return connect;
}
public static void disconnect(){
try {
connect.close();
session.disconnect();
instance = null;
System.out.println("Deconnexion r�ussie");
} catch (Exception e) {
System.out.println(e.getMessage());
}
}
I have another config.properties file in which I specified the login and password.
But what if now I wanna change those values in a controller for example after a user authenticates ?
I would say that the easiest way to solve this problems will be to make as many clases as roles you want to create (of course if you want to leave your constructor private).
In my opinion it should look something like this:
ConnexionMySQLuser;
ConnexionMySQLmanager;
ConnexionMySQLadmin;
Of course if you have numerous of roles it can create problems in future.
I'm working with Java and mysql for database and I ran into a weird problem:
One of my clients have a very unstable connection and sometimes packet loss can be high. Ok that's not software's fault I know, but I went there to test and, when the program calls "DriverManager.getConnection()" and the network connection gets unstable, that line gets to lock the application (or the given thread) by several minutes. I have added some logics of course to use another datasource for caching data locally then saving to the network host when possible, but, I can't often let the program hang for longer than 10s (And this method doesn't seem to have any timeout specification).
So, I came out with a workaround like this:
public class CFGBanco implements Serializable {
public String driver = "com.mysql.jdbc.Driver";
public String host;
public String url = "";
public String proto = "jdbc:mysql://";
public String database;
public String user;
public String password;
}
private static java.sql.Connection Connect(HostConfig dataHost) throws java.sql.SQLException, ClassNotFoundException
{
dataHost.url = dataHost.proto+dataHost.host;
if(dataHost.database != null && !dataHost.database.equals("")) dataHost.url += "/"+dataHost.database;
java.lang.Class.forName(dataHost.driver);
ArrayList<Object> lh = new ArrayList<>();
lh.add(0, null);
Thread ConThread = new Thread(()-> {
try {
lh.add(0, java.sql.DriverManager.getConnection(
dataHost.url, dataHost.user, dataHost.password));
} catch(Exception x ) {
System.out.println(x.getMessage());
}
}, "ConnThread-"+SessId);
ConThread.start();
Thread TimeoutThread = new Thread(() -> {
int c = 0;
int delay = 100;
try {
try {
do {
try {
if(t.isAlive())
Thread.sleep(delay);
else
break;
} catch(Exception x) {}
} while((c+=delay) < 10000);
} catch(Exception x){}
} finally {
try {
t.stop();
} catch(Exception x){}
}
}, "ConTimeout-"+SessId);
TimeoutThread.start();
try {
ConThread.join();
} catch(Exception x) {}
if(lh.get(0) == null)
throw new SQLException();
return (Connection) lh.get(0);
}
I call getConnection from another thread, then make a secondary "timeout" thread to watch it and then Join the calling thread to the ConThread.
I have been getting results close to expected, indeed, but it got me wondering:
Is there a better way to do this? Does the creation of 2 threads eat up much on system resources, enough to make this approach unpractical?
You need connection pooling. Pool in the connection and reuse it rather than recreating everytime. One such library for DB connection pooling is DBCP by Apache
It will take care of when connection gets dropped off and so on. You could have validation Query and it would query DB say before borrowing connection from the pool and once it validates successfully, it will fire your actual query.
I am developing a simple CRUD application, using JDBC to establish connection and perform basic CRUD operations. In that process, created a DatabaseListener to create a connection object at startup and storing it in the context attribute for reuse.
Below is the code.
import javax.servlet.ServletContext;
import javax.servlet.ServletContextEvent;
import javax.servlet.ServletContextListener;
import org.apache.log4j.Logger;
public class DatabaseInitListner implements ServletContextListener {
private static final Logger LOG = Logger.getLogger(DatabaseInitListner.class);
private DBUtil databaseUtil = null;
#Override
public void contextDestroyed(ServletContextEvent event) {
databaseUtil.closeConnection();
}
#Override
public void contextInitialized(ServletContextEvent contextinitEvent) {
ServletContext servletContext = contextinitEvent.getServletContext();
String database = servletContext.getInitParameter("db_name");
String url = servletContext.getInitParameter("db_url")
+ database;
String username = servletContext.getInitParameter("db_user");
String password = servletContext.getInitParameter("db_password");
String driverName = servletContext.getInitParameter("db_driver");
databaseUtil = new DBUtil(url, username, password,
driverName);
servletContext.setAttribute("databaseSingleConnectionObject",
databaseUtil.getConnection());
}
}
public class DBUtil {
private Connection connection = null;
private static final Logger LOG = Logger.getLogger(DatabaseUtil.class);
public DatabaseUtil(String url, String username, String password,
String driver) {
try {
Class.forName(driver);
this.connection = DriverManager.getConnection(url, username,
password);
LOG.debug("Connection Established... ");
} catch (ClassNotFoundException | SQLException e) {
LOG.error("Could not create connection... ", e);
}
}
public Connection getConnection() {
return connection;
}
public void closeConnection() {
if (connection != null) {
try {
connection.close();
} catch (SQLException e) {
LOG.error("Unable to close connection... ", e);
}
}
}
}
I am accessing the connection in servlets like this
Connection jdbcConnection = (Connection) getServletContext().getAttribute("databaseSingleConnectionObject");
I am not sure if this is right approach. What are the effects of single database connection?
When you use a single database connection like this you make your application slow and brittle.
Slow: because the connection implementation is synchronized, each user has to wait until the connection is free. If one user's query takes a while to come back that directly increases the time any other concurrent users spend waiting. If there were multiple connections available from a pool then the time spent by one user would not impact other users nearly as greatly (unless a query's results take all the JVM's memory or a big query bogs down the database server).
Brittle: The connection is a network connection, they tend to go down. Without a provision to create new connections any kind of timeout, network hiccup, or period of database non-availability (such as taking the database offline for maintenance) is going to require an application restart. Using a connection pool will mean your application will be able to survive these episodes and recover without outside intervention.
This will not be threadsafe, and if it were, performance would be really poor.
Look into using a Connection Pool, like DBCP or C3PO
You should let your application server manage database connection. Add a JNDI datasource in its configuration file and make a lookup from your application to get a connection when needed (for instance when you instantiate a class that must access your database).
You may configure the datasource to manage a connection pool so that each user session will get its own.
Depending on the AS you use run a search with keywords 'JNDI' and 'datasource' and you will get further details about the AS configuration and how to implement it in your application.
I have a method in one of the classes in my code base that for the life of me, I cannot get into with my junit tests.
Basically this class is called when I request a database connection, if a stale connection is returned, a new connection is established
Here is the snippet of the mthod in my class (trimmed down for this purpose)
public class TCSOracleDataSourceWrapper extends OracleDataSource {
private static final int STALE_CONNECTION_EX_CODE = 17143;
private OracleConnectionCacheManager cacheManager;
private String cacheName;
/** Local log variable **/
private final Log logger = LogFactory.getLog(getClass());
/**
* Class constructor
* #throws SQLException
*/
public TCSOracleDataSourceWrapper() throws SQLException {
super();
}
private static final long serialVersionUID = 1L;
#Override
/**
* Get a connection but if the connection is stale then refresh all DB connections
*
*/
public final Connection getConnection() throws SQLException {
logger.debug("Retrieving a database connection from the pool");
Connection connection = null;
try{
connection = super.getConnection();
}
catch(SQLException e)
{
if(e.getErrorCode() == STALE_CONNECTION_EX_CODE)
{
logger.error("Stale Oracle connection found in the Connection Pool. Refreshing invalid DB connections.");
//refresh invalid connections
cacheManager.refreshCache(cacheName, OracleConnectionCacheManager.REFRESH_INVALID_CONNECTIONS);
//now try to get the connection again
connection = super.getConnection();
}
else
{
throw e;
}
}
return connection;
}}
Any idea how I can ensure my junit tests execute the if statement?
I am currently using EasyMock and Powermock but I cannot find a way to get into this if statment using these tools
All help is greatly appreciated
Thank you
Damien
You should refactor your class to become a proxy for another data source, rather than inherit from one. This way you can easily inject into it a mock data source instead of the real one.
import javax.sql.DataSource;
public class TCSOracleDataSourceWrapper implements DataSource {
...
private DataSource wrappedDataSource;
...
public TCSOracleDataSourceWrapper(DataSource ds) {
wrappedDataSource = ds;
}
...
public final Connection getConnection() throws SQLException {
...
Connection connection = null;
try{
connection = ds.getConnection();
}
catch(SQLException e)
{
...
}
return connection;
}
}
One idea springs to mind: use aggregation rather than inheritance. This problem and others like it would go away because you can then mock the aggregated object to have whatever behavior you want. I don't see another way of getting in there right off hand. In fact, the name TCSOracleDataSourceWrapper already indicates that it's wrapping a data source (aggregation), when it actually isn't.
One quick workaround is to factor out the super.getConnection() call to a new private / protected method. Once you make that change it would be easy to mock the getBaseConnection method using power mock. This is short term fix, like the other answers suggest it is better to use delegation instead of inheritance for the wrapper implementation.
Connection getBaseConnection() throws SQLException {
return super.getConnection();
}
public final Connection getConnection() throws SQLException {
logger.debug("Retrieving a database connection from the pool");
Connection connection = null;
try{
connection = getBaseConnection();
}
catch(SQLException e)
{
if(e.getErrorCode() == STALE_CONNECTION_EX_CODE)
{
logger.error("Stale Oracle connection found in the Connection Pool. Refreshing invalid DB connections.");
//refresh invalid connections
cacheManager.refreshCache(cacheName, OracleConnectionCacheManager.REFRESH_INVALID_CONNECTIONS);
//now try to get the connection again
connection = getBaseConnection();
}
else
{
throw e;
}
}
return connection;
}