Mongo DB pool connection error - java

I am using mongoDb with java. I am getting the following error while inserting data. Any help would be appreciated. Thanx
Jul 4, 2012 1:45:32 PM org.xsocket.connection.HandlerAdapter performOnData
WARNING: [6f829b013850ff7914137a5cceC291] closing connection. Error occured by performing onData of com.avaya.onex.hss.requesthandlers.ResponseHandler#18746603 java.lang.OutOfMemoryError: Java heap space
Jul 4, 2012 1:45:25 PM com.mongodb.DBPortPool gotError
WARNING: emptying DBPortPool to localhost:27017 b/c of error
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at org.bson.io.PoolOutputBuffer.pipe(PoolOutputBuffer.java:129)
at com.mongodb.OutMessage.pipe(OutMessage.java:111)
at com.mongodb.DBPort.go(DBPort.java:119)
at com.mongodb.DBPort.go(DBPort.java:89)
at com.mongodb.DBPort.say(DBPort.java:84)
at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:153)
at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:138)
at com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:261)
at com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:211)
at com.mongodb.DBCollection.insert(DBCollection.java:57)
at com.mongodb.DBCollection.insert(DBCollection.java:102)
at com.avaya.onex.hss.requesthandlers.DatabaseConnection.dbWrite(DatabaseConnection.java:50)
at com.avaya.onex.hss.requesthandlers.ResponseHandler.handleResponse(ResponseHandler.java:232)
at com.avaya.onex.hss.requesthandlers.ResponseHandler.onData(ResponseHandler.java:104)
at org.xsocket.connection.HandlerAdapter.performOnData(HandlerAdapter.java:242)
at org.xsocket.connection.HandlerAdapter.access$200(HandlerAdapter.java:42)
at org.xsocket.connection.HandlerAdapter$PerformOnDataTask.run(HandlerAdapter.java:210)
at org.xsocket.SerializedTaskQueue.performPendingTasks(SerializedTaskQueue.java:161)
at org.xsocket.SerializedTaskQueue.access$100(SerializedTaskQueue.java:40)
at org.xsocket.SerializedTaskQueue$MultithreadedTaskProcessor.run(SerializedTaskQueue.java:189)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
This is the class where I have dbWrite function which I call from other classes for inserting data in mongodb.
package com.avaya.onex.hss.requesthandlers;
import com.mongodb.BasicDBObject;
import com.mongodb.DB;
import com.mongodb.DBCollection;
import com.mongodb.Mongo;
import com.mongodb.MongoOptions;
import com.mongodb.WriteConcern;
public class DatabaseConnection {
private static DatabaseConnection con = null;
static DB db=null;
static Mongo m;
//private static int count=0;
private DatabaseConnection(){
}
public void dbConnect() {
try{
// connect to the local database server
MongoOptions options = new MongoOptions();
options.connectionsPerHost = 9000000;
options.maxWaitTime = 1000000;
options.socketKeepAlive = true;
options.threadsAllowedToBlockForConnectionMultiplier = 1000;
m = new Mongo("localhost", options);
//m.dropDatabase("ClientSimulator");
db = m.getDB( "ClientSimulator" );
}catch(Exception e){
e.printStackTrace();
}
}
public static synchronized DatabaseConnection getInstanceClass(){
if(con==null)
con=new DatabaseConnection();
return con;
}
public DB getDatabaseObject(){
return db;
}
public synchronized void dbWrite(DBCollection coll, BasicDBObject obj){
coll.insert(obj);
try {
Thread.currentThread().sleep(100);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public synchronized void dbRead(DBCollection coll){
}
public synchronized void dbUpdate(DBCollection coll) {
}
public synchronized void dbCollDrop(DBCollection coll){
coll.drop();
}
}

Try this.
options.connectionsPerHost = 100;
options.maxWaitTime = 2000;
options.socketKeepAlive = true;
options.threadsAllowedToBlockForConnectionMultiplier = 50;
connectionsPerHost is the connectionPool. If you put 9000000 it will try to create 90000000 threads and connection object to handle your request. I believe that it is by far more than necessary. You have to reduce that number, and you might also remove the sleep(100) in your insert function.
If java tries to create that many connections. Mongodb server might not be able to handle that much connection or you'd have to change the config to allow that much connections.
Keep in mind that connectionsPerHost is not the maximum quantity of threads. You could have 1000000 threads inserting but only 100 connections will be used and will be shared for all the threads once they are free. maxWaitTime of 2 second should be more than enough. If an insert takes more than 2 seconds, you might consider upgrading your server.

Driver can't remove dropped socket from connection from pool, you need use try catch to let the driver know the socket is dropped.

Related

Does MariaDB disconnect automatically or Should i have to disconnect Manually?

I got to use MariaDB for my University Project.
it's my first time doing it, so I dont't know well how to use and code JDBC Driver and mariaDB.
Now I'm implementing the code in many places while looking at examples.
As I see, All the examples seems to creating Statement and making connection by using "DriverManager.getConnection"
Now I have a question.
I'm going to create a DBmanager Class that can connect, create tables, execute queries, and execute the code that updates data on tables in a single line.
I thought all the examples would run alone in one method and came from different places, so I could only try a new connection and create a code that would not close. But I have a gut feeling that this will be a problem.
Is there any way I can leave a connection connected at a single connection to send a command, and disconnect it to DB.disconnect()? And I'd appreciate it if you could tell me whether what I'm thinking is right or wrong.
The code below is the code I've written so far.
I am sorry if you find my English difficult to read or understand. I am Using translator, So, my English could not be display as I intended.
import java.sql.*;
import java.util.Properties;
public class DBManager {
/*********INNITIAL DEFINES********/
final static private String HOST="sumewhere.azure.com";//Azure DB URL
final static private String USER="id#somewhere";//root ID
final static private String PW="*****";//Server Password
final static private String DRIVER="org.mariadb.jdbc.Driver";//DB Driver info
private String database="user";
/***************API***************/
void setDB(String databaseinfo){
database=databaseinfo;
}
private void checkDriver() throws Exception
{
try
{
Class.forName("org.mariadb.jdbc.Driver");
}
catch (ClassNotFoundException e)
{
throw new ClassNotFoundException("MariaDB JDBC driver NOT detected in library path.", e);
}
System.out.println("MariaDB JDBC driver detected in library path.");
}
public void checkOnline(String databaseinfo) throws Exception
{
setDB(databaseinfo);
this.checkDriver();
Connection connection = null;
try
{
String url = String.format("jdbc:mariadb://%s/%s", HOST, database);
// Set connection properties.
Properties properties = new Properties();
properties.setProperty("user", USER);
properties.setProperty("password", PW);
properties.setProperty("useSSL", "true");
properties.setProperty("verifyServerCertificate", "true");
properties.setProperty("requireSSL", "false");
// get connection
connection = DriverManager.getConnection(url, properties);
}
catch (SQLException e)
{
throw new SQLException("Failed to create connection to database.", e);
}
if (connection != null)
{
System.out.println("Successfully created connection to database.");
}
else {
System.out.println("Failed to create connection to database.");
}
System.out.println("Execution finished.");
}
void makeCcnnection() throws ClassNotFoundException
{
// Check DB driver Exists
try
{
Class.forName("org.mariadb.jdbc");
}
catch (ClassNotFoundException e)
{
throw new ClassNotFoundException("MariaDB JDBC driver NOT detected in library path.", e);
}
System.out.println("MariaDB JDBC driver detected in library path.");
Connection connection = null;
}
public void updateTable(){}
public static void main(String[] args) throws Exception {
DBManager DB = new DBManager();
DB.checkOnline("DB");
}
}
For a studying project it's okay to give a connection from your DB Manager to client code and close it there automatically using try-with-resources construction.
Maybe you will find it possible to check Connection Pool tools and apply it further in your project or use as example (like HikariCP, here is a good introduction).
Read about Java try with resources. I think that this link could be usefull for your problem.
JDBC with try with resources

Is MariaDB JDBC driver ignoring connection timeout?

In the application we are working on, the user can connect to external RDBMSs by entering an arbitrary JDBC connection URL into a text field. One of our customers reported that our application server freezes (indefinitly) at 0% CPU when he accidentally tried to connect to a Microsoft SQL Server with a MySQL JDBC URL.
The following Java snippet illustrates the situation:
public static void main(String[] args){
// note: the application running on localhost:1433 is ACTUALLY
// an MS SQL Server instance!
String jdbcUrl = "jdbc:mysql://localhost:1433/my-db";
// enable JDBC Driver Manager logging
DriverManager.setLogWriter(new PrintWriter(System.err));
// set a timeout of 5 seconds for connecting (which is blissfully ignored!)
DriverManager.setLoginTimeout(5);
// open the connection (which should fail, but freezes instead)
try (Connection c = DriverManager.getConnection(jdbcUrl)){
System.out.println("This should never be reached due to wrong JDBC URL.");
}catch(Exception e){
System.out.println("This is expected (but never printed).");
}
System.out.println("This is never printed either.");
}
To run the snippet:
have a SQL Server instance running on localhost:1433 (content does not matter)
have the MariaDB JDBC driver version 2.2.5. (latest) on your classpath.
Questions:
1) Could this be a bug in the MariaDB JDBC driver? A google search revealed nothing in this regard.
2) How should I work around this issue? I don't want my server to freeze when the user accidentally inserts an invalid JDBC URL.
I tried several other JDBC drivers (MySQL, DB2, Oracle...) and they all handle this issue gracefully, only the MariaDB JDBC driver freezes the JVM.
Here's what I did to resolve the issue. The trick is to add a socketTimeout to the connection. To fix the program in the question, it is enough to modify the JDBC URL to be:
jdbc:mysql://localhost:1433/my-db?socketTimeout=2000
This answer to a related question was the hint I needed.
Answer 1: Yes, it is a bug. They missed to use the login timeout in the implementation of mariadb jdbc driver.
Answer 2: I worked around by using a task that that wraps the getConnection method. This task is stopped after a defined login time if it hasn't finished. Here is my implementation.
import java.sql.Connection;
import java.sql.DriverManager;
import java.util.Properties;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.FutureTask;
import java.util.concurrent.TimeUnit;
import org.mariadb.jdbc.util.DefaultOptions;
public class ConnectionTest {
private static final String CONNECTION_STRING = "jdbc:mariadb://localhost:3306/test";
private static final String USER = "root";
private static final String PW = "";
private static final int LOGIN_TIMEOUT_SEC = 2;
public static void main(String[] args) throws Exception {
var test = new ConnectionTest();
Connection connection = test.getConnection();
if(connection != null && connection.isValid(LOGIN_TIMEOUT_SEC)) {
System.out.println("Connected!");
}
}
private Connection getConnection() throws Exception {
ConnEstablishSync sync = new ConnEstablishSync();
Properties conProps = new Properties();
conProps.setProperty(DefaultOptions.USER.getOptionName(), USER);
conProps.setProperty(DefaultOptions.PASSWORD.getOptionName(), PW);
FutureTask<Connection> task = new FutureTask<>(() -> {
Connection c = DriverManager.getConnection(CONNECTION_STRING, conProps);
if(sync.canceled && c != null) {
c.close();
c = null;
}
return c;
});
Connection connection = null;
ExecutorService executor = Executors.newSingleThreadExecutor();
try {
executor.submit(task);
connection = task.get(LOGIN_TIMEOUT_SEC, TimeUnit.SECONDS);
} finally {
sync.canceled = true;
task.cancel(true);
executor.shutdown();
}
return connection;
}
private static class ConnEstablishSync {
private volatile boolean canceled = false;
}
}

Timing out DriverManager.getConnection()?

I'm working with Java and mysql for database and I ran into a weird problem:
One of my clients have a very unstable connection and sometimes packet loss can be high. Ok that's not software's fault I know, but I went there to test and, when the program calls "DriverManager.getConnection()" and the network connection gets unstable, that line gets to lock the application (or the given thread) by several minutes. I have added some logics of course to use another datasource for caching data locally then saving to the network host when possible, but, I can't often let the program hang for longer than 10s (And this method doesn't seem to have any timeout specification).
So, I came out with a workaround like this:
public class CFGBanco implements Serializable {
public String driver = "com.mysql.jdbc.Driver";
public String host;
public String url = "";
public String proto = "jdbc:mysql://";
public String database;
public String user;
public String password;
}
private static java.sql.Connection Connect(HostConfig dataHost) throws java.sql.SQLException, ClassNotFoundException
{
dataHost.url = dataHost.proto+dataHost.host;
if(dataHost.database != null && !dataHost.database.equals("")) dataHost.url += "/"+dataHost.database;
java.lang.Class.forName(dataHost.driver);
ArrayList<Object> lh = new ArrayList<>();
lh.add(0, null);
Thread ConThread = new Thread(()-> {
try {
lh.add(0, java.sql.DriverManager.getConnection(
dataHost.url, dataHost.user, dataHost.password));
} catch(Exception x ) {
System.out.println(x.getMessage());
}
}, "ConnThread-"+SessId);
ConThread.start();
Thread TimeoutThread = new Thread(() -> {
int c = 0;
int delay = 100;
try {
try {
do {
try {
if(t.isAlive())
Thread.sleep(delay);
else
break;
} catch(Exception x) {}
} while((c+=delay) < 10000);
} catch(Exception x){}
} finally {
try {
t.stop();
} catch(Exception x){}
}
}, "ConTimeout-"+SessId);
TimeoutThread.start();
try {
ConThread.join();
} catch(Exception x) {}
if(lh.get(0) == null)
throw new SQLException();
return (Connection) lh.get(0);
}
I call getConnection from another thread, then make a secondary "timeout" thread to watch it and then Join the calling thread to the ConThread.
I have been getting results close to expected, indeed, but it got me wondering:
Is there a better way to do this? Does the creation of 2 threads eat up much on system resources, enough to make this approach unpractical?
You need connection pooling. Pool in the connection and reuse it rather than recreating everytime. One such library for DB connection pooling is DBCP by Apache
It will take care of when connection gets dropped off and so on. You could have validation Query and it would query DB say before borrowing connection from the pool and once it validates successfully, it will fire your actual query.

Single database connection for web application

I am developing a simple CRUD application, using JDBC to establish connection and perform basic CRUD operations. In that process, created a DatabaseListener to create a connection object at startup and storing it in the context attribute for reuse.
Below is the code.
import javax.servlet.ServletContext;
import javax.servlet.ServletContextEvent;
import javax.servlet.ServletContextListener;
import org.apache.log4j.Logger;
public class DatabaseInitListner implements ServletContextListener {
private static final Logger LOG = Logger.getLogger(DatabaseInitListner.class);
private DBUtil databaseUtil = null;
#Override
public void contextDestroyed(ServletContextEvent event) {
databaseUtil.closeConnection();
}
#Override
public void contextInitialized(ServletContextEvent contextinitEvent) {
ServletContext servletContext = contextinitEvent.getServletContext();
String database = servletContext.getInitParameter("db_name");
String url = servletContext.getInitParameter("db_url")
+ database;
String username = servletContext.getInitParameter("db_user");
String password = servletContext.getInitParameter("db_password");
String driverName = servletContext.getInitParameter("db_driver");
databaseUtil = new DBUtil(url, username, password,
driverName);
servletContext.setAttribute("databaseSingleConnectionObject",
databaseUtil.getConnection());
}
}
public class DBUtil {
private Connection connection = null;
private static final Logger LOG = Logger.getLogger(DatabaseUtil.class);
public DatabaseUtil(String url, String username, String password,
String driver) {
try {
Class.forName(driver);
this.connection = DriverManager.getConnection(url, username,
password);
LOG.debug("Connection Established... ");
} catch (ClassNotFoundException | SQLException e) {
LOG.error("Could not create connection... ", e);
}
}
public Connection getConnection() {
return connection;
}
public void closeConnection() {
if (connection != null) {
try {
connection.close();
} catch (SQLException e) {
LOG.error("Unable to close connection... ", e);
}
}
}
}
I am accessing the connection in servlets like this
Connection jdbcConnection = (Connection) getServletContext().getAttribute("databaseSingleConnectionObject");
I am not sure if this is right approach. What are the effects of single database connection?
When you use a single database connection like this you make your application slow and brittle.
Slow: because the connection implementation is synchronized, each user has to wait until the connection is free. If one user's query takes a while to come back that directly increases the time any other concurrent users spend waiting. If there were multiple connections available from a pool then the time spent by one user would not impact other users nearly as greatly (unless a query's results take all the JVM's memory or a big query bogs down the database server).
Brittle: The connection is a network connection, they tend to go down. Without a provision to create new connections any kind of timeout, network hiccup, or period of database non-availability (such as taking the database offline for maintenance) is going to require an application restart. Using a connection pool will mean your application will be able to survive these episodes and recover without outside intervention.
This will not be threadsafe, and if it were, performance would be really poor.
Look into using a Connection Pool, like DBCP or C3PO
You should let your application server manage database connection. Add a JNDI datasource in its configuration file and make a lookup from your application to get a connection when needed (for instance when you instantiate a class that must access your database).
You may configure the datasource to manage a connection pool so that each user session will get its own.
Depending on the AS you use run a search with keywords 'JNDI' and 'datasource' and you will get further details about the AS configuration and how to implement it in your application.

Java - Code Coverage

I have a method in one of the classes in my code base that for the life of me, I cannot get into with my junit tests.
Basically this class is called when I request a database connection, if a stale connection is returned, a new connection is established
Here is the snippet of the mthod in my class (trimmed down for this purpose)
public class TCSOracleDataSourceWrapper extends OracleDataSource {
private static final int STALE_CONNECTION_EX_CODE = 17143;
private OracleConnectionCacheManager cacheManager;
private String cacheName;
/** Local log variable **/
private final Log logger = LogFactory.getLog(getClass());
/**
* Class constructor
* #throws SQLException
*/
public TCSOracleDataSourceWrapper() throws SQLException {
super();
}
private static final long serialVersionUID = 1L;
#Override
/**
* Get a connection but if the connection is stale then refresh all DB connections
*
*/
public final Connection getConnection() throws SQLException {
logger.debug("Retrieving a database connection from the pool");
Connection connection = null;
try{
connection = super.getConnection();
}
catch(SQLException e)
{
if(e.getErrorCode() == STALE_CONNECTION_EX_CODE)
{
logger.error("Stale Oracle connection found in the Connection Pool. Refreshing invalid DB connections.");
//refresh invalid connections
cacheManager.refreshCache(cacheName, OracleConnectionCacheManager.REFRESH_INVALID_CONNECTIONS);
//now try to get the connection again
connection = super.getConnection();
}
else
{
throw e;
}
}
return connection;
}}
Any idea how I can ensure my junit tests execute the if statement?
I am currently using EasyMock and Powermock but I cannot find a way to get into this if statment using these tools
All help is greatly appreciated
Thank you
Damien
You should refactor your class to become a proxy for another data source, rather than inherit from one. This way you can easily inject into it a mock data source instead of the real one.
import javax.sql.DataSource;
public class TCSOracleDataSourceWrapper implements DataSource {
...
private DataSource wrappedDataSource;
...
public TCSOracleDataSourceWrapper(DataSource ds) {
wrappedDataSource = ds;
}
...
public final Connection getConnection() throws SQLException {
...
Connection connection = null;
try{
connection = ds.getConnection();
}
catch(SQLException e)
{
...
}
return connection;
}
}
One idea springs to mind: use aggregation rather than inheritance. This problem and others like it would go away because you can then mock the aggregated object to have whatever behavior you want. I don't see another way of getting in there right off hand. In fact, the name TCSOracleDataSourceWrapper already indicates that it's wrapping a data source (aggregation), when it actually isn't.
One quick workaround is to factor out the super.getConnection() call to a new private / protected method. Once you make that change it would be easy to mock the getBaseConnection method using power mock. This is short term fix, like the other answers suggest it is better to use delegation instead of inheritance for the wrapper implementation.
Connection getBaseConnection() throws SQLException {
return super.getConnection();
}
public final Connection getConnection() throws SQLException {
logger.debug("Retrieving a database connection from the pool");
Connection connection = null;
try{
connection = getBaseConnection();
}
catch(SQLException e)
{
if(e.getErrorCode() == STALE_CONNECTION_EX_CODE)
{
logger.error("Stale Oracle connection found in the Connection Pool. Refreshing invalid DB connections.");
//refresh invalid connections
cacheManager.refreshCache(cacheName, OracleConnectionCacheManager.REFRESH_INVALID_CONNECTIONS);
//now try to get the connection again
connection = getBaseConnection();
}
else
{
throw e;
}
}
return connection;
}

Categories