I am trying to run few queries using a multithreaded approach, however I think I am doing something wrong because my program takes about five minute to run a simple select statement like
SELECT * FROM TABLE WHERE ID = 123'
My implementation is below and I am using one connection object.
In my run method
public void run() {
runQuery(conn, query);
}
runQuery method
public void runQuery(Connection conn, String queryString){
Statement statement;
try {
statement = conn.createStatement();
ResultSet rs = statement.executeQuery(queryString);
while (rs.next()) {}
} catch (SQLException e) {
e.printStackTrace();
}
}
Finally in the main method, I start the threads using the snippet below.
MyThread bmthread = new MyThread(conn, query);
ArrayList<Thread> allThreads = new ArrayList<>();
double start = System.currentTimeMillis();
int numberOfThreads = 1;
for(int i=0; i<=numberOfThreads; i++){
Thread th = new Thread(bmthread);
th.setName("Thread "+i);
System.out.println("Starting Worker "+th.getName());
th.start();
allThreads.add(th);
}
for(Thread t : allThreads){
try {
t.join();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
double end = System.currentTimeMillis();
double total = end - start;
System.out.println("Time taken to run threads "+ total);
Update : I am now using separate connection for each thread.
ArrayList<Connection> sqlConn = new ArrayList<>();
for(int i =0; i<10; i++){
sqlConn.add(_ut.initiateConnection(windowsAuthURL, driver));
}
loop:
MyThread bmthread = new MyThread(sqlConn.get(i), query);
As rohivats and Asaph said, one connection must be used by one and only one thread, that said, consider using a database connection pool. Taking into account that c3p0, DBCP and similars are almost abandoned, I would use HikariCP which is really fast and reliable.
If you want something very simple you could implement a really simple connection pool using a thread safe collection (such as LinkedList), for example:
public class CutrePool{
String connString;
String user;
String pwd;
static final int INITIAL_CAPACITY = 50;
LinkedList<Connection> pool = new LinkedList<Connection>();
public String getConnString() {
return connString;
}
public String getPwd() {
return pwd;
}
public String getUser() {
return user;
}
public CutrePool(String connString, String user, String pwd) throws SQLException {
this.connString = connString;
for (int i = 0; i < INITIAL_CAPACITY; i++) {
pool.add(DriverManager.getConnection(connString, user, pwd));
}
this.user = user;
this.pwd = pwd;
}
public synchronized Connection getConnection() throws SQLException {
if (pool.isEmpty()) {
pool.add(DriverManager.getConnection(connString, user, pwd));
}
return pool.pop();
}
public synchronized void returnConnection(Connection connection) {
pool.push(connection);
}
}
As you can see getConnection and returnConnection methods are synchronized to be thread safe. Get a connection (conn = pool.getConnection();) and don't forget to return/free a connection after being used (pool.returnConnection(conn);)
Don't use the same connection object in all threads. Give each thread a dedicated database connection.
One Connection can only execute one query at a time. You need multiple connections available to execute database operations in parallel. Try using a DataSource with a connection pool, and make each thread request a connection from the pool.
Related
I want to use Executors to speed up the import of a list of available data to the database
public static void main(String[] args) {
methodOpenDatabaseConnection();
ExecutorService newFixedThreadPool = Executors.newScheduledThreadPool(100);
for (long i = 0; i < 1000; i++) {
newFixedThreadPool.execute(new Runnable() {
#Override
public void run( {
methodSaveDataToDatabaseMysql();
}
});
}
newFixedThreadPool.shutdown();
newFixedThreadPool.shutdown();
newFixedThreadPool.awaitTermination(60, TimeUnit.SECONDS);
if (newFixedThreadPool.isShutdown()) {
methodCloseConnectionDatabase();
}
}
public void methodSaveDataToDatabaseMysql() {
try {
preparedStatement = connection.prepareStatement("INSERT INTO `demo`"
+ " (`name`,`name2`,`name3`) VALUES (?,?,?)");
preparedStatement.setString(1, "name1");
preparedStatement.setString(2, "name2");
preparedStatement.setString(3, "name3");
preparedStatement.executeUpdate();
} catch (SQLException ex) {
Logger.getLogger(WebsiteResourcesImpl1.class.getName()).log(Level.SEVERE, null, ex);
}
}
It generated an error,If I add synchronized in the save method.It will succeed but the speed does not improve
java.sql.SQLException: No value specified for parameter 3
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:569)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:537)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:527)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:512)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:480)
at com.mysql.cj.jdbc.PreparedStatement.checkAllParametersSet(PreparedStatement.java:2159)
I think there is no problem with the save statement, because it can still save data successfully but not stable in ExecutorService
Based upon this code preparedStatement = connection.prepareStatement... then preparedStatement is not thread safe.
Consider to make it local to the methodSaveDataToDatabaseMysql method
I have an ExecutorService (thread pool size = 4) that is handling a number of Callables. Each of them opens a database connection (Hikari Connection pool) and closes it again.
If I now call shutdownNow() on the ExecutorService I do also wait for the termination of the currently running tasks. However, eventhough awaitTermination does not produce a timeout - thus all running tasks should have been terminated, and all database operations should have finished - I get an org.h2.jdbc.JdbcSQLException stating the following:
General error: "java.lang.IllegalStateException: Reading from nio:database.mv.db failed; file length -1 read length 256 at 711665 [1.4.196/1]"; SQL statement: SELECT * FROM PERSON WHERE id = ? [50000-196]
In addition, I do close the Hikari connection pool far later than shutting down the ExecutorService. Do you have any ideas what I could search for?
EDIT:
Here is the basic code structure - I think I have mentioned all necessary items. Note, that the exception mentioned does not get thrown every time - but most of the time:
class DatabaseManager {
private HikariDataSource datasource;
private static DatabaseManager instance;
public static DatabaseManager getInstance() {
if (instance == null) {
instance = new DatabaseManager();
}
return instance;
}
public Connection getConnection() { datasource.getConnection(); }
private DatabaseManager() {
// initialize with parameters for a H2 database
this.datasource = new HikariDataSource();
}
public void shutdown() {
if (this.datasource != null) {
this.datasource.shutdown();
}
this.datasource = null;
}
}
class SimpleCallable extends Callable<SomeType> {
String information;
public SomeCallable(String info) { this.information = info; }
public SomeType call() {
// omitted try-catch
Connection connection = DatabaseManager.getInstance().getConnection();
// doing some things with connection (reading and writing data), the Method invoked is static and synchronized
// within this method the exception mentioned above is thrown
SomeType someType = SomeTypeHelper.transferToDB(connection, information);
connection.close();
return someType;
}
}
class SimpleTask extends Runnable {
public void run() {
ExecutorService service = new Executors.newFixedThreadPool(4);
for (i=0; i<1000; i++) {
SimpleCallable callable = new SimpleCallable("random text");
FutureTask task = new FutureTask(callable);
service.submit(task);
}
try {
Thread.sleep(1000);
}
catch (InterruptedException e) {
// nothing to do
}
service.shutdownNow();
try {
if (!service.awaitTermination(60, TimeUnit.SECONDS)) {
System.out.println("timeout"); // but will never be printed
}
}
catch (InterruptedException e) {
// nothing to do
}
}
}
class App {
public static void main(String[] args) {
SimpleTask task = new SimpleTask();
new Thread(task).start();
DatabaseManager.getInstance().shutdown();
}
}
A full day of googling this problem has left me more confused than ever so I'm appealing to SO for help. I'm trying to use pooled connections to a mysql DB but I'm clearly misunderstanding something. Below are snippets of code from an application that scans a folder for new directories that represent "jobs"; when found, database objects are created for each folder found. I based the _insert() method on a pattern I found on SO. My understanding is that the connections are properly closed and returned to the connection pool. However, I noticed that, after adding 8 objects, the code would hang on getConnection(). I found somewhere that the default number of active connections was 8, so I added the debug line where I limit the number of active connections to 2. Sure enough, only two objects get added before the code hangs.
What's going on? What do I need to change to make these connections get freed and added back to the pool? I found one post that mentioned the PoolableConnection class but I'm confused by the documentation as well as by the fact that most other examples I've found don't seem to use it.
The Scanner class that creates Job objects in the database based on folders found in a particular directory on disk:
public class Scanner extends Thread {
public void run() {
syncJobs();
}
void syncJobs(List<String> folderNames) {
for (String folderName : folderNames) {
Job job = addJobToDB(folderName);
}
}
Job addJobToDB(String folderName ) {
Job job = new Job();
job.name = folderName;
job.save();
return job;
}
}
There's an abstract base class for all objects (each objects overrides _insert):
public abstract class DBObject {
private final int insert() {
return _insert();
}
public final void save() {
if (id == 0)
id = insert();
else
update();
}
}
And there's the actual Job object (with only the insert method shown):
public class Job extends DBObject {
public int _insert() {
String query = "insert into jobs (name) values (?)";
Connection conn = null;
PreparedStatement ps = null;
ResultSet rs = null;
int id = 0;
try {
conn = Database.getConnection();
ps = conn.prepareStatement(query, Statement.RETURN_GENERATED_KEYS);
ps.setInt(1, id);
ps.executeUpdate();
rs = ps.getGeneratedKeys();
rs.next();
id = rs.getInt(1);
} catch (Exception e) {
System.out.println(e.getMessage());
} finally {
DbUtils.closeQuietly(rs);
DbUtils.closeQuietly(ps);
DbUtils.closeQuietly(conn);
}
return id;
}
}
And, lastly, the Database object that provides connections:
import org.apache.commons.dbcp.BasicDataSource;
public final class Database {
private static final BasicDataSource dataSource = new BasicDataSource();
static {
dataSource.setUrl("jdbc:mysql://localhost:3306/dbName?useUnicode=true&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC");
dataSource.setUsername("user");
dataSource.setPassword("****");
// This line added for debugging: sure enough, only 2 objects are created.
dataSource.setMaxActive(2);
}
public static Connection getConnection() throws SQLException {
return dataSource.getConnection();
}
}
I am attempting to use connection pooling for Executor Service.
I am facing some problem when connection pool config is initialSize=3, maxToal=5, maxIdle=5.
I need to process 10 services at a time for every minute. But its picking only 5 services for every minute.
If i configure initialSize=3, maxToal=10, maxIdle=10 then its picking 10 services for every minute..
I am new to multithreading and connection. Below is my code snippet. Please provide suggestion.
public class TestScheduledExecutorService {
public static void main (String a[]) {
ScheduledExecutorService service = null;
try {
TestObject runnableBatch = new TestObject() {
public void run() {
testMethod ();
}
};
service = Executors.newSingleThreadScheduledExecutor();
service.scheduleAtFixedRate(runnableBatch, 0, 30, TimeUnit.SECONDS);
} catch (Exception e) {
e.printStackTrace();
}
}
}
public class TestObject implements Runnable{
public void testMethod (int inc) {
ExecutorService service = null;
try {
service = Executors.newFixedThreadPool(10);
for (int i = 0; i < 10; i++) {
service.submit(new TestService());
}
} catch (Exception e) {
e.printStackTrace();
}
}
#Override
public void run() {
}
}
public class TestService implements Callable{
Connection conn = null;
public void process(Connection conn) {
try {
if (conn != null) {
System.out.println("Thread & Connection pool conn : "+Thread.currentThread() + " :: " +conn);
// service process here
} else {
System.out.println("Connection pool conn is null : ");
}
} catch (Exception e) {
e.printStackTrace();
} finally {
}
}
#Override
public Object call() throws Exception {
ConnectionPoolTest cp = ConnectionPoolTest.getInstance();
BasicDataSource bds = cp.getBasicDataSource();
conn = bds.getConnection();
System.out.println(" call() "); **// it prints only 5 times for every minute eventhough total services are 10**
process(conn);
return null;
}
}
public class ConnectionPoolTest {
private static ConnectionPoolTest dataSource = new ConnectionPoolTest();
private static BasicDataSource basicDataSource = null;
private ConnectionPoolTest() {
}
public static ConnectionPoolTest getInstance() {
if (dataSource == null)
dataSource = new ConnectionPoolTest();
return dataSource;
}
public BasicDataSource getBasicDataSource() throws Exception {
try {
basicDataSource = new BasicDataSource();
basicDataSource.setInitialSize(3);
basicDataSource.setMaxTotal(10);
basicDataSource.setMaxIdle(10);
} catch (Exception e) {
throw e;
}
return basicDataSource;
}
}
For Executor Service
initialSize : Specified Number of Threads to spin , when New executor is created.
maxTotal : Number of Threads that can exist at max peak load.
maxIdle : Number of Thread that are kept active even if load goes below threshold.
As you mentioned, you want to pick up 10 number of tasks in parallel, we should have maxTotal set at 10. intialSize can be configured to a number that you think is optimal at the start , lets say 3 - 5. maxIdle is the number of threads you want to keep active , we generally assume how many threads are required if tasks are submitted. though there is no standard recomendation, vaues might be determined a number of various factors like .
Distribution of task submitted during the minute
Duration of Task
Urgency of executing those tasks in parallel.
As you mentioned you need 10 parallel tasks, then you will have to configure 10 as maxTotal, considering your task distribution and Duration causes overlap. If duration is pretty small , and distribution is even you can also survive with a lower number too.
This is the code that I am using to implement the queue.
Here queue poll is always returning null even when queue is not empty.
Runnable runnable = new Runnable() {
#Override
public void run() {
service.schedule(runnable, 500, TimeUnit.MILLISECONDS);
process();
}
public void process() {
try {
String tt = nextItem();
//System.out.println("SQ:"+tt);
} catch (Exception e) {//Catch exception if any
System.out.println("2Error: " + e.getMessage());
}
}
};
public String nextItem() {
Object poll;
try {
synchronized (queue) {
System.out.println("SQ:" + queue.poll());
//if (poll != null) {
// return poll.toString();
//} else {
return "";
//}
}
} catch (Exception e) {
e.printStackTrace();
return "";
}
}
public void run() {
try {
Class.forName("com.mysql.jdbc.Driver");
String url =
"jdbc:mysql://1xxx:3306/ayan";
Connection con =
DriverManager.getConnection(
url, "[user]", "[pass]");
Queue queue = new LinkedList();
service = Executors.newScheduledThreadPool(1000);
service.schedule(runnable, 0, TimeUnit.MILLISECONDS);
while (true) {
Statement statement = con.createStatement();
statement.setFetchSize(1);
ResultSet resultSet = statement.executeQuery("SELECT * from query_q");
while (resultSet.next()) {
// process results. each call to next() should fetch the next row
String id = resultSet.getString("id");
String query = resultSet.getString("query");
String msisdn = resultSet.getString("msisdn");
String pass = id + "|" + query + "|" + msisdn;
System.out.println("MQ:" + pass);
//String str = "foo";
//Queue<Character> charsQueue = new LinkedList<Character>();
boolean inserted = false;
for (char c : pass.toCharArray()) {
inserted = queue.offer(c);
}
if (inserted != false) {
// Statement stats = con.createStatement();
//stats.executeUpdate("delete from query_q where id=" + id);
}
}
Thread.sleep(10000);
}
//con.close();
}
LinkedList is the only non-thread safe Queue. Any other implementation would have been a better choice. Your offer is not synchronized. ;)
An ExecutorService has a built in queue. You can make use of that and not create a queue of your own at all. Just execute(Runnable) tasks as you need something to be done.
That's because you are not synchronizing the queue for your queue.offer(). You need to synchronize all access to the queue.
The simplest way to do this is to use a LinkedBlockingQueue which will take care of all the synchronization for you.
Note that you call offer() and poll() on different queue's - offer()'s queue is a local variable, whereas the poll()'s one is probably a field:
Queue queue = new LinkedList();
Also, syncrhonization is required, as suggested in other answers.