We Keep Coding

SortedSet not adding new constructed objects from certain sql queried results data

The answer to the following described issue may be as simple as that I am not using SortedSet correctly, but I wouldn't know if that is the case.
void SQLRankGuildsByPoints(final CallbackReturnIntegerStringSortedSet callback)
{
java.sql.Connection cn = null;
try {
cn = DataSource.getConnection();
if(cn != null)
{
PreparedStatement query = cn.prepareStatement("SELECT GuildName, TotalActivityPoints FROM Guilds");
ResultSet result = query.executeQuery();
SortedSet<Pair_IntString> GuildsRanking = new TreeSet(new ComparatorGuildsRanking());
while(result.next())
{
int resultInt = result.getInt("TotalActivityPoints");
String resultString = result.getString("GuildName");
GuildsRanking.add(new Pair_IntString(resultInt, resultString));
}
Bukkit.getScheduler().runTask(MainClassAccess, new Runnable() { //Callback to main thread
#Override
public void run() {
callback.onDone(GuildsRanking);
}
});
}
} catch (SQLException e) {
System.err.print(e);
} finally {
try {
cn.close();
} catch (SQLException e) {
System.err.print(e);
}
}
}
All 8 results from the Guilds table are present in "result" ResultSet.
GuildsRanking.add() isn't adding the new custom Pair_IntString object constructed with the query results, specifically for guilds "test" and "lilo" in Guilds table.
SQLRankGuildsByPoints method finishes it's execution, calling back the GuildsRanking SortedSet without 2 of the iterated results.
This behaviour is unintended and I can't find an explanation for it.
The comparator used for TreeSet:
public class ComparatorGuildsRanking implements Comparator<Pair_IntString> {
#Override
public int compare(Pair_IntString intStr1, Pair_IntString intStr2) {
return intStr2.integer.compareTo(intStr1.integer);
}
}
Custom Pair_IntString class:
public class Pair_IntString {
public Integer integer;
public String string;
Pair_IntString(Integer i, String s)
{
integer = i;
string = s;
}
}
No error messages with the skipped add iterations.

ThreadPoolTaskExecutor add tasks upon available active threads

I am trying to call a REST service asynchronously using ThreadPoolTaskExecutor. I have to call the same service asynchronously for the given input. I have a thread pool of a certain size and want to send a request to the service once the active thread count is less than the pool size. I am not sure how I could do that. Here is the sample code.
public class CallableWrkr implements Callable<Response>{
public CallableWrkr(Map<String,String> headers,
String hostName, Transfer transfer, String name) {
this.transfer = transfer;
this.hostName = hostName;
this.name = name;
}
#Override
public Response call() throws Exception {
System.out.println(name + 1);
Thread.sleep((new Random().nextInt(5000)) + 500);
CoreServicesImpl service = new CoreServicesImpl();
service.setHost(hostName);
Response resp = service.process( headers,
transfer);
System.out.println(name + 2);
return resp;
}
}
I want to do some thing like below when I have a list of transfer objects.
ApplicationContext context = new ClassPathXmlApplicationContext("applicationContext.xml");
ThreadPoolTaskExecutor taskExecutor = (ThreadPoolTaskExecutor) context.getBean("taskExecutor");
for (;;) {
int count = taskExecutor.getActiveCount();
System.out.println("Active Threads : " + count);
if(count < taskExecutor.getMaxPoolSize())
{
CallableWrkr callableTask = new CallableWrkr(headers transfers.get(i), new Integer(threadNumber).toString());
Future<Response> result = taskExecutor.submit(callableTask);
futureList.add(result);
}
else {
wait();
}
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
if (count == 0) {
taskExecutor.shutdown();
break;
}
}
Not sure if I could do that. Is there a queue I can add the list to?
Thanks.

How to give message when Threadpool Executor is completed?

I am trying to give a pop up alert message when my ThreadpoolExecutor is finished executing. It is searching email addresses from websites, once it is done I want a alert message as "Completed". Here is my Thread :-
public class Constant
{
public static final int NUM_OF_THREAD = 60;
public static final int TIME_OUT = 10000;
}
ThreadPoolExecutor poolMainExecutor = (ThreadPoolExecutor) Executors.newFixedThreadPool
(Constant.NUM_OF_THREAD);
Here is my Searching Operation class :-
class SearchingOperation implements Runnable {
URL urldata;
int i;
Set<String> emailAddresses;
int level;
SearchingOperation(URL urldata, int i, Set<String> emailAddresses, int level) {
this.urldata = urldata;
this.i = i;
this.emailAddresses = emailAddresses;
this.level = level;
if (level != 1)
model.setValueAt(urldata.getProtocol() + "://" + urldata.getHost() + "/contacts", i, 3);
}
public void run() {
BufferedReader bufferreader1 = null;
InputStreamReader emailReader = null;
System.out.println(this.i + ":" + poolMainExecutor.getActiveCount() + ":" + level + ";" + urldata.toString());
try {
if (level < 1) {
String httpPatternString = "https?:\\/\\/(www\\.)?[-a-zA-Z0-9#:%._\\+~#=]{2,256}\\.[a-z]{2,6}\\b([-a-zA-Z0-9#:%_\\+.~#?&//=]*)";
String httpString = "";
BufferedReader bufferreaderHTTP = null;
InputStreamReader httpReader = null;
try {
httpReader = new InputStreamReader(urldata.openStream());
bufferreaderHTTP = new BufferedReader(httpReader
);
StringBuilder rawhttp = new StringBuilder();
while ((httpString = bufferreaderHTTP.readLine()) != null) {
rawhttp.append(httpString);
}
if (rawhttp.toString().isEmpty()) {
return;
}
List<String> urls = getURL(rawhttp.toString());
for (String url : urls) {
String fullUrl = getMatchRegex(url, httpPatternString);
if (fullUrl.isEmpty()) {
if (!url.startsWith("/")) {
url = "/" + url;
}
String address = urldata.getProtocol() + "://" + urldata.getHost() + url;
fullUrl = getMatchRegex(address, httpPatternString);
}
if (!addressWorked.contains(fullUrl) && fullUrl.contains(urldata.getHost())) {
addressWorked.add(fullUrl);
sendToSearch(fullUrl);
}
}
} catch (Exception e) {
//System.out.println("652" + e.getMessage());
//e.printStackTrace();
return;
} finally {
try {
if (httpReader != null)
bufferreaderHTTP.close();
} catch (Exception e) {
// e.printStackTrace();
}
try {
if (httpReader != null)
httpReader.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
String someString = "";
emailReader = new InputStreamReader(urldata.openStream());
bufferreader1 = new BufferedReader(
emailReader);
StringBuilder emailRaw = new StringBuilder();
while ((someString = bufferreader1.readLine()) != null) {
if (someString.contains("#")) {
emailRaw.append(someString).append(";");
}
}
//Set<String> emailAddresses = new HashSet<String>();
String emailAddress;
//Pattern pattern = Pattern
//.compile("\\b[a-zA-Z0-9.-]+#[a-zA-Z0-9.-]+\\.[a-zA-Z0-9.-]+\\b");
Pattern
pattern = Pattern
.compile("\\b[a-zA-Z0-9.-]+#[a-zA-Z0-9.-]+\\.[a-zA-Z0-9.-]+\\b");
Matcher matchs = pattern.matcher(emailRaw);
while (matchs.find()) {
emailAddress = (emailRaw.substring(matchs.start(),
matchs.end()));
// //System.out.println(emailAddress);
if (!emailAddresses.contains(emailAddress)) {
emailAddresses.add(emailAddress);
// //System.out.println(emailAddress);
if (!foundItem.get(i)) {
table.setValueAt("Found", i, 4);
foundItem.set(i, true);
}
String emails = !emailAddresses.isEmpty() ? emailAddresses.toString() : "";
model.setValueAt(emails, i, 2);
model.setValueAt("", i, 3);
}
}
} catch (Exception e) {
//System.out.println("687" + e.getMessage());
} finally {
try {
if (bufferreader1 != null)
bufferreader1.close();
} catch (Exception e) {
e.printStackTrace();
}
try {
if (emailReader != null)
emailReader.close();
} catch (Exception e) {
e.printStackTrace();
}
Thread.currentThread().interrupt();
return;
}
}
After this the final snippet :-
private void sendToSearch(String address) throws Throwable {
SearchingOperation operation = new SearchingOperation(new URL(address), i,
emailAddresses, level + 1);
//operation.run();
try {
final Future handler = poolMainExecutor.submit(operation);
try {
handler.get(Constant.TIME_OUT, TimeUnit.MILLISECONDS);
} catch (TimeoutException e) {
e.printStackTrace();
handler.cancel(false);
}
} catch (Exception e) {
//System.out.println("Time out for:" + address);
} catch (Error error) {
//System.out.println("Time out for:" + address);
} finally {
}
}

Implement Callable<Void> instead of Runnable and wait for all the task to terminate by calling Future<Void>.get():
class SearchingOperation implements Callable<Void>
{
public Void call() throws Exception
{
//same code as in run()
}
}
//submit and wait until the task complete
Future<Void> future = poolMainExecutor.submit(new SearchingOperation()).get();

Use ThreadPoolExecutor.awaitTermination():
Blocks until all tasks have completed execution after a shutdown request, or the timeout occurs, or the current thread is interrupted, whichever happens first.
As in your code, you create your ThreadPoolExecutor first
ThreadPoolExecutor poolMainExecutor = (ThreadPoolExecutor) Executors.newFixedThreadPool(Constant.NUM_OF_THREAD);
Then, you need to add Tasks to it:
poolMainExecutor.execute(myTask);
poolMainExecutor.submit(myTask);
execute will return nothing, while submit will return a Future object. Tasks must implement Runnable or Callable. An object of SearchingOperation is a task for example. The thread pool will execute the tasks in parallel, but each task will be executed by one thread. That means to effectively use NUM_OF_THREAD Threads you need to add at least NUM_OF_THREAD Tasks.
(Optional) Once you got all jobs to work, shutdown your pool. This will prevent new tasks from being submitted. It won't affect running tasks.
poolMainExecutor.shutdown();
At the end, you need to wait for all Tasks to complete. The easiest way is by calling
poolMainExecutor.awaitTermination(Integer.MAX_VALUE, TimeUnit.DAYS);
You should adjust the amount of time you want to wait for the tasks to finish before throwing an exception.
Now that the work is done, notify the user. A simple way is to call one of the Dialog presets from JOptionPane, like:
JOptionPane.showMessageDialog(null, "message", "title", JOptionPane.INFORMATION_MESSAGE);
It will popup a little window with title "title", the message "message", an "information" icon and a button to close it.

This code can be used., it will check whether the execution is completed in every 2.5 seconds.
do {
System.out.println("In Progress");
try {
Thread.sleep(2500);
} catch (InterruptedException e) {
e.printStackTrace();
}
} while (poolMainExecutor.getActiveCount() != 0);
System.out.println("Completed");

JDBC and Multithreading

I am trying to run few queries using a multithreaded approach, however I think I am doing something wrong because my program takes about five minute to run a simple select statement like
SELECT * FROM TABLE WHERE ID = 123'
My implementation is below and I am using one connection object.
In my run method
public void run() {
runQuery(conn, query);
}
runQuery method
public void runQuery(Connection conn, String queryString){
Statement statement;
try {
statement = conn.createStatement();
ResultSet rs = statement.executeQuery(queryString);
while (rs.next()) {}
} catch (SQLException e) {
e.printStackTrace();
}
}
Finally in the main method, I start the threads using the snippet below.
MyThread bmthread = new MyThread(conn, query);
ArrayList<Thread> allThreads = new ArrayList<>();
double start = System.currentTimeMillis();
int numberOfThreads = 1;
for(int i=0; i<=numberOfThreads; i++){
Thread th = new Thread(bmthread);
th.setName("Thread "+i);
System.out.println("Starting Worker "+th.getName());
th.start();
allThreads.add(th);
}
for(Thread t : allThreads){
try {
t.join();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
double end = System.currentTimeMillis();
double total = end - start;
System.out.println("Time taken to run threads "+ total);
Update : I am now using separate connection for each thread.
ArrayList<Connection> sqlConn = new ArrayList<>();
for(int i =0; i<10; i++){
sqlConn.add(_ut.initiateConnection(windowsAuthURL, driver));
}
loop:
MyThread bmthread = new MyThread(sqlConn.get(i), query);

As rohivats and Asaph said, one connection must be used by one and only one thread, that said, consider using a database connection pool. Taking into account that c3p0, DBCP and similars are almost abandoned, I would use HikariCP which is really fast and reliable.
If you want something very simple you could implement a really simple connection pool using a thread safe collection (such as LinkedList), for example:
public class CutrePool{
String connString;
String user;
String pwd;
static final int INITIAL_CAPACITY = 50;
LinkedList<Connection> pool = new LinkedList<Connection>();
public String getConnString() {
return connString;
}
public String getPwd() {
return pwd;
}
public String getUser() {
return user;
}
public CutrePool(String connString, String user, String pwd) throws SQLException {
this.connString = connString;
for (int i = 0; i < INITIAL_CAPACITY; i++) {
pool.add(DriverManager.getConnection(connString, user, pwd));
}
this.user = user;
this.pwd = pwd;
}
public synchronized Connection getConnection() throws SQLException {
if (pool.isEmpty()) {
pool.add(DriverManager.getConnection(connString, user, pwd));
}
return pool.pop();
}
public synchronized void returnConnection(Connection connection) {
pool.push(connection);
}
}
As you can see getConnection and returnConnection methods are synchronized to be thread safe. Get a connection (conn = pool.getConnection();) and don't forget to return/free a connection after being used (pool.returnConnection(conn);)

Don't use the same connection object in all threads. Give each thread a dedicated database connection.

One Connection can only execute one query at a time. You need multiple connections available to execute database operations in parallel. Try using a DataSource with a connection pool, and make each thread request a connection from the pool.

How pass these new messages to another class

Now basically I have created three classes.
public void run() {
int seqId = 0;
while(true) {
List<KamMessage> list = null;
try {
list = fullPoll(seqId);
} catch (Exception e1) {
e1.printStackTrace();
}
if (!list.isEmpty()) {
seqId = list.get(0).getSequence();
incomingMessages.addAll(list);
System.out.println("waiting 3 seconds");
System.out.println("new incoming message");
}
try {
Thread.sleep(3000);
System.out.println("new incoming message");
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
public List<KamMessage> fullPoll(int lastSeq) throws Exception {
Statement st = dbConnection.createStatement();
ResultSet rs = st.executeQuery("select * from msg_new_to_bde where ACTION = 804 and SEQ >" +
lastSeq + "order by SEQ DESC");
List<KamMessage> pojoCol = new ArrayList<KamMessage>();
while (rs.next()) {
KamMessage filedClass = convertRecordsetToPojo(rs);
pojoCol.add(filedClass);
}
for (KamMessage pojoClass : pojoCol) {
System.out.print(" " + pojoClass.getSequence());
System.out.print(" " + pojoClass.getTableName());
System.out.print(" " + pojoClass.getAction());
System.out.print(" " + pojoClass.getKeyInfo1());
System.out.print(" " + pojoClass.getKeyInfo2());
System.out.println(" " + pojoClass.getEntryTime());
}
return pojoCol;
}
The following are the classes:
1.Poller- does the Polling and Passes the new data from db to controller
2.Controller- this class has a thread Pool, which simultaneously calls the Poller and has the new data to be requested from processor
3.Processor- this class has to look for new data, process it and return it to controller.
So now my problem is how to implement the third phase...
Here is my controller class:
public class RunnableController {
/** Here This Queue initializes the DB and have the collection of incoming message
*
*/
private static Collection<KpiMessage> incomingQueue = new ArrayList<KpiMessage>();
private Connection dbConncetion;
public ExecutorService threadExecutor;
private void initializeDb()
{
//catching exception must be adapted - generic type Exception prohibited
DBhandler conn = new DBhandler();
try {
dbConncetion = conn.initializeDB();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
private void initialiseThreads()
{
try {
threadExecutor = Executors.newFixedThreadPool(10);
PollingSynchronizer read = new PollingSynchronizer(incomingQueue, dbConncetion);
threadExecutor.submit(read);
}catch (Exception e){
e.printStackTrace();
}
}
#SuppressWarnings("unused")
private void shutDownThreads()
{
try {
threadExecutor.shutdown();
//DB handling should be moved to separate DB class
dbConncetion.close();
}catch (Exception e){
e.printStackTrace();
}
}
/** Here This Queue passes the messages and have the collection of outgoing message
*
*/
//private Collection<KpiMessage> outgingQueue = new ArrayList<KpiMessage>();
//have to implement something here for future
public static void main(String[] args) throws InterruptedException {
RunnableController controller = new RunnableController();
System.out.println(incomingQueue.size());
controller.initializeDb();
controller.initialiseThreads();
Thread.sleep(3000);
System.out.println("Polling");
}
}

I would recommend using a BlockingQueue for doing so, instead of a simple ArrayList. Just change the type of your incomingQueue variable. Then you can have another thread (or a thread pool) doing something like
//pseudocode
while (true) {
// it polls data from the incomingQueue that shares with the producers
KpiMessage message = this.incomingQueue.take()
//Then process the message and produces an output... you can put that output in a different queue as well for other part of the code to pick it up
}
A good example on BlockingQueues can be found here http://www.javamex.com/tutorials/blockingqueue_example.shtml