We Keep Coding

Java ScheduledThreadPoolExecutor: Can Runnables in a List be executed after shutdownNow()?

I can't execute Runnables(delayed tasks in a queue), that have been returned in an list of Runnables after invoking of shutdownNow() on ScheduledThreadPoolExecutor object.
I've tried some ways to do it: you can get list size, one of the Runnable objects itself, invoke isDone() query, but I haven't coped to run them.
CAN they be executed and HOW if possible?
See please code below. Thank you.
import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.FutureTask;
import java.util.concurrent.ScheduledThreadPoolExecutor;
import java.util.concurrent.ThreadPoolExecutor;
public class ExecuteExisitingDelayedTasksAfterShutdownPolicy1 {
private static int count = 0;
private static class Task implements Runnable {
private String name;
public Task(String name) {
this.name = name;
count++;
}
#Override
public void run() {
try {
Thread.sleep(500);
} catch (InterruptedException e) {
return;
}
System.out.printf("\n%s: " + getName(), Thread.currentThread().getName());
}
public String getName() {
return name;
}
}
public static void main(String[] args) throws InterruptedException, ExecutionException {
ScheduledThreadPoolExecutor stpe = new ScheduledThreadPoolExecutor(10, new ThreadPoolExecutor.DiscardPolicy());
stpe.setExecuteExistingDelayedTasksAfterShutdownPolicy(true);
List<Runnable> queue = null;
for (int i = 0; i < 100; i++) {
stpe.execute(new Task("Task " + count));
if (i == 50) {
Thread.sleep(1000);
queue = stpe.shutdownNow();
System.out.print("\nQueue SIZE: " + queue.size());
}
}
Thread.sleep(3000);
System.out.print("\n" + queue.get(0));
#SuppressWarnings("unchecked")
FutureTask<Task> ftask = (FutureTask<Task>) queue.get(0);
ExecutorService ses = Executors.newSingleThreadExecutor();
/**
* all of the next.. doesn't work: tasks returned in a queue are likely
to be
* unrunnable
*/
ftask.get().run();
System.out.println(ftask.get().name);
ses.execute(ftask);
queue.get(0).run();
}
}

Executors not completing all the task

Hi I am trying to run below code, and after executor is terminated I am expecting the count of remaining task to be 0, but for some reason it's more then 100 when it satisfy below condition.
while(executor.isTerminated()) {
System.out.println("Total Task Remaining : " + ExecutorServiceExample.task.size());
System.out.println("*** Executor Terminated ***");
break;
}
Code Snippet.
package test;
import java.util.HashSet;
import java.util.Set;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
public class ExecutorServiceExample {
public static volatile Set<String> task = new HashSet<String>();
public static void main(String args[]) {
ExecutorService executor = Executors.newFixedThreadPool(2000);
for (int i = 0; i < 10000; i++) {
String name = "task#" + i;
task.add(name);
Runnable runner = new TaskPrint(name);
executor.execute(runner);
}
try {
executor.shutdown();
executor.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
if (executor.isTerminated()) {
System.out.println("Total Task Remaining : " + ExecutorServiceExample.task.size());
System.out.println("*** Executor Terminated ***");
}
} catch (InterruptedException ignored) {
}
}
}
class TaskPrint implements Runnable {
private final String name;
public TaskPrint(String name) {
this.name = name;
}
public void run() {
ExecutorServiceExample.task.remove(name);
}
}
Something strange with the result based on the number of tasks.
Output for 100 tasks.
Total Task Remaining : 0
*** Executor Terminated ***
Output for 1000 tasks.
Total Task Remaining : 0
*** Executor Terminated ***
Output for 10000 tasks.
Total Task Remaining : -27
*** Executor Terminated ***
Output for 100000 tasks.
Total Task Remaining : 1205
*** Executor Terminated ***

HashSet is not thread safe. You can create a synchronizedSet with
public static volatile Set<String> task = Collections.synchronizedSet(new HashSet<String>());

How to give file as input and work in multiple threads?

I have this code to find out how to get the status code from a URL:
import java.io.IOException;
import java.net.HttpURLConnection;
import java.net.URL;
/**
* #author Crunchify.com
*
*/
class j {
public static void main(String args[]) throws Exception {
String[] hostList = { "http://example.com", "http://example2.com","http://example3.com" };
for (int i = 0; i < hostList.length; i++) {
String url = hostList[i];
String status = getStatus(url);
System.out.println(url + "\t\tStatus:" + status);
}
}
public static String getStatus(String url) throws IOException {
String result = "";
try {
URL siteURL = new URL(url);
HttpURLConnection connection = (HttpURLConnection) siteURL
.openConnection();
connection.setRequestMethod("HEAD");
connection.connect();
int code = connection.getResponseCode();
result = Integer.toString(code);
} catch (Exception e) {
result = "->Red<-";
}
return result;
}
}
I have checked it for small input it works fine. But I have millions of domains which I need to scan. I have a file containing it.
I want to know how I can give file as an input to this code.
I want the code to work in Multiple Threads. Say Thread count should be more than 20000, so that my output will be faster.
How I can write the out to another file?
Kindly help me. If possible I would like to know which the Bandwidth Savvy method to do the same job. I want to make the code faster anyways. how I can do these thing with the code I have?
Java Version:
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

This does what you want:
Input list file (c://lines.txt)
http://www.adam-bien.com/
http://stackoverflow.com/
http://www.dfgdfgdfgdfgdfgertwsgdfhdfhsru.de
http://www.google.de
The Thread:
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.concurrent.Callable;
public class StatusThread implements Callable<String> {
String url;
public StatusThread(String url) {
this.url = url;
}
#Override
public String call() throws Exception {
String result = "";
try {
URL siteURL = new URL(url);
HttpURLConnection connection = (HttpURLConnection) siteURL.openConnection();
connection.setRequestMethod("HEAD");
connection.connect();
int code = connection.getResponseCode();
result = Integer.toString(code);
} catch (Exception e) {
result = "->Red<-";
}
return url + "|" + result;
}
}
And the main program:
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.stream.Stream;
public class CallableExample {
public static void main(String[] args) throws IOException {
// Number of threads
int numberOfThreads = 10;
// Input file
String sourceFileName = "c://lines.txt"; // Replace by your own
String targetFileName = "c://output.txt"; // Replace by your own
// Read input file into List
ArrayList<String> urls = new ArrayList<>();
try (Stream<String> stream = Files.lines(Paths.get(sourceFileName ))) {
stream.forEach((string) -> {
urls.add(string);
});
} catch (IOException e) {
e.printStackTrace();
}
// Create thread pool
ThreadPoolExecutor executor = (ThreadPoolExecutor) Executors.newFixedThreadPool(numberOfThreads);
List<Future<String>> resultList = new ArrayList<>();
// Launch threads
for(String url : urls) {
StatusThread statusGetter = new StatusThread(url);
Future<String> result = executor.submit(statusGetter);
resultList.add(result);
}
// Use results
FileWriter writer;
writer = new FileWriter(targetFileName);
for (Future<String> future : resultList) {
try {
String oneResult = future.get().split("\\|")[0] + " -> " + future.get().split("\\|")[1];
// Print the results to the console
System.out.println(oneResult);
// Write the result to a file
writer.write(oneResult + System.lineSeparator());
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
}
}
writer.close();
// Shut down the executor service
executor.shutdown();
}
}
Don't forget to:
Create your input file and point to it (c://lines.txt)
Change the number of threads to get the best result

You will have issues sharing a file across threads. Much better to read the file and then spawn a thread to process each record in the file.
Creating a thread is none trivial resource wise so a thread pool would be useful so threads can be reused.
Do you want all threads to write to a single file?
I would do that using a shared list between the threads and the writer. others may have a better idea.
How to do all this depends on Java version.

You can use the ExecutorService and set the thread number to use.
The ExecutorService instance will handle for your the threads management.
You just need to provide it the tasks to execute and invoking all tasks executions.
When all the task are performed you can get the result.
In the call() method of The Callable implementation we return a String with a separator to indicate the url and the response code of the request.
For example : http://example3.com||301, http://example.com||200, etc...
I have not written the code to read a file and store in another file the result of the tasks. You should not have great difficulty to implement it.
Here is the main class :
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
public class Main {
public static void main(String[] args) throws InterruptedException {
String[] hostList = { "http://example.com", "http://example2.com", "http://example3.com" };
int nbThreadToUse = Runtime.getRuntime().availableProcessors() - 1;
ExecutorService executorService = Executors.newFixedThreadPool(nbThreadToUse);
Set<Callable<String>> callables = new HashSet<Callable<String>>();
for (String host : hostList) {
callables.add(new UrlCall(host));
}
List<Future<String>> futures = executorService.invokeAll(callables);
for (Future<String> future : futures) {
try {
String result = future.get();
String[] keyValueToken = result.split("\\|\\|");
String url = keyValueToken[0];
String response = keyValueToken[1];
System.out.println("url=" + url + ", response=" + response);
} catch (ExecutionException e) {
e.printStackTrace();
}
}
executorService.shutdown();
}
}
Here is UrlCall, the Callable implementation to perform a call to the url.
UrlCall takes in its constructor the url to test.
import java.io.IOException;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.concurrent.Callable;
public class UrlCall implements Callable<String> {
private String url;
public UrlCall(String url) {
this.url = url;
}
#Override
public String call() throws Exception {
return getStatus(url);
}
private String getStatus(String url) throws IOException {
try {
URL siteURL = new URL(url);
HttpURLConnection connection = (HttpURLConnection) siteURL.openConnection();
connection.setRequestMethod("HEAD");
connection.connect();
int code = connection.getResponseCode();
return url + "||" + code;
} catch (Exception e) {
//FIXME to log of course
return url + "||exception";
}
}
}

I agree with Thread pool approach exposed here.
Multi-threading consists in exploiting the time the others threads spend to wait (I guess int his case: the distant site response). It does not multiply processing power. Then about 10 threads seem reasonable (more depending on hardware).
An important point that seem to have been neglected in answer I read is that OP talk about millions of domains. Then I would discourage loading whole file in memory in a list iterated over afterwards. I would rather merge all in a single loop (file reading), instead of 3 (read, ping, write).
stream.forEach((url) -> {
StatusThread statusGetter = new StatusThread(url, outputWriter);
Future<String> result = executor.submit(statusGetter);
});
outputWriter would be a type with a synchronized method to write into an output stream.

IB Java API: Extracting ticker data (real time bars) for multiple contracts

I'm doing some self-learning and experimentation with algorithmic trading and the IB API. I decided to use Java but I'm open to switching to C++. I went through an online tutorial that walks you through the code shown below but was wondering about extending it past just one stock. I want to go through all SP500 stocks and check ticker data to make decisions based on that.
The code below will create a contract for and get data for Microsoft but I'd like to get data for all 500 stocks. All of the other methods defined in the EWrapper interface were left out of the post for more ease of readability.
I'm thinking that I need to store the ticker symbols in a file, parse this, and add each contract one by one to a vector. However, I'm not sure about how to monitor the data after that. It would be nice if I could just sequentially loop through each ticker and make a request for data but I believe the stream is processed on an asynchronous thread (correct me if wrong.)
So how do I go through all 500 stocks and check their ticker data?
Code snippets and explanations would be appreciated. Thanks!
// Import Java utilities and Interactive Brokers API
import java.util.Vector;
import com.ib.client.Contract;
import com.ib.client.ContractDetails;
import com.ib.client.EClientSocket;
import com.ib.client.EWrapper;
import com.ib.client.Execution;
import com.ib.client.Order;
import com.ib.client.OrderState;
import com.ib.client.TagValue;
import com.ib.client.CommissionReport;
import com.ib.client.UnderComp;
// RealTimeBars Class is an implementation of the
// IB API EWrapper class
public class RealTimeBars implements EWrapper
{
// Keep track of the next ID
private int nextOrderID = 0;
// The IB API Client Socket object
private EClientSocket client = null;
public RealTimeBars ()
{
// Create a new EClientSocket object
client = new EClientSocket (this);
// Connect to the TWS or IB Gateway application
// Leave null for localhost
// Port Number (should match TWS/IB Gateway configuration
client.eConnect (null, 7496, 0);
// Pause here for connection to complete
try
{
// Thread.sleep (1000);
while (! (client.isConnected()));
} catch (Exception e) {
e.printStackTrace ();
};
// Create a new contract
Contract contract = new Contract ();
contract.m_symbol = "MSFT";
contract.m_exchange = "SMART";
contract.m_secType = "STK";
contract.m_primaryExch = "NASDAQ";
contract.m_currency = "USD";
// Create a TagValue list
Vector<TagValue> realTimeBarsOptions = new Vector<TagValue>();
// Make a call to start off data retrieval
client.reqRealTimeBars(0, contract,
5, // Bar Size 5 seconds
"TRADES", // whatToShow
false, // useRTH
realTimeBarsOptions);
// At this point our call is done and any market data events
// will be returned via the realtimeBar method
}
public static void main (String args[])
{
try
{
// Create an instance
// At this time a connection will be made
// and the request for market data will happen
RealTimeBars myData = new RealTimeBars();
}
catch (Exception e)
{
e.printStackTrace ();
}
}
}

I don't know how this will work for all 500, but you can try. The data is from https://raw.githubusercontent.com/datasets/s-and-p-500-companies/master/data/constituents.csv
SP
package sp;
import com.ib.client.Contract;
import com.ib.client.EClientSocket;
import com.ib.client.EWrapper;
import java.util.Arrays;
import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;
public class SP {
//just a sample, like this so you can just use Files.lines instead.
private static List<String> lines = Arrays.asList(new String[]{
"Symbol,Name,Sector",
"MMM,3M Company,Industrials",
"ABT,Abbott Laboratories,Health Care",
"ABBV,AbbVie,Health Care",
"ACN,Accenture plc,Information Technology",
"ATVI,Activision Blizzard,Information Technology",
"AYI,Acuity Brands Inc,Industrials",
"ADBE,Adobe Systems Inc,Information Technology",
"AAP,Advance Auto Parts,Consumer Discretionary",
"AES,AES Corp,Utilities",
"AET,Aetna Inc,Health Care",
"AMG,Affiliated Managers Group Inc,Financials",
"AFL,AFLAC Inc,Financials",
"A,Agilent Technologies Inc,Health Care",
"APD,Air Products & Chemicals Inc,Materials",
"AKAM,Akamai Technologies Inc,Information Technology",
});
public static void main(String[] args) throws InterruptedException{
EWrapper wrapper = new Wrapper();
EClientSocket socket = new EClientSocket(wrapper);
socket.eConnect("", 4001, 123);
//supposedly gives frozen last recorded value, not working!
socket.reqMarketDataType(2);
AtomicInteger tickerId = new AtomicInteger(0);
lines.stream().skip(1).forEach(line -> {
//new cont for every request
Contract cont = new Contract();
cont.m_currency = "usd";
cont.m_exchange = "smart";
cont.m_secType = "stk";
cont.m_symbol = line.split(",")[0];
Data data = new Data(cont, socket);
});
//need you own logic for when to end program
//Thread.sleep(5000);//this thread, Socket starts a reader thread
//socket.eDisconnect();
}
}
Wrapper
package sp;
import com.ib.client.CommissionReport;
import com.ib.client.Contract;
import com.ib.client.ContractDetails;
import com.ib.client.EWrapper;
import com.ib.client.Execution;
import com.ib.client.Order;
import com.ib.client.OrderState;
import com.ib.client.TickType;
import com.ib.client.UnderComp;
import java.util.HashMap;
import java.util.Map;
public class Wrapper implements EWrapper{
public Map<Integer, Data> dataMap = new HashMap<>();
public Map<Integer, Strat> orderMap = new HashMap<>();
//reqMktData snapshots are received here
#Override
public void tickPrice(int tickerId, int field, double price, int canAutoExecute) {
if (field == TickType.LAST) {
//if you just want the last price
dataMap.get(tickerId).dataRecd(price);
}
}
#Override
public void execDetails(int reqId, Contract contract, Execution execution) {
orderMap.get(execution.m_orderId).exec(execution);
}
//snip
}
Data
package sp;
import com.ib.client.Contract;
import com.ib.client.EClientSocket;
import java.util.ArrayList;
import java.util.List;
import java.util.Timer;
import java.util.TimerTask;
public class Data {
final Contract cont;
private final EClientSocket socket;
private final Strat strat;
private static int nextId = 1; //auto increment for each request
private final int myId;
List<Double> prices = new ArrayList<>();
double lastPrice = -1;
public Data(Contract cont, EClientSocket socket) {
this.cont = cont;
this.socket = socket;
strat = new Strat(this, socket);
myId = nextId++;
((Wrapper) socket.wrapper()).dataMap.put(myId, this);
reqData();
// //call every 10 min
// Timer timer = new Timer();
// timer.schedule(new TimerTask() {
// #Override
// public void run() {
// reqData();
// }
// }, 10 * 60 * 1000);
}
private void reqData(){
socket.reqMktData(myId, cont, "", false /* true */, null);
}
public void dataRecd(double last){
lastPrice = last;
prices.add(last);
strat.check();
}
}
Strat
package sp;
import com.ib.client.EClientSocket;
import com.ib.client.Execution;
public class Strat {
public static final int NULL=0, LOOK=1<<0, LONG=1<<1, SHORT=1<<2, WAIT_FILL=1<<3, WAIT_CANCEL=1<<4;
public int sysState = NULL;
private final Data data;
private final EClientSocket socket;
private static int nextOrderId = 1;
Strat(Data data, EClientSocket socket) {
this.data = data;
this.socket = socket;
sysState = LOOK;
}
void check() {
System.out.println("should I buy? "+ data.cont.m_symbol + " # " + data.lastPrice);
/*if (false && sysState & LOOK == LOOK) {
((Wrapper) socket.wrapper()).orderMap.put(nextOrderId, this);
socket.placeOrder(nextOrderId++, data.cont, new Order());
sysState = WAIT_FILL;
nextOrderId++;
}*/
}
public void exec(Execution exec){
//will be called by wrapper after an exec.
//sysState = LONG; //or whatever
}
}

Why is this code executing sequentially?

Below code :
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
public class ThreadTest {
private static int counter = 0;
private static ExecutorService executorService = Executors.newCachedThreadPool();
private static List<Integer> intValues = new ArrayList<Integer>();
public static void main(String args[]){
for(int counter = 0; counter < 10; ++counter){
intValues.add(testCallback());
}
for(int i : intValues){
System.out.println(i);
}
System.exit(0);
}
public static Integer testCallback() {
Future<Integer> result = executorService.submit(new Callable<Integer>() {
public Integer call() throws Exception {
counter += 1;
Thread.sleep(500);
return counter;
}
});
try {
return result.get();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
return null;
}
}
Outputs :
1
2
3
4
5
6
7
8
9
10
This program takes approx 5 seconds to run. I am trying to execute multiple invocations of testCallback method in a seperate thread so I would expect this method to run in 10 threads concurrently where each thread uses approx 500 miliseconds of time. So over all I expet the program to run in < 1 second.
Why is counter not being invoked in seperate threads concurrently ?

result.get();
This is a blocking call that waits for the task to complete.
Therefore, you're waiting for each task to finish before starting the next one.