We Keep Coding

Synchronized thread get/create via factory

I want to have multiple threads, each of them linked to a "group", which will do operations. Each of these threads, will in a loop look into a queue of Operations to do, and it is not empty, they will take next Operation and process it.
It is as simple as this:
public class GroupThreadManager {
private static ConcurrentHashMap<Long, GroupThread> threads = new ConcurrentHashMap<>();
private synchronized static void newOperation(Operation op) throws IOException {
final Long idGroup = op.getIdGroup();
if (!threads.containsKey(idGroup )) {
threads.put(idGroup , new GroupThread());
}
threads.get(idGroup ).start(op);
}
private synchronized static void interrupThread(Long id) {
if(threads.remove(id) == null) {
log.info("THIS SHOULDNT HAVE HAPPENED!!!!!");
}
}
}
public class GroupThread implements Runnable {
private Thread worker;
private ConcurrentLinkedQueue<Operation> operations = new ConcurrentLinkedQueue<>();
private Long idGroup;
public void start(Operation op) throws IOException {
addOperation(op);
if (worker == null) {
idGroup = op.getIdGroup();
worker = new Thread(this);
worker.start();
}
}
public synchronized void addOperation(Operation op) {
operations .add(user);
}
private synchronized int size() {
return operations.size();
}
public void run() {
while (size() > 0) {
operations.poll()
.compute() // do something here
}
GroupThreadManager.interrupThread(idUser);
}
}
If the run method was implemented with while (true) I would have no problem. The issue comes when I want that thread to process all the operations it has, and whenever it becomes out of operations, I want to make that Thread end. I have been trying to make proper synchronization to create/get the Thread from the GroupThreadManager, but I always come into either deadlocks, either Thread ending with new Operations to proceed due to missing synchronization.
The idea is that from another part of the program I can just call GroupThreadManager.newOperation(new Operation()) and this manager automatically gives me the correct thread for that groupId (contained in Operation), creating it, giving me the existing one, or stopping & deleting it when it detects there are no new operations for it

Externally lock Cache for updates

I have 2 caches which are updated individually form various parts of my code. Every some time (e.g. 12 hours) I want to make sure they are synced. An external class is responsible for starting and executing this task. How can I make sure other classes are not working with the caches when this happens?
My thinking is using some ReadWriteLock in each cache and exposing lock/unlock methods.
Class Cache {
private final ReadWriteLock lock = new ReentrantReadWriteLock();
public void put(String id, Object object) {
lock.readLock().lock();
try {
// put in cache
} finally {
lock.readLock().unlock();
}
}
public Object get(String id) {
lock.readLock().lock();
try {
// get from cache
} finally {
lock.readLock().unlock();
}
}
public void lock() {
lock.writeLock().lock();
}
public void unlock() {
lock.writeLock().unlock();
}
}
And this is the code for the sync class
Class Synchronizer {
Cache cache1 = new Cache();
Cache cache2 = new Cache();
public void syncCaches() {
cache1.lock();
cache2.lock();
try {
// do sync
} finally {
cache1.unlock();
cache2.unlock();
}
}
}
This works, but I think it's a misuse of the Read/Write lock architecture and I couldn't find libraries or something else what might work.
Any ideas are welcome!

Java concurrency exercise. Asynchronous download

I'm doing an exercise about Java concurrency using wait, notify to study for an exam.
The exam will be written, so the code does have to be perfect since we can't try to compile and check errors.
This is the text of the exercise:
General idea:
when the downloader is instanced the queue and the hashmap are created and passed to all the threads. (shared data)
the download method add the url to the queue and call notifyAll to wake up the Downloader Threads.
the getData method waits until there are data in the hashmap for the provided url. When data are available it returns to the caller.
the downloader thread runs an infinity loop. It waits until an url is present in the queue. When it receives an url it downloads it and puts the bytes in the hashmap calling notifyAll to wake up a possible user waiting in getData method.
This is the code that I produced:
public class Downloader{
private Queue downloadQueue;
private HashMap urlData;
private final static THREADS_NUMBER;
public Downloader(){
this.downloadQueue = new Queue();
this.urlData = new HashMap();
for(int i = 0; i < THREADS_NUMBER; i++){
new DownTh(this.downloadQueue, this.urlData).start();
}
}
void syncronized download(String URL){
downloadQueue.add(url);
notifyAll();
return;
}
byte[] syncronized getData(String URL){
while(urlData.get(URL) == null ){
wait()
}
return urlData.get(URL);
}
}
public class DownTh extend Thread{
private Queue downloadQueue;
private HashMap urlData;
public DownTh(Queue downloadQueue, HashMap urlData){
this.downloadQueue = downloadQueue
this.urlData = urlData;
}
public void run(){
while(true){
syncronized{
while(queue.isEmpty()){
wait()
}
String url = queue.remove();
urlData.add(url, Util.download(url))
notifyAll()
}
}
}
}
Can you help me and tell me if the logic is right?

Let's assume for a second that all those great classes in Java that handle synchronization do not exist, because this is a synthetic task, and all you got to handle is sychronized, wait and notify.
The first question to answer in simple words is: "Who is going to wait on what?"
The download thread is going to wait for an URL to download.
The caller is going to wait for the result of that download thread.
What does this mean in detail? We need at least one synchronization element between the caller and the download thread (your urlData), also there should be one data object handling the download data itself for convenience, and to check whether or not the download has yet been completed.
So the detailed steps that will happen are:
Caller requests new download.create: DownloadResultwrite: urlData(url -> DownloadResult)wake up 1 thread on urlData.
Thread X must find data to download and process it or/then fall asleep again.read: urlData (find first unprocessed DownloadResult, otherwise wait on urlData)write: DownloadResult (acquire it)write: DownloadResult (download result)notify: anyone waiting on DownloadResultrepeat
Caller must be able to asynchronously check/wait for download result.read: urlDataread: DownloadResult (wait on DownloadResult if required)
As there are reads and writes from different threads on those objects, synchronization is required when accessing the objects urlData or DownloadResult.
Also there will be a wait/notify association:
caller -> urlData -> DownTh
DownTh -> DownloadResult -> caller
After careful analysis the following code would fulfill the requirements:
public class DownloadResult {
protected final URL url; // this is for convenience
protected boolean inProgress;
protected byte[] result;
public DownloadResult(final URL url) {
this.url = url;
this.inProgress = false;
}
/* Try to lock this against tother threads if not already acquired. */
public synchronized boolean acquire() {
if (this.inProgress == false) {
this.inProgress = true;
return true;
} else {
return false;
}
}
public void download() {
final byte[] downloadedBytes = Util.download(this.url); // note how this is done outside the synchronized block to avoid unnecessarily long blockings
synchronized (this) {
this.result = downloadedBytes;
this.notifyAll(); // wake-up ALL callers
}
}
public synchronized byte[] getResult() throws InterruptedException {
while (this.result == null) {
this.wait();
}
return this.result;
}
}
protected class DownTh extends Thread {
protected final Map<URL, DownloadResult> urlData;
public DownTh(final Map<URL, DownloadResult> urlData) {
this.urlData = urlData;
this.setDaemon(true); // this allows the JVM to shut down despite DownTh threads still running
}
protected DownloadResult getTask() {
for (final DownloadResult downloadResult : urlData.values()) {
if (downloadResult.acquire()) {
return downloadResult;
}
}
return null;
}
#Override
public void run() {
DownloadResult downloadResult;
try {
while (true) {
synchronized (urlData) {
while ((downloadResult = this.getTask()) == null) {
urlData.wait();
}
}
downloadResult.download();
}
} catch (InterruptedException ex) {
// can be ignored
} catch (Error e) {
// log here
}
}
}
public class Downloader {
protected final Map<URL, DownloadResult> urlData = new HashMap<>();
// insert constructor that creates the threads here
public DownloadResult download(final URL url) {
final DownloadResult result = new DownloadResult(url);
synchronized (urlData) {
urlData.putIfAbsent(url, result);
urlData.notify(); // only one thread needs to wake up
}
return result;
}
public byte[] getData(final URL url) throws InterruptedException {
DownloadResult result;
synchronized (urlData) {
result = urlData.get(url);
}
if (result != null) {
return result.getResult();
} else {
throw new IllegalStateException("URL " + url + " not requested.");
}
}
}
In real Java things would be done differently, by using Concurrent classes and/or Atomic... classes, so this is just for educational purposes. For further reading see "Callable Future".

How to check the size() or isEmpty() for ConcurrentLinkedQueue

I am trying to prototype a simple structure for a Web crawler in Java. Until now the prototype is just trying to do the below:
Initialize a Queue with list of starting URLs
Take out a URL from Queue and submit to a new Thread
Do some work and then add that URL to a Set of already visited URLs
For the Queue of starting URLs, I am using a ConcurrentLinkedQueue for synchronizing.
To spawn new Threads I am using ExecutorService.
But while creating a new Thread, the application needs to check if the ConcurrentLinkedQueue is empty or not. I tried using:
.size()
.isEmpty()
But both seem not to be returning the true state of ConcurrentLinkedQueue.
The problem is in below block:
while (!crawler.getUrl_horizon().isEmpty()) {
workers.submitNewWorkerThread(crawler);
}
And because of this, ExecutorService creates all the Threads in its limit, even if the input is only 2 URLs.
Is there a problem with the way multi-threading is being implemented here? If not, what is the better way to check the state of ConcurrentLinkedQueue?
Starting class for the application:
public class CrawlerApp {
private static Crawler crawler;
public static void main(String[] args) {
crawler = = new Crawler();
initializeApp();
startCrawling();
}
private static void startCrawling() {
crawler.setUrl_visited(new HashSet<URL>());
WorkerManager workers = WorkerManager.getInstance();
while (!crawler.getUrl_horizon().isEmpty()) {
workers.submitNewWorkerThread(crawler);
}
try {
workers.getExecutor().shutdown();
workers.getExecutor().awaitTermination(10, TimeUnit.MINUTES);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
private static void initializeApp() {
Properties config = new Properties();
try {
config.load(CrawlerApp.class.getClassLoader().getResourceAsStream("url-horizon.properties"));
String[] horizon = config.getProperty("urls").split(",");
ConcurrentLinkedQueue<URL> url_horizon = new ConcurrentLinkedQueue<>();
for (String link : horizon) {
URL url = new URL();
url.setURL(link);
url_horizon.add(url);
}
crawler.setUrl_horizon(url_horizon);
} catch (IOException e) {
e.printStackTrace();
}
}
}
Crawler.java which maintains the Queue of URLs and Set of already visited URLs.
public class Crawler implements Runnable {
private ConcurrentLinkedQueue<URL> url_horizon;
public void setUrl_horizon(ConcurrentLinkedQueue<URL> url_horizon) {
this.url_horizon = url_horizon;
}
public ConcurrentLinkedQueue<URL> getUrl_horizon() {
return url_horizon;
}
private Set<URL> url_visited;
public void setUrl_visited(Set<URL> url_visited) {
this.url_visited = url_visited;
}
public Set<URL> getUrl_visited() {
return Collections.synchronizedSet(url_visited);
}
#Override
public void run() {
URL url = nextURLFromHorizon();
scrap(url);
addURLToVisited(url);
}
private URL nextURLFromHorizon() {
if (!getUrl_horizon().isEmpty()) {
URL url = url_horizon.poll();
if (getUrl_visited().contains(url)) {
return nextURLFromHorizon();
}
System.out.println("Horizon URL:" + url.getURL());
return url;
}
return null;
}
private void scrap(URL url) {
new Scrapper().scrap(url);
}
private void addURLToVisited(URL url) {
System.out.println("Adding to visited set:" + url.getURL());
getUrl_visited().add(url);
}
}
URL.java is just a class with private String url and overriden hashCode() and equals().
Also, Scrapper.scrap() just has dummy implementation until now:
public void scrap(URL url){
System.out.println("Done scrapping:"+url.getURL());
}
WorkerManager to create Threads:
public class WorkerManager {
private static final Integer WORKER_LIMIT = 10;
private final ExecutorService executor = Executors.newFixedThreadPool(WORKER_LIMIT);
public ExecutorService getExecutor() {
return executor;
}
private static volatile WorkerManager instance = null;
private WorkerManager() {
}
public static WorkerManager getInstance() {
if (instance == null) {
synchronized (WorkerManager.class) {
if (instance == null) {
instance = new WorkerManager();
}
}
}
return instance;
}
public Future submitNewWorkerThread(Runnable run) {
return executor.submit(run);
}
}

Problem
The reason why you end up creating more Threads than there are URLs in the queue is because it is possible (and in fact likely) that none of the Threads of the Executor start until you go through the while loop a lot of times.
Whenever working with threads you should always keep in mind that the threads are scheduled independently and run at their own pace except when you explicitly synchronize them. In this case, the threads can start at any time after the submit() call, even though it seems you'd like each one to start and go past nextURLFromHorizon before a next iteration in your while loop.
Solution
Consider dequeuing the URL from the queue before submitting the Runnable to the Executor. I also suggest defining a CrawlerTask that is submitted to the Executor once, rather than a Crawler that is submitted repeatedly. In such design you wouldn't even need a thread-safe container for the URLs to-be-scraped.
class CrawlerTask extends Runnable {
URL url;
CrawlerTask(URL url) { this.url = url; }
#Override
public void run() {
scrape(url);
// add url to visited?
}
}
class Crawler {
ExecutorService executor;
Queue urlHorizon;
//...
private static void startCrawling() {
while (!urlHorizon.isEmpty()) {
executor.submit(new CrawlerTask(urlHorizon.poll());
}
// ...
}
}

How to retry function request after a certain time

How do I make it retry the send attempt if user data is null. Max 2 retries, 1 retry after 10 seconds?
public class UserHandler {
private List users = new ArrayList();
public void addUser(username) {} //adds user
public Userdata findUser(username) {} //finds user
public void sendTo(String username, String message) {
Userdata user = findUser(username);
if(user != null) {
Out out = new Out(user.getClientSocket());
out.println(message);
}
}
}
Do I really have to manually insert a thread and sleep it inside sendTo()?
EDIT: the server uses java 1.4.2

You're got more of an architectural problem to solve first. In a single-threaded program the sequence is normally:
Do stuff;
Call sendTo();
Do more stuff.
You have to work out if what you want is:
Do stuff;
Call sendTo();
If (2) fails, wait 10 seconds and sendTo() again;
If (3) fails, throw an error;
Do more stuff.
The point being that this is still synchronous. If so you'll need a thread. You should use the Java 5 Executors.
public void sendTo(final String username, final String message) {
if (!internalSendTo(username, message)) {
// attempt resend
ExecutorService exec = Executors.newSingleThreadExecutor();
final AtomicBoolean result = new AtomicBoolean(false);
exec.submit(new Runnable() {
boolean b = internalSendto(username, message);
result.set(b);
});
try {
exec.awaitTermination(10, TimeUnit.SECONDS);
} catch (InterruptedException e) {
// still didn't work
} finally {
exec.shutdownNow();
}
}
}
private boolean internalSendTo(String username, String message) {
Userdata user = findUser(username);
boolean success = false;
if (user != null) {
Out out = new Out(user.getClientSocket());
// do the communication here
success = true;
}
return success;
}
Now that's just a rough sketch of how it might work. It should give you some appreciation for the issues however.
Do you want this or do you want:
Do stuff;
Call sendTo();
If (2) fails, queue the send and keep going;
Do more stuff.
Basically this is the asynchronous approach. If you go this way you then have to answer questions like:
What happens if after 10+ seconds (or some arbitrary interval) it still hasn't worked?
What processes attempt the sendTo() calls?
What if they block/die?
Do I need multiple senders?
etc
Basically it gets much more complicated.

I would recommend to use AOP and Java annotations. Try a read-made mechanism from jcabi-aspects:
#RetryOnFailure(attempts = 3)
public void sendTo(String username, String message) {
// try to do it
}