We Keep Coding

"Run"ing vs. Single stepping in Java Eclipse -

I am trying to update a large (~150,000 line) Java/Eclipse program from 2009 that has several threads, and communicates with a TCP-IP telescope. I wrote new code to do the communications, and that all works fine when separately tested. This is on a Mac - the original worked on Mac OSX 10.6 and nothing later - I'm writing this for current Mac OS 12.xx
In the overall program, there is a thread that receives mouse clicks from the GUI and initiates operation of a thread with runs the communications code. The communications thread stays alive, using parameters passed to it to determine which communication function to execute.
Here's some of the code:
class ActualTask {
ActualTask () {
// PRAA Dec 7 2021
System.out.println("What thread are we in? it's: "+ Thread.currentThread().getName());
while (true) // this is true so we can keep reusing this task!
{
// PRAA Dec 7 2021
//System.out.println("What thread are we in? it's: "+ Thread.currentThread().getName());
if (running)
{
// PRAA Dec 7 2021
System.out.println("Just before call to run() - What thread are we in? it's: "+ Thread.currentThread().getName());
run();
// PRAA Dec 7 2021
System.out.println("What thread are we in? it's: "+ Thread.currentThread().getName());
running = false;
}
}
}
// ------------------------------------
void run()
{
done = false;
successful = true;
telescopeException = null;
// PRAA Dec 7 2021
System.out.println("In run() - thread is: "+ Thread.currentThread().getName());
try
{ // *** execute the telescope task ***
TelescopeGlobals.telescopeProgressTaskData.setBackgroundColor(Color.red); // red when active
telescopeTask.execute();
done = true;
TelescopeGlobals.telescopeProgressTaskData.setBackgroundColor(Color.white); // white when NOT active
}
catch (TelescopeException te)
{
// Clear progress bar before throwing up dialog
TelescopeGlobals.telescopeProgressTaskData.setBackgroundColor(Color.white); // white when NOT active
TelescopeGlobals.telescopeProgressTaskData.clear();
done = true;
successful = false;
telescopeException = te;
Reporting.selctiveAlert(te.getMessage() + "\n" + te.getWhereExceptionOccured());
TelescopeGlobals.telescopeProgressTaskData.setText1("Error: " + te.getMessage());
TelescopeGlobals.telescopeProgressTaskData.setText2("Where: " + te.getWhereExceptionOccured());
}
done = true;
}
}
When I am in "debug" mode, and I single-step through the communications code, things usually execute properly - however, when I hit "Run", things don't go so well -
the first call works ok
the second call appears to do nothing - that is, it doesn't seem to be executing, and none of my print statements print
subsequent calls only print an error message saying the previous task has not completed.
So I'm wondering - what is the difference between the "run" and "debug" modes - why would one work and the other behave differently - and
Does all of the "SwingWorker" stuff from 2009 still work?
Should creating the communications thread be done differently? It is just created once, as shown in the code snippet below:
// Static member above of HighLevelTaskWorker makes sure that one command is dispatched to the telescope at a time.
class HighLevelTaskWorker { // SwingWorker wrapper class
private lib.SwingWorker worker = null;
private boolean done;
private boolean successful;
private TelescopeException telescopeException;
private boolean running;
private TelescopeTask telescopeTask;
public HighLevelTaskWorker()
{
}
public void start(TelescopeTask telescopeTask) {
this.telescopeTask = telescopeTask;
// PRAA Dec 7 2021
System.out.println("What thread are we in? it's: "+ Thread.currentThread().getName());
done = false;
successful = false;
telescopeException = null;
running = true;
if (worker == null) // no need to create this tread over and over again, think of all that Garbage Collecting and tread creation!
{
worker = new lib.SwingWorker() {
public Object construct() {
return new ActualTask();
}
};
worker.start();
}
}
Thanks so much for any help !!!

Scala Process never end when applying Future and ThreadPools for Multi-threading

Here's the code snippet:
object Test {
def main(args: Array[String]): Unit = {
// MARK: parallelization
val pool = Executors.newFixedThreadPool(3)
implicit val xc = ExecutionContext.fromExecutorService(pool)
var taskQueue = new ArrayBuffer[Future[Unit]]()
for (i <- 0 until 10) {
try {
taskQueue += Future {
print(s"in_${i}\n")
Thread.sleep(1000)
}
} catch {
case t: Throwable => {
print(t)
}
}
}
val importTasks = Future.sequence(taskQueue)
importTasks.onSuccess { case res => print("finishOnSuccess") }
Await.result(importTasks, Duration.Inf)
}
}
it will hang forever after all jobs are executed with the following output:
in_1
in_2
in_0
in_4
in_3
in_5
in_8
in_6
in_7
in_9
finishOnSuccess
I tried to call System.exit(0) at the end of main method but still no use. Any suggestions? Thanks!

You need to shutdown ExecutorService of pool by pool.shutdown() after Await. Since the newFixedThreadPool is creating non-daemon thread by defaultThreadFactory.
and this Post explain more about Daemon thread and Non-Daemon thread:
What is Daemon thread in Java?

Scala server with Spray Akka overthreading

I'm using spray.io and akka.io on my freebsd 1CPU/2GB server and I'm facing
threading problems. I've started to notice it when I got OutOfMemory exception
because of "can't create native thread".
I check Thread.activeCount() usually and I see it grows enormously.
Currently I use these settings:
myapp-namespace-akka {
akka {
loggers = ["akka.event.Logging$DefaultLogger"]
loglevel = "DEBUG"
stdout-loglevel = "DEBUG"
actor {
deployment {
default {
dispatcher = "nio-dispatcher"
router = "round-robin"
nr-of-instances = 1
}
}
debug {
receive = on
autoreceive = on
lifecycle = on
}
nio-dispatcher {
type = "Dispatcher"
executor = "fork-join-executor"
fork-join-executor {
parallelism-min = 8
parallelism-factor = 1.0
parallelism-max = 16
task-peeking-mode = "FIFO"
}
shutdown-timeout = 4s
throughput = 4
throughput-deadline-time = 0ms
attempt-teamwork = off
mailbox-requirement = ""
}
aside-dispatcher {
type = "Dispatcher"
executor = "fork-join-executor"
fork-join-executor {
parallelism-min = 8
parallelism-factor = 1.0
parallelism-max = 32
task-peeking-mode = "FIFO"
}
shutdown-timeout = 4s
throughput = 4
throughput-deadline-time = 0ms
attempt-teamwork = on
mailbox-requirement = ""
}
}
}
}
I want nio-dispatcher to be my default non blocking (lets say single thread)
dispatcher. And I execute all my futures (db, network queries) on aside-dispatcher.
I get my contexts through my application as follows:
trait Contexts {
def system: ActorSystem
def nio: ExecutionContext
def aside: ExecutionContext
}
object Contexts {
val Scope = "myapp-namespace-akka"
}
class ContextsImpl(settings: Config) extends Contexts {
val System = "myapp-namespace-akka"
val NioDispatcher = "akka.actor.nio-dispatcher"
val AsideDispatcher = "akka.actor.aside-dispatcher"
val Settings = settings.getConfig(Contexts.Scope)
override val system: ActorSystem = ActorSystem(System, Settings)
override val nio: ExecutionContext = system.dispatchers.lookup(NioDispatcher)
override val aside: ExecutionContext = system.dispatchers.lookup(AsideDispatcher)
}
// Spray trait mixed to service actors
trait ImplicitAsideContext {
this: EnvActor =>
implicit val aside = env.contexts.aside
}
I think that I did mess up with configs or implementations. Help me out here.
Usually I see thousands of threads for now on my app until it crashes (I set
freebsd pre process limit to 5000).

If your app indeed starts so many threads this can be usually tracked back to blocking behaviours inside ForkJoinPools (bad bad thing to do!), I explained the issue in detail in the answer here: blocking blocks, so you may want to read up on it there and verify what threads are being created in your app and why – the ForkJoinPool does not have a static upper limit.

Does Akka has a ExecutorCompletionService equivalent where Futures are queued by their completion time?

With java I can create an ExecutorCompletionService with an executor and a bunch of tasks. This class arranges that submitted tasks are, upon completion, placed on a queue accessible using take.
https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ExecutorCompletionService.html
Does Akka has something similar for managing Futures returned by actors?

This answer is for Scala only. In scala there is sequence/firstCompletedOf to compose futures, which returns you new future completing after all/one of the underlying futures isCompleted (which is equivalent to examples from CompletionService's api docs). Such solution is more safe than ecs.take().get() as there is no blocking if you use onComplete listener; however, if you still want some blocking waiter - use Await.result. So, no need for CompletionService as list of futures is flexible enough and much more safe. Equivalent of first example:
val solvers: List[() => Int] = ...
val futures = solvers.map(s => Future {s()}) //run execution
(Future sequence futures) onComplete { results: Seq[Int] =>
results.map(use)
}
Another example is cancelling the task:
val solvers: List[Future => Int] = ... //some list of functions(tasks), Future is used to check if task was interrupted
val (futures, cancels): solvers.map(cancellableFuture) //see https://stackoverflow.com/questions/16020964/cancellation-with-future-and-promise-in-scala
(Future firstCompletedOf futures) onComplete { result: Int =>
cancels.foreach(_())
use(result)
}
Talking about Java, Akka has adaptation of scala's futures: http://doc.akka.io/docs/akka/snapshot/java/futures.html
If you just want to sequentially process results on their completion, you may use actor for that:
val futures: List[Future]
futures.map(_ pipeTo actor) //actor's mailbox is used as queue
To model completion queue's behavior (which is not recommended):
import scala.concurrent._
import duration._
import scala.concurrent.ExecutionContext.Implicits.global //some execution context
class Queue[T](solvers: Seq[() => T]) extends Iterator[T]{
case class Result(f: Future[Result], r: T)
var futures: Set[Future[Result]] = solvers map {s =>
lazy val f: Future[Result] = Future{Result(f, s())}
f
} toSet
def hasNext() = futures.nonEmpty
def next() = {
val result = Await.result((Future firstCompletedOf futures.toSeq), Duration.Inf)
futures -= result.f
result.r
}
}
scala> val q = new Queue(List(() => 1, () => 2, () => 3, () => 4))
q: Queue[Int] = non-empty iterator
scala> q.next
res14: Int = 2
scala> q.next
res15: Int = 1
scala> q.foreach(println)
4
3

Maybe this probable solution without using ExecutorCompletionService will help you:
import java.util.concurrent.atomic.AtomicLong
import java.util.concurrent._
import scala.concurrent.duration._
import scala.util._
import scala.concurrent.{ExecutionContextExecutorService, ExecutionContext, Future}
class BatchedIteratorsFactory[S,R](M: Int, timeout: Duration) {
implicit val ec = ExecutionContext.fromExecutor(Executors.newCachedThreadPool())
val throttlingQueue = new LinkedBlockingQueue[Future[R]](M) // Can't put more than M elements to the queue
val resultQueue = new LinkedBlockingQueue[Try[R]](M)
val jobCounter = new AtomicLong(0)
def iterator(input: Iterator[S])(job: S => R): Iterator[Try[R]] = {
val totalWork = Future(input.foreach { elem =>
jobCounter.incrementAndGet
throttlingQueue.put(Future { job(elem) } andThen {
case r => resultQueue.put(r); throttlingQueue.poll() // the order is important here!
})
})
new Iterator[Try[R]] {
override def hasNext: Boolean = jobCounter.get != 0 || input.hasNext
override def next(): Try[R] = {
jobCounter.decrementAndGet
Option(resultQueue.poll(timeout.toMillis, TimeUnit.MILLISECONDS)).getOrElse(
throw new TimeoutException(s"No task has been completed within ${timeout.toMillis} ms!")
)
}
}
}
}
So you can use it like this:
val job = { (elem: Int) =>
val result = elem * elem
Thread.sleep(1000L) // some possibel computation...
result
}
val src = Range(1, 16).toIterator
val it = new BatchedIteratorsFactory[Int, Int](M = 3, timeout = 4 seconds)
.iterator(src)(job)

Substitute for thread executor pool in Scala

My application requires that I have multiple threads running fetching data from various HDFS nodes. For that I am using the thread executor pool and forking threads.
Forking at :
val pathSuffixList = fileStatuses.getOrElse("FileStatus", List[Any]()).asInstanceOf[List[Map[String, Any]]]
pathSuffixList.foreach(block => {
ConsumptionExecutor.execute(new Consumption(webHdfsUri,block))
})
My class Consumption :
class Consumption(webHdfsUri: String, block:Map[String,Any]) extends Runnable {
override def run(): Unit = {
val uriSplit = webHdfsUri.split("\\?")
val fileOpenUri = uriSplit(0) + "/" + block.getOrElse("pathSuffix", "").toString + "?op=OPEN"
val inputStream = new URL(fileOpenUri).openStream()
val datumReader = new GenericDatumReader[Void]()
val dataStreamReader = new DataFileStream(inputStream, datumReader)
// val schema = dataStreamReader.getSchema()
val dataIterator = dataStreamReader.iterator()
while (dataIterator.hasNext) {
println(" data : " + dataStreamReader.next())
}
}
}
ConsumptionExecutor :
object ConsumptionExecutor{
val counter: AtomicLong = new AtomicLong()
val executionContext: ExecutorService = Executors.newCachedThreadPool(new ThreadFactory {
def newThread(r: Runnable): Thread = {
val thread: Thread = new Thread(r)
thread.setName("ConsumptionExecutor-" + counter.incrementAndGet())
thread
}
})
executionContext.asInstanceOf[ThreadPoolExecutor].setMaximumPoolSize(200)
def execute(trigger: Runnable) {
executionContext.execute(trigger)
}
}
However I want to use Akka streaming/ Akka actors where in I don't need to give a fixed thread pool size and Akka takes care of everything.
I am pretty new to Akka and the concept of Streaming and actors . Can someone give me any leads in the form of a sample code to fit my use case?
Thanks in advance!

An idea would be to create a (subclass) instance of ActorPublisher for each HDFS node that you are reading from, and then Merge them in as multiple Sources in a FlowGraph.
Something like this pseudo-code, where the details of the ActorPublisher sources are left out:
val g = PartialFlowGraph { implicit b =>
import FlowGraphImplicits._
val in1 = actorSource1
val in2 = actorSource2
// etc.
val out = UndefinedSink[T]
val merge = Merge[T]
in1 ~> merge ~> out
in2 ~> merge
// etc.
}
This can be improved for a collection of actor sources by just iterating over them and adding an edge to the merge for each one, but this gives the idea.