We Keep Coding

How to block until next data is emitted from hot Flux?

I have some function that return some Flux<Integer>. This flux is hot, it is being emitting live data. After some time of execution, I want to block until the next Integer is emitted, and assign to a variable. This Integer may not be the first and will not be the last.
I considered blockFirst(), but this would block indefinitely as the Flux has already emitted an Integer. I do not care if the Integer is the first or last in the Flux, I just want to block till the next Integer is emitted and assign it to a variable.
How would I do this? I think I could subscribe to the Flux, and after the first value, unsubscribe, but I am hoping there is a better way to do this.

It depends on the replay/buffering behavior of your hot flux. Both blockFirst() and next() operator do the same things: they wait for the first value received in the current subscription.
It is very important to understand that, because in the case of hot fluxes, subscription is independent of source data emission. The first value is not necessarily the first value emitted upstream. It is the first value received by your current subscriber, and that depends on the upstream flow behaviour.
In case of hot fluxes, how they pass values to the subscribers depends both on their buffering and broadcast strategies. For this answer, I will focus only on the buffering aspect:
If your hot flux does not buffer any emitted value (Ex: Flux.share(), Sinks.multicast().directBestEffort()), then both blockFirst() and next().block() operators meet your requirement: wait until the next emitted live data in a blocking fashion.
NOTE: next() has the advantage to allow to become non-blocking if replacing block with cache and subscribe
If your upstream flux does buffer some past values, then your subscriber / downsream flow will not only receive live stream. Before it, it will receive (part of) upstream history. In such case, you will have to use a more advanced strategy to skip values until the one you want.
From your question, I would say that skipping values until an elapsed time has passed can be done using skipUntilOther(Mono.delay(wantedDuration)).
But be careful, because the delay starts from your subscription, not from upstream subscription (to do so, you would require the upstream to provide timed elements, and switch to another strategy).
It is also important to know that Reactor forbids calling block() from some Threads (the one used by non-elastic schedulers).
Let's demonstrate all of that with code. In the below program, there's 4 examples:
Use next/blockFirst directly on a non-buffering hot flux
Use skipUntilOther on a buffering hot flux
Show that blocking can fail sometimes
Try to avoid block operation
All examples are commented for clarity, and launched sequentially in a main function:
import java.time.Duration;
import java.util.concurrent.CountDownLatch;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
public class HotFlux {
public static void main(String[] args) throws Exception {
System.out.println("== 1. Hot flux without any buffering");
noBuffer();
System.out.println("== 2. Hot flux with buffering");
withBuffer();
// REMINDER: block operator is not always accepted by Reactor
System.out.println("== 3. block called from a wrong context");
blockFailsOnSomeSchedulers();
System.out.println("== 4. Next value without blocking");
avoidBlocking();
}
static void noBuffer() {
// Prepare a hot flux thanks to share().
var hotFlux = Flux.interval(Duration.ofMillis(100))
.share();
// Prepare an operator that fetch the next value from live stream after a delay.
var nextValue = Mono.delay(Duration.ofMillis(300))
.then(hotFlux.next());
// Launch live data emission
var livestream = hotFlux.subscribe(i -> System.out.println("Emitted: "+i));
try {
// Trigger value fetching after a delay
var value = nextValue.block();
System.out.println("next() -> " + value);
// Immediately try to block until next value is available
System.out.println("blockFirst() -> " + hotFlux.blockFirst());
} finally {
// stop live data production
livestream.dispose();
}
}
static void withBuffer() {
// Prepare a hot flux replaying all values emitted in the past to each subscriber
var hotFlux = Flux.interval(Duration.ofMillis(100))
.cache();
// Launch live data emission
var livestream = hotFlux.subscribe(i -> System.out.println("Emitted: "+i));
try {
// Wait half a second, then get next emitted value.
var value = hotFlux.skipUntilOther(Mono.delay(Duration.ofMillis(500)))
.next()
.block();
System.out.println("skipUntilOther + next: " + value);
// block first can also be used
value = hotFlux.skipUntilOther(Mono.delay(Duration.ofMillis(500)))
.blockFirst();
System.out.println("skipUntilOther + blockFirst: " + value);
} finally {
// stop live data production
livestream.dispose();
}
}
public static void blockFailsOnSomeSchedulers() throws InterruptedException {
var hotFlux = Flux.interval(Duration.ofMillis(100)).share();
var barrier = new CountDownLatch(1);
var forbiddenInnerBlock = Mono.delay(Duration.ofMillis(200))
// This block will fail, because delay op above is scheduled on parallel scheduler by default.
.then(Mono.fromCallable(() -> hotFlux.blockFirst()))
.doFinally(signal -> barrier.countDown());
forbiddenInnerBlock.subscribe(value -> System.out.println("Block success: "+value),
err -> System.out.println("BLOCK FAILED: "+err.getMessage()));
barrier.await();
}
static void avoidBlocking() throws InterruptedException {
var hotFlux = Flux.interval(Duration.ofMillis(100)).share();
var asyncValue = hotFlux.skipUntilOther(Mono.delay(Duration.ofMillis(500)))
.next()
// time wil let us verify that the value has been fetched once then cached properly
.timed()
.cache();
asyncValue.subscribe(); // launch immediately
// Barrier is required because we're in a main/test program. If you intend to reuse the mono in a bigger scope, you do not need it.
CountDownLatch barrier = new CountDownLatch(2);
// We will see that both subscribe methods return the same timestamped value, because it has been launched previously and cached
asyncValue.subscribe(value -> System.out.println("Get value (1): "+value), err -> barrier.countDown(), () -> barrier.countDown());
asyncValue.subscribe(value -> System.out.println("Get value (2): "+value), err -> barrier.countDown(), () -> barrier.countDown());
barrier.await();
}
}
This program gives the following output:
== 1. Hot flux without any buffering
Emitted: 0
Emitted: 1
Emitted: 2
Emitted: 3
next() -> 3
Emitted: 4
blockFirst() -> 4
== 2. Hot flux with buffering
Emitted: 0
Emitted: 1
Emitted: 2
Emitted: 3
Emitted: 4
Emitted: 5
skipUntilOther + next: 5
Emitted: 6
Emitted: 7
Emitted: 8
Emitted: 9
Emitted: 10
Emitted: 11
skipUntilOther + blockFirst: 11
== 3. block called from a wrong context
BLOCK FAILED: block()/blockFirst()/blockLast() are blocking, which is not supported in thread parallel-6
== 4. Next value without blocking
Get value (1): Timed(4){eventElapsedNanos=500247504, eventElapsedSinceSubscriptionNanos=500247504, eventTimestampEpochMillis=1674831275873}
Get value (2): Timed(4){eventElapsedNanos=500247504, eventElapsedSinceSubscriptionNanos=500247504, eventTimestampEpochMillis=1674831275873}

Does windowAll operator in Flink scale down the parallelization to 1?

I have a stream in Flink which sends cubes from a source, make a transformation on the cube(adding 1 to each element in the cube), then lastly send it downstream to print the throughput of every second.
The stream is parallelized over 4 threads.
If I understand correctly the windowAll operator is a non-parallel transformation and should therefore scale down the parallelization back to 1, and by using it together with TumblingProcessingTimeWindows.of(Time.seconds(1)), sum the throughput of all parallelized subtasks within the latest second and print it. I'm unsure if I get correct output since the throughput every second is printed like this:
1> 25
2> 226
3> 354
4> 372
1> 382
2> 403
3> 363
...
Question: Does the stream printer print the throughput from each thread(1,2,3 & 4), or is it only that it chooses e.g. thread 3 to print the throughput sum of all the subtasks on?
When I set the parallelism of the environment to 1 in the beginningenv.setParallelism(1), I don't get the "x> " before the throughput, but I seem to get the same(or even better) throughput as when it is set to 4. Like this:
45
429
499
505
1
503
524
530
...
Here is a code-snippet of the program:
imports...
public class StreamingCase {
public static void main(String[] args) throws Exception {
int parallelism = 4;
final StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime);
env.setParallelism(parallelism);
DataStream<Cube> start = env
.addSource(new CubeSource());
DataStream<Cube> adder = start
.map(new MapFunction<Cube, Cube>() {
#Override
public Cube map(Cube cube) throws Exception {
return cube.cubeAdd(1);
}
});
DataStream<Integer> throughput = ((SingleOutputStreamOperator<Cube>) adder)
.windowAll(TumblingProcessingTimeWindows.of(Time.seconds(1)))
.apply(new AllWindowFunction<Cube, Integer, TimeWindow>() {
#Override
public void apply(TimeWindow tw,
Iterable<Cube> values,
Collector<Integer> out) throws Exception {
int sum = 0;
for (Cube c : values)
sum++;
out.collect(sum);
}
});
throughput.print();
env.execute("Cube Stream of Sweetness");
}
}

If the parallelism of the environment is set to 3 and you are using a WindowAll operator, only the window operator runs in parallelism 1. The sink will still be running with parallelism 3. Hence, the plan looks as follows:
In_1 -\ /- Out_1
In_2 --- WindowAll_1 --- Out_2
In_3 -/ \- Out_3
The WindowAll operator emits its output to its subsequent tasks using a round-robin strategy. That's the reason for the different threads emitting the result records of program.
When you set the environment parallelism to 1, all operators run with a single task.

Flux.generate with Consumer and parallel

I have simple Flux
Flux<Long> flux = Flux.generate(
AtomicLong::new,
(state, sink) -> {
long i = state.getAndIncrement();
sink.next(i);
if (i == 3) sink.complete();
return state;
}, (state) -> System.out.println("state: " + state));
Which works as expected in a single thread:
flux.subscribe(System.out::println);
The output is
0 1 2 3 state: 4
But when I switch to parallel:
flux.parallel().runOn(Schedulers.elastic()).subscribe(System.out::println);
The Consumer which should print state: Number isn't invoked. I just see:
0 3 2 1
Is it a bug or expected feature?

I'm not a reactive expert but after digging into the source code it seems that the behavior is by design; it seems that creating a ParallelFlux has the side effect of blocking the call of the State Consumer; if you want to go parallel and getting the State Consumer invoked you can use:
flux.publishOn(Schedulers.elastic()).subscribe(System.out::println);

How to achieve a guaranteed sleep time on a thread

I have a requirement for a class method to be called every 50 milliseconds. I don't use Thread.sleep because it's very important that it happens as precisely as possible to the milli, whereas sleep only guarantees a minimum time. The basic set up is this:
public class ClassA{
public void setup(){
ScheduledExecutorService se = Executors.newScheduledThreadPool(20);
se.scheduleAtFixedRate(this::onCall, 2000, 50, TimeUnit.MILLISECONDS);
}
protected void onCall(Event event) {
// do something
}
}
Now this by and large works fine. I have put System.out.println(System.nanoTime) in onCall to check its being called as precisely as I hope it is. I have found that there is a drift of 1-5 milliseconds over the course of 100s of calls, which corrects itself now and again.
A 5 ms drift unfortunately is pretty hefty for me. 1 milli drift is ok but at 5ms it messes up the calculation I'm doing in onCall because of states of other objects. It would be almost OK if I could get the scheduler to auto-correct such that if it's 5ms late on one call, the next one would happen in 45ms instead of 50.
My question is: Is there a more precise way to achieve this in Java? The only solution I can think of at the moment is to call a check method every 1ms and check the time to see if its at the 50ms mark. But then I'd need to maintain some logic if, on the off-chance, the precise 50ms interval is missed (49,51).
Thanks

Can I achieve a guaranteed sleep time on a thread?
Sorry, but No.
There is no way to get reliable, precise delay timing in a Java SE JVM. You need to use a Real time Java implementation running on a real time operating system.
Here are a couple of reasons why Java SE on a normal OS cannot do this.
At certain points, the GC in a Java SE JVM needs to "stop the world". While this is happening, no user thread can run. If your timer goes off in a "stop the world" pause, it can't be scheduled until the pause is over.
Scheduling of threads in a JVM is actually done by the host operating system. If the system is busy, the host OS may decide not to schedule the JVM's threads when your application needs this to happen.
The java.util.Timer.scheduleAtFixedRate approach is probably as good as you will get on Java SE. It should address long-term drift, but you can't get rid of the "jitter". And that jitter could easily be hundreds of milliseconds ... or even seconds.
Spinlocks won't help if the system is busy and the OS is preempting or not scheduling your threads. (And spinlocking in user code is wasteful ...)

According to the comment, the primary goal is not to concurrently execute multiple tasks at this precise interval. Instead, the goal is to execute a single task at this interval as precisely as possible.
Unfortunately, neither the ScheduledExecutorService nor any manual constructs involving Thread#sleep or LockSupport#parkNanos are very precise in that sense. And as pointed out in the other answers: There may always be influencing factors that are beyond your control - namely, details of the JVM implementation, garbage collection, JIT runs etc.
Nevertheless, a comparatively simple approach to achieve a high precision here is busy waiting. (This was already mentioned in an answer that is now deleted). But of course, this has several caveats. Most importantly, it will burn processing resources of one CPU. (And on a single-CPU-system, this may be particularly bad).
But in order to show that it may be far more precise than other waiting approaches, here is a simple comparison of the ScheduledExecutorService approach and the busy waiting:
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
public class PreciseSchedulingTest
{
public static void main(String[] args)
{
long periodMs = 50;
PreciseSchedulingA a = new PreciseSchedulingA();
a.setup(periodMs);
PreciseSchedulingB b = new PreciseSchedulingB();
b.setup(periodMs);
}
}
class CallTracker implements Runnable
{
String name;
long expectedPeriodMs;
long baseTimeNs;
long callTimesNs[];
int numCalls;
int currentCall;
CallTracker(String name, long expectedPeriodMs)
{
this.name = name;
this.expectedPeriodMs = expectedPeriodMs;
this.baseTimeNs = System.nanoTime();
this.numCalls = 50;
this.callTimesNs = new long[numCalls];
}
#Override
public void run()
{
callTimesNs[currentCall] = System.nanoTime();
currentCall++;
if (currentCall == numCalls)
{
currentCall = 0;
double maxErrorMs = 0;
for (int i = 1; i < numCalls; i++)
{
long ns = callTimesNs[i] - callTimesNs[i - 1];
double ms = ns * 1e-6;
double errorMs = ms - expectedPeriodMs;
if (Math.abs(errorMs) > Math.abs(maxErrorMs))
{
maxErrorMs = errorMs;
}
//System.out.println(errorMs);
}
System.out.println(name + ", maxErrorMs : " + maxErrorMs);
}
}
}
class PreciseSchedulingA
{
public void setup(long periodMs)
{
CallTracker callTracker = new CallTracker("A", periodMs);
ScheduledExecutorService se = Executors.newScheduledThreadPool(20);
se.scheduleAtFixedRate(callTracker, periodMs,
periodMs, TimeUnit.MILLISECONDS);
}
}
class PreciseSchedulingB
{
public void setup(long periodMs)
{
CallTracker callTracker = new CallTracker("B", periodMs);
Thread thread = new Thread(new Runnable()
{
#Override
public void run()
{
while (true)
{
long periodNs = periodMs * 1000 * 1000;
long endNs = System.nanoTime() + periodNs;
while (System.nanoTime() < endNs)
{
// Busy waiting...
}
callTracker.run();
}
}
});
thread.setDaemon(true);
thread.start();
}
}
Again, this should be taken with a grain of salt, but the results on My Machine® are as follows:
A, maxErrorMs : 1.7585339999999974
B, maxErrorMs : 0.06753599999999693
A, maxErrorMs : 1.7669149999999973
B, maxErrorMs : 0.007193999999998368
A, maxErrorMs : 1.7775299999999987
B, maxErrorMs : 0.012780999999996823
showing that the error for the waiting times is in the range of few microseconds.
In order to apply such an approach in practice, a more sophisticated infrastructure would be necessary. E.g. the bookkeeping that is necessary to compensate for waiting times that have been too high. (I think they can't be too low). Also, all this still does not guarantee a precisely timed execution. But it may be an option to consider, at least.

If you really have hard time constraints, you want to use a real-time operating system. General computing does not have hard time constraints; if your OS goes to virtual memory in one of your intervals, then you can miss your sleep interval. The real-time OS will make the tradeoff that you may get less done, but that work will can be better scheduled.
If you need to do this on a normal OS, you can spinlock instead of sleeping. This is really inefficient, but if you really have hard time constraints, it's the best way to approximate that.

That will be hard - think about GC... What I would do is to grab time with nanoTime, and use it in calculations. Or in other words I would get exact time and use it in calculations.

Yes (assuming you only want to prevent long term drifts and don't worry about each delay individually). java.util.Timer.scheduleAtFixedRate:
...In fixed-rate execution, each execution is scheduled relative to the scheduled execution time of the initial execution. If an execution is delayed for any reason (such as garbage collection or other background activity), two or more executions will occur in rapid succession to "catch up." In the long run, the frequency of execution will be exactly the reciprocal of the specified period (assuming the system clock underlying Object.wait(long) is accurate). ...
Basically, do something like this:
new Timer().scheduleAtFixedRate(new TimerTask() {
#Override
public void run() {
this.onCall();
}
}, 2000, 50);

Best method for parallel log aggregation

My program needs to analyze a bunch of log files daily which are generated on a hourly basis from each application server.
So if I have 2 app servers I will be processing 48 files (24 files * 2 app servers).
file sizes range 100-300 mb. Each line in every file is a log entry which is of the format
[identifier]-[number of pieces]-[piece]-[part of log]
for example
xxx-3-1-ABC
xxx-3-2-ABC
xxx-3-3-ABC
These can be distributed over the 48 files which I mentioned, I need to merge these logs like so
xxx-PAIR-ABCABCABC
My implementation uses a thread pool to read through files in parallel and then aggregate them using a ConcurrentHashMap
I define a class LogEvent.scala
class LogEvent (val id: String, val total: Int, var piece: Int, val json: String) {
var additions: Long = 0
val pieces = new Array[String](total)
addPiece(json)
private def addPiece (json: String): Unit = {
pieces(piece) = json
additions += 1
}
def isDone: Boolean = {
additions == total
}
def add (slot: Int, json: String): Unit = {
piece = slot
addPiece(json)
}
The main processing happens over multiple threads and the code is something on the lines of
//For each file
val logEventMap = new ConcurrentHashMap[String, LogEvent]().asScala
Future {
Source.fromInputStream(gis(file)).getLines().foreach {
line =>
//Extract the id part of the line
val idPart: String = IDPartExtractor(line)
//Split line on '-'
val split: Array[String] = idPart.split("-")
val id: String = split(0) + "-" + split(1)
val logpart: String = JsonPartExtractor(line)
val total = split(2) toInt
val piece = split(3) toInt
def slot: Int = {
piece match {
case x if x - 1 < 0 => 0
case _ => piece - 1
}
}
def writeLogEvent (logEvent: LogEvent): Unit = {
if (logEvent.isDone) {
//write to buffer
val toWrite = id + "-PAIR-" + logEvent.pieces.mkString("")
logEventMap.remove(logEvent.id)
writer.writeLine(toWrite)
}
}
//The LOCK
appendLock {
if (!logEventMap.contains(id)) {
val logEvent = new LogEvent(id, total, slot, jsonPart)
logEventMap.put(id, logEvent)
//writeLogEventToFile()
}
else {
val logEvent = logEventMap.get(id).get
logEvent.add(slot, jsonPart)
writeLogEvent(logEvent)
}
}
}
}
The main thread blocks till all the futures complete
Using this approach I have been able to cut the processing time from an hour+ to around 7-8 minutes.
My questions are as follows -
Can this be done in a better way, I am reading multiple files using different threads and I need to lock at the block where the aggregation happens, are there better ways of doing this?
The Map grows very fast in memory, any suggestions for off heap storage for such a use case
Any other feedback.
Thanks

A common way to do this is to sort each file and then merge the sorted files. The result is a single file that has the individual items in the order that you want them. Your program then just needs to do a single pass through the file, combining adjacent matching items.
This has some very attractive benefits:
The sort/merge is done by standard tools that you don't have to write
Your aggregator program is very simple. Or, there might even be a standard tool that will do it.
Memory requirements are lessened. The sort/merge programs know how to manage memory, and your aggregation program's memory requirements are minimal.
There are, of course some drawbacks. You'll use more disk space and the process will be somewhat slower due to the I/O cost.
When I'm faced with something like this, I almost always go with using the standard tools and a simple aggregator program. The increased performance I get from a custom program just doesn't justify the time it takes to develop the thing.

For this sort of thing, if you can, use Splunk, if not, copy what it does which is index the log files for aggregation on demand at a later point.
For off heap storage, look at distributed caches - Hazelcast or Coherence. Both support provide java.util.Map implementations that are stored over multiple JVMs.