Those who are familiar with lmax ring buffer (disruptor) know that one of the biggest advanatages of that data structure is that it batches incomming events and when we have a consumer that can take advantage of batching that makes the system automatically adjustable to the load, the more events you throw at it the better.
I wonder couldnt we achieve the same effect with an Observable (targeting the batching feature). I've tried out Observable.buffer but this is very different, buffer will wait and not emit the batch while the expected number of events didnt arrive. what we want is quite different.
given the subsriber is waiting for a batch from Observable<Collection<Event>>, when a single item arrives at stream it emits a single element batch which gets processed by subscriber, while it is processing other elements are arriving and getting collected into next batch, as soon as subscriber finishes with the execution it gets the next batch with as many events as had arrived since it started last processing...
So as a result if our subscriber is fast enough to process one event at a time it will do so, if load gets higher it will still have the same frequency of processing but more events each time (thus solving backpressure problem)... unlike buffer which will stick and wait for batch to fill up.
Any suggestions? or shall i go with ring buffer?
RxJava and Disruptor represent two different programming approaches.
I'm not experienced with Disruptor but based on video talks, it is basically a large buffer where producer emit data like a firehose and consumers spin/yield/block until data is available.
RxJava, on the other hand, aims at non-blocking event delivery. We too have ringbuffers, notably in observeOn which acts as the async-boundary between producers and consumers, but these are much smaller and we avoid buffer overflows and buffer bloat by applying the co-routines approach. Co-routines boil down to callbacks sent to your callbacks so yo can callback our callbacks to send you some data at your pace. The frequency of such requests determines the pacing.
There are data sources that don't support such co-op streaming and require one of the onBackpressureXXX operators that will buffer/drop values if the downstream doesn't request fast enough.
If you think you can process data in batches more efficiently than one-by-one, you can use the buffer operator which has overloads to specify time duration for the buffers: you can have, for example, 10 ms worth of data, independent of how many values arrive in this duration.
Controlling the batch-size via request frequency is tricky and may have unforseen consequences. The problem, generally, is that if you request(n) from a batching source, you indicate you can process n elements but the source now has to create n buffers of size 1 (because the type is Observable<List<T>>). In contrast, if no request is called, the operator buffers the data resulting in longer buffers. These behaviors introduce extra overhead in the processing if you really could keep up and also has to turn the cold source into a firehose (because otherwise what you have is essentially buffer(1)) which itself can now lead to buffer bloat.
Related
I have a datastream in which the order of the events is important. The time characteristic is set to EventTime as the incoming records have a timestamp within them.
In order to guarantee the ordering, I set the parallelism for the program to 1. Could that become a problem, performance wise, when my program gets more complex?
If I understand correctly, I need to assign watermarks to my events, if I want to keep the stream ordered by timestamp. This is quite simple. But I'm reading that even that doesn't guarantee order? Later on, I want to do stateful computations over that stream. So, for that I use a FlatMap function, which needs the stream to be keyed. But if I key the stream, the order is lost again. AFAIK this is because of different stream partitions, which are "caused" by parallelism.
I have two questions:
Do I need parallelism? What factors do I need to consider here?
How would I achieve "ordered parallelism" with what I described above?
Several points to consider:
Setting the parallelism to 1 for the entire job will prevent scaling your application, which will affect performance. Whether this actually matters depends on your application requirements, but it would certainly be limitation, and could be a problem.
If the aggregates you've mentioned are meant to be computed globally across all the event records then operating in parallel will require doing some pre-aggregation in parallel. But in this case you will then have to reduce the parallelism to 1 in the later stages of your job graph in order to produce the ultimate (global) results.
If on the other hand these aggregates are to be computed independently for each value of some key, then it makes sense to consider keying the stream and to use that partitioning as the basis for operating in parallel.
All of the operations you mention require some state, whether computing max, min, averages, or uptime and downtime. For example, you can't compute the maximum without remembering the maximum encountered so far.
If I understand correctly how Flink's NiFi source connector works, then if the source is operating in parallel, keying the stream will result in out-of-order events.
However, none of the operations you've mentioned require that the data be delivered in-order. Computing uptime (and downtime) on an out-of-order stream will require some buffering -- these operations will need to wait for out-of-order data to arrive before they can produce results -- but that's certainly doable. That's exactly what watermarks are for; they define how long to wait for out-of-order data. You can use an event-time timer in a ProcessFunction to arrange for an onTimer callback to be called when all earlier events have been processed.
You could always sort the keyed stream. Here's an example.
The uptime/downtime calculation should be easy to do with Flink's CEP library (which sorts its input, btw).
UPDATE:
It is true that after applying a ProcessFunction to a keyed stream the stream is no longer keyed. But in this case you could safely use reinterpretAsKeyedStream to inform Flink that the stream is still keyed.
As for CEP, this library uses state on your behalf, making it easier to develop applications that need to react to patterns.
I have a basic understanding of a producer-consumer problem in a bounded buffer, but I am not able to relate it to this analogy: Describe how a highway off-ramp onto a local road is a good example of a producer/consumer
relationship with a bounded buffer. In particular, discuss how the designers might choose the size
of the off-ramp.
Things I know: (I feel this is just the basics, but anyway)
In a bounded buffer, we need a thread-safe consumer and producer.
A producer can execute only when the buffer is not full.
A consumer can execute only when the buffer is not empty.
At a time either producer can execute or consumer can execute.
I'm assuming the highway to be the producer for vehicles. These vehicles are going to a local road through an off-ramp. Hence off-ramp is the buffer, and the local road is the consumer. Am I taking it in the right way? but I've no idea how to relate it to the size of the off-ramp!
If the producer is the highway, the off-ramp is the bounded buffer, and the local road is the consumer then you might think of the data throughput as a form of traffic flow.
If the highway feeds the off-ramp faster than vehicles can enter the side street then your off-ramp will fill up. If the highway feeds the off-ramp slower than the vehicles can enter the side street the side street will sometimes have no traffic from the highway. If the off-ramp fills up some of the traffic will either back up onto the highway or will have to use an alternate highway exit.
The producer-consumer pattern has similar behaviors. If the producer writes data to the bounded buffer faster than the consumer reads data from the bounded buffer the buffer will fill. When the buffer fills either the producer must wait for space in the buffer to write new values to the buffer. When the buffer empties the consumer must wait for data to be available in the buffer before the consumer can read from the buffer.
This is where the analogy starts to fail us. Producer-consumer patterns can be blocking in an attempt to ensure all data produced by the producer is consumed by the consumer. However, it is also possible to design a producer-consumer pattern that recognizes that the producer will always be faster than the consumer and therefore the consumer will never process all the data produced in a continuous system. In this case the producer must be able to over-write some of the data before the consumer reads the data. This is known as a sampling system. The consumer only reads a sample of the data, not every data value.
An example of a sampling system might be an engine control system. The sensors monitoring cylinder pressure may be able to produce data at a rate of one reading per milisecond, or 1000 times per second, but the engine valve controls may only be able to respond to the cylinder pressure at a rate of 500 times per second. If the bounded buffer were infinitely large the consumer (the valve controls) would always be dealing with very old data, causing the engine to be responding to cylinder pressures from multiple seconds in the past. The engine would be out of control. If, instead, the valve controls only read the most recent data, ignoring any data between the last reading and the current reading, the engine could maintain proper control. It would never be dealing with stale data.
A producer-consumer using a sampling buffer often only needs a single element buffer. The producer writes to the buffer at its natural rate. The consumer reads the data at its natural rate. With a single element buffer the consumer is guaranteed to only see the most recent value in the buffer. All intermediate values are not even seen by the consumer.
The main benefit of reactive programming is that it is fault-tolerant and can process a lot more events than a blocking implementation, despite the fact that the processing will usually happen slower.
What I don't fully understand is how and where the events are stored. I know that there is an event buffer and it can be tweaked but that buffer can easily overload the memory if the queue is unbound, can't it? Can this buffer flush onto disk? Isn't it a rist to have it in-memory? Can it be configured similarly to Lagom event-sourcing or persistent Akka actors where events can be stored in DB?
The short answer is no, this buffer cannot be persisted. At least in reference implementation.
The internal in-memory buffer can hold up to 128 emited values by default, but there are some points. First of all, there is a backpressure — situatuion when the source emits items faster than observer or operator consumes them. Thus, when this internal buffer is overloaded you get a MissingBackpressureException and there are no any disk or some other way to persist it. However you can tweak the behavior, for instance keep only latest emit or just drop new emits. There are special operators for that — onBackpressureBuffer, onBackpressureDrop, onBackpressureLatest.
RxJava2 introduces a new type — Flowable which supports backpressure by default and gives more ways to tweak internal buffer.
Rx is a way to process data streams and you should care if you can consume all the items and how to store them if you can't.
One of the main advantages of rxjava is contract and there are ways to create your own operators or use some extensions like rxjava-extras
One EventHandler(DatabaseConsumer) of the Disruptor calls stored procedures in database, which is so slow that it blocks the Disruptor for some time.
Since I need the Disruptor keep running without blocking. I am thinking adding an extra queue so that EventHandler could serve as Producer and another new-created thread could serve as Consumer to handle database's work, which could be asynchronous without affecting the Disruptor
Here is some constrain:
The object that Disruptor passed to the EventHandler is around 30KB and the number of this object is about 400k. In theory, the total size of the objects that needs to be handled is around 30KBX400K =12GB. So the extra queue should be enough for them.
Since performance matters, GC pause should be avoided.
The heap size of the Java program is only 2GB.
I'm thinking text file as a option. EventHandler(Producer) writes the object to the file and Consumer reads from them and call stored procedure. The problem is how to handle the situation that it reach to the end of the file and how to know the new coming line.
Anyone who has solve this situation before? Any advice?
The short answer is size your disruptor to cope with the size of your bursts not your entire volume, bare in mind the disruptor can just contain a reference to the 30kb object, the entire object does not need to be in the ring buffer.
With any form of buffering before your database will require the memory for buffering the disruptor offers you the option of back pressure on the rest of the system when the database has fallen too far behind. That is to say you can slow the inputs to the disruptor down.
The other option for spooling to files is to look at Java Chronicle which uses memory mapped files to persist things to disk.
The much more complicated answer is take advantage of the batching effects of the disruptor so that your DB can catch up. I.e. using a EventHandler which collects events a batch of events together and submits them to the database as one unit.
This practice allows the EventHandler to become more efficient as things back up thus increasing throughput.
Short answer: don't use disruptor. Use a distributed MQ with retransmission support.
Long answer: If you have fast producers with slow consumers you will need some sort of retransmission mechanism. I don't think you can escape from that unless you can tolerate nasty blocks (i.e. huge latencies) in your system. That's when distributed MQs (Messaging Queues) come to play. Disruptor is not a distributed MQ, but you could try to implement something similar. The idea is:
All messages are sequenced and processed in order by the consumer
If the queue gets full, messages are dropped
If the consumer detects a message gap it will request a retransmission of the lost messages, buffering the future messages until it receives the gap
With that approach the consumer can be as slow as it wants because it can always request the retransmission of any message it lost at any time. What we are missing here is the retransmission entity. In a distributed MQ that will be a separate and independent node persisting all messages to disk, so it can replay back any message to any other node at any time. Since you are not talking about an MQ here, but about disruptor, then you will have to somehow implement that retransmission mechanism yourself on another thread. This is a very interesting problem without an easy answer or recipe. I would use multiple disruptor queues so your consumer could do something like:
Read from the main channel (i.e. main disruptor queue)
If you detect a sequence gap, go to another disruptor queue connected to the replayer thread. You will actually need two queues there, one to request the missing messages and another one to receive them.
The replayer thread would have another disruptor queue from where it is receiving all messages and persisting it to disk.
You are left to make sure your replayer thread can write messages fast enough to disk. If it cannot then there is no escape besides blocking the whole system. Fortunately disk i/o can be done very fast if you know what you are doing.
You can forget all I said if you can just afford to block the producers if the consumers are slow. But if the producers are getting messages from the network, blocking them will eventually give you packet drops (UDP) and probably an IOException (TCP).
As you can see this is a very interesting question with a very complicated answer. At Coral Blocks we have experience developing distributed MQs like that on top of CoralReactor. You can take a look in some of the articles we have on our website.
I was curious regarding the most common (or recommended) implementations of disruptor about the journaling step. And the most common questions of mine are:
how it is actually implemented (by example)?
Is it wise to use JPA?
What DB is commonly used (by the community that has already implement projects with disruptor)?
Is it wise to be used at the intermediate handlers (of EventProcessors) so the State of each message should be saved, rather than before and after the business logic process?
By the way (I am sorry, I know this is not related with the journalling step), what is the right way to delete a message from the RingBuffer during an eventHandler process (assuming that the message is dead/expired and should be removed by the whole procedure). I was wondering something similar as the Dead Letter Channel pattern.
Cheers!
The Disruptor is usually used for low latency, high throughput processing. E.g. millions of messages with a typical latency in the hundreds of micro-seconds. As very few databases can handle this sort of rate of updates with reasonably bounded delays, journaling is often done to a raw file with replication to a second (or third) system.
For reporting purposes, a system reads this file or listens to messages and updates a database as quickly as it can but this is taken out of the critical path.
An entry is dead in the ring buffer when every event processor has processed it.
The slot a message uses is not available until every event processor has processed it and all the message before it. Deleting a message would be too expensive, both in terms of performance and impact on the design
Every event processors sees every message. As this happens concurrently, there is little cost in doing this, but it quite normal for event processors to ignore messages as a result. (possibly most messages)