I have a simple setup to a problem but the solution seems to be more complicated.
Setup: I have a hot observable which originates from a scanner that will emit every number as a different element and an R when a code is complete.
Problem: From this I want a hot observable that emits every full code as 1 element.
I tried playing around with different flatMap, takeUntil and groupByoperators but haven't been able to come to a solution.
You can use the buffer operator.
PublishSubject<Token<Integer>> s = PublishSubject.create();
Observable<Token<Integer>> markers = s.filter(x->x.isMarker());
s.buffer(markers).subscribe(
v->{
Optional<Integer> reduce = v.stream()
.filter(t->!t.isMarker())
.map(t->(ValueToken<Integer>)t)
.map(ValueToken::get)
.reduce((a,b)->a+b);
reduce.ifPresent(System.out::println);
}
);
s.onNext(value(12));
s.onNext(value(13));
s.onNext(marker()); // will emit 25
s.onNext(value(10));
s.onNext(value(7));
s.onNext(marker()); // will emit 17
s.onNext(value(10));
s.onNext(value(7)); // Not emitting yet
I made a class to wrap both values and markers in the flow.
public abstract class Token<T> {
private static final MarkerToken MARKER = new MarkerToken<>();
public boolean isMarker() {
return false;
}
public static <T> MarkerToken<T> marker() {
return MARKER;
}
public static <T> ValueToken<T> value(T o) {
return new ValueToken<>(o);
}
public static class ValueToken<T> extends Token<T> {
T value;
public ValueToken(T value) {
this.value = value;
}
public T get() {
return value;
}
}
public static class MarkerToken<T> extends Token<T> {
public boolean isMarker() {
return true;
}
}
}
update (using scan)
The previous method would emit also on the closing of the stream, with this solution you can emit only complete buffers.
The message class function as an accumulator, it will accumulate tokens until closing marker is accumulated.
When this happens the next message will start from scratch.
The presence of the closing mark as last element marks the message as complete.
public static class Message<T> {
List<Token<T>> tokens = new ArrayList<>();
public Message<T> append(Token<T> t) {
Message<T> mx = new Message<T>();
if(!isComplete()) {
mx.tokens.addAll(tokens);
}
mx.tokens.add(t);
return mx;
}
public boolean isComplete() {
int n = tokens.size();
return n>0 && tokens.get(n-1).isMarker();
}
public Optional<List<Token<T>>> fullMessage(){
return isComplete() ? Optional.of(tokens):Optional.empty();
}
}
Scanning the source you emit a message for each token emitted, then you filter out incomplete message and emit just the one marked as complete.
s.scan(new Message<Integer>(), (a, b) -> a.append(b))
.filter(Message::isComplete)
.map(Message::fullMessage)
.map(Optional::get).subscribe(v -> {
System.out.println(v);
});
s.onNext(value(12));
s.onNext(value(13));
s.onNext(marker());// [V(12), V(13), MARKER]
s.onNext(value(10));
s.onNext(value(7));
s.onNext(marker()); // [V(10), V(7), MARKER]
s.onNext(value(10));
s.onNext(value(127));
s.onComplete(); // Not emitting incomplete messages on the closing of the subject.
Related
In the following code, tick emits a new object every three seconds. I'm trying to count the number of emitted objects every second using groupedWithin (which ignores empty groups). Is there any way in Akka Streams for the following code to print 0 in periods when tick does not emit any objects?
Source.tick(Duration.ZERO, Duration.ofSeconds(3), new Object())
.groupedWithin(Integer.MAX_VALUE, Duration.ofSeconds(1))
.map(List::size)
.runWith(Sink.foreach(e -> System.out.println(e)), materializer);
In other words, I'd like the output of this code to be this sequence: 1 0 0 1 0 0 1 ... (every second) instead of 1 1 1 ... (every three seconds).
EDIT: This is the best workaround I have come up with so far (using keepAlive to send some special objects if the upstream is idle):
Source.tick(Duration.ZERO, Duration.ofSeconds(3), new Object())
.keepAlive(Duration.ofSeconds(1), KeepAliveElement::new)
.groupedWithin(Integer.MAX_VALUE, Duration.ofSeconds(1))
.map(lst -> lst.stream().filter(e -> !(e instanceof KeepAliveElement)).collect(Collectors.toList()))
.map(List::size)
.runWith(Sink.foreach(e -> System.out.println(e)), materializer);
Is there any better way to do this?
I thought this would be of normal difficulty, I was wrong. One thing I wanted to do is to ensure that the flow counting items that pass through the stream does not keep a reference to each item it sees: if many items pass in the aggregation period, you will end up with an unnecessarily big list in memory (even if only for a second) and the performance penalty to add (many) items to it. The following solution, although complex, keeps only a counter.
NOTE: Although I tested the happy scenario, I cannot say this is battle-proven, so use with caution!
Based on Akka's GroupedWeightedWithin and the documentation here:
public class CountInPeriod<T> extends GraphStage<FlowShape<T, Integer>> {
public Inlet<T> in = Inlet.<T>create("CountInPeriod.in");
public Outlet<Integer> out = Outlet.<Integer>create("CountInPeriod.out");
private FlowShape<T, Integer> shape = FlowShape.of(in, out);
private Duration duration;
public CountInPeriod(Duration duration) {
this.duration = duration;
}
#Override
public GraphStageLogic createLogic(Attributes inheritedAttributes) {
return new TimerGraphStageLogic(shape) {
private int counter = 0;
private int bufferPushCounter = -1;
{
setHandler(in, new AbstractInHandler() {
#Override public void onPush() throws Exception, Exception {
grab(in);
counter++;
pull(in);
}
});
setHandler(out, new AbstractOutHandler() {
#Override public void onPull() throws Exception, Exception {
if (bufferPushCounter >= 0) {
push(out, bufferPushCounter);
bufferPushCounter = -1;
}
}
});
}
#Override
public void preStart() throws Exception, Exception {
scheduleWithFixedDelay(CountInPeriod.class, duration, duration);
pull(in);
}
#Override
public void onTimer(Object timerKey) throws Exception, Exception {
if (isAvailable(out)) emitCounter();
else bufferPush();
}
private void emitCounter() {
push(out, counter);
counter = 0;
bufferPushCounter = -1;
}
private void bufferPush() {
bufferPushCounter = counter;
counter = 0;
}
};
}
#Override
public FlowShape<T, Integer> shape() {
return shape;
}
}
Test code:
public class GroupTicked {
final static ActorSystem as = ActorSystem.create("as");
public static void main(String... args) throws Exception {
CompletionStage<Done> done = Source.tick(Duration.ZERO, Duration.ofSeconds(3), new Object())
.take(7) // to finish in finite time...
.via(new CountInPeriod<>(Duration.ofSeconds(1)))
.runWith(Sink.foreach(e -> System.out.println(System.currentTimeMillis() + " -> " + e)), as);
done.thenAccept(x -> as.terminate());
}
}
I created a factory pattern in my class.
In this class I injected classes which implements Command interface based on incoming String parameter.
Factory class
#Component
#RequiredArgsConstructor
public class CommandFactory {
private final ACommand aCommand;
private final BCommand bCommand;
private final CCommand cCommand;
private final DCommand dCommand;
private final ECommand eCommand;
private final FCommand fCommand;
public Command createCommand(String content) {
if (aCommand.isMatching(content)) {
return aCommand;
} else if (bCommand.isMatching(content)) {
return bCommand;
} else if (cCommand.isMatching(content)) {
return cCommand;
} else if (dCommand.isMatching(content)) {
return dCommand;
} else if (eCommand.isMatching(content)) {
return eCommand;
} else if (fCommand.isMatching(content)) {
return fCommand;
} else {
return null;
}
}
In isMatching() method there are different regex'es and I try to figure out how this incoming String should be processed.
I am looking for a cleaner way to get rid of these sequential if statements. Because whenever I create a new class into this factory I add another if statement.
Maybe Stream can help?
Stream<Command> stream = Stream.of(aCommand, bCommand, cCommand ...);
return stream.filter(x -> x.isMatching(content)).findFirst().orElse(null);
Now every time you add a new class, you just add a new object to the first line.
If you want to get rid of the sequential if statements you can use streams (like user Sweeper suggested) or loops and I would also suggest to return and optional which makes null handling clearer for the client.
Here are two suggested options to get rid of if else repetitions one with loops another with streams:
import java.util.Arrays;
import java.util.List;
import java.util.Optional;
public class CommandPatternExample {
private List<Command> candidates = Arrays.asList(new ACommand(), new BCommand(), new CCommand());
public Optional<Command> createCommand(String content) {
for(Command command : candidates) {
if(command.isMatching(content)) {
return Optional.of(command);
}
}
return Optional.empty();
}
public Optional<Command> createCommandStream(String content) {
return candidates.stream().filter(c -> c.isMatching(content)).findFirst();
}
}
interface Command<T> {
void execute(T obj);
boolean isMatching(String s);
}
class ACommand implements Command<String> {
#Override
public void execute(String obj) {
}
#Override
public boolean isMatching(String s) {
return "A".equals(s);
}
}
class BCommand implements Command<String> {
#Override
public void execute(String obj) {
}
#Override
public boolean isMatching(String s) {
return "B".equals(s);
}
}
class CCommand implements Command<String> {
#Override
public void execute(String obj) {
}
#Override
public boolean isMatching(String s) {
return "C".equals(s);
}
}
Map might be a good idea. Meaning if you place your command instances into a map as values where your key would be something that you could match against incoming String. Then instead of sequential search with Efficiency O(n) you can get much better performance O(1). This is a short answer.
Besides that There is an open source java library MgntUtils (wriiten by me) that contains some utility called "Self-instantiating factories" Basically it manages and the Factory for you. All you will need to do is to create a class that implements a certain interface and the utility will add it for you into a map based factory. It might be useful to you. Here is the link to an article that explains about the utilities in the library as well as where to get the library (Github and Maven central). In the article look for the paragraph "Lifecycle management (Self-instantiating factories)". Also library comes with a detailed written javadoc and code example for that feature.
I was trying to create a parser based on Rovo's example (see below)
creating a simple rule engine in java.
The source code of Rovo's example:
https://github.com/RovoMe/RuleBasedEngine/tree/master/src/main/java/at/rovo/test/rulebasedengine
When I tried to add parenthesis function(I tried to create a parenthesis operator that extends Operation) to change the order of performance of the expression. It did not work as what I intended. I would like to know if anyone could give me a logic about how to achieve this goal?
My method is to push the left parenthesis to the stack until the iterator reads the right parenthesis. At that point, I will pop the expression and the left parenthesis and evaluate it the expression. However ,it does not work as what I think.
RightParenthesis
public class RightParenthesis extends Operation {
public RightParenthesis() {
super(")");
}
public RightParenthesis getOperator() {
return new RightParenthesis();
}
public int parse (String tokens[], int pos , Deque<Expression> deque) {
// Get expression in side the parenthesis
this.leftOperand =deque.pop();
deque.pop();
deque.push(this.leftOperand);
return pos+1;
}
public boolean interpret(Map<String,?> InputValueMap) {
return true;
} }
LeftParenthesis
public class LeftParenthesis extends Operation {
public LeftParenthesis() {
super("(");
}
public LeftParenthesis getOperator() {
return new LeftParenthesis();
}
public int parse (String[] tokens, int pos , Deque<Expression> deque) {
deque.push(this);
return pos+1;
}
public boolean interpret(Map<String,?> InputValueMap) {
return true;
}
public String toString() {
return this.getSymbol();
} }
I'm trying to generate a aggregate view of consecutive market data, which means we need to calculate the sum value every 2 message. say the data coming in as:
(V0,T0),(V1,T1),(V2,T2),(V3,T3)....
V means value T means timestamp when we receive the data.
We need to generate the sum for every 2 points say:
(R1=Sum(V0,V1),T1),(R2=Sum(V1,V2),T2),(R3=Sum(V2,V3),T3),....
Any suggestion how can we do this by using aggregator2 or we need to write a processor for this?
You are right, aggregator2 component is the good way to go. I would try something like that:
from("somewhere").split(body().tokenize("),")).streaming()
.aggregate(new ValueAggregationStrategy()).completionTimeout(1500)
.to("whatYouWant");
class ValueAggregationStrategy implements AggregationStrategy {
public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
if (oldExchange == null) {
return newExchange;
}
String oldBody = oldExchange.getIn().getBody(String.class);
String newBody = newExchange.getIn().getBody(String.class);
oldExchange.getIn().setBody(extractValue(oldBody) + extractValue(newBody));
return oldExchange;
}
public int extractValue(String body) {
// Do the work "(V0,T0" -> "V0"
}
}
NB: It would be easier to parse if you could have a format like that: V0,T0;V1,T1...
For more information: here is an article wrote by Claus Ibsen on parsing large file with Camel
After reading the source code of Aggregator, it turns out that camel only aggregate one message to one group, we have to build a "aggregator" for this purpose. here is the code:
public abstract class GroupingGenerator<I> implements Processor {
private final EvictingQueue<I> queue;
private final int size;
public int getSize() {
return size;
}
public GroupingGenerator(int size) {
super();
this.size = size;
this.queue = EvictingQueue.create(size);
}
#SuppressWarnings("unchecked")
#Override
public void process(Exchange exchange) throws Exception {
queue.offer((I) exchange.getIn().getBody());
if (queue.size() != size) {
exchange.setProperty(Exchange.ROUTE_STOP, true);
return;
} else {
processGroup(queue, exchange);
}
}
protected abstract void processGroup(Collection<I> items, Exchange exchange);
}
Consider the following code:
CLASS AuditProgressReport:
public class AuditProgressReport
{
private List<AuditProgressReport> audit_progress_reports = null;
private String name = null;
private String description = null;
private int compliant;
private int non_compliant;
private int not_completed ;
/**
*
*/
public AuditProgressReport()
{
super();
}
public AuditProgressReport(
String name_param,
int compliant_param,
int non_compliant_param,
int not_completed_param)
{
super();
this.name = name_param;
this.compliant = compliant_param;
this.non_compliant = non_compliant_param;
this.not_completed = not_completed_param;
}
public void addToCompliant(int compl_to_add_param)
{
this.compliant += compl_to_add_param;
}
public void addToNonCompliant(int non_compl_to_add_param)
{
this.non_compliant += non_compl_to_add_param;
}
public void addToNotCompleted(int not_compl_param)
{
this.not_completed += not_compl_param;
}
public void setAuditProgressReports(List<AuditProgressReport> report_category_nodes_param)
{
this.audit_progress_reports = report_category_nodes_param;
}
public List<AuditProgressReport> getAuditProgressReports()
{
return this.audit_progress_reports;
}
public void setCompliant(int compliantParam)
{
this.compliant = compliantParam;
}
public int getCompliant()
{
return this.compliant;
}
public void setNonCompliant(int nonCompliantParam)
{
this.non_compliant = nonCompliantParam;
}
public int getNonCompliant()
{
return this.non_compliant;
}
public void setNotCompleted(int notCompletedParam)
{
this.not_completed = notCompletedParam;
}
public int getNotCompleted()
{
return this.not_completed;
}
public void setName(String name_param)
{
this.name = name_param;
}
public String getName()
{
return this.name;
}
public void setDescription(String description_param)
{
this.description = description_param;
}
public String getDescription()
{
return this.description;
}
#Override
public String toString()
{
return ("Compliant["+this.compliant+
"] Non-Compliant["+this.non_compliant+
"] Not-Completed["+this.not_completed+"]");
}
}
And CLASS Tester:
public class Tester
{
public static void main(String[] args)
{
List<AuditProgressReport> main_level = new ArrayList<AuditProgressReport>();
AuditProgressReport ar_1_1 = new AuditProgressReport("ar_1_1",0,0,0);
AuditProgressReport ar_1_2 = new AuditProgressReport("ar_1_2",0,0,0);
AuditProgressReport ar_1_1_1 = new AuditProgressReport("ar_1_1_1",0,0,0);
AuditProgressReport ar_1_1_2 = new AuditProgressReport("ar_1_1_2",15,65,20);
AuditProgressReport ar_1_1_3 = new AuditProgressReport("ar_1_1_3",20,30,50);
AuditProgressReport ar_1_1_1_1 = new AuditProgressReport("ar_1_1_1_1",5,5,90);
AuditProgressReport ar_1_1_1_2 = new AuditProgressReport("ar_1_1_1_2",55,5,40);
AuditProgressReport ar_1_1_1_3 = new AuditProgressReport("ar_1_1_1_3",35,35,30);
List<AuditProgressReport> arl_1_1_1 = new ArrayList<AuditProgressReport>();
arl_1_1_1.add(ar_1_1_1_1);
arl_1_1_1.add(ar_1_1_1_2);
arl_1_1_1.add(ar_1_1_1_3);
ar_1_1_1.setAuditProgressReports(arl_1_1_1);
List<AuditProgressReport> arl_1_1 = new ArrayList<AuditProgressReport>();
arl_1_1.add(ar_1_1_1);
arl_1_1.add(ar_1_1_2);
arl_1_1.add(ar_1_1_3);
AuditProgressReport ar_1_2_1 = new AuditProgressReport("ar_1_2_1",10,30,60);
AuditProgressReport ar_1_2_2 = new AuditProgressReport("ar_1_2_2",20,20,60);
List<AuditProgressReport> arl_1_2 = new ArrayList<AuditProgressReport>();
arl_1_2.add(ar_1_2_1);
arl_1_2.add(ar_1_2_2);
ar_1_1.setAuditProgressReports(arl_1_1);
ar_1_2.setAuditProgressReports(arl_1_2);
main_level.add(ar_1_1);
main_level.add(ar_1_2);
Tester tester = new Tester();
for(AuditProgressReport prog_rep : main_level)
{
tester.populateParents(prog_rep, null);
}
//TODO Now check the values...
}
private void populateParents(
AuditProgressReport audit_progress_param,
AuditProgressReport parent_param)
{
List<AuditProgressReport> audit_progress =
audit_progress_param.getAuditProgressReports();
System.out.println("name["+audit_progress_param.getName()+"]");
if(parent_param != null)
{
int compl = audit_progress_param.getCompliant();
int nonCompl = audit_progress_param.getNonCompliant();
int notCompleted = audit_progress_param.getNotCompleted();
parent_param.addToCompliant(compl);
parent_param.addToNonCompliant(nonCompl);
parent_param.addToNotCompleted(notCompleted);
}
if(audit_progress != null && ! audit_progress.isEmpty())
{
for(AuditProgressReport prog_rep : audit_progress)
{
this.populateParents(prog_rep,audit_progress_param);
}
}
}
}
When you run this, you will note that the values of the parent elements in the list is updated with the sum of the values in the child list.
The problem I am facing is that I want to have it updated all the way through the tree instead of just the immediate parent.
Is there a pattern that would help me achieve this?
See illustration below:
Like others suggested I would use the Observer pattern. Each parent node listens for changes on the childrens.
But my solution differs from that of #zmf because if you have a a big tree with lot of children node and at each update you have to sum each value, you would spend a lot of processing time.
What if you send only the difference between the old value and the new value each time you update a child node. Let's make an example. You start with this tree:
[12]--+--[10]-----[10]
|
+--[ 2]--+--[ ]
|
+--[ 2]
and you update a children like this
[12]--+--[10]-----[10]
|
+--[ 2]--+--[ 3]
|
+--[ 2]
the node that gets updated with the value "3" send its change to the parent with the method call parent.updateNode(3). The parent have only to sum its current value (in this example "2") with the value it receives from the child node. So it will update to the value "5"
[12]--+--[10]-----[10]
|
+--[ 5]--+--[ 3]
|
+--[ 2]
the node with the new value "5" will call parent.updateNode(3) and the final solution will be
[15]--+--[10]-----[10]
|
+--[ 5]--+--[ 3]
|
+--[ 2]
IMHO this solution is better because each updateNode() method have only to sum its own current value with the change received from its child node and call its parent with the same value received. You do not have to get the value from each one of your children and sum all the values. This will save you a lot of time if you have a big tree. So in this example when you change the value from 0 to 3. You will get 2 call to parent.updateNode(3) and each parent will get updated.
public void updateNode(int value) {
if (value != this.value) {
this.value = value;
if (getParent() != null) {
int sum = 0;
for (Node n : getParent().getChildren()) {
sum += n.getValue();
}
getParent.updateNode(sum);
}
}
}
Other poster's suggested the use of the Observer pattern. The Observer pattern is a subset of a Pub/Sub pattern. I recommend using this over an Observer pattern.
The main difference between an Observer pattern and a Pub/Sub pattern is that in an Observer pattern, an Observer is both a publisher of ChangeEvents and a dispatcher of messages. It's essentially making every Observable into an EventDispatcher. In a traditional Pub/Sub pattern, Observables are only publisher's of ChangeEvents. ChangeEvents are published to a separate EventDispatchingService which handles what Subscribers the Events need to be sent to.
Attempting to track global changes with an Observer pattern is difficult to do. For example, if you want to count the number of times time the addToCompliant() method was called, you would have to add the Observer on every instance of the Observable. With an Event Pub/Sub, your observer class can just subscribe to listen on the type of ChangeEvent and it will receive all of them. The best (IMHO) Event Pub/Sub library I've used is Google Guava's Event Bus. In your particular case, I'd do something like the following.
public class EventBusSingleton {
public static final EventBus INSTANCE = new EventBus("My Event Bus");
}
public class ComplianceChange {
private AuditProgressReport changedReport;
private int delta;
public ComplianceChange(AuditProgressReport changedReport, int delta) {
this.changedReport = changedReport;
this.delta = delta;
}
...
}
public class AuditProgressReport {
...
private AuditProgressReport parent;
public AuditProgressReport getParent() {
return parent;
}
public void addToCompliant(int delta) {
this.compliant += delta;
ComplianceChange change = new ComplianceChange(this, delta);
EventBusSingleton.INSTANCE.post(change);
}
...
}
public class ComplianceChangeHandler {
#Subscribe
public void notifyParent(ComplianceChange event) {
AuditProgressReport parent = event.getChangedReport().getParent();
int delta = event.getDelta();
parent.addToCompliant(delta);
}
#Subscribe
public void somethingElse(ComplianceChange event) {
// Do Something Else
}
}
// Somewhere during initialization
EventBusSingleton.INSTANCE.register(new ComplianceChangeHandler());
Based on your class name, I guess you want to see your audit progression live when running. So my hypothesis:
the tree structure does not change too much, almost fixed after creation
node values change often, counters initial states are 0
Here is an efficient implementation:
each node maintains the full list of its parent nodes
nodes are inserted with 0 value
when a node value is changed or simply increased, the parents' values from the node's list are updated by applying the delta between the previous node value
As a consequence the structure is always up-to-date, node insertion is still possible and does not impact the existing nodes.
If many audit threads run concurrently and report values into the structure, you have to take care to concurrency issues and use AtomicInteger as counter holders.
This is a pragmatic design and sincerely I have not found any matching pattern. Like for sort algorithms, trying to use patterns in such a context may be counter-productive.