We Keep Coding

Is there any way to get Akka Streams' groupedWithin to emit empty groups?

In the following code, tick emits a new object every three seconds. I'm trying to count the number of emitted objects every second using groupedWithin (which ignores empty groups). Is there any way in Akka Streams for the following code to print 0 in periods when tick does not emit any objects?
Source.tick(Duration.ZERO, Duration.ofSeconds(3), new Object())
.groupedWithin(Integer.MAX_VALUE, Duration.ofSeconds(1))
.map(List::size)
.runWith(Sink.foreach(e -> System.out.println(e)), materializer);
In other words, I'd like the output of this code to be this sequence: 1 0 0 1 0 0 1 ... (every second) instead of 1 1 1 ... (every three seconds).
EDIT: This is the best workaround I have come up with so far (using keepAlive to send some special objects if the upstream is idle):
Source.tick(Duration.ZERO, Duration.ofSeconds(3), new Object())
.keepAlive(Duration.ofSeconds(1), KeepAliveElement::new)
.groupedWithin(Integer.MAX_VALUE, Duration.ofSeconds(1))
.map(lst -> lst.stream().filter(e -> !(e instanceof KeepAliveElement)).collect(Collectors.toList()))
.map(List::size)
.runWith(Sink.foreach(e -> System.out.println(e)), materializer);
Is there any better way to do this?

I thought this would be of normal difficulty, I was wrong. One thing I wanted to do is to ensure that the flow counting items that pass through the stream does not keep a reference to each item it sees: if many items pass in the aggregation period, you will end up with an unnecessarily big list in memory (even if only for a second) and the performance penalty to add (many) items to it. The following solution, although complex, keeps only a counter.
NOTE: Although I tested the happy scenario, I cannot say this is battle-proven, so use with caution!
Based on Akka's GroupedWeightedWithin and the documentation here:
public class CountInPeriod<T> extends GraphStage<FlowShape<T, Integer>> {
public Inlet<T> in = Inlet.<T>create("CountInPeriod.in");
public Outlet<Integer> out = Outlet.<Integer>create("CountInPeriod.out");
private FlowShape<T, Integer> shape = FlowShape.of(in, out);
private Duration duration;
public CountInPeriod(Duration duration) {
this.duration = duration;
}
#Override
public GraphStageLogic createLogic(Attributes inheritedAttributes) {
return new TimerGraphStageLogic(shape) {
private int counter = 0;
private int bufferPushCounter = -1;
{
setHandler(in, new AbstractInHandler() {
#Override public void onPush() throws Exception, Exception {
grab(in);
counter++;
pull(in);
}
});
setHandler(out, new AbstractOutHandler() {
#Override public void onPull() throws Exception, Exception {
if (bufferPushCounter >= 0) {
push(out, bufferPushCounter);
bufferPushCounter = -1;
}
}
});
}
#Override
public void preStart() throws Exception, Exception {
scheduleWithFixedDelay(CountInPeriod.class, duration, duration);
pull(in);
}
#Override
public void onTimer(Object timerKey) throws Exception, Exception {
if (isAvailable(out)) emitCounter();
else bufferPush();
}
private void emitCounter() {
push(out, counter);
counter = 0;
bufferPushCounter = -1;
}
private void bufferPush() {
bufferPushCounter = counter;
counter = 0;
}
};
}
#Override
public FlowShape<T, Integer> shape() {
return shape;
}
}
Test code:
public class GroupTicked {
final static ActorSystem as = ActorSystem.create("as");
public static void main(String... args) throws Exception {
CompletionStage<Done> done = Source.tick(Duration.ZERO, Duration.ofSeconds(3), new Object())
.take(7) // to finish in finite time...
.via(new CountInPeriod<>(Duration.ofSeconds(1)))
.runWith(Sink.foreach(e -> System.out.println(System.currentTimeMillis() + " -> " + e)), as);
done.thenAccept(x -> as.terminate());
}
}

Abstract Generic Object class

Hi
In the game Minecraft you have to send data between the client and the server in order to sync stuff, one of the things that need to be synced in particular circumstances is TileEntities with a render. The data is stored on the server and sent to the client which is thereafter used for render.
You create a class that contains the data that needs to be synced and attach it to a channel creation API called "SimpleImpl". My Network object and a Message Object:
public class IntercraftPacketHandler
{
private static int index = 1;
private static final ResourceLocation CHANNEL_NAME = new ResourceLocation(Reference.MODID,"network");
private static final String PROTOCOL_VERSION = new ResourceLocation(Reference.MODID,"1").toString();
public static SimpleChannel getNetworkChannel()
{
final SimpleChannel channel = NetworkRegistry.ChannelBuilder.named(CHANNEL_NAME)
.clientAcceptedVersions(version -> true)
.serverAcceptedVersions(version -> true)
.networkProtocolVersion(() -> PROTOCOL_VERSION)
.simpleChannel();
// Sync Capability Identity Hidden data message.
channel.messageBuilder(MessageIdentityHidden.class,index)
.encoder(MessageIdentityHidden::encode)
.decoder(MessageIdentityHidden::decode)
.consumer(MessageIdentityHidden::handle)
.add(); index++;
// Send TreeTapTileEntity data to client.
channel.messageBuilder(MessageTreeTap.class,index)
.encoder(MessageTreeTap::encode)
.decoder(MessageTreeTap::decode)
.consumer(MessageTreeTap::handle)
.add(); index++;
// Send ChunkLoaderTileEntity data to client.
channel.messageBuilder(MessageChunkLoader.class,index)
.encoder(MessageChunkLoader::encode)
.decoder(MessageChunkLoader::decode)
.consumer(MessageChunkLoader::handle)
.add(); index++;
return channel;
}
}
public class MessageChunkLoader
{
private BlockPos pos;
private boolean canLoad;
public MessageChunkLoader(BlockPos pos,boolean canLoad)
{
this.pos = pos;
this.canLoad = canLoad;
}
public void handle(Supplier<NetworkEvent.Context> ctx)
{
ctx.get().enqueueWork(() -> {
try {
ChunkLoaderBaseTileEntity tile = (ChunkLoaderBaseTileEntity) Minecraft.getInstance().world.getTileEntity(pos);
tile.canLoad = canLoad;
} catch (NullPointerException err) {
System.out.println(String.format("Could not find ChunkLoaderTileEntity at %s %s %s!",pos.getX(),pos.getY(),pos.getZ()));
}
});
}
public static void encode(MessageChunkLoader message, PacketBuffer buffer)
{
buffer.writeBlockPos(message.pos);
buffer.writeBoolean(message.canLoad);
}
public static MessageChunkLoader decode(final PacketBuffer buffer)
{
return new MessageChunkLoader(buffer.readBlockPos(),buffer.readBoolean());
}
}
I then initialize it in my main mod class used by objects in my mod project.
#Mod(Reference.MODID)
public class IntercraftCore
{
public static final SimpleChannel NETWORK = IntercraftPacketHandler.getNetworkChannel();
...
The problem and this post's question; right now I create a new message class from the formula I follow in MessageChunkLoader (public static encode & decode method and a handle method). I would like to create a more generic class for creating message classes for TileEntities, but I'm having problems with that. Here's the current class:
public abstract class MessageTileEntity<T extends TileEntity>
{
protected final BlockPos pos;
protected final Class<T> clazz;
public MessageTileEntity(BlockPos pos, Class<T> clazz)
{
this.pos = pos;
this.clazz = clazz;
}
public abstract void handle(Supplier<NetworkEvent.Context> ctx);
protected T getTileEntity()
{
try {
return clazz.cast(Minecraft.getInstance().world.getTileEntity(pos));
} catch (NullPointerException e) {
System.out.println(String.format("Could not find %s at [%d %d %d]!",clazz.getSimpleName(),pos.getX(),pos.getY(),pos.getZ()));
throw e;
} catch (ClassCastException e) {
System.out.println(String.format("TileEntity at [%d %d %d] is not %s!",pos.getX(),pos.getY(),pos.getZ(),clazz.getSimpleName()));
throw e;
}
}
public static void encode(MessageTileEntity message, PacketBuffer buffer)
{
}
public static MessageTileEntity decode(final PacketBuffer buffer)
{
return null;
}
}
The main problem is I lack the proper Java skills to make it like I want it to function. The method handle is easy as it's non-static and needs to be custom to every TileEntity message, but the methods encode and decode which needs to be static gives me problems. I have no idea what I'm trying to achieve is possible, asking won't hurt. Maybe the solution is easier than I think.

Take elements until a certain character and group them with RxJava

I have a simple setup to a problem but the solution seems to be more complicated.
Setup: I have a hot observable which originates from a scanner that will emit every number as a different element and an R when a code is complete.
Problem: From this I want a hot observable that emits every full code as 1 element.
I tried playing around with different flatMap, takeUntil and groupByoperators but haven't been able to come to a solution.

You can use the buffer operator.
PublishSubject<Token<Integer>> s = PublishSubject.create();
Observable<Token<Integer>> markers = s.filter(x->x.isMarker());
s.buffer(markers).subscribe(
v->{
Optional<Integer> reduce = v.stream()
.filter(t->!t.isMarker())
.map(t->(ValueToken<Integer>)t)
.map(ValueToken::get)
.reduce((a,b)->a+b);
reduce.ifPresent(System.out::println);
}
);
s.onNext(value(12));
s.onNext(value(13));
s.onNext(marker()); // will emit 25
s.onNext(value(10));
s.onNext(value(7));
s.onNext(marker()); // will emit 17
s.onNext(value(10));
s.onNext(value(7)); // Not emitting yet
I made a class to wrap both values and markers in the flow.
public abstract class Token<T> {
private static final MarkerToken MARKER = new MarkerToken<>();
public boolean isMarker() {
return false;
}
public static <T> MarkerToken<T> marker() {
return MARKER;
}
public static <T> ValueToken<T> value(T o) {
return new ValueToken<>(o);
}
public static class ValueToken<T> extends Token<T> {
T value;
public ValueToken(T value) {
this.value = value;
}
public T get() {
return value;
}
}
public static class MarkerToken<T> extends Token<T> {
public boolean isMarker() {
return true;
}
}
}
update (using scan)
The previous method would emit also on the closing of the stream, with this solution you can emit only complete buffers.
The message class function as an accumulator, it will accumulate tokens until closing marker is accumulated.
When this happens the next message will start from scratch.
The presence of the closing mark as last element marks the message as complete.
public static class Message<T> {
List<Token<T>> tokens = new ArrayList<>();
public Message<T> append(Token<T> t) {
Message<T> mx = new Message<T>();
if(!isComplete()) {
mx.tokens.addAll(tokens);
}
mx.tokens.add(t);
return mx;
}
public boolean isComplete() {
int n = tokens.size();
return n>0 && tokens.get(n-1).isMarker();
}
public Optional<List<Token<T>>> fullMessage(){
return isComplete() ? Optional.of(tokens):Optional.empty();
}
}
Scanning the source you emit a message for each token emitted, then you filter out incomplete message and emit just the one marked as complete.
s.scan(new Message<Integer>(), (a, b) -> a.append(b))
.filter(Message::isComplete)
.map(Message::fullMessage)
.map(Optional::get).subscribe(v -> {
System.out.println(v);
});
s.onNext(value(12));
s.onNext(value(13));
s.onNext(marker());// [V(12), V(13), MARKER]
s.onNext(value(10));
s.onNext(value(7));
s.onNext(marker()); // [V(10), V(7), MARKER]
s.onNext(value(10));
s.onNext(value(127));
s.onComplete(); // Not emitting incomplete messages on the closing of the subject.

Spring batch, logging Skipped records / lines

In order to be able to log all skipped record / lines (from Flat file), I have linked Item (Data model) to FieldSet.getValues() in FieldSetMapper<T> implementation.
Here record is my Data model.
public void loadRecord(FieldSet fieldSet){
record.setFieldSet(fieldSet.getValues());
}
SB provides record/line for onSkipInRead in the FlatFileParseException. To have the same for onSkipInWrite and onSkipInProcess, I came up with the above solution.
Also, record does not store all the fields from the flat file, hence I can't each line using just the Data model.
Is this a good way to acheive this ?
Are there any other ways to do this ?
Is the record / line corresponding to an Item stored in any context ?
Thanks for the help !

Now I understand better your problem :)
Save the whole line into your domain object and not the FieldSet using a custom LineMapper.From LineMapper javadoc:
Interface for mapping lines (strings) to domain objects typically used
to map lines read from a file to domain objects on a per line basis.
Implementations of this interface perform the actual work of parsing a
line without having to deal with how the line was obtained.
So with a delegate is pretty easy solve your problem:
class StoreLineLineMapper implements LineMapper<DomainObject> {
private LineMapper<DomainObject> delegate;
public DomainObject mapLine(java.lang.String line, int lineNumber) throws java.lang.Exception {
DomainObject record = delegate.mapLine(line, lineNumber);
record.setLineInfo(new LineInfo(line, lineNumber));
return record;
}
}

The complete code for this use case:
public class UserMapper implements FieldSetMapper<User> {
#Override
public User mapFieldSet(FieldSet fieldSet) throws BindException {
User user = new User();
String ped = fieldSet.readString("Position effective date");
user.setId(fieldSet.readInt("User ID"));
user.setFn(fieldSet.readString("First Name"));
user.loadRecord(fieldSet); // store fieldSet for regenerating line in future
return user;
}
}
This will regenerate line from fieldSet (Delimiter is TAB):
class User{
#Override
public String toString() {
return StringUtils.arrayToDelimitedString(fieldSet,
DelimitedLineTokenizer.DELIMITER_TAB);
}
}
Log lines in SkipListener like this:
class SkipListener {
public static final Logger logger = LoggerFactory.getLogger(SkipListener.class);
#OnSkipInWrite
public void onSkipInWrite(Object item, Throwable t) {
onSkip(item);
}
#OnSkipInRead
public void onSkipInRead(Throwable t) {
if (t instanceof FlatFileParseException) {
FlatFileParseException ffpe = (FlatFileParseException) t;
onSkip(ffpe.getInput());
}
}
#OnSkipInProcess
public void onSkipInProcess(Object item, Throwable t) {
onSkip(item);
}
public void onSkip(Object item) {
logger.info(item);
}
}

Populate parent List elements based on child values

Consider the following code:
CLASS AuditProgressReport:
public class AuditProgressReport
{
private List<AuditProgressReport> audit_progress_reports = null;
private String name = null;
private String description = null;
private int compliant;
private int non_compliant;
private int not_completed ;
/**
*
*/
public AuditProgressReport()
{
super();
}
public AuditProgressReport(
String name_param,
int compliant_param,
int non_compliant_param,
int not_completed_param)
{
super();
this.name = name_param;
this.compliant = compliant_param;
this.non_compliant = non_compliant_param;
this.not_completed = not_completed_param;
}
public void addToCompliant(int compl_to_add_param)
{
this.compliant += compl_to_add_param;
}
public void addToNonCompliant(int non_compl_to_add_param)
{
this.non_compliant += non_compl_to_add_param;
}
public void addToNotCompleted(int not_compl_param)
{
this.not_completed += not_compl_param;
}
public void setAuditProgressReports(List<AuditProgressReport> report_category_nodes_param)
{
this.audit_progress_reports = report_category_nodes_param;
}
public List<AuditProgressReport> getAuditProgressReports()
{
return this.audit_progress_reports;
}
public void setCompliant(int compliantParam)
{
this.compliant = compliantParam;
}
public int getCompliant()
{
return this.compliant;
}
public void setNonCompliant(int nonCompliantParam)
{
this.non_compliant = nonCompliantParam;
}
public int getNonCompliant()
{
return this.non_compliant;
}
public void setNotCompleted(int notCompletedParam)
{
this.not_completed = notCompletedParam;
}
public int getNotCompleted()
{
return this.not_completed;
}
public void setName(String name_param)
{
this.name = name_param;
}
public String getName()
{
return this.name;
}
public void setDescription(String description_param)
{
this.description = description_param;
}
public String getDescription()
{
return this.description;
}
#Override
public String toString()
{
return ("Compliant["+this.compliant+
"] Non-Compliant["+this.non_compliant+
"] Not-Completed["+this.not_completed+"]");
}
}
And CLASS Tester:
public class Tester
{
public static void main(String[] args)
{
List<AuditProgressReport> main_level = new ArrayList<AuditProgressReport>();
AuditProgressReport ar_1_1 = new AuditProgressReport("ar_1_1",0,0,0);
AuditProgressReport ar_1_2 = new AuditProgressReport("ar_1_2",0,0,0);
AuditProgressReport ar_1_1_1 = new AuditProgressReport("ar_1_1_1",0,0,0);
AuditProgressReport ar_1_1_2 = new AuditProgressReport("ar_1_1_2",15,65,20);
AuditProgressReport ar_1_1_3 = new AuditProgressReport("ar_1_1_3",20,30,50);
AuditProgressReport ar_1_1_1_1 = new AuditProgressReport("ar_1_1_1_1",5,5,90);
AuditProgressReport ar_1_1_1_2 = new AuditProgressReport("ar_1_1_1_2",55,5,40);
AuditProgressReport ar_1_1_1_3 = new AuditProgressReport("ar_1_1_1_3",35,35,30);
List<AuditProgressReport> arl_1_1_1 = new ArrayList<AuditProgressReport>();
arl_1_1_1.add(ar_1_1_1_1);
arl_1_1_1.add(ar_1_1_1_2);
arl_1_1_1.add(ar_1_1_1_3);
ar_1_1_1.setAuditProgressReports(arl_1_1_1);
List<AuditProgressReport> arl_1_1 = new ArrayList<AuditProgressReport>();
arl_1_1.add(ar_1_1_1);
arl_1_1.add(ar_1_1_2);
arl_1_1.add(ar_1_1_3);
AuditProgressReport ar_1_2_1 = new AuditProgressReport("ar_1_2_1",10,30,60);
AuditProgressReport ar_1_2_2 = new AuditProgressReport("ar_1_2_2",20,20,60);
List<AuditProgressReport> arl_1_2 = new ArrayList<AuditProgressReport>();
arl_1_2.add(ar_1_2_1);
arl_1_2.add(ar_1_2_2);
ar_1_1.setAuditProgressReports(arl_1_1);
ar_1_2.setAuditProgressReports(arl_1_2);
main_level.add(ar_1_1);
main_level.add(ar_1_2);
Tester tester = new Tester();
for(AuditProgressReport prog_rep : main_level)
{
tester.populateParents(prog_rep, null);
}
//TODO Now check the values...
}
private void populateParents(
AuditProgressReport audit_progress_param,
AuditProgressReport parent_param)
{
List<AuditProgressReport> audit_progress =
audit_progress_param.getAuditProgressReports();
System.out.println("name["+audit_progress_param.getName()+"]");
if(parent_param != null)
{
int compl = audit_progress_param.getCompliant();
int nonCompl = audit_progress_param.getNonCompliant();
int notCompleted = audit_progress_param.getNotCompleted();
parent_param.addToCompliant(compl);
parent_param.addToNonCompliant(nonCompl);
parent_param.addToNotCompleted(notCompleted);
}
if(audit_progress != null && ! audit_progress.isEmpty())
{
for(AuditProgressReport prog_rep : audit_progress)
{
this.populateParents(prog_rep,audit_progress_param);
}
}
}
}
When you run this, you will note that the values of the parent elements in the list is updated with the sum of the values in the child list.
The problem I am facing is that I want to have it updated all the way through the tree instead of just the immediate parent.
Is there a pattern that would help me achieve this?
See illustration below:

Like others suggested I would use the Observer pattern. Each parent node listens for changes on the childrens.
But my solution differs from that of #zmf because if you have a a big tree with lot of children node and at each update you have to sum each value, you would spend a lot of processing time.
What if you send only the difference between the old value and the new value each time you update a child node. Let's make an example. You start with this tree:
[12]--+--[10]-----[10]
|
+--[ 2]--+--[ ]
|
+--[ 2]
and you update a children like this
[12]--+--[10]-----[10]
|
+--[ 2]--+--[ 3]
|
+--[ 2]
the node that gets updated with the value "3" send its change to the parent with the method call parent.updateNode(3). The parent have only to sum its current value (in this example "2") with the value it receives from the child node. So it will update to the value "5"
[12]--+--[10]-----[10]
|
+--[ 5]--+--[ 3]
|
+--[ 2]
the node with the new value "5" will call parent.updateNode(3) and the final solution will be
[15]--+--[10]-----[10]
|
+--[ 5]--+--[ 3]
|
+--[ 2]
IMHO this solution is better because each updateNode() method have only to sum its own current value with the change received from its child node and call its parent with the same value received. You do not have to get the value from each one of your children and sum all the values. This will save you a lot of time if you have a big tree. So in this example when you change the value from 0 to 3. You will get 2 call to parent.updateNode(3) and each parent will get updated.

public void updateNode(int value) {
if (value != this.value) {
this.value = value;
if (getParent() != null) {
int sum = 0;
for (Node n : getParent().getChildren()) {
sum += n.getValue();
}
getParent.updateNode(sum);
}
}
}

Other poster's suggested the use of the Observer pattern. The Observer pattern is a subset of a Pub/Sub pattern. I recommend using this over an Observer pattern.
The main difference between an Observer pattern and a Pub/Sub pattern is that in an Observer pattern, an Observer is both a publisher of ChangeEvents and a dispatcher of messages. It's essentially making every Observable into an EventDispatcher. In a traditional Pub/Sub pattern, Observables are only publisher's of ChangeEvents. ChangeEvents are published to a separate EventDispatchingService which handles what Subscribers the Events need to be sent to.
Attempting to track global changes with an Observer pattern is difficult to do. For example, if you want to count the number of times time the addToCompliant() method was called, you would have to add the Observer on every instance of the Observable. With an Event Pub/Sub, your observer class can just subscribe to listen on the type of ChangeEvent and it will receive all of them. The best (IMHO) Event Pub/Sub library I've used is Google Guava's Event Bus. In your particular case, I'd do something like the following.
public class EventBusSingleton {
public static final EventBus INSTANCE = new EventBus("My Event Bus");
}
public class ComplianceChange {
private AuditProgressReport changedReport;
private int delta;
public ComplianceChange(AuditProgressReport changedReport, int delta) {
this.changedReport = changedReport;
this.delta = delta;
}
...
}
public class AuditProgressReport {
...
private AuditProgressReport parent;
public AuditProgressReport getParent() {
return parent;
}
public void addToCompliant(int delta) {
this.compliant += delta;
ComplianceChange change = new ComplianceChange(this, delta);
EventBusSingleton.INSTANCE.post(change);
}
...
}
public class ComplianceChangeHandler {
#Subscribe
public void notifyParent(ComplianceChange event) {
AuditProgressReport parent = event.getChangedReport().getParent();
int delta = event.getDelta();
parent.addToCompliant(delta);
}
#Subscribe
public void somethingElse(ComplianceChange event) {
// Do Something Else
}
}
// Somewhere during initialization
EventBusSingleton.INSTANCE.register(new ComplianceChangeHandler());

Based on your class name, I guess you want to see your audit progression live when running. So my hypothesis:
the tree structure does not change too much, almost fixed after creation
node values change often, counters initial states are 0
Here is an efficient implementation:
each node maintains the full list of its parent nodes
nodes are inserted with 0 value
when a node value is changed or simply increased, the parents' values from the node's list are updated by applying the delta between the previous node value
As a consequence the structure is always up-to-date, node insertion is still possible and does not impact the existing nodes.
If many audit threads run concurrently and report values into the structure, you have to take care to concurrency issues and use AtomicInteger as counter holders.
This is a pragmatic design and sincerely I have not found any matching pattern. Like for sort algorithms, trying to use patterns in such a context may be counter-productive.