We Keep Coding

Buffer and flush Apache Beam streaming data

I have a streaming job that with initial run will have to process large amount of data. One of DoFn calls remote service that supports batch requests, so when working with bounded collections I use following approach:
private static final class Function extends DoFn<String, Void> implements Serializable {
private static final long serialVersionUID = 2417984990958377700L;
private static final int LIMIT = 500;
private transient Queue<String> buffered;
#StartBundle
public void startBundle(Context context) throws Exception {
buffered = new LinkedList<>();
}
#ProcessElement
public void processElement(ProcessContext context) throws Exception {
buffered.add(context.element());
if (buffered.size() > LIMIT) {
flush();
}
}
#FinishBundle
public void finishBundle(Context c) throws Exception {
// process remaining
flush();
}
private void flush() {
// build batch request
while (!buffered.isEmpty()) {
buffered.poll();
// do something
}
}
}
Is there a way to window data so the same approach can be used on unbounded collections?
I've tried following:
pipeline
.apply("Read", Read.from(source))
.apply(WithTimestamps.of(input -> Instant.now()))
.apply(Window.into(FixedWindows.of(Duration.standardMinutes(2L))))
.apply("Process", ParDo.of(new Function()));
but startBundle and finishBundle are called for every element. Is there a chance to have something like with RxJava (2 minute windows or 100 element bundles):
source
.toFlowable(BackpressureStrategy.LATEST)
.buffer(2, TimeUnit.MINUTES, 100)

This is a quintessential use case for the new feature of per-key-and-windows state and timers.
State is described in a Beam blog post, while for timers you'll have to rely on the Javadoc. Nevermind what the javadoc says about runners supporting them, the true status is found in Beam's capability matrix.
The pattern is very much like what you have written, but state allows it to work with windows and also across bundles, since they may be very small in streaming. Since state must be partitioned somehow to maintain parallelism, you'll need to add some sort of key. Currently there is no automatic sharding for this.
private static final class Function extends DoFn<KV<Key, String>, Void> implements Serializable {
private static final long serialVersionUID = 2417984990958377700L;
private static final int LIMIT = 500;
#StateId("bufferedSize")
private final StateSpec<Object, ValueState<Integer>> bufferedSizeSpec =
StateSpecs.value(VarIntCoder.of());
#StateId("buffered")
private final StateSpec<Object, BagState<String>> bufferedSpec =
StateSpecs.bag(StringUtf8Coder.of());
#TimerId("expiry")
private final TimerSpec expirySpec = TimerSpecs.timer(TimeDomain.EVENT_TIME);
#ProcessElement
public void processElement(
ProcessContext context,
BoundedWindow window,
#StateId("bufferedSize") ValueState<Integer> bufferedSizeState,
#StateId("buffered") BagState<String> bufferedState,
#TimerId("expiry") Timer expiryTimer) {
int size = firstNonNull(bufferedSizeState.read(), 0);
bufferedState.add(context.element().getValue());
size += 1;
bufferedSizeState.write(size);
expiryTimer.set(window.maxTimestamp().plus(allowedLateness));
if (size > LIMIT) {
flush(context, bufferedState, bufferedSizeState);
}
}
#OnTimer("expiry")
public void onExpiry(
OnTimerContext context,
#StateId("bufferedSize") ValueState<Integer> bufferedSizeState,
#StateId("buffered") BagState<String> bufferedState) {
flush(context, bufferedState, bufferedSizeState);
}
private void flush(
WindowedContext context,
BagState<String> bufferedState,
ValueState<Integer> bufferedSizeState) {
Iterable<String> buffered = bufferedState.read();
// build batch request from buffered
...
// clear things
bufferedState.clear();
bufferedSizeState.clear();
}
}
Taking a few notes here:
State replaces your DoFn's instance variables, since
instance variables have no cohesion across windows.
The buffer and the size are just initialized as needed instead
of #StartBundle.
The BagState supports "blind" writes, so there doesn't need to be
any read-modify-write, just committing the new elements in the same
way as when you output.
Setting a timer repeatedly for the same time is just fine;
it should mostly be a noop.
#OnTimer("expiry") takes the place of #FinishBundle, since
finishing a bundle is not a per-window thing but an artifact of
how a runner executes your pipeline.
All that said, if you are writing to an external system, perhaps you would want to reify the windows and re-window into the global window before just doing writes where the manner of your write depends on the window, since "the external world is globally windowed".

The documentation for apache beam 0.6.0 says that StateId is "Not currently supported by any runner."

How can I switch perspective programmatically after starting an E4 application?

Scenario
I have a pure E4 application in which I want to select the initial perspective for the user depending on the user's roles. I therefore have a perspective to start with which contains one part only. In that part, I use the #PostConstruct-Method to check the user's roles and afterwards trigger the command for switching perspective:
Initial View
#Inject
private IEclipseContext eclipseContext;
#PostConstruct
public void initialize() {
// checking credentials and retrieving roles come here which is pretty long
// that's why switching perspective is a seperate method
// and EclipseContext is injected to instance instead of method
this.switchPerspective(_usersInitialPerspectiveId)
}
private void switchPerspective(String pTargetPerspectiveId) {
final ECommandService _commandService = this.eclipseContext.get(ECommandService.class);
final EHandlerService _handlerService = this.eclipseContext.get(EHandlerService.class);
final Map<String, Object> _commandParameter = new HashMap<>();
_commandParameter.put(PluginIdConstants.ID_OF_PARAMETER_FOR_SWITCH_PERSPEKTIVE,
pZielPerspektiveId);
final ParameterizedCommand _switchPerspectiveCommand =
_commandService.createCommand(COMMAND_ID_FOR_SWITCH_PERSPECTIVE,
_commandParameter);
_handlerService.executeHandler(_switchPerspectiveCommand);
}
For switching perspective from here, I use the exact same handler as from menu items configured in Application.e4xmi, which looks like this:
Perspective Switch Handler
#Execute
public void execute(final MWindow pWindow,
final EPartService pPartService,
final EModelService pModelService,
#Named(PluginIdConstants.ID_OF_PARAMETER_FOR_SWITCH_PERSPEKTIVE)
final String pPerspectiveId) {
final List<MPerspective> _perspectives =
pModelService.findElements(pWindow, pPerspectiveId, MPerspective.class, null);
if (!(_perspectives.isEmpty())) {
// Show perspective for looked up id
pPartService.switchPerspective(_perspectives.get(0));
}
}
The Problem
The problem is pretty simple: When using the above handler triggered by a menu item, it works as intended and switches perspective. But using the same handler by my initial view (triggering it programmatically) does not switch perspective. I debugged the code to check if the handler gets identical information in both cases and it does.
Maybe my application has not finished starting and that's why the handler has no effect, but if this is the problem, how can I check this?
Any ideas on what I maybe missed are welcome!

Based on Christoph Keimel's hint I could create a working solution (thank you very much!). Here's the code that solves the problem:
#ProcessAdditions
private void switchPerspective(final MApplication pApplication,
final IApplicationContext pApplicationContext,
final EModelService pModelService) {
final MWindow _window =
(MWindow) pModelService.find(PluginIdConstants.WINDOW_ID_FOR_MAIN, pApplication);
final String _appName = pApplicationContext.getBrandingName();
initializeWindowTitle(_window, _appName);
final MPerspectiveStack pPerspectiveStack =
(MPerspectiveStack) pModelService.find(PluginIdConstants.PERSPECTIVE_STACK_ID_FOR_MAIN,
pAnwendung);
for (final MPerspective _perspective : pPerspectiveStack.getChildren()) {
if (_perspektive.getElementId().equalsIgnoreCase(this.startingPerspectiveId)) {
pPerspectiveStack.setSelectedElement(_perspective);
break;
}
}
}
On how to register a LifeCycleHandler you can take a look at Lars Vogel's Tutorial.
My main problem finding this solution was how to access the perspective stack. As the UI is not up while the method annotated with ProcessAdditions is running, I have to access the application model via the MApplication type - which is the root element of my application model. Combining the EModelService I can access all UI elements I want and manipulate them accordingly.
Injecting any UI element like the MPerspectiveStack or the MWindow leads to a skipped method as these result in null values due to not being initalized yet.

Creating Annotations in Java like Mockito's before

In processing I have several functions that change the properties of applet to draw stuff, for instance:
public void resetBackground(PApplet pApplet){
pApplet.fill(175);
pApplet.noStroke();
pApplet.rect(0,0,100,100);
}
But I want these functions to preserve the state of the pApplet after the function call, for that I have something like:
public void resetBackground(PApplet pApplet){
SaveAndRestoreDefaults saveAndRestoreDefaults = new SaveAndRestoreDefaults(pApplet);
// Code that changes state.
saveAndRestoreDefaults.restoreOriginals();
}
Now this works for me but I would like this not to clutter my code here but rather be annotation driven, something like:
#PreserveState
public void resetBackground(){
// code that changes state.
}
I have done a little research on it but it seems to be not an easy task. The googling took me to AOP and I don't want to spend time to learn that. Is there an easier way to achieve the same?
Thanks :)

I'd strongly recommend staying in Processing, instead of reaching into the underlying virtual machine API (just because you run it in java, doesn't mean every implementation of Processing has a JVM. Processing.js comes to mind).
Just make a state class and keep track that way:
class SketchState {
color background_color, stroke_color, fill_color;
SketchState(color bg, color st, color fl) {
sketch = s; background_color = bg; stroke_color = st; fill_color = fl;
}
}
ArrayList<SketchState> stateCache = new ArrayList<SketchState>();
void cacheState() {
stateCache.add(new SketchState(...));
}
void restoreState() {
SketchState rest = stateCache.remove(stateCache.size()-1);
background(rest.background_color);
stroke(rest.stroke_color);
fill(rest.fill_color);
}
and add whatever other state aspects you want saved to that class.

java callback, unscribe listener inside of interface implementation

I would like to implement system of callbacks which looks like this (pseudo code):
final Listener listener = ListenerCtrl.addListener(new Listener() {
void onNotify(String response){
ListenerCtrl.unsetListener(listener);
} }
This code mean that after received message, i want to unscribe from future notifications. I found very attractive have this action inside of callback.
Here is my actual implementation:
final WebServiceMsgListener wml = new WebServiceMsgListener()
{
public void onMsgNotify(JSONObject response, int ecode)
{
Log.v(TAG, "getSetStateProgressBar MSG_MGT_STATICINFO: onMsgNotify ecode" +
ecode);
authDelegate.unsetMsgListener(wml);
}
};
authDelegate.addMsgListener(NAOMsg.MSG_MGT_STATICINFO, wml);
Unfortunately, my current implementation show me eclipse error:"The local variable wml may not have been initialized"
Question: how I can get round this, to finally unscribe inside of callback and dont have this error ?

Change your code to:
authDelegate.unsetMsgListener(this);
this refers to the current object (whose onMsgNotify() is being executed at the time this statement is executed).
Note: Although, the variable wml is available to the new object, it has not yet been initialized at the time of the creation of the object, hence the error. It is initialized right after the object is fully created.

Using Enabled Property on a JButton in netbeans

I am trying to setup a button similar to the save button with the default CRUD database template (where the button only becomes active if a variable is true). I have looked at the code for the save button and worked out that i need:
A variable to link it with (saveNeeded in their case)
An action to run
I have recreated both of these on another button but it never seams to get enabled. I have print statements on 2 other buttons i am using to set the variable i have my button linked to to true and false so i can see the value is changing.
Is there some crucial step i am missing? this seems like it should be fairly straight forward.
One other thing, if i manualy change the variable to true in my constructor to true and run the application it enables the button and false disables it so that part is working, just not the change.
Any help would be appreciated as i have spent the last few hours trying and can not figure it out
Thanks

The variable or "property" needs to be watched somehow, perhaps by using a PropertyChangeSupport object and allowing other objects to add a PropertyChangeListener to it, making it a "bound property". There's a special version of this for Swing applications that takes care with the Swing event thread, SwingPropertyChangeSupport, and you may wish to use it.
Edit
You asked
Thanks for the reply, i assume that would be what firePropertyChange("saveNeeded", !saveNeeded, saveNeeded); is doing but waht is this doing? does this just notify the program or do i need to catch an handle this somewhere. This is based off the pre generated code so im not sure if it added something in the background.
The class that holds the watched variable would need a private SwingPropertyChangeSupport field. You would give it a public addPropertyChangeListener method where you'd allow other classes to listen to its bound properties, something like this (if the property were a String):
import java.beans.PropertyChangeEvent;
import java.beans.PropertyChangeListener;
import javax.swing.event.SwingPropertyChangeSupport;
public class Foo {
public static final String MY_BOUND_PROPERTY = "My Bound Property";
private SwingPropertyChangeSupport spcSupport = new SwingPropertyChangeSupport(
this);
private String myBoundProperty;
public void addPropertyChangeListener(PropertyChangeListener listener) {
spcSupport.addPropertyChangeListener(listener);
}
public void removePropertyChangeListener(PropertyChangeListener listener) {
spcSupport.removePropertyChangeListener(listener);
}
public String getMyBoundProperty() {
return myBoundProperty;
}
public void setMyBoundProperty(String myBoundProperty) {
Object oldValue = this.myBoundProperty;
Object newValue = myBoundProperty;
this.myBoundProperty = myBoundProperty;
PropertyChangeEvent pcEvent = new PropertyChangeEvent(this,
MY_BOUND_PROPERTY, oldValue, newValue);
spcSupport.firePropertyChange(pcEvent);
}
}
Then any class that would like to listen for changes would simply add a PropertyChangeListener to an object of this class and respond to changes as it saw fit.