I am trying to make use of Java 8 and streams and one of the things I am trying to replace is a system we have where we
Use an aspect to measure call latency (per config period of time) to out webservices and then
Feed those results into a Complex Event Processor (esper) so that
We can send out alert notifications
So, one step at a time. For the first step, I need to produce a stream (I think) that allows me to feed those latency numbers into existing listeners. Understanding that, getting the next number in series might have to wait until there is a call.
How can I do that? Here is the latency aspect with comments.
public class ProfilingAspect {
private ProfilingAction action;
public ProfilingAspect(ProfilingAction action) {
this.action = action;
}
public Object doAroundAdvice(ProceedingJoinPoint jp) throws Throwable{
long startTime = System.currentTimeMillis();
Object retVal = null;
Throwable error = null;
try{
retVal = jp.proceed();
}catch (Throwable t){
error = t;
}
Class withinType = jp.getSourceLocation().getWithinType();
String methodName = jp.getSignature().getName();
long endTime = System.currentTimeMillis();
long runningTime = endTime - startTime;
// Let the IntStream know we have a new latency. Or really, we have an object
// stream with all this extra data
action.perform(withinType, methodName, jp.getArgs(), runningTime, error);
if( error != null ){
throw error;
}
return retVal;
}
}
Ok, I have a working example. It doesn't handle the situation where I have to buffer up results though is the stream isn't being read fast enough. I am open to some improvement
public class LatencySupplier implements Supplier<SomeFancyObject> {
private Random r = new Random();
#Override
public SomeFancyObject get() {
try {
Thread.sleep(100 + r.nextInt(1000));
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
return new SomeFancyObject(10 + r.nextInt(1000));
}
}
public class SomeFancyObject {
private static String[] someGroups = {"Group1","Group2","Group3"};
private final String group;
private int value;
public SomeFancyObject(int value) {
this.value = value;
this.group = WSRandom.selectOne(someGroups);
}
public String getGroup() {
return group;
}
public int getValue() {
return value;
}
#Override
public String toString() {
return value + "";
}
}
My next step is to create a stream by time so I can do avg/5 min, etc.
public class Sample {
public static void main(String[] args) throws InterruptedException {
Stream<SomeFancyObject> latencyStream = Stream.generate(new LatencySupplier());
Map<Object,List<SomeFancyObject>> collect = latencyStream.limit(10).collect(Collectors.groupingBy(sfo -> sfo.getGroup()));
System.out.println(collect);
Object o = new Object();
synchronized (o){
o.wait();
}
}
}
Related
I am trying to create a TumblingWindow on a stream of continuous data and create aggregates within the window. But for some reason, the getResult() does not get called.
public class MyAggregator implements AggregateFunction<Event, MyMetrics, MyMetrics> {
#Override
public MyMetrics createAccumulator() {
return new MyMetrics(0L, 0L);
}
#Override
public MyMetrics add(Event value, MyMetrics accumulator) {
Instant previousValue = ...;
if (previousValue != null) {
Long myWay = ...;
accumulator.setMyWay(myWay);
}
return accumulator;
}
#Override
public MyMetrics getResult(MyMetrics accumulator) {
System.out.println("Inside getResult()");
return accumulator;
}
#Override
public MyMetrics merge(MyMetrics acc1, MyMetrics acc2) {
return new MyMetrics(
acc1.getMyWay() + acc2.getMyWay());
}
}
Note: event.getClientTime() returns an Instant object.
private WatermarkStrategy getWatermarkStrategy() {
return WatermarkStrategy
.<MyEvent>forBoundedOutOfOrderness(Duration.ofMinutes(10))
.withTimestampAssigner(
(event, timestamp) ->
event.getClientTime().toEpochMilli()
);
}
public static void main(String[] args) {
DataStream<MyEvent> watermarkedData = actuals
.assignTimestampsAndWatermarks(
getWatermarkStrategy()
).name("addWatermark");
final OutputTag<MyEvent> lateOutputTag = new OutputTag<MyEvent>("late-data"){};
SingleOutputStreamOperator<OutputModel> output_data = watermarkedData
.keyBy("input_key")
.window(TumblingEventTimeWindows.of(Time.hours(1)))
.sideOutputLateData(lateOutputTag)
.aggregate(
new MyAggregator(),
).name("AggregationRollUp");
output_data.addSink(new PrintSinkFunction<>());
}
Any pointers as to what I am missing here would be helpful.
First check the timing of the data to see if it meets the window trigger conditions
Second may be you can do a test by reducing the window size from 1h to 1min and reducing the watermark region from 10min to 30s
//How can we handle the below code in functional way like collect all invalid sort params, prepare message with all them listed and finally throw InvalidSortParam exception.
DO you think the below query does it?
public Sort resolveArgument() {
Sort sort = sortHandlerMethodArgumentResolver.resolveArgument();
List<Sort.Order> orders = sort.stream().collect(Collectors.toList());
List<String> invalidSortList = orders.stream().map(Sort.Order::getProperty)
.filter(property -> !allowedSortParams.contains(property))
.collect(Collectors.toList());
if (orders.isEmpty()) {
sort = Sort.by(Sort.Direction.DESC, defaultSortParam);
} else {
if (orders.size() > sortMaxCount) {
throw new InvalidSortException(INVALID_SIZE_PARAMS);
} else {
if (!invalidSortList.isEmpty()) {
throw new InvalidSortException(invalidSortList.stream()
.collect(Collectors.joining(",")) + INVALID_SORT_PARAMS);
}
}
}
return sort;
}
First of all - you do not take advantage of the streaming functionality, you divided it into two separate steps - loading data, and when it is completed - transforming it. You could merge it into a single pipeline. Secondly - throwing exceptions is not a functional way. You should use some type to return a state e.g with some Try implementation.
final class Try<T> {
private final Exception e;
private final T t;
public Try(Exception e) {
this.e = e;
t = null;
}
public Try(T t) {
this.t = t;
e = null;
}
public Exception getE() {
return e;
}
public T getT() {
return t;
}
}
and the code itself could look something like
public Try<Sort> resolveArgument() {
int cnt = getCount();
return cnt == 0 ? Sort.by(Sort.Direction.DESC, defaultSortParam) : someErrorHandlingLogic(cnt);
}
private static void getCount() {
sort.stream().map(Sort.Order::getProperty)
.filter(property -> !allowedSortParams.contains(property))
.count();
}
If you're interested in the Functional Programming paradigm in java - I recommend great presentation https://dev.tube/video/YnzisJh-ZNI
public class FlinkWindowTest {
public static long timestamp = 1496301598L;
public static void main(String[] args) throws Exception {
// get the execution environment
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
// get input data by connecting to the socket
SourceFunction<String> out = new OutSource();
DataStream<String> text = env.addSource(out);
// parse the data
DataStream<WordWithCount> windowCounts = text
.flatMap(new FlatMapFunction<String, WordWithCount>() {
public void flatMap(String value, Collector<WordWithCount> out) {
for (String word : value.split(" ")) {
out.collect(new WordWithCount(word, 1L));
}
}
});
//assign timestamp
windowCounts = windowCounts.assignTimestampsAndWatermarks(new MyTimestampExtractor(Time.seconds(0)));
windowCounts.keyBy(new MyKeySelector())
.join(windowCounts)
.where(new MyKeySelector()).equalTo(new MyKeySelector())
.window(TumblingEventTimeWindows.of(Time.seconds(10)))
.apply(new JoinFunction<WordWithCount, WordWithCount, Object>() {
public Object join(WordWithCount wordWithCount, WordWithCount wordWithCount2) throws Exception {
System.out.println("start join");
System.out.println(wordWithCount.toString());
System.out.println(wordWithCount2.toString());
WordWithCount wordWithCount3 = new WordWithCount(wordWithCount.word, wordWithCount.count + wordWithCount2.count);
System.out.println(wordWithCount3.toString());
return wordWithCount3;
}
});
env.execute("Window WordCount");
}
public static class MyKeySelector implements KeySelector<WordWithCount, String> {
public String getKey (WordWithCount wordWithCount) throws Exception {
return wordWithCount.word;
}
}
public static class MyTimestampExtractor extends BoundedOutOfOrdernessTimestampExtractor<WordWithCount> {
public MyTimestampExtractor(Time maxOutOfOrderness) {
super(maxOutOfOrderness);
}
public long extractTimestamp(WordWithCount wordWithCount) {
return wordWithCount.getTimeStamp();
}
}
public static class OutSource implements SourceFunction<String> {
private String[] str = {
"aa ff","bb gg","cc hh","dd kk"
};
public void run(SourceContext<String> sourceContext) throws Exception {
int index =0;
while (true) {
if(index == str.length)
index = 0;
sourceContext.collect(str[index]);
index++;
}
}
public void cancel() {
}
}
// Data type for words with count and timestamp
public static class WordWithCount {
public String word;
public long count;
public WordWithCount() {}
public long getTimeStamp() {
return timestamp;
}
public WordWithCount(String word, long count) {
this.word = word;
this.count = count;
++timestamp;
}
#Override
public String toString() {
return word + " : " + count;
}
}
}
This class is a demo. I create a SourceFunction to emit strings, then cut them to words. Finally I use join operation to join the stream itself. I don't care the count result.
The question is that there is no output in my JoinFunction class. I think the output should be
start join
aa : 1
aa : 1
aa : 2
start join
........
but now there is no output, because elements are in the window and not emitted to the join function.
I don't have ideas about this situation. If there is anyone have advice, please tell me here. I expect replies by all.
:)
You forgot to the set the time characteristic to event time:
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
Is it possible for a variable in java to revert back to its previous state after a time limit.For example:If I have a boolean variable say x which was initialized to false and later the value was changed to true.Is it possible for this variable to revert back to its original value (ie false) after a time limit.Can this be achieved without using a timer or are there any design patterns to achieve this functionality.
Find a small example of a class which keeps the state and based on the timeout it negates the state.
See this as a PoC. You might need to do some improvements for concurrent access or visibility if you want to use it in a multithreaded application.
public static void main(String[] args) throws Exception {
VolatileState volatileState = new VolatileState();
System.out.println("initial: " + volatileState.getState());
volatileState.setState(true, 5);
System.out.println("valid : " + volatileState.getState());
TimeUnit.SECONDS.sleep(10);
System.out.println("reset : " + volatileState.getState());
}
The class which keeps the state for a given time.
class VolatileState {
private long timeSet = Long.MAX_VALUE;
private long timeToLive = 0;
private boolean state;
/**
* Keep the given state for <code>timeToLife</code> seconds.
*/
void setState(boolean state, long timeToLive) {
this.timeSet = System.currentTimeMillis();
this.timeToLive = TimeUnit.MILLISECONDS.toSeconds(timeToLive);
}
boolean getState() {
if (System.currentTimeMillis() > timeSet + timeToLive ) {
state = !state;
System.out.println("state reset to " + state);
}
return state;
}
}
You can use something like this:
class RevertingBoolean {
private long timeToRevert = Long.MAX_VALUE;
private boolean value;
private boolean defaultValue;
void setDefaultValue(boolean value) {
this.defaultValue = value;
}
void setRevertAfter(long revertAfter) {
this.timeToRevert = System.currentTimeMillis() + revertAfter;
}
void setValue(boolean value) {
this.value = value;
}
boolean getValue() {
if (System.currentTimeMillis() > timeToRevert) {
this.value = this.defaultValue;
timeToRevert = Long.MAX_VALUE;
}
return this.value;
}
}
Usage:
RevertingBoolean myBool = new RevertingBoolean();
myBool.setDefaultValue(false);
myBool.setValue(false);
myBool.setRevertAfter(10000); // Revert value in 10 seconds
myBool.getValue(); // false
myBool.setValue(true);
myBool.getValue(); // true
// ... 10 seconds later ...
myBool.getValue(); // false
You can create a class, e.g., BooleanReverts, which has members for the new value, old value to which it reverts, and time to revert, and with setter methods that set up reversion. You probably want a separate thread to manage all the timers and execute callbacks to effect the reversion.
here is a simple generic class and a Test case with a boolean type (but you can use with any type) of a class that shoud match your requirements
package test;
public class TimeValueVar<T> {
private T originalValue;
private T currentValue;
private long duration;
private long creationTime;
public TimeValueVar(T value, long duration) {
originalValue=value;
currentValue=value;
this.duration=duration;
creationTime=System.currentTimeMillis();
}
public void set(T value) {
currentValue=value;
}
public T get() {
if ((System.currentTimeMillis()-creationTime)>duration) {
return originalValue;
}
return currentValue;
}
}
here is the test case
package test;
import java.util.concurrent.TimeUnit;
public class TesttimeBasedValue {
public static void main(String[] args) throws Exception {
long duration =2000;
TimeValueVar<Boolean> boolVal=new TimeValueVar<>(true,duration);
System.out.println("original: " + boolVal.get());
boolVal.set(false);
System.out.println("before duration : " + boolVal.get());
TimeUnit.MILLISECONDS.sleep(duration+1);
System.out.println("after duration : " + boolVal.get());
}
}
I created a class which is used as a cache provider. It uses a Map, timestamped map entries and it spawns a Thread which performs cleanup every so often. This class is used in a web application. This web application had a problem where POST would take 30 seconds. I traced the problem to this cache class, eliminating it resolves the problem.
I have tried my best to find the error in this class but I can't. Please help me out here.
Assume User class is some kind of POJO describing the user.
public class UserStore implements Thread.UncaughtExceptionHandler {
private static volatile UserStore instance;
private static Thread cleanUpThread;
private static Map<String, TimeStampedToken<User>> tokenMap = new HashMap<String, TimeStampedToken<User>>();
public static UserStore getInstance() {
if (instance == null) {
synchronized(UserStore.class) {
if (instance == null) {
instance = new UserStore();
cleanUpThread = new Thread(new CleanUpWorker());
cleanUpThread.setUncaughtExceptionHandler(instance);
cleanUpThread.start();
}
}
}
return instance;
}
public void uncaughtException(Thread thread, Throwable throwable) {
if (throwable instanceof ThreadDeath) {
cleanUpThread = new Thread(new CleanUpWorker());
cleanUpThread.setUncaughtExceptionHandler(this);
cleanUpThread.start();
throw (ThreadDeath)throwable;
}
}
private static class CleanUpWorker implements Runnable {
private static final long CLEANUP_CYCLE_MS = 300000;
private static final long OBJECT_LIVE_TIME = 299900;
public void run() {
long sleepRemaining;
long sleepStart = System.currentTimeMillis();
sleepRemaining = CLEANUP_CYCLE_MS;
while (true) {
try {
sleepStart = System.currentTimeMillis();
Thread.sleep(sleepRemaining);
cleanUp();
sleepRemaining = CLEANUP_CYCLE_MS;
} catch (InterruptedException e) {
sleepRemaining = System.currentTimeMillis() - sleepStart;
}
}
}
private void cleanUp() {
Long currentTime = System.currentTimeMillis();
synchronized(tokenMap) {
for (String user : tokenMap.keySet()) {
TimeStampedToken<User> tok = tokenMap.get(user);
if (tok.accessed + OBJECT_LIVE_TIME < currentTime) {
tokenMap.remove(user);
}
}
}
}
}
public void addToken(User tok) {
synchronized(tokenMap) {
tokenMap.put(tok.getUserId(), new TimeStampedToken<User>(tok));
}
}
public User getToken(String userId) {
synchronized(tokenMap) {
TimeStampedToken<User> user = tokenMap.get(userId);
if (user != null) {
user.accessed = System.currentTimeMillis();
return user.payload;
} else {
return null;
}
}
}
private static class TimeStampedToken<E> {
public TimeStampedToken(E payload) {
this.payload = payload;
}
public long accessed = System.currentTimeMillis();
public E payload;
}
}
Here is how I would approach it. With multi-threaded code, simplicity is often the best approach as its more likely to work.
(the third parameter trueof the LinkedHashMap means that iterators over this Map follow the order of access rather than order of insertion)
public enum UserStore {
;
interface User {
String getUserId();
}
// a LRU cache with a timestamp.
private static final Map<String, TimeStampedToken<User>> tokenMap = new LinkedHashMap<String, TimeStampedToken<User>>(16, 0.7f, true);
private static final long OBJECT_LIVE_TIME = 299900;
public static synchronized void addToken(User tok) {
final long now = System.currentTimeMillis();
// clean up as we go
for (Iterator<Map.Entry<String, TimeStampedToken<User>>> iter = tokenMap.entrySet().iterator(); iter.hasNext(); ) {
final Map.Entry<String, TimeStampedToken<User>> next = iter.next();
if (next.getValue().accessed + OBJECT_LIVE_TIME >= now)
// the map is ordered by access time so there are no more to clean up.
break;
iter.remove();
}
// add a new entry
tokenMap.put(tok.getUserId(), new TimeStampedToken<User>(tok, now));
}
public static synchronized User getToken(String userId) {
final long now = System.currentTimeMillis();
TimeStampedToken<User> user = tokenMap.get(userId);
if (user == null)
return null;
user.accessed = now;
return user.payload;
}
static class TimeStampedToken<E> {
long accessed;
final E payload;
TimeStampedToken(E payload, long now) {
this.payload = payload;
accessed = now;
}
}
}
This line looks weird to me...
sleepRemaining = System.currentTimeMillis() - sleepStart;
...surely it should be...
sleepRemaining = CLEANUP_CYCLE_MS - (System.currentTimeMillis() - sleepStart);