We Keep Coding

How to use non-keyed state with Kafka Consumer in Flink?

I'm trying to implement (just starting work with Java and Flink) a non-keyed state in KafkaConsumer object, since in this stage no keyBy() in called. This object is the front end and the first module to handle messages from Kafka.
SourceOutput is a proto file representing the message.
I have the KafkaConsumer object :
public class KafkaSourceFunction extends ProcessFunction<byte[], SourceOutput> implements Serializable
{
#Override
public void processElement(byte[] bytes, ProcessFunction<byte[], SourceOutput>.Context
context, Collector<SourceOutput> collector) throws Exception
{
// Here, I want to call to sorting method
collector.collect(output);
}
}
I have an object (KafkaSourceSort) that do all the sorting and should keep the unordered message in priorityQ in the state and also responsible to deliver the message if it comes in the right order thru the collector.
class SessionInfo
{
public PriorityQueue<SourceOutput> orderedMessages = null;
public void putMessage(SourceOutput Msg)
{
if(orderedMessages == null)
orderedMessages = new PriorityQueue<SourceOutput>(new SequenceComparator());
orderedMessages.add(Msg);
}
}
public class KafkaSourceState implements Serializable
{
public TreeMap<String, SessionInfo> Sessions = new TreeMap<>();
}
I read that I need to use a non-keyed state (ListState) which should contain a map of sessions while each session contains a priorityQ holding all messages related to this session.
I found an example so I implement this:
public class KafkaSourceSort implements SinkFunction<KafkaSourceSort>,
CheckpointedFunction
{
private transient ListState<KafkaSourceState> checkpointedState;
private KafkaSourceState state;
#Override
public void snapshotState(FunctionSnapshotContext functionSnapshotContext) throws Exception
{
checkpointedState.clear();
checkpointedState.add(state);
}
#Override
public void initializeState(FunctionInitializationContext context) throws Exception
{
ListStateDescriptor<KafkaSourceState> descriptor =
new ListStateDescriptor<KafkaSourceState>(
"KafkaSourceState",
TypeInformation.of(new TypeHint<KafkaSourceState>() {}));
checkpointedState = context.getOperatorStateStore().getListState(descriptor);
if (context.isRestored())
{
state = (KafkaSourceState) checkpointedState.get();
}
}
#Override
public void invoke(KafkaSourceState value, SinkFunction.Context contex) throws Exception
{
state = value;
// ...
}
}
I see that I need to implement an invoke message which probably will be called from processElement() but the signature of invoke() doesn't contain the collector and I don't understand how to do so or even if I did OK till now.
Please, a help will be appreciated.
Thanks.

A SinkFunction is a terminal node in the DAG that is your job graph. It doesn't have a Collector in its interface because it cannot emit anything downstream. It is expected to connect to an external service or data store and send data there.
If you share more about what you are trying to accomplish perhaps we can offer more assistance. There may be an easier way to go about this.

Spring Webflux - Proper way to throw checked custom exception (not RuntimeException)

May I ask what is the proper way to throw checked custom exception in Spring webflux please?
I would like to insist, it is about checked custom exception, like MyException.java, not something like RuntimeException, and it is about throwing exception, not handling exception.
I tried the following :
#Controller
#SpringBootApplication
public class QuestionHowToThrowException {
public static void main(String[] args) {
SpringApplication.run(QuestionHowToThrowException.class);
}
#PostMapping(path = "/question", consumes = MediaType.APPLICATION_JSON_VALUE, produces = MediaType.APPLICATION_JSON_VALUE)
public Mono<ResponseEntity<QuestionResponse>> question(#RequestBody QuestionRequest questionRequest) {
Mono<FirstStep> firstStepMono = WebClient.create().post().uri("http://firstWebService:8111/getFirstStep")
.body(questionRequest.getThing(), String.class).retrieve().bodyToMono(FirstStep.class);
Mono<SecondStep> secondStepMono = firstStepMono.map(oneFirstStep -> getSecondStepFromFirstStepAfterCheck(oneFirstStep));
return secondStepMono.map(oneSecondStep -> ResponseEntity.ok(new QuestionResponse(oneSecondStep.getSecondThing())));
}
private SecondStep getSecondStepFromFirstStepAfterCheck(FirstStep firstStep) throws MyException {
if (firstStep.getThingNeedsToCheckCanThrowException().equals("exception")) {
throw new MyException("exception");
} else {
return new SecondStep(firstStep.getThingNeedsToCheckCanThrowException() + "good");
}
}
public class QuestionRequest {
private String thing;
public String getThing() {
return thing;
}
}
public class QuestionResponse {
private String response;
public QuestionResponse(String response) {
this.response = response;
}
}
public class FirstStep {
private String thingNeedsToCheckCanThrowException;
public String getThingNeedsToCheckCanThrowException() {
return thingNeedsToCheckCanThrowException;
}
}
public class SecondStep {
private String secondThing;
public SecondStep(String secondThing) {
this.secondThing = secondThing;
}
public String getSecondThing() {
return secondThing;
}
}
}
This is not possible, since there in an unhandled exception in getSecondStepFromFirstStepAfterCheck method.
If I throw and propagate, private SecondStep getSecondStepFromFirstStepAfterCheck(FirstStep firstStep) throws MyException the lambda caller method is not happy.
What is the cleanest and proper way to throw custom exception in webflux please?
Thank you

Reading through your sample code, it looks like you are trying to introduce some error handling with on your Mono.
You can create an unchecked exception by extending the RuntimeException class. If you want a checked exception that enforces handling, you can simply extend Exception.
public class MyException extends RuntimeException {
public MyException(String msg) {
super(s);
}
}
The cleanest way to throw an exception with the Reactor project is really just to throw it. There are error handling functions that allow you to provide different flows to certain error cases.
The good news is you have several options that provides some flow control for error handling.
Project Reactor provides several of these methods on the Mono object.
doOnError(),onErrorContinue(),onErrorReturn(),onErrorStop(),onErrorMap()
I am not entirely sure what you are trying to achieve with the following sample code.
return Mono.error(new MyException("exception"));
} else {
return Mono.just(new SecondStep(firstStep.getThingNeedsToCheckCanThrowException() + "good"));
But this looks like a good case for a onErrorMap() since it looks like you are trying to translate some exception here
return Mono.just(new SecondStep(firstStep.getThingNeedsToCheckCanThrowException() + "good")
.onErrorMap(e -> "translated result");

For our applications, we have our custom base exception extend from RuntimeException. We then have standard exception handling that looks for our custom exception for special handling before returning results back to the end user. This allows us to use normal throws mechanics since we want all exceptions thrown to ripple up the top level of the call.
For performance concerns webflux and reactive are slightly lower performance on a per call basis especially for calls that don't need to do any parallelization. However once load is put onto the system it tends to become more performant primarily related to garbage collection. Overhead from the difference between map and flatMap should be negligible at best.

Mocking Network response for unit test case

I am trying to create a test case for AlbumsController which fetches data from the network and parses it.
AlbumService: Responsible for network call and getting data. ( Assume it can be anything that performs some async task and gives callback accordingly so as to avoid "Server" specific mocking solutions )
public class AlbumsController {
public void getAlbums(final ServiceCallback<AlbumsWrapper> albumsServiceCallback) {
new AlbumsService().fetchAlbums(new ServiceCallback<NetworkResponse>() {
#Override
public void onSuccess(NetworkResponse response) {
// parse response
}
#Override
public void onFailure(NetworkResponse error) {
// do something for Failure
}
});
}
public class AlbumControllerTest {
#Mock
private ServiceCallback<NetworkResponse> serviceCallback;
#Captor
private ArgumentCaptor<ServiceCallback<AlbumsWrapper>> albumsWrapper;
#Captor
private ArgumentCaptor<ServiceCallback<NetworkResponse>> networkResponseArgumentCaptor;
#Before
public void init() {
MockitoAnnotations.initMocks(this);
}
#Test
public void parseAlbums_EmptyList_ReturnsTrue() {
// below are different ways I am trying with no success so far.
AlbumsController albumsController = new AlbumsController();
albumsController.getAlbums(albumsWrapper.capture());
NetworkResponse response = new NetworkResponse();
networkResponseArgumentCaptor.capture();
networkResponseArgumentCaptor.getValue().onSuccess(response);
}
}
Aim:
To mock AlbumService inside getAlbums method so that instead of getting data from the server I should be able to call onSuccess() with my test data as an argument. And is it possible to get the list after parsing so as that I can assert base on the data in the list?
I don't want to move code responsible for parsing to some method and make that public. My intention here is to understand how to handle such a case.
Open for refactoring of code if that's what is required to be taken care of in TDD.
Full solution or pointers to look for both are appreciated.
Libraries
mockito-core:2.2.9
junit:4.12

If small refactoring is the option then:
1) Move the new AlbumsService() to a package level method:
AlbumService createAlbumService(){
return new AlbumService();
}
...
public void getAlbums(final ServiceCallback<AlbumsWrapper> albumsServiceCallback) {
createAlbumService().fetchAlbums(new ServiceCallback<NetworkResponse>()
package visibility is enough as the test class will be in the same package as the AlbumController.
2) Spy the AlbumController:
#Spy
private AlbumsController albumsControllerSpy = new AlbumController();
#Mock
private AlbumService albumServiceMock;
3) Make the createAlbumService() method return your mock:
#Test
public void parseAlbums_EmptyList_ReturnsTrue() {
doReturn(albumServiceMock).when(albumControllerSpy).createAlbumService();
...

Receive custom exceptions from Restlet Framework in GWT client

I want to use Exception handling with the Restlet Framework and GWT clients.
The Restlet Framework supports the concept of annotated exceptions as described in this post;
http://restlet.com/company/blog/2015/12/21/exception-handling-with-restlet-framework/
In my project i created a LocationNameException
#Status(value = 409)
public class LocationNameException extends Exception
{
...
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
public LocationNameException(String pMessage, Throwable pCause)
{
super(pMessage, pCause);
}
}
And use this in my ServerResource;
#Override
#Transactional(rollbackOn = LocationNameException.class)
public LocationDto postLocation(LocationDto pLocationDto) throws LocationNameException
{
...
Location lLocation = new Location(pLocationDto);
try
{
LocationDao lLocationDao = getLocationDao();
lLocationDao.persist(lLocation);
}
catch (PersistenceException pPersistenceException)
{
throw new LocationNameException("Location requires unique Name", pPersistenceException);
}
...
return lLocationDto;
}
With the interface
public interface LocationListServerResourceInt
{
...
#Post
LocationDto postLocation(LocationDto pLocationDto) throws LocationNameException;
...
}
This works, in the case of an exception the call returns code 409;
And at the GWT client side onFailure() is called;
private class PostLocationCallback implements AsyncCallback<LocationDto>
{
...
#Override
public void onFailure(Throwable pCaught)
{
mCallback.onFailure(pCaught, mLocationDto);
}
}
But parameter pCaught only contains a ResourceException with the status code 409.
My LocationNameException isn't included in the underlying cause stack.
I need this LocationNameException for appropriate error message handling.
The reason is the generated ServerResourceProxy LocationListServerResourceProxyImpl by the Restlet GWT ClientProxyGenerator;
public void postLocation(LocationDto param1, final LocationDto> callback)
{
...
public void handle(Request request, Response response)
{
if (getClientResource().getStatus().isError())
{
callback.onFailure(new ResourceException(getClientResource().getStatus()));
}
else
{
...
}
I think i have to rewrite the Post method in the ClientProxyGenerator;
The LocationNameException is present in the Response data so the Basic approach using the getResponseEntity() method of the ClientResource class should be possible.
Is this the way to go? Or can i catch the LocationNameException exception somewhere else as suggested by Catching annotated exceptions?
It is really hard to try a different approach because of the generated code. Is there an easy way to circumvent the code generator with custom classes?

As already mentioned the LocationNameException is present in the Response data.
Therefore we can get it, just like a normal entity;
...
public void handle(Request request, Response response)
{
if (getClientResource().getStatus().isError())
{
LocationNameException lLocationNameException = null;
boolean serializationError = false;
try
{
if (response.isEntityAvailable())
{
if (MediaType.APPLICATION_JAVA_OBJECT_GWT.equals(response.getEntity().getMediaType()))
{
lLocationNameException = new ObjectRepresentation<LocationNameException>(
response.getEntity().getText(),
(SerializationStreamFactory) MyLocationListServerResourceProxyImpl.this, false)
.getObject();
}
else
{
throw new IOException("Can't parse the enclosed LocationNameException.");
}
}
}
catch (Throwable e)
{
serializationError = true;
callback.onFailure(new ResourceException(e));
}
if (!serializationError)
{
callback.onFailure(lLocationNameException);
}
}
else
{
...
The ClientProxyGenerator needs to know the exception type (in this case LocationNameException). Therefore we specify the exception in the ClientProxy interface;
#Post
void postLocation(LocationDto pLocationDto, AsyncCallback<LocationDto> pResult) throws LocationNameException;
And use getExceptionTypes() or getGenericExceptionTypes() in the ClientProxyGenerator;
Class<?>[] exceptionTypes = method.getExceptionTypes();
java.lang.reflect.Type[] genericExceptionTypes = method.getGenericExceptionTypes();
Of course not all REST methods use a custom exception. When getExceptionTypes() returns an empty list we just return the good old status code;
callback.onFailure(new ResourceException(getClientResource().getStatus()));

With help of Jerome Louvel and Thierry Boileau i created a new ClientProxyGenerator() that supports custom exceptions towards a GWT client;
Just specify the exception from the interface in the ServerResourceProxy (ClientProxy)
and voilà
It is possible to use this custom ClientProxyGenerator() in your project right away.
Download custom ClientProxyGenerator
And place it in a package on the server (for example package com.ludus.server.util)
In GWT module XML change the ClientProxyGenerator to the new version on the server;
And you're ready to go with your custom exceptions, but it would be nice if this extension would be integrated in the Restlet framework.

Join two streams using a count-based window

I am new to Flink Streaming API and I want to complete the following simple (IMO) task. I have two streams and I want to join them using count-based windows. The code I have so far is the following:
public class BaselineCategoryEquiJoin {
private static final String recordFile = "some_file.txt";
private static class ParseRecordFunction implements MapFunction<String, Tuple2<String[], MyRecord>> {
public Tuple2<String[], MyRecord> map(String s) throws Exception {
MyRecord myRecord = parse(s);
return new Tuple2<String[], myRecord>(myRecord.attributes, myRecord);
}
}
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment environment = StreamExecutionEnvironment.createLocalEnvironment();
ExecutionConfig config = environment.getConfig();
config.setParallelism(8);
DataStream<Tuple2<String[], MyRecord>> dataStream = environment.readTextFile(recordFile)
.map(new ParseRecordFunction());
DataStream<Tuple2<String[], MyRecord>> dataStream1 = environment.readTextFile(recordFile)
.map(new ParseRecordFunction());
DataStreamSink<Tuple2<String[], String[]>> joinedStream = dataStream1
.join(dataStream)
.where(new KeySelector<Tuple2<String[],MyRecord>, String[]>() {
public String[] getKey(Tuple2<String[], MyRecord> recordTuple2) throws Exception {
return recordTuple2.f0;
}
}).equalTo(new KeySelector<Tuple2<String[], MyRecord>, String[]>() {
public String[] getKey(Tuple2<String[], MyRecord> recordTuple2) throws Exception {
return recordTuple2.f0;
}
}).window(TumblingProcessingTimeWindows.of(Time.seconds(1)))
.apply(new JoinFunction<Tuple2<String[],MyRecord>, Tuple2<String[],MyRecord>, Tuple2<String[], String[]>>() {
public Tuple2<String[], String[]> join(Tuple2<String[], MyRecord> tuple1, Tuple2<String[], MyRecord> tuple2) throws Exception {
return new Tuple2<String[], String[]>(tuple1.f0, tuple1.f0);
}
}).print();
environment.execute();
}
}
My code works without errors, but it does not produce any results. In fact, the call to apply method is never called (verified by adding a breakpoint on debug mode). I think, the main reason for the previous is that my data do not have a time attribute. Therefore, windowing (materialized through window) is not done properly. Therefore, my question is how can I indicate that I want my join to take place based on count-windows. For instance, I want the join to materialize every 100 tuples from each stream. Is the previous feasible in Flink? If yes, what should I change in my code to achieve it.
At this point, I have to inform you that I tried to call the countWindow() method, but for some reason it is not offered by Flink's JoinedStreams.
Thank you

Count-based joins are not supported. You could emulate count-based windows, by using "event-time" semantics and apply a unique seq-id as timestamp to each record. Thus, a time-window of "5" would be effectively a count-window of 5.