We Keep Coding

Apache Flink join different DataStreams on specific key

I have two DataStreams, the first one called DataStream<String> source which receive records from a message broker, and the second one is a SingleOutputOperator<Event> events, which is the result of mapping the source into Event.class.
I have a uses cases that needs to use SingleOutputOperator<Event> events and other that uses DataStream<String> source. In one of the use cases that use DataStream<String> source, I need to join the SingleOutputOperator<String> result after apply some filters and to avoid to map the source again into Event.class as I already have that operation done and that Stream, I need to search each record into the SingleOutputOperator<String> result into the SingleOutputOperator<Event> events and the apply another map to export a SingleOutputOperator<EventOutDto> out.
This is the idea as example:
DataStream<String> source = env.readFrom(source);
SingleOutputOperator<Event> events = source.map(s -> mapper.readValue(s, Event.class));
public void filterAndJoin(DataStream<String> source, SingleOutputOperator<Event> events){
SingleOutputOperator<String> filtered = source.filter(s -> new FilterFunction());
SingleOutputOperator<EventOutDto> result = (this will be the result of search each record
based on id in the filtered stream into the events stream where the id must match and return the event if found)
.map(event -> new EventOutDto(event)).addSink(new RichSinkFunction());
}
I have this code:
filtered.join(events)
.where(k -> {
JsonNode tree = mapper.readTree(k);
String id = "";
if (tree.get("Id") != null) {
id = tree.get("Id").asText();
}
return id;
})
.equalTo(e -> {
return e.Id;
})
.window(TumblingEventTimeWindows.of(Time.seconds(1)))
.apply(new JoinFunction<String, Event, BehSingleEventTriggerDTO>() {
#Override
public EventOutDto join(String s, Event event) throws Exception {
return new EventOutDto(event);
}
})
.addSink(new SinkFunction());
In the above code all works fine, the ids are the same, so basically the where(id).equalTo(id) should work, but the process never reaches the apply function.
Observation: Watermark are assigned with the same timestamp
Questions:
Any idea why?
Am I explained myself fine?

I solved the join by doing this:
SingleOutputStreamOperator<ObjectDTO> triggers = candidates
.keyBy(new KeySelector())
.intervalJoin(keyedStream.keyBy(e -> e.Id))
.between(Time.milliseconds(-2), Time.milliseconds(1))
.process(new new ProcessFunctionOne())
.keyBy(k -> k.otherId)
.process(new ProcessFunctionTwo());

Extracting Timestamp from producer message

I really need help!
I can't extract the timestamp for a message sent by a producer. In my project I work with Json, I have a class in which I define the keys and one in which I define the values ​​of the message that I will send via a producer on a "Raw" topic. I have 2 other classes that do the same thing for the output message that my consumer will read on the topic called "Tdt". In the main class KafkaStreams.java I define the stream and map the keys and values. Starting Kafka locally, I start a producer who writes a message on the "raw" topic with keys and values, then on another shell the consumer starts reading the exit message on the "tdt" topic. How do I get the event timestamp? I need to know the timestamp in which the message was sent by the producer. Do I need a TimestampExtractor?
Here is my main class kafkastreams (my application works great, I just need the timestamp)
#Bean("app1StreamTopology")
public KStream<LibAssIbanRawKey, LibAssIbanRawValue> kStream() throws ParseException {
JsonSerde<Dwsitspr4JoinValue> Dwsitspr4JoinValueSerde = new JsonSerde<>(Dwsitspr4JoinValue.class);
KStream<LibAssIbanRawKey, LibAssIbanRawValue> stream = defaultKafkaStreamsBuilder.stream(inputTopic);
stream.peek((k,v) -> logger.info("Debug3 Chiave descrizione -> ({})",v.getCATRAPP()));
GlobalKTable<Integer, Dwsitspr4JoinValue> categoriaRapporto = defaultKafkaStreamsBuilder
.globalTable(temptiptopicname,
Consumed.with(Serdes.Integer(), Dwsitspr4JoinValueSerde)
// .withOffsetResetPolicy(Topology.AutoOffsetReset.EARLIEST)
);
logger.info("Debug3 Chiave descrizione -> ({})",categoriaRapporto.toString()) ;
stream.peek((k,v) -> logger.info("Debug4 Chiave descrizione -> ({})",v.getCATRAPP()) );
stream
.join(categoriaRapporto, (k, v) -> v.getCATRAPP(), (valueStream, valueGlobalKtable) -> {
// Value mapping
LibAssIbanTdtValue newValue = new LibAssIbanTdtValue();
newValue.setDescrizioneRidottaCodiceCategoriaDelRapporto(valueGlobalKtable.getDescrizioneRidotta());
newValue.setDescrizioneEstesaCodiceCategoriaDelRapporto(valueGlobalKtable.getDescrizioneEstesa());
newValue.setIdentificativo(valueStream.getAUD_CCID());
.
.
.//Other Value Mapped
.
.
.map((key, value) -> {
// Key mapping
LibAssIbanTdtKey newKey = new LibAssIbanTdtKey();
newKey.setData(dtf.format(localDate));
newKey.setIdentificatoreUnivocoDellaRigaDiTabella(key.getTABROWID());
return KeyValue.pair(newKey, value);
}).to(outputTopic, Produced.with(new JsonSerde<>(LibAssIbanTdtKey.class), new JsonSerde<>(LibAssIbanTdtValue.class)));
return stream;
}
}

Yes you need a TimestampExtractor.
public class YourTimestampExtractor implements TimestampExtractor {
#Override
public long extract(ConsumerRecord<Object, Object> consumerRecord, long l) {
// do whatever you want with the timestamp available with consumerRecord.timestamp()
...
// return here the timestamp you want to use (here default)
return consumerRecord.timestamp();
}
}
You'll need to tell kafka stream what extractor to use under the key StreamsConfig.DEFAULT_TIMESTAMP_EXTRACTOR_CLASS_CONFIG

Clear and populate multiple sheets with data in a single request

I need to periodically clear the data in multiple sheets and re-populate them with data (via the Google Sheets API v4). To do this, I'm executing 2 separate requests for each sheet (1 clear & 1 update). This is kind of a slow process when the user is sitting there waiting for it. It seems to me that each new request significantly adds to the completion time. If I could wrap all these into a single batch-command request, it might help a lot.
I'm currently doing this for each sheet...
service.spreadsheets()
.values()
.clear(idSpreadsheet, sheetTitle + "!$A$1:$Y", new ClearValuesRequest())
.execute();
service.spreadsheets()
.values()
.update(idSpreadsheet, range, new ValueRange().setValues(values))
.setValueInputOption("USER_ENTERED")
.execute();
I don't see a way to just wrap a bunch of generic commands into a single batch request. I see that DeleteDimensionRequest and AppendCellsRequest can be wrapped into a batch, but I can't really find a good AppendCellsRequest example (and it seems that people recommend my current values().update() method anyway).
Can anyone recommend a good way to streamline this? Or am I already doing it the best way?

Still don't know if I'm doing it the BEST way, but I was able to accomplish my goal of clearing and populating multiple sheets with data in a single batch request. The trick was to not use the clear() method but instead, overwrite the the sheet with blank data using RepeatCellRequest. Also, now using AppendCellsRequest instead of update(). These two requests can be wrapped in a batch request.
My early tests with 3 sheets, shows about a 25% performance improvement. Not spectacular, but it helps.
List<Request> requests = new ArrayList<Request>();
for (SheetData mySheet : sheetDatas)
{
List<List<Object>> values = mySheet.getValues();
Request clearSheetRequest = new Request()
.setRepeatCell(new RepeatCellRequest()
.setRange(new GridRange()
.setSheetId(mySheet.getSheetId())
)
.setFields("*")
.setCell(new CellData())
);
List<RowData> preppedRows = new ArrayList<RowData>();
for (List<Object> row : values)
{
RowData preppedRow = new RowData();
List<CellData> cells = new ArrayList<CellData>();
for (Object value : row)
{
CellData cell = new CellData();
ExtendedValue userEnteredValue = new ExtendedValue();
if (value instanceof String)
{
userEnteredValue.setStringValue((String) value);
}
else if (value instanceof Double)
{
userEnteredValue.setNumberValue((Double) value);
}
else if (value instanceof Integer)
{
userEnteredValue.setNumberValue(Double.valueOf((Integer) value).doubleValue());
}
else if (value instanceof Boolean)
{
userEnteredValue.setBoolValue((Boolean) value);
}
cell.setUserEnteredValue(userEnteredValue);
cells.add(cell);
}
preppedRow.setValues(cells);
preppedRows.add(preppedRow);
}
Request appendCellsRequest = new Request().setAppendCells(
new AppendCellsRequest()
.setSheetId(mySheet.getSheetId())
.setRows(preppedRows)
.setFields("*")
);
requests.add(clearSheetRequest);
requests.add(appendCellsRequest);
}
BatchUpdateSpreadsheetRequest batch = new BatchUpdateSpreadsheetRequest().setRequests(requests);
BatchUpdateSpreadsheetResponse batchResponse = service.spreadsheets().batchUpdate(idSpreadsheet, batch).execute();

neo4j Java beginner. Cypher Execution Engine

The user logs in on the website and creates different events. This event is saved into the neo4j database as a node and I make the "EVENT_CREATOR" realtionship between the user and the event node.
I am trying to implement pagination for all the user's events on my website (using Play2 framework) and I need for example if user accesses the first page, I load the first ten events; 2nd page to load the 10th- 20th events, and so on...
this is my query:
match(n);
...
skip k;
limit 10;
return n;
At the moment I am getting all the events created by the user and add them to the array list.
private static List<PublicEvent> getEvents(int page, int pageSize) {
List<PublicEvent> events = new ArrayList<PublicEvent>();
GraphDatabaseService db = Neo4JHelper.getDatabase();
try (Transaction tx = db.beginTx()) {
Index<Node> userIndex = db.index().forNodes(ModelIndex.Users);
IndexHits<Node> userNodes = userIndex.get(ModelGraphProperty.UserProfile.UserName, SessionUtilities.getCurrentUser());
Node me = userNodes.next(); //current logged in user
PagingIterator paginator = new PagingIterator(me.getRelationships(GraphRelation.RelTypes.EVENT_CREATOR).iterator(), pageSize); // get all the events that were created by this user
paginator.page(page);
// adding all the created events by this user to an array
if (paginator.hasNext()) {
Relationship eventCreator = (Relationship)paginator.next();
Node event = eventCreator.getOtherNode(me);
events.add(new PublicEvent(event));
}
tx.success();
}
db.shutdown();
return events;
}
I want to update the code to run Cypher queries and I add the following lines of code (using the example https://www.tutorialspoint.com/neo4j/neo4j_cypher_api_example.htm )
GraphDatabaseService db = Neo4JHelper.getDatabase();
ExecutionEngine execEngine = new ExecutionEngine(db); //HERE I GET AN ERROR
ExecutionResult execResult = execEngine.execute("MATCH (n) RETURN n");
String results = execResult.dumpToString();
System.out.println(results);
it is expecting a second parameter: logger. What is the error or is there anything I am doing wrong?

RestGraphDatabase db= (RestGraphDatabase)Neo4JHelper.getDatabase();
RestCypherQueryEngine engine=new RestCypherQueryEngine(db.getRestAPI());
Map<String, Object> params = new HashMap<String, Object>();
params.put( "id", eventId );
String query="match (s) where id(s) = {id} return s;";
QueryResult result=engine.query(query,params);
if(result.iterator().hasNext()) {
//HERE PUT WHATEVER YOU NEED
}
Take a look at the documentation:
https://neo4j.com/docs/java-reference/current/

Using ElasticSearch's script_upsert to create a document

According to the official documentation Update API - Upserts one can use scripted_upsert in order to handle update (for existing document) or insert (for new document) form within the script. The thing is they never show how the script should look to do that. The Java - Update API Doesn't have any information on the ScriptUpsert uses.
This is the code I'm using:
//My function to build and use the upsert
public void scriptedUpsert(String key, String parent, String scriptSource, Map<String, ? extends Object> parameters) {
Script script = new Script(scriptSource, ScriptType.INLINE, null, parameters);
UpdateRequest request = new UpdateRequest(index, type, key);
request.scriptedUpsert(true);
request.script(script);
if (parent != null) {
request.parent(parent);
}
this.bulkProcessor.add(request);
}
//A test call to validate the function
String scriptSource = "if (!ctx._source.hasProperty(\"numbers\")) {ctx._source.numbers=[]}";
Map<String, List<Integer>> parameters = new HashMap<>();
List<Integer> numbers = new LinkedList<>();
numbers.add(100);
parameters.put("numbers", numbers);
bulk.scriptedUpsert("testUser", null, scriptSource, parameters);
And I'm getting the following exception when "testUser" documents doesn't exists:
DocumentMissingException[[user][testUser]: document missing
How can I make the scriptUpsert work from the Java code?

This is how a scripted_upsert command should look like (and its script):
POST /sessions/session/1/_update
{
"scripted_upsert": true,
"script": {
"inline": "if (ctx.op == \"create\") ctx._source.numbers = newNumbers; else ctx._source.numbers += updatedNumbers",
"params": {
"newNumbers": [1,2,3],
"updatedNumbers": [55]
}
},
"upsert": {}
}
If you call the above command and the index doesn't exist, it will create it, together with the newNumbers values in the new documents. If you call again the exact same command the numbers values will become 1,2,3,55.
And in your case you are missing "upsert": {} part.

As Andrei suggested I was missing the upsert part, changing the function to:
public void scriptedUpsert(String key, String parent, String scriptSource, Map<String, ? extends Object> parameters) {
Script script = new Script(scriptSource, ScriptType.INLINE, null, parameters);
UpdateRequest request = new UpdateRequest(index, type, key);
request.scriptedUpsert(true);
request.script(script);
request.upsert("{}"); // <--- The change
if (parent != null) {
request.parent(parent);
}
this.bulkProcessor.add(request);
}
Fix it.