We Keep Coding

Set WatermarkStrategy for Event Timestamps

I am trying to do a windowed aggregation query on a data stream that contains over 40 attributes in Flink. The stream's schema contains an epoch timestamp which I want to use for the WatermarkStrategy so I can actually define tumbling windows over it.
I know from the docs, that you can define a Timestamp using the SQL Api in an CREATE TABLE-query by first using TO_TIMESTAMP_LTZ on the epochs to convert it to a proper timestamp which can be used in the following WATERMARK FOR-statement. Since I got a really huge schema tho, I want to deserialise and provide the schema NOT by completely writing the complete CREATE TABLE-statement containing all columns BUT by using a custom class derived from the proto file that cointains the schema. As far as I know, this is only possible by providing a deserializer for the KafkaSourceBuilder and calling the results function of the stream on the class derived from the protofile with protoc. Which means, that I have to define the table using the stream api.
Inspired by the answer to this question, I do it like this:
WatermarkStrategy watermarkStrategy = WatermarkStrategy
.<Row>forBoundedOutOfOrderness(Duration.ofSeconds(10))
.withTimestampAssigner( (event, ts) -> (Long) event.getField("ts"));
tableEnv.createTemporaryView(
"bidevents",
stream
.returns(BiddingEvent.BidEvent.class)
.map(e -> Row.of(
e.getTracking().getCampaign().getId(),
e.getTracking().getAuction().getId(),
Timestamp.from(Instant.ofEpochSecond(e.getTimestamp().getMilliseconds() / 1000))
)
)
.returns(Types.ROW_NAMED(new String[] {"campaign_id", "auction_id", "ts"}, Types.STRING, Types.STRING, Types.SQL_TIMESTAMP))
.assignTimestampsAndWatermarks(watermarkStrategy)
);
tableEnv.executeSql("DESCRIBE bidevents").print();
Table resultTable = tableEnv.sqlQuery("" +
"SELECT " +
" TUMBLE_START(ts, INTERVAL '1' DAY) AS window_start, " +
" TUMBLE_END(ts, INTERVAL '1' DAY) AS window_end, " +
" campaign_id, " +
" count(distinct auction_id) auctions " +
"FROM bidevents " +
"GROUP BY TUMBLE(ts, INTERVAL '1' DAY), campaign_id");
DataStream<Row> resultStream = tableEnv.toDataStream(resultTable);
resultStream.print();
env.execute();
I get this error:
Caused by: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Window aggregate can only be defined over a time attribute column, but TIMESTAMP(9) encountered.
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372) ~[flink-dist-1.15.1.jar:1.15.1]
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) ~[flink-dist-1.15.1.jar:1.15.1]
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) ~[flink-dist-1.15.1.jar:1.15.1]
at org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:291) ~[flink-dist-1.15.1.jar:1.15.1]
This seems kind of logical, since in line 3 I cast a java.sql.Timestamp to a Long value, which it is not (but also the stacktrace does not indicate that an error occurred during the cast). But when I do not convert the epoch (in Long-Format) during the map-statement to a Timestamp, I get this exception:
"Cannot apply '$TUMBLE' to arguments of type '$TUMBLE(<BIGINT>, <INTERVAL DAY>)'"
How can I assign the watermark AFTER the map-statement and use the column in the later SQL Query to create a tumbling window?
======UPDATE=====
Thanks to a comment from David, I understand that I need the column to be of type TIMESTAMP(p) with precision p <= 3. To my understanding this means, that my timestamp may not be more precise than having full milliseconds. So i tried different ways to create Java Timestamps (java.sql.Timestamps and java.time.LocaleDateTime) which correspond to the Flink timestamps.
Some examples are:
1 Trying to convert epochs into a LocalDateTime by setting nanoseconds (the 2nd parameter of ofEpochSecond to 0):
LocalDateTime.ofEpochSecond(e.getTimestamp().getMilliseconds() / 1000, 0, ZoneOffset.UTC )
2 After reading the answer from Svend in this question who uses LocalDateTime.parse on timestamps that look like this "2021-11-16T08:19:30.123", I tried this:
LocalDateTime.parse(
DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss").format(
LocalDateTime.ofInstant(
Instant.ofEpochSecond(e.getTimestamp().getMilliseconds() / 1000),
ZoneId.systemDefault()
)
)
)
As you can see, the timestamps even only have seconds-granularity (which i checked when looking at the printed output of the stream I created) which I assume should mean, they have a precision of 0. But actually when using this stream for defining a table/view, it once again has the type TIMESTAMP(9).
3 I also tried it with the sql timestamps:
new Timestamp(e.getTimestamp().getMilliseconds() )
This also did not change anything. I somehow always end up with a precision of 9.
Can somebody please help me how I can fix this?

Ok, I found the solution to the problem. If you got a stream containing a timestamp which you want to define as event time column for watermarks, you can use this function:
Table inputTable = tableEnv.fromDataStream(
stream,
Schema.newBuilder()
.column("campaign_id", "STRING")
.column("auction_id", "STRING")
.column("ts", "TIMESTAMP(3)")
.watermark("ts", "SOURCE_WATERMARK()")
.build()
);
The important part is, that you can "cast" the timestamp ts from TIMESTAMP(9) "down" to TIMESTAMP(3) or any other precision below 4 and you can set the column to contain the water mark.
Another mention that seems important to me is, that only Timestamps of type java.time.LocalDateTime actually worked for later use as watermarks for tumbling windows.
Any other attempts to influence the precision of the timestamps by differently creating java.sql.Timestamp or java.time.LocalDateTime failed. This seemed to be the only viable way.

Does presence of "|" [Bitwise Or] in query PARAM value have impact on performance Java Hibernate

Recently I encountered a strange issue where I had a table called person which has columns personid, name, age, username, ...
There are around 200 entries for username which as '|' in value.
i.e. for ex :
1] abc | def
2] mno | pqr
and so on.
I Have return an Spring Boot based Rest API which deletes person based on username but I am observing that the API takes more than expected time when we try to delete person whose username value has "|" in it.
i.e. time taken to delete 10 records for username with "|" is app 10 mins but the usernames without "|" values takes process transaction with in 4 min.
I tried to google a lot to find the impact of it but could not find any corresponding proof on it causing performance issue. Hence I am reaching out here .
Query used:
select * from person where username = trim (:name)
API Details:
Takes List of Usernames to be deleted or inactivated as input and have some set of basic validation before performing delete.

How to use SQLITE Recursion to check at least once daily event in android?

I am working on an android app. It has a database that stores some events(say, when the user presses a button, that special event is stored in the database) and timestamp. I want to know if the user has pressed that button daily at least once for 5 consecutive days.
I came across this stackoverflow answer that tells me to use recursion of SQLite. I tried to use the SQLite Recursion to 'loop for 5 days' but I am getting an error. I don't understand what I am doing wrong. Help.
Here is my code:
WITH RECURSIVE
recursiveDailyEvents (startTimeMillis, endTimeMillis, eventCount) AS
(
1612244382565, 1612330782565, (select unique count from event_tracking_table where event_tracking_table_event_id = 'post_image' and 1612244382565 <= event_tracking_table_timestamp and event_tracking_table_timestamp <= 1612330782565 )
UNION ALL startTimeMillis + 86400000, endTimeMillis + 86400000 FROM recursiveDailyEvents
Limit 5)
select * from recursiveDailyEvents;
);
This is the error from sqlite browser:
near "1612244382565": syntax error: WITH RECURSIVE
recursiveDailyEvents (startTimeMillis, endTimeMillis, eventCount) AS
(
1612244382565
But I was expecting a table with startTimeMillis, endTimeMillis, and a count (1 or 0).
What am I doing wrong? Or how should I write this recursion?
Edit
Here is some sample data
event_tracking_table_row_id| event_tracking_table_event_id| event_tracking_table_timestamp
1|app_open|1612169104224
2|post_image|1612169437373
3|post_image|1612169738068
4|app_open|1612170216320
5|post_image|1612170507935
6|app_open|1612689116738
7|post_image|1612689316673
8|post_video|1612689579697
9|post_video|1612689609683
10|app_open|1612689664683
... ... ...
Here, event_tracking_table_event_id is in millisecond.
Expected output
If I understand correctly, the recursion should generate a table of start time millisecond, end time millisecond, and eventCount between those time limits.
So today (2 February 20201) at some time, the epoch time was 1612244382565, and after 24 hours, the end time is 1612330782565, and so on.
1612244382565 , 1612330782565, 1 // 1st day start time, end time, event count
1612330782565 , 1612417182565, 0 // 2nd day start time, end time, event count
... ... // 5 rows for 5 consecutive days.
I am trying my best to be as clear as possible.

If you want the starting and ending time of each day and whether the button was clicked, you can do it with conditional aggregation:
SELECT date(event_tracking_table_timestamp / 1000, 'unixepoch', 'localtime') day,
MIN(event_tracking_table_timestamp) min_time,
MAX(event_tracking_table_timestamp) max_time,
MAX(event_tracking_table_event_id = 'post_image') event,
FROM event_tracking_table
GROUP BY day
If you want the number of times the button was clicked for each day:
SELECT date(event_tracking_table_timestamp / 1000, 'unixepoch', 'localtime') day,
MIN(event_tracking_table_timestamp) min_time,
MAX(event_tracking_table_timestamp) max_time,
SUM(event_tracking_table_event_id = 'post_image') event_count
FROM event_tracking_table
GROUP BY day
If you want the rows for the last 5 days, add a WHERE clause before GROUP BY day:
WHERE date(event_tracking_table_timestamp / 1000, 'unixepoch', 'localtime') >=
date('now', '-5 days', 'localtime')
See the demo.

How to sum inserted rows in tjavaflex component when iterating between inputs?

In Talend (open studio for data integration) 7.0.1 (and earlier versions). I use tJavaFlex to log how many rows have been inserted into a database.
Talend Job In detail:
1. Split large file into multiple smaller
2. Iterate between smaller files, insert them into database
3. Log how many rows have been inserted
The logging part logs every iteration, to look like this:
2019-01-31 09:39:20 |Stage_SalesInvoiceLine | Rows inserted 5000
2019-01-31 09:39:25 |Stage_SalesInvoiceLine | Rows inserted 5000
2019-01-31 09:39:32 |Stage_SalesInvoiceLine | Rows inserted 5000
I need help figuring out how to get it to look like this:
2019-01-31 09:39:32 |Stage_SalesInvoiceLine | Rows inserted 15000
tJavaFlex behaviour when changing loop position I've tried to look here for an answer, but did not manage to solve my problem
Current code in tJavaFlex Main code part (start and end parts are empty)
Integer Inserted = ((Integer)globalMap.get("tJDBCOutput_6_NB_LINE"));
String InsertedS = "Rows inserted " + Integer.toString(Inserted);
row19.TimeStamp = TalendDate.getDate("yyyy-MM-dd HH:mm:ss ");
row19.LogRow = "Stage_SalesInvoiceLine | " + InsertedS;

If you use local variables in tJavaFlex they will get reset at each iteration. Instead, you could define a global variable before the start of your subjob, increment it inside tJavaFlex, and retrieve its value after you've done all your inserts.
tSetGlobalVar (NB_INSERTS set to 0)
|
OnSubjobOK
|
database inserts -- OnComponentOK -- tJavaFlex
|
tFixedFlowInput -- tFileOutputDelimited
In the above tJavaFlex, you can increment your variable in the main part:
globalMap.put("NB_INSERTS", (Integer)globalMap.get("NB_INSERTS") + (Integer)globalMap.get("tJDBCOutput_1_NB_LINE_INSERTED"))
in tFixedFlowInput: "Rows inserted " + (Integer)globalMap.get("NB_INSERTS")

How to check string contain a year sarting from 20.. or 19

I have a string, I want to check whether that string contain string like bellow, and get that string no(eg:2012) if exist. (I am trying to get years of a string)
"from 20##" , "From 20##" , "from 19##" or "From 19##"
and
"to 20##" , "To 20##" , "to 19##" or "To 19##" # can be any one digit no
please tell me how to do this

Use Regular Expression to search the pattern.
You can replace any digit by "[/d]" in the regular expression.
So, if you are searching for 20##, your search pattern will be 20[\d][\d].
Have a look at this tutorial is you still have doubts.

All you really need is
int year = Integer.parseInt(str.substring(str.length()-4)
and you are free to check that year >= 1900 && year < 2100.
If you need to assert that the string matches your pattern, you can test for
str.matches("([Ff]rom|[Tt]o)\\s+\\d{4}")