Kafka Streams API GroupBy behaviour - java

So I've been trying to aggregate some stream data to a KTable using Kafka stream. My JSON from the topic looks like
{
"id": "d04a6184-e805-4ceb-9aaf-b2ab0139ee84",
"person": {
"id": "d04a6184-e805-4ceb-9aaf-b2ab0139ee84",
"createdBy": "user",
"createdDate": "2023-01-01T00:28:58.161Z",
"name": "person 1",
"description": "test1"
}
}....
KStream<Object, String> firstStream = builder.stream("topic-1").mapValues(value -> {
JSONObject json = new JSONObject(String.valueOf(value));
JSONObject json2 = new JSONObject(json.getJSONObject("person").toString());
return json2.toString();
});
I get something like
null{"createdDate":"2023-01-01T00:28:58.161Z","createdBy":"user","name":"person 1","description":"test1","id":"d04a6184-e805-4ceb-9aaf-b2ab0139ee84"}
null{"createdDate":"2023-01-01T00:29:07.862Z","createdBy":"user","name":"person 2","description":"test 2","id":"48d8b895-eb27-4977-9dbc-adb8fbf649d8"}
null{"createdDate":"2023-01-01T00:29:12.261Z","createdBy":"anonymousUser","name":"person 2","description":"test 2 updated","id":"d8b895-eb27-4977-9dbc-adb8fbf649d8"}
I want to group this data in such a way such that
person 1 will hold one JSON associated with it
person 2 will hold a List of both JSON associated with it
I have checked this Kafka Streams API GroupBy behaviour which describes the same problem but the solution given there doesn't work for me. Do I have to perform any extra operations? Please help

In order to groupBy, you need a pairing key. So, use map to extract the name of each person.
Then, as the linked answer says, you need to aggregate after grouping to "combine data per person", across events.
By the way, you should setup the Streams config with JsonSerde for values rather than String Serde in order to reduce the need to manually parse each event.

Related

Serialize a hashmap to a json in a certain feilds in JAVA

I need to serialize a map to a json in a certain order.
This is the map
HashMap<String, String> dataMap = {
"CompanyCode": "4",
"EntyyCode": "2002296",
"SubEntityCode": "000",
"ContractNumber": "52504467115",
"Progressive Contract": "0",
"DocumentNumber": "200003333494028",
"LogonUserName": "AR333",
"Progressive Title": "0"
}
This is the json model I would like:
{
"Policy": {
"ContractNumber": "52504467115",
"ProgressiveContract": "0"
},
"Title": {
"LogonUserName": "AR333",
"ProgressiveTitle": "0"
},
"BusinessChannel": {
"CompanyCode": "4",
"EntyyCode": "2002296",
"SubEntityCode": "000"
},
"Document": {
"DocumentNumber": "200003333494028"
}
}
I need to convert this map into a JSON string. I know that this can be done using Jackson as below:
new ObjectMapper().writeValueAsString(map);
How do I do this using Jackson? Or is there any other way to do this in Java?
Thank you
First of all, the solution you request contains a second problem: partition. Not only must the items contain a particular order, but they must also somehow be divided over different categories. In Java, these categories usually correspond to their own classes or, since recently, records. Then the top level class (corresponding to the unnamed outer object of the JSON) determines ordering, as so (the name Contract is my choice):
record Contract(
Policy policy,
Title title,
BusinessChannel businessChannel,
Document document )
{
}
with each of the properties of Contract having their own class, e.g.:
record Policy( String contractNumber, int progressiveContract )
etc.
Serializing Contract then recursively serializes each of its parameters, with the required outcome as the result.
This would be the 'standard' way.
So, since you start with a HashMap, which by contract offers no guarantee of ordering, let alone an easy way to partition its contents into sub-objects, you could try two things:
Rethink the use of a map. Switching to the class structure takes care of the structure automatically.
Manually stream and convert the values in order (or use e.g. a TreeMap with custom Comparator) and then partition the values themselves. This probably requires more work than a map saves.

Rest assured and Java: how to get wanted JSON object/body?

Lets assume I have a GET request that returns something like the following:
[
{
"id": 1,
"name": "Mark"
},
{
"id": 2,
"name": "Steve"
},
{
"id": 3,
"name": "Bill"
}
]
How can I return the wanted object from a List? or something that contains maybe this JSON as a String or what is the correct approach to get only one of the items from the response for example lets say i need to get Bills info only so i want to parse that JSON to get only this:
{
"id": 3,
"name": "Bill"
}
And no, I do not want to do this parsing in the GET request URL. I need to be able to get it from the list of everything that the GET request returns.
"$[2]"
In JsonPath, $ represents the anonymous root of the queried JSON, for cases like this one when you need to refer to it directly instead of stuff like "stuff.things[8]"
In this case, the array you're trying to analyze is the anonymous root, so you refer to it as $. Then you want the element at index 2 of this array, so it's $[2]
I did another solution that actually fits exactly for my needs. I deserialized the JSON response into a List of POJO classes that the JSON body corresponds like this:
List<MyEntity> myList = new ArrayList<>();
myList = given().
contentType(ContentType.JSON).
when().
get(getURL).
then().
extract().
body().
jsonPath().
getList(".", MyEntity.class);
This way I get a List of the initialized MyEntity classes and I can just do anything I need to, for example I just modify the values like this:
myList.get(0).setName("newName");
Then I can just send them back with a POST/PUT calls or something like that. Works like a charm!

Map inside a Map with Jackson with JSF

I'm using Jackson to read/write datas from/into json files and I have an issue with the User POJO. It has a Map wich is supposed to be the ways to contact the User (so it can have from 0 to 7, depending on the Enum). I want to be able to put ways to contact using a form in JSF.
I tried something like value="#{config.user.contacts[EMAIL_PRO]}"
where of course EMAIL_PRO is an Enum (later, the user should be able to chose the Enum himself, but right now I try simple).
But when I do so, the error is
Null key for a Map not allowed in JSON
wich I understand, 'cause my debug says that the value returned is{null = null}. Now first question : since the map is empty, is JSF supposed to work simply like that ? The key "EMAIL_PRO" doesnt exists yet, but shouldn't JSF make the work done for me, and put right value with the key ?
The other question is much more about Jackson and Maps. As I said, my POJO User contains a Map, and the json file is a Map himself (containing multiple users).
Is it really possible to write a Map into this file using Jackson where the Map is Map<String, Object> and the Object contains a Map<Enum, Object> ? And if yes, how ?
Thanks for the help
PS: I cannot change either my APIs or my POJOs.
I think this is a repeated post, see How to convert hashmap to JSON object in Java
And as it says on one of the responses:
Map<String, Object> data = new HashMap<String, Object>();
data.put( "name", "Mars" );
data.put( "age", 32 );
data.put( "city", "NY" );
JSONObject json = new JSONObject();
json.putAll( data );
System.out.printf( "JSON: %s", json.toString(2) );
output:
JSON: {
"age": 32,
"name": "Mars",
"city": "NY"
}
You can also try to use Google's GSON.Google's GSON is the best library available to convert Java Objects into their JSON representation

How to/best practice on how to differentiate between serialised objects stored within JSON format?

Suppose that I serialise two different objects and save them to a directory.
Problem: Upon application start up, parsing the JSON files are not a problem - since GSON is employed, I can write my own serialisers and deserialisers for both of the JSON files for their respective objects to be constructed.
But the problem is, how can I differentiate between the numerous JSON files in terms of what they store within them, so I can apply the correct deserialiser to it.
Thank you, best.
Consider standardizing your JSON structure to include document type. You can even store the target object type in that field. Good practice is to include document version number as well. Example below shows two different versions of the 'account' document and a transaction document. All three can be stored in, say, the same Couchbase bucket. The way to differentiate between different documents would be to look at the "doc_type" field and the document version (if required). From the GSON serializer selection standpoint, you can look at at the "doc_type" in a switch/if-else statement or store the target object type in place of "account" or "transaction" and then, at the expense of performance, dynamically parse JSON to POJO.
{
"doc_type": "account",
"doc_ver": 1,
"content": {
"accnt_no": "12321645645484",
"name": "Name or alias",
"email": "Email address",
"password": "Password in raw format",
"exp_date": "06/10/2017"
}
}
{
"doc_type": "account",
"doc_ver": 2,
"content": {
"accnt_no": "12321645645484",
"name": "customer name",
"email": "customer email",
"password": "pass",
"timezone": "customer timezone",
"ip": "IP address",
"spoken_languages": [ "EN", "RU" ],
"exp_date": "06/10/2017"
}
}
{
"doc_type": "transaction",
"doc_ver": 1,
"content": {
"accnt_no": "12321645645484",
"tran_date": "06/04/2017",
"tran_time": "09:15:84.953"
}
}
Hope this helps.
I think that the best way is parse JSON to a HashMap<String, Object> with multiple level. GSON will parse your JSON to HashMap with key is object name and value is an object (This object will belong to 3 type: HashMap for a object in JSON, List for an array in JSON and String for a string in JSON). To using this HashMap you need to iterate through the HashMap using a recursive method.

Elasticsearch Java API do not return fields of hits

I have a problem with the java api for elasticsearch.
When I do a search like this:
MatchQueryBuilder query = QueryBuilders.matchQuery("_type", "booking");
SearchResponse searchResponse = client.prepareSearch().setQuery(query).execute().actionGet();
for (SearchHit hit : searchResponse.getHits()){
Map<String, SearchHitField> fields = hit.getFields();
System.out.println(fields.size());
}
my response never has fields, can somebody help me ?
I'am using:
elasticsearch java api 1.4.0
elasticsearch 1.4.0
and my data looks like
{
"_index": "bookings",
"_type": "booking",
"_id": "50245171",
"_score": 1,
"_source": {
"field1": "value1",
"field2": "value2",
"field3": "value3",
...
}
}
Have you tried adding .addFields() to your query?
SearchResponse searchResponse = client.prepareSearch().setQuery(query).addFields("field1", "field2",...).execute().actionGet();
I'm not sure about all details, but I believe elasticsearch tries to send you as few data as possible. Which makes sense, since it should be fast and light.
In any case, are you by any chance indexing Booking objects? Because if you need the whole object again, you could also just fetch the source and transform that back into its original Booking object. For example:
ObjectMapper mapper = new ObjectMapper();
Booking booking = mapper.readValue(hit.getSourceAsString(), Booking.class);
ObjectMapper is from com.fasterxml.jackson.databind.ObjectMapper (should be included with elasticsearch I believe).
the fields are not responding as fields....
with hit.getSource() i got my information
The fields section of a hit includes the fields you explicitly ask for, by default that is none. You can use addField() or addFields() to add one or more fields to the request. This should populate the fields in the response for you.
The source in each hit contains the entire original object, so may well have data you do not need. Returning the source in this manner is often turned off to prevent sending this excess data around.
QueryBuilders.matchQuery()
seems to work with fields only
"_type" is a build-in properties of the index...
Try something like that :
client.prepareSearch("bookings")
.setTypes("booking")
.setQuery(query)
.execute()
.actionGet();
Your query must be based on your document fields

Categories