We Keep Coding

Kafka Stream with multiple consumers / processors not persisting data concurrently

I'm new with Kafka and want to persist data from kafka topics to database tables (each topic flow to a specific table). I know Kafka connect exists and can be used to achieve this but there are reasons why this approach is preferred.
Unfortunately only one topic is writing the database. Kafka seems to not process() all processors concurrently. Either MyFirstData is writing to database or MySecondData but never but at the same time.
According the my readings the is the option overriding init() from of kafka stream Processor interface which offers context.forward() not sure if this will help and how to use it in my used case.
I use Spring Cloud Stream (but got the same behaviour with Kafka DSL and Processor API implementations)
My code snippet:
Configuring the consumers:
#Configuration
#RequiredArgsConstructor
public class DatabaseProcessorConfiguration {
private final MyFirstDao myFirstDao;
private final MySecondDao mySecondDao;
#Bean
public Consumer<KStream<GenericData.Record, GenericData.Record>> myFirstDbProcessor() {
return stream -> stream.process(() -> {
return new MyFirstDbProcessor(myFirstDao);
});
}
#Bean
public Consumer<KStream<GenericRecord, GenericRecord>> mySecondDbProcessor() {
return stream -> stream.process(() -> new MySecondDbProcessor(mySecondDao));
}
}
This MyFirstDbProcessor and MySecondDbProcessor is analog to this.
#Slf4j
#RequiredArgsConstructor
public class MyFirstDbProcessor implements Processor<GenericData.Record, GenericData.Record, Void, Void> {
private final MyFirstDao myFirstDao;
#Override
public void process(Record<GenericData.Record, GenericData.Record> record) {
CdcRecordAdapter adapter = new CdcRecordAdapter(record.key(), record.value());
MyFirstTopicKey myFirstTopicKey = adapter.getKeyAs(MyFirstTopicKey.class);
MyFirstTopicValue myFirstTopicValue = adapter.getValueAs(MyFirstTopicValue.class);
MyFirstData data = PersistenceMapper.map(myFirstTopicKey, myFirstTopicValue);
switch (myFirstTopicValue.getCrudOperation()) {
case UPDATE, INSERT -> myFirstDao.persist(data);
case DELETE -> myFirstDao.delete(data);
default -> System.err.println("unimplemented CDC operation streamed by kafka");
}
}
}
My Dao implementations: I try an implementation of MyFirstRepository with JPARepository and ReactiveCrudRepository but same behaviour. MySecondRepository is implemented analog to MyFirstRepository.
#Component
#RequiredArgsConstructor
public class MyFirstDaoImpl implements MyFirstDao {
private final MyFirstRepository myFirstRepository;
#Override
public MyFirstData persist(MyFirstData myFirstData) {
Optional<MyFirstData> dataOptional = MyFirstRepository.findById(myFirstData.getId());
if (dataOptional.isPresent()){
var data = dataOptional.get();
myFirstData.setCreatedDate(data.getCreatedDate());
}
return myFirstRepository.save(myFirstData);
}
#Override
public void delete(MyFirstData myFirstData) {
System.out.println("delete() from transaction detail dao called");
MyFirstRepository.delete(myFirstData);
}
}

how to show the number of calls per minute on grafana using micrometer

I have a simple Spring Boot application. I want to monitor application metrics using grafana and prometheus. To do this, I added the actuator and micrometer-registry-prometheus to the application
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
I have written two methods for counter and timer
#Service
public class MicrometerServiceImpl implements MicrometerService {
#Autowired
private MeterRegistry meterRegistry;
#Override
public void addCounterMetric(String statName, String description, double count, String... tags){
Counter.builder(statName)
.description(description)
.tags(tags)
.register(meterRegistry)
.increment(count);
}
#Override
public void addTimerMetric(String statName, String description, long duration, String... tags){
Timer.builder(statName)
.description(description)
.tags(tags)
.register(meterRegistry)
.record(duration, TimeUnit.MILLISECONDS);
}
}
And added a method call to the controller
#RequestMapping(path = "/addMetric")
public String callMetric(HttpServletRequest request) {
long start = System.currentTimeMillis();
//some work
micrometerService.addCounterMetric("CONTROLLER_COUNTER", "controller with COUNTER", 1);
micrometerService.addTimerMetric("CONTROLLER_TIMER", "controller with Timer",
System.currentTimeMillis() - start);
return "addMetric";
}
I would like the counter value to be reset to zero every minute, that is, the number of calls per minute is counted. Is it possible to do this using micrometer or grafana? Now I see that the counter is only increasing.
I know there are metrics that count the number of http requests. But I need this for custom metrics, when the metric is considered under certain conditions (for example, the number of errors during processing)

Spring-data-aerospike | how to achieve rollback in case whole transaction is rolled back

Use Case :-
We are using Spring-data-aerospike to get and save the aerospike record.
Problem :- We are performing SAVE into two different aerospike sets and both of these SAVES should happen in an transactional manner i.e. if 2nd write fails, then the first write should also get rolled back.
Here is code-snippet looks like :-
#Document(collection = "cust", expiration = 90, expirationUnit = TimeUnit.DAYS)
public class Customer {
#Id
#Field(value = "PK")
private String custId;
#Field(value = "mobileNumber")
private String mobileNumber;
}
#Document(collection = "custDetails", expiration = 90, expirationUnit = TimeUnit.DAYS)
public class CustomerDetails {
#Id
#Field(value = "PK")
private String custDetailsId;
#Field(value = "addnDetails")
private Map additionalDetails;
}
#Repository
public interface CustomerRepository extends AerospikeRepository<Customer, String> {}
#Repository
public interface CustomerDetailsRepository extends AerospikeRepository<CustomerDetails, String>{}
#Autowired
AerospikeTemplate aerospikeTemplate;
This is what we want to achieve, which is not working right now :--
#Transactional(isolation = Isloation.SERIALIZABLE, rollbackFor=Exception.class)
public ResponseDTO<String> updateCustomer(CustomerUpdateRequest custUpdateReqDTO) {
Optional<Customer> cust = customerRepository.findById(custUpdateReqDTO.getCustId());
// Update Business logic of Customer Record.
aerospikeTemplate.update(cust);
Optional<CustomerDetails> custDet = customerDetailsRepository.findById(custUpdateReqDTO.getCustDetId());
// Update Business logic of CustomerDetails Record.
aerospikeTemplate.update(custDet);
}
Here is dependencies looks like :-
<dependency>
<groupId>com.aerospike</groupId>
<artifactId>aerospike-client</artifactId>
<version>4.1.3</version>
</dependency>
<dependency>
<groupId>com.aerospike</groupId>
<artifactId>spring-data-aerospike</artifactId>
<version>${aerospike.data.version}</version>
<scope>system</scope>
<systemPath>${basedir}/lib/spring-data-aerospike-2.0.0.RELEASE.jar</systemPath>
</dependency>
<dependency>
<groupId>com.aerospike</groupId>
<artifactId>aerospike-client</artifactId>
</dependency>
<dependency>
<groupId>com.aerospike</groupId>
<artifactId>aerospike-helper-java</artifactId>
<version>1.2.2</version>
</dependency>
Questions :-
We are aware that spring transactional annotations works with RDBMS !! Whats the way to get the transactional property working here in this scenario ??
Any help or suggestions shall be highly appreciated !

Aerospike has transaction support for single record only. You will need to perform insert-rollback logic inside your application for multiple records.

Multiple Consumers Spring Kafka

Note to duplicate markers: I DID check out the other question, but it does not answer my specific question below.
So imagine I have a Kafka topic on a single server with only one partition. So it is much similar to a queue.
Now lets assume I want 100 listeners waiting to accept values from the queue.
So by design, if all 100 consumers are in a single group, the contents from the log (or queue here) will be distributed among the consumers. So the operation will be over in 1/100th of the time.
The problem is that the Spring Kafka listener is only configured with the topic name.
#Service
public class Consumer {
#KafkaListener(topics = "${app.topic}")
public void receive(#Payload String message,
#Headers MessageHeaders headers) {
System.out.println("Received message="+message);
headers.keySet().forEach(key -> System.out.println(key+"->"+headers.get(key)));
}
}
I can seem to get Kafka to spawn up a 100 consumers for processing messages from the "queue" (logs).
How can it be done?

Check out this answer for an understanding of Kafka consumers In Apache Kafka why can't there be more consumer instances than partitions?
To properly distribute messages amongst a single consumer group you must have more than one partition. Once you find the correct partition amount for your load I would use spring cloud streaming to better manage your concurrency and consumer group assignment.
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-stream-kafka</artifactId>
</dependency>
Sample of sink
#SpringBootApplication
#EnableBinding(Sink.class)
public class LoggingConsumerApplication {
public static void main(String[] args) {
SpringApplication.run(LoggingConsumerApplication.class, args);
}
#StreamListener(Sink.INPUT)
public void handle(Person person) {
System.out.println("Received: " + person);
}
public static class Person {
private String name;
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public String toString() {
return this.name;
}
}
}
Concurrency settings
cloud:
stream:
bindings:
input:
destination: <topic-name>
group: <consumer-group>
consumer:
headerMode: raw
partitioned: true
concurrency: 20
More information available here https://cloud.spring.io/spring-cloud-stream/

Spring MVC 3: return a Spring-Data Page as JSON

I have a data access layer made with Spring-Data. I'm now creating a web application on top of it. This one controller method should return a Spring-Data Page formatted as JSON.
Such a Page is a List with additional Paging info like total amount of records and so forth.
Is that possible and if yes how?
And directly related to that can I define the mapping of property names? Eg. meaning I would need define how the paging info properties are named in JSON (differently than in page). Is this possible and how?

There's support for a scenario like this upcoming in Spring HATEOAS and Spring Data Commons. Spring HATEOAS comes with a PageMetadata object that essentially contains the same data as a Page but in a less enforcing manner, so that it can be more easily marshaled and unmarshaled.
Another aspect of the reason we implement this in combination with Spring HATEOAS and Spring Data commons is that there's little value in simply marshaling the page, it's content and the metadata but also want to generate the links to maybe existing next or previous pages, so that the client doesn't have to construct URIs to traverse these pages itself.
An example
Assume a domain class Person:
class Person {
Long id;
String firstname, lastname;
}
as well as it's corresponding repository:
interface PersonRepository extends PagingAndSortingRepository<Person, Long> { }
You can now expose a Spring MVC controller as follows:
#Controller
class PersonController {
#Autowired PersonRepository repository;
#RequestMapping(value = "/persons", method = RequestMethod.GET)
HttpEntity<PagedResources<Person>> persons(Pageable pageable,
PagedResourcesAssembler assembler) {
Page<Person> persons = repository.findAll(pageable);
return new ResponseEntity<>(assembler.toResources(persons), HttpStatus.OK);
}
}
There's probably quite a bit to explain here. Let's take it step by step:
We have a Spring MVC controller getting the repository wired into it. This requires Spring Data being set up (either through #Enable(Jpa|Mongo|Neo4j|Gemfire)Repositories or the XML equivalents). The controller method is mapped to /persons, which means it will accept all GET requests to that method.
The core type returned from the method is a PagedResources - a type from Spring HATEOAS that represents some content enriched with Links plus a PageMetadata.
When the method is invoked, Spring MVC will have to create instances for Pageable and PagedResourcesAssembler. To get this working you need to enable the Spring Data web support either through the #EnableSpringDataWebSupport annotation about to be introduced in the upcoming milestone of Spring Data Commons or via standalone bean definitions (documented here).
The Pageable will be populated with information from the request. The default configuration will turn ?page=0&size=10 into a Pageable requesting the first page by a page size of 10.
The PageableResourcesAssembler allows you to easily turn a Page into a PagedResources instances. It will not only add the page metadata to the response but also add the appropriate links to the representation based on what page you access and how your Pageable resolution is configured.
A sample JavaConfig configuration to enable this for JPA would look like this:
#Configuration
#EnableWebMvc
#EnableSpringDataWebSupport
#EnableJpaRepositories
class ApplicationConfig {
// declare infrastructure components like EntityManagerFactory etc. here
}
A sample request and response
Assume we have 30 Persons in the database. You can now trigger a request GET http://localhost:8080/persons and you'll see something similar to this:
{ "links" : [
{ "rel" : "next", "href" : "http://localhost:8080/persons?page=1&size=20 }
],
"content" : [
… // 20 Person instances rendered here
],
"pageMetadata" : {
"size" : 20,
"totalElements" : 30,
"totalPages" : 2,
"number" : 0
}
}
Note that the assembler produced the correct URI and also picks up the default configuration present to resolve the parameters into a Pageable for an upcoming request. This means, if you change that configuration, the links will automatically adhere to the change. By default the assembler points to the controller method it was invoked in but that can be customized by handing in a custom Link to be used as base to build the pagination links to overloads of the PagedResourcesAssembler.toResource(…) method.
Outlook
The PagedResourcesAssembler bits will be available in the upcoming milestone release of the Spring Data Babbage release train. It's already available in the current snapshots. You can see a working example of this in my Spring RESTBucks sample application. Simply clone it, run mvn jetty:run and curl http://localhost:8080/pages.

Oliver, your answer is great and I mark it as answer. Here just for completeness what I came up with for the mean time which might be useful for someone else.
I use JQuery Datatables as my grid/table widget. It sends very specific parameter to server and excepts a very specific response: see http://datatables.net/usage/server-side.
To achieve this is created a custom helper object reflecting what datatables expects. Note that getter and setter must be named like they are else the produced json is wrong (case sensitive property names and datatables uses this "pseudo Hungarian notation"...).
public class JQueryDatatablesPage<T> implements java.io.Serializable {
private final int iTotalRecords;
private final int iTotalDisplayRecords;
private final String sEcho;
private final List<T> aaData;
public JQueryDatatablesPage(final List<T> pageContent,
final int iTotalRecords,
final int iTotalDisplayRecords,
final String sEcho){
this.aaData = pageContent;
this.iTotalRecords = iTotalRecords;
this.iTotalDisplayRecords = iTotalDisplayRecords;
this.sEcho = sEcho;
}
public int getiTotalRecords(){
return this.iTotalRecords;
}
public int getiTotalDisplayRecords(){
return this.iTotalDisplayRecords;
}
public String getsEcho(){
return this.sEcho;
}
public List<T> getaaData(){
return this.aaData;
}
}
The second part is a method in the according controller:
#RequestMapping(value = "/search", method = RequestMethod.GET, produces = "application/json")
public #ResponseBody String search (
#RequestParam int iDisplayStart,
#RequestParam int iDisplayLength,
#RequestParam int sEcho, // for datatables draw count
#RequestParam String search) throws IOException {
int pageNumber = (iDisplayStart + 1) / iDisplayLength;
PageRequest pageable = new PageRequest(pageNumber, iDisplayLength);
Page<SimpleCompound> page = compoundService.myCustomSearchMethod(search, pageable);
int iTotalRecords = (int) (int) page.getTotalElements();
int iTotalDisplayRecords = page.getTotalPages() * iDisplayLength;
JQueryDatatablesPage<SimpleCompound> dtPage = new JQueryDatatablesPage<>(
page.getContent(), iTotalRecords, iTotalDisplayRecords,
Integer.toString(sEcho));
String result = toJson(dtPage);
return result;
}
private String toJson(JQueryDatatablesPage<?> dt) throws IOException {
ObjectMapper mapper = new ObjectMapper();
mapper.registerModule(new Hibernate4Module());
return mapper.writeValueAsString(dt);
}
compoundService is backed by a Spring-Data repository. It manages transactions and method level security. toJSON() method uses Jackson 2.0 and you need to register the appropriate module to the mapper, in my case for hibernate 4.
In case you have bidirectional relationships, you need to annotate all your entity classes with
#JsonIdentityInfo(generator=ObjectIdGenerators.IntSequenceGenerator.class, property="jsonId")
This enables Jackson 2.0 to serialize circular dependencies (was not possible in earlier version and requires that your entities are annotated).
You will need to add following dependencies:
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.2.1</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-hibernate4</artifactId>
<version>2.2.1</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.2.1</version>
<type>jar</type>
</dependency>

Using Spring Boot (and for Mongo DB) I was able to do the following with successful results:
#RestController
#RequestMapping("/product")
public class ProductController {
//...
#RequestMapping(value = "/all", method = RequestMethod.GET, produces = {MediaType.APPLICATION_JSON_VALUE })
HttpEntity<PagedResources<Product>> get(#PageableDefault Pageable p, PagedResourcesAssembler assembler) {
Page<Product> product = productRepository.findAll(p);
return new ResponseEntity<>(assembler.toResource(product), HttpStatus.OK);
}
}
and the model class is like this:
#Document(collection = "my_product")
#Data
#ToString(callSuper = true)
public class Product extends BaseProduct {
private String itemCode;
private String brand;
private String sku;
}