I am writing a Java based Kafka Consumer application. I am utilizing kafka-clients, Spring Kafka and Spring boot for my application. While Spring boot lets me easily write Kafka Consumers (without really writing the ConcurrentKafkaListenerContainerFactory, ConsumerFactory etc), I want to be able to define / customize some of the properties for these consumers. However, I could not find out an easy way to do it using Spring boot. For eg: some of the properties that I would be interested in setting up are -
ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG
ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG
I took a look at the Spring Boot pre-defined properties here.
Also, based on an earlier question here, I want to setup the concurrency on the consumers, but cannot find a configuration, application.properties driven way to do that using Spring Boot.
An obvious way is to define the ConcurrentKafkaListenerContainerFactory, ConsumerFactory classes again in my Spring Context and work from there. I wanted to understand if there is a cleaner way of doing that, especially since I am using Spring Boot.
Versions-
kafka-clients - 0.10.0.0-SASL
spring-kafka - 1.1.0.RELEASE
spring boot - 1.5.10.RELEASE
At the URL you cited, scroll down to
spring.kafka.listener.concurrency= # Number of threads to run in the listener containers.
spring-kafka - 1.1.0.RELEASE
I recommend upgrading to at least 1.3.5; it has a much simpler threading model, thanks to KIP-62.
EDIT
With Boot 2.0, you can set arbitrary producer, consumer, admin, common properties, as described in the boot documentation.
spring.kafka.consumer.properties.heartbeat.interval.ms
With Boot 1.5 there is only spring.kafka.properties as described here.
This sets the properties for both producers and consumers, but you may see some noise in the log about unused/unsupported properties for the producer.
Alternatively, you can simply override Boot's consumer factory and add properties as needed...
#Bean
public ConsumerFactory<?, ?> kafkaConsumerFactory(KafkaProperties properties) {
Map<String, Object> consumerProps = properties.buildConsumerProperties();
consumerProps.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 5_000);
return new DefaultKafkaConsumerFactory<Object, Object>(consumerProps);
}
I googled for it and found https://github.com/spring-projects/spring-kafka/issues/604. The issue had been closed citing https://docs.spring.io/spring-boot/docs/2.0.0.RELEASE/reference/htmlsingle/#boot-features-kafka-extra-props. But is for spring boot version 2.0.
Related
I have an application built with spring boot 1.x and the whole process to expose metric is different from v2.x of spring boot.
Currently, I am using a cache in my application where i check for key existence in bulk and my requirement is that the counterService that increments a metric should increment it by some amount rather than by 1 so that I can leverage the bulk part atleast.
How do i achieve that?
One way i have done before is including Dropwizard metrics in spring boot. But I want to work under the scope of Spring boot actuators.
You may just have to implement your own CounterService using the DefaultCounterService as a template: https://github.com/spring-projects/spring-boot/blob/v1.5.17.RELEASE/spring-boot-actuator/src/main/java/org/springframework/boot/actuate/metrics/writer/DefaultCounterService.java
Then add methods something to the tune of increment(String metricName, int delta).
Then register an instance of the bean in your configuration - and it'll override the DefaultCounterService.
You'll need to refer to the instance by your specific type name so you can access the methods you added.
We are externalizing configuration of our microservices (spring boot based) using spring cloud.
As per my understanding on Spring Cloud, to enable the beans loading refreshed/updated values from Config server we need to do 2 things in Spring Cloud Client:
add #RefreshScope on the beans reading values from property files
using #Value
add spring actuator to provide /refresh endpoint to
refresh the context.
Scenario:
We have 100s of classes reading values from property file using #Value.
I have to mark all these beans refresh enabled using #RefreshScope annotation.
How can I avoid putting #RefreshScope annotation on all these classes.
Is there any shortcut or spring cloud feature to get around this situation.
You may want to look into Spring Boot feature called #ConfigurationProperties. It is designed to better organize several external configuration options.
According this Github issue, it should work for spring-cloud without #RefreshScope usage.
EDIT (reaction on comment): Maybe you are missing point of #ConfigurationProperties. With this annotation, you wouldn't use it in other configuration classes. You would have dedicated class (or few classes) only for reading and providing properties. Other configuration classes would inject this configuration holder bean.
You could encapsulate your #Values into one (or several) ConfigurationService bean which is #RefreshScoped and autowire this service into your classes instead. That way you only have a small amount of request scoped beans and your services can stay singletons.
I want to replace Basic Authentication for Spring Cloud Config Server with oAuth implementation. Let's leave Config Server alone for now and focus on changes for Config Client. Obviously I don't want to write my own implementation for whole thing, but instead execute my own logic and fallback on standard Config Client. Also I have to pack my changes into library since I will use it in multiple micro-services.
Long story short I want to achieve following:
1a. Create custom Starter which will contain Spring Cloud Config Client as dependency. Is it even doable or necessary?
or
1b. Create custom Starter with only my custom logic which will be executed before Spring Cloud Config Client. In this case each micro-service will have Spring Cloud Config Client and custom Starter as dependencies. How can I manage execution order and inject custom logic results into Config Client?
2.Introduce new bootstrap settings. e.g. spring.cloud.config.custom.username and spring.cloud.config.custom.password (Optional).
3.Introduce custom annotation for custom Starter. e.g. #enableCustomConfigClient (Optional).
I started with building custom Starter with following code in /resources/META-INF/spring.factories:
# Bootstrap components
org.springframework.cloud.bootstrap.BootstrapConfiguration=\
com.example.greeter.config.ConfigClientBootstrapConfiguration
But this code invoked after profile is set, not the first thing like Config Client does.
Any suggestions and especially code samples are appreciated. Thanks!
Posting approach I chose for future reference.
Create new package which will be executed on top of / before Spring Cloud Config Client. Two main features here:
Create file src/main/resources/META-INF/spring.factories with org.springframework.cloud.bootstrap.BootstrapConfiguration={YOUR_CLASS}
In {YOUR_CLASS} apply custom logic. Don't forget to use #org.springframework.core.annotation.Order({YOUR_PRECEDENCE}) and fact that Ordered.LOWEST_PRECEDENCE will be executed first
Build jar from previous step and include it into your project (as local file or via artifactory)
Add Custom logic to Spring Cloud Config Server so it can use JWT.
Working example is here: https://github.com/ka4ok85/spring-cloud-config-client-jwt
Would it be possible to set up a DataSource using Spring Cloud in which open JDBC connections could be injected into all of my Spring Boot applications?
Something kind of like a JNDI server lookup? If so, can someone provide some examples or a description on how to use this type of configuration?
You could use a Spring Cloud bootstrap configuration to create a DataSource. I don't see much value in doing it that way over a normal Spring Boot autoconfiguration though. Link: http://docs.spring.io/spring-boot/docs/current-SNAPSHOT/reference/htmlsingle/#boot-features-developing-auto-configuration.
One solution I found out was to set all datasources info into the properties files which will supplied by Spring Cloud Config Server to applications clients. So the aplications clients which creates DataSources gets from remote properties those values.
I am no fan of gross over abstractions, And i think Spring has committed a major felony.
But I'm willing to overlook it this time if someone can explain the algorithm behind the 'auto' configuration.
Having a look at spring's own javadocs, It doesn't give much away other than saying that it will intelligently guess what you need and something to do about conditional beans.
Does someone know what algorithm is used to determine what needs to be loaded?
In my experience as a Spring Boot user the basic factors for Spring Boot to decide on what auto-configurations will be enabled are:
1) The classes present on the classpath. For example if RabbitMQ and Spring AMQP classes are present, then the RabbitAutoConfiguration will be enabled. The corresponding annotation is #ConditionalOnClass,
2) The presence or not of user defined beans. For example, if all the Spring Data JPA is present on the classpath, Spring Boot will register a LocalContainerEntityManagerFactoryBean bean only if the user has not already done so. The beans registered by the user will 'override' the default ones. The relevant annotation is #ConditionalOnMissingBean
As #DaveSyer mentions, you can of course use Spring Boot without #EnableAutoConfiguration if you want to include the relevant configuration on your own. Or you could use the less drastic solution of the exclude field of #EnableAutoConfiguration. If for example you want Spring Boot to autoconfigure everything except ActiveMQ, you would use #EnableAutoConfiguration(exclude=ActiveMQAutoConfiguration.class)
In my opinion, there is absolutely no felony here! You can use what you want from Spring Boot. When you don't want something it has to offer, you can easily opt out partially or completely!
Also if you want to get a look under the covers, you can add the property
logging.level.org.springframework.boot=DEBUG
to application.properties and Spring Boot will gladly give a detailed report of what was auto-configured and what wasn't
There is some documentation in the Spring Boot Reference Guide. It's not terribly complicated, and I hardly think it's a felony to just include a bunch of #Configuration that you might have written anyway (because that's all it does). Feel free not to use #EnableAutoConfiguration if you prefer to include the individual configurations individually.