We Keep Coding

Spring Boot Kafka StreamsConfig or ConsumerConfig from application.yaml not applying

I have a very simple spring boot project with a KTable and I want to customize my configuration in application.yml, but the config seems to not be applied. This is my configuration file application.yml
spring:
kafka:
bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS:localhost:9092}
streams:
application-id: ${APPLICATION_ID:train-builder-processor}
buffered-records-per-partition: 50
consumer:
auto-offset-reset: earliest
max-poll-records: ${MAX_POLL_RECORDS:50}
max-poll-interval-ms: ${KAFKA_CONSUMER_MAX_POLL_INTERVAL_MS:1000}
properties:
spring:
json:
trusted:
packages:
- com.example.kafkastream
However, when starting the application the log outputs the following:
2022-03-03 08:20:06.992 INFO 32989 --- [ main] s.r.s.m.t.TrainBuilderApplication : Starting TrainBuilderApplication using Java 16.0.2 on MAPFVFG90ZQQ05P with PID 32989 (/Users/xxx/dev/train-builder-processor/target/classes started by xxx in /Users/xxx/dev/train-builder-processor)
2022-03-03 08:20:06.995 DEBUG 32989 --- [ main] s.r.s.m.t.TrainBuilderApplication : Running with Spring Boot v2.6.3, Spring v5.3.15
2022-03-03 08:20:06.995 INFO 32989 --- [ main] s.r.s.m.t.TrainBuilderApplication : No active profile set, falling back to default profiles: default
2022-03-03 08:20:08.856 INFO 32989 --- [ main] org.apache.kafka.streams.StreamsConfig : StreamsConfig values:
acceptable.recovery.lag = 10000
application.id = test.train-builder-processor
application.server =
bootstrap.servers = [localhost:9092]
buffered.records.per.partition = 1000
... (a bunch of other configs)
ConsumerConfig:
...
max.poll.interval.ms = 300000
max.poll.records = 1000
...
Below is the simple application class I'm using:
#EnableKafka
#EnableKafkaStreams
#SpringBootApplication
public class TrainBuilderApplication {
...
#Autowired
private TrainIdMapper trainIdMapper;
#Autowired
private TrainBuilder trainBuilder;
public static void main(String[] args) {
SpringApplication.run(TrainBuilderApplication.class, args);
}
#Bean
public KTable<String, Train> trainTable(StreamsBuilder kStreamBuilder) {
return kStreamBuilder
.stream(Pattern.compile(sourceTopicsPattern), Consumed.with(Serdes.String(), myJsonSerde))
.map(trainIdMapper)
.filter((key, value) -> key != null)
.groupByKey(Grouped.with(Serdes.String(), mySerde))
.aggregate(() -> null, trainBuilder, trainStore);
}
}
The values from my application.yml seems to be ignored. What could be the cause of this? What am I missing? Thanks in advance!

So I figured it out with the help of How do I properly externalize spring-boot kafka-streams configuration in a properties file?.
Apparently, consumer and producer configs are completely separated from streams config when using a KStream. To set specific properties for the consumer of the kafka stream one must use "additional properties" like so:
spring:
kafka:
bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS,localhost:9092}
streams:
application-id: ${APPLICATION_ID:train-builder-processor}
cache-max-size-buffering: 1048576
cleanup.on-shutdown: ${CLEANUP_ON_SHUTDOWN:false}
properties:
max:
poll:
records: 50
which was a bit unintuitive, but it works. Hope this can help someone in the future!

Unable to find a #SpringBootConfiguration when doing spring cloud contract test

I have a spring boot project as producer which have API endpoints, created via RouterFunction.
That API endpoint call handler which call a service and then make a call to JPA repository.
We have used liquibase to create db tables and for writing testcases we have used postgressql testcontainer.
I have written base class for producer as below:
#SpringBootTest
public abstract class RestBase extends TestBase {
#Autowired
private AccountHandler accountHandler;
#BeforeEach
public void setup() {
RestAssuredWebTestClient.webTestClient(WebTestClient.bindToRouterFunction(
new RouterConfig(accountHandler).routes()).build());
}
}
However, when I am trying to run mvn clean install, below stack trace is coming.
[INFO] Running com.example.RestTest
2021-09-02 13:46:26.207 INFO 25420 --- [ main] .b.t.c.SpringBootTestContextBootstrapper : Neither #ContextConfiguration nor #ContextHierarchy found for test class [com.example.RestTest], using SpringBootContextLoader
2021-09-02 13:46:26.208 INFO 25420 --- [ main] o.s.t.c.support.AbstractContextLoader : Could not detect default resource locations for test class [com.example.RestTest]: no resource found for suffixes {-context.xml, Context.groovy}.
2021-09-02 13:46:26.208 INFO 25420 --- [ main] t.c.s.AnnotationConfigContextLoaderUtils : Could not detect default configuration classes for test class [com.example.RestTest]: RestTest does not declare any static, non-private, non-final, nested classes annotated with #Configuration.
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.127 s <<< FAILURE! - in com.example.RestTest
[ERROR] com.example.RestTest Time elapsed: 0.127 s <<< ERROR!
java.lang.IllegalStateException: Unable to find a #SpringBootConfiguration, you need to use #ContextConfiguration or #SpringBootTest(classes=...) with your test
2021-09-02 13:46:26.346 INFO 25420 --- [ main] c.e.e.c.ContainerEnvironmentResource : Stopping all test containers.
2021-09-02 13:46:26.778 INFO 25420 --- [ionShutdownHook] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
2021-09-02 13:46:26.781 INFO 25420 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
2021-09-02 13:46:26.795 INFO 25420 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] RestTest » IllegalState Unable to find a #SpringBootConfiguration, you need to...
How to solve this issue? Is there any issue with my base test class?
Router Config class looks like below:
#Configuration
#RequiredArgsConstructor
public class RouterConfig {
private final AccountHandler accountHandler;
#Bean
public RouterFunction<ServerResponse> routes (){
return route()
.GET("/accounts",accountHandler::retrieveAccountByFilter)
.build();
}
}

I mean literally it is said right there in the error message what you should do. Just provide the configuration class in #SpringBootTest annotation. Also for contract tests you should not start any databases or repositories. You should mock them out.

Make sure that the classes you are testing are in the same namespace as your #SpringBootApplication class, or are in namespaces UNDER the namespace that contains your #SpringBootApplication class.
For example, if your app class is in com.example.demo.MyApplication but the classes you are testing is com.example.MyClass S
Or if they need to be in separate namespace trees, you may try adding the list of app classes to your #SpringBootTest annotation, e.g.
#SpringBootTest(classes=YouAppClassName.class)
public abstract class RestBase extends TestBase {
...
}

Kafka transactional producer throws 'Timeout expired while initializing transactional state in 60000ms'

I have configured a Kafka ProducerFactory with a transactionIdPrefix, in order to enable transaction synchronization using #Transactional (see Spring documentation on producer-only transactions).
I'm running an EmbeddedKafka in my integration test, to see how it behaves.
The logs show the following :
DEBUG 8384 --- [ad | producer-1] o.a.k.clients.producer.internals.Sender :
[Producer clientId=producer-1, transactionalId=tx-0-0]
Sending transactional request (type=FindCoordinatorRequest, coordinatorKey=tx-0-0, coordinatorType=TRANSACTION) to node 127.0.0.1:61445 (id: -1 rack: null)
DEBUG 8384 --- [ad | producer-1] o.a.k.c.p.internals.TransactionManager :
[Producer clientId=producer-1, transactionalId=tx-0-0]
Enqueuing transactional request (type=FindCoordinatorRequest, coordinatorKey=tx-0-0, coordinatorType=TRANSACTION)
Timeout expired while initializing transactional state in 60000ms.
This is thrown when DefaultKafkaProducerFactory executes newProducer.initTransactions().
My configuration is the following :
IntegrationTest
#EmbeddedKafka(brokerProperties = { "transaction.state.log.replication.min.isr=1", "transaction.state.log.replication.factor=1" })
ProducerConfig
#Bean
public ProducerFactory<String, String> transactionalProducerFactory() {
Map<String, Object> configuration = new HashMap<>();
configuration.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, properties.getBootstrapServers());
configuration.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
configuration.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
String transactionIdPrefix = "tx-0-";
configuration.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
DefaultKafkaProducerFactory<String, String> factory = new DefaultKafkaProducerFactory<>(configuration);
factory.setTransactionIdPrefix(transactionIdPrefix);
return factory;
}
#Bean
public KafkaTemplate<String, String> transactionalKafka() {
return new KafkaTemplate<>(transactionalProducerFactory());
}
Spring-Kafka version : 2.2.7.RELEASE
I don't see how to move forward, I think that I followed every step from the documentation and the communication between the Kafka client and the broker should be fine during transaction initialization.
Could anyone please help me fix this?

I could solve the problem thanks to the embedded kafka server logs.
Property transaction.state.log.min.isr defaulted to 2, I had to overwrite it with transaction.state.log.min.isr = 1 to fix the server error. After that my integration test passed.

Spring Boot test fails in Maven, works in IntelliJ

I have made a Spring Boot test for testing JMS consumption.
The test looks like this:
#RunWith(SpringRunner.class)
#SpringBootTest(classes = Application.class)
#DirtiesContext(classMode = DirtiesContext.ClassMode.BEFORE_EACH_TEST_METHOD)
public class UpdateThingByJmsIntegrationTest {
#Test
#Rollback(false)
public void updateThingByJmsUpdatesDatabase() throws InterruptedException {
final Thing thing = new ThingBuilder().withId(null).build();
final TransactionTemplate transactionTemplate = new TransactionTemplate(transactionManager);
transactionTemplate.setPropagationBehavior(TransactionDefinition.PROPAGATION_REQUIRES_NEW);
transactionTemplate.execute(transactionStatus -> {
thingRepository.save(thing);
return thing;
});
final String xml = String.format(
"<thingDto><id>%s</id><name>something else</name><location>somewhere</location></thingDto>",
thing.getId());
jmsMessagingTemplate.convertAndSend(thingUpdateQueue, xml);
Thread.sleep(1500L);
final Thing updatedThing = thingRepository.getOne(thing.getId());
assertNotNull(updatedThing);
assertEquals("something else", updatedThing.getName());
assertEquals("somewhere", updatedThing.getLocation());
}
So, I save a Thing in the database, then send a JMS message to update the Thing. Since JMS consumption happens in a separate thread from the test itself, I wait, and then try to verify that the Thing has been updated.
This works just fine in IntelliJ, but when running it with Maven it fails, due to the thread consuming the JMS message not being able to find the Thing in the database.
I have tried to output the object hashcode (identifier) of the ThingRepository in both the test and the code consuming the JMS message, and they come out differently. With IntelliJ they are the same. I suspect this might be part of the problem, but I'm not sure how to avoid it.
I also checked the log output in IntelliJ vs Maven, and I find that maven outputs these lines before the test is even run, which IntelliJ does not. Don't know if it is relevant.
2019-05-13 09:48:53.983 INFO 9271 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Shutting down ExecutorService 'applicationTaskExecutor'
2019-05-13 09:48:53.995 INFO 9271 --- [ main] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
2019-05-13 09:48:53.996 INFO 9271 --- [ main] .SchemaDropperImpl$DelayedDropActionImpl : HHH000477: Starting delayed evictData of schema as part of SessionFactory shut-down'
2019-05-13 09:48:54.000 INFO 9271 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-3 - Shutdown initiated...
2019-05-13 09:48:54.001 INFO 9271 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-3 - Shutdown completed.
But why would I get a different repository-object in the test and the class under test?
Update:
Turns out this only happens when running the test in question in the same run as another test. In this other test, I have:
#RunWith(SpringRunner.class)
#SpringBootTest(classes = Application.class)
#DirtiesContext(classMode = DirtiesContext.ClassMode.BEFORE_EACH_TEST_METHOD)
public class OtherIntegrationTest {
#MockBean
private ThingRepository thingRepository;
It seems this "bleeds" through to my other test, making the context use a mock while my test uses the real deal. Any way to avoid this, or do I have to find an alternative to using #MockBean?

This could be caused by lack of proper test isolation. If the updateThingByJmsUpdatesDatabase test is working by itself and fails when run as part of the test suite during build e.g. when tests are run with mvn clean install.
You should verify this by running this single test using Maven:
mvn test -Dtest=ClassName.updateThingByJmsUpdatesDatabase

Spring Boot Controller not mapping

I have used STS and now I am using IntelliJ Ultimate Edition but I am still getting the same output. My controller is not getting mapped thus showing 404 error. I am completely new to Spring Framework.
DemoApplication.java
package com.webservice.demo;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.ComponentScan;
#SpringBootApplication
public class DemoApplication {
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
}
HelloController.java
package com.webservice.demo;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
#RestController
public class HelloController {
#RequestMapping("/hello")
public String sayHello(){
return "Hey";
}
}
Console Output
com.webservice.demo.DemoApplication : Starting DemoApplication on XFT000159365001 with PID 11708 (started by Mayank Khursija in C:\Users\Mayank Khursija\IdeaProjects\demo)
2017-07-19 12:59:46.150 INFO 11708 --- [ main] com.webservice.demo.DemoApplication : No active profile set, falling back to default profiles: default
2017-07-19 12:59:46.218 INFO 11708 --- [ main] ationConfigEmbeddedWebApplicationContext : Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext#238e3f: startup date [Wed Jul 19 12:59:46 IST 2017]; root of context hierarchy
2017-07-19 12:59:47.821 INFO 11708 --- [ main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat initialized with port(s): 8211 (http)
2017-07-19 12:59:47.832 INFO 11708 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2017-07-19 12:59:47.832 INFO 11708 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet Engine: Apache Tomcat/8.5.15
2017-07-19 12:59:47.944 INFO 11708 --- [ost-startStop-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2017-07-19 12:59:47.944 INFO 11708 --- [ost-startStop-1] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 1728 ms
2017-07-19 12:59:47.987 INFO 11708 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'characterEncodingFilter' to: [/*]
2017-07-19 12:59:48.510 INFO 11708 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Registering beans for JMX exposure on startup
2017-07-19 12:59:48.519 INFO 11708 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Starting beans in phase 0
2017-07-19 12:59:48.634 INFO 11708 --- [ main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8211 (http)
2017-07-19 12:59:48.638 INFO 11708 --- [ main] com.webservice.demo.DemoApplication : Started DemoApplication in 2.869 seconds (JVM running for 3.44)

I too had the similar issue and was able to finally resolve it by correcting the source package structure following this
Your Controller classes are not scanned by the Component scanning. Your Controller classes must be nested below in package hierarchy to the main SpringApplication class having the main() method, then only it will be scanned and you should also see the RequestMappings listed in the console output while Spring Boot is getting started.
Tested on Spring Boot 1.5.8.RELEASE
But in case you prefer to use your own packaging structure, you can always use the #ComponentScan annotation to define your basePackages to scan.

Because of DemoApplication.class and HelloController.class in the same package
Locate your main application class in a root package above other classes
Take look at Spring Boot documentation Locating the Main Application Class
Using a root package also allows component scan to apply only on your
project.
For example, in your case it looks like below:
com.webservice.demo.DemoApplication
com.webservice.demo.controller.HelloController

In my case, it was missing the dependency from pom.xml, otherwise everything compiled just fine. The 404 and missing mappings info from Spring logs were the only hints.
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>

I also had trouble with a similar issue and resolved it using the correct package structure as per below. After correction, it is working properly.
e.g.
Spring Application Main Class is in package com.example
Controller Classes are in package com.example.controller

Adding #ComponentScan(com.webservice) in main class above #SpringBootApplication will resolve your problem. Refer below code
package com.webservice.demo;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.ComponentScan;
#ComponentScan(com.webservice)
#SpringBootApplication
public class DemoApplication {
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
}

In my case, I was using #Controller instead of #RestController with #RequestMapping

In my opinion, this visibility problem comes when we leave the component scan to Spring which has a particular way of looking for the classes using standard convention.
In this scenario as the Starter class(DemoApplication)is in com.webservice.demo package, putting Controller one level below will help Spring to find the classes using the default component scan mechanism. Putting HelloController under com.webservice.demo.controller should solve the issue.

It depends on a couple of properties:
server.contextPath property in application properties. If it's set to any value then you need to append that in your request url. If there is no such property then add this line in application.properties server.contextPath=/
method property in #RequestMapping, there does not seem to be any value and hence, as per documentation, it should map to all the methods. However, if you want it to listen to any particular method then you can set it to let's say method = HttpMethod.GET

I found the answer to this. This was occurring because of security configuration which is updated in newer versions of Spring Framework. So i just changed my version from 1.5.4 to 1.3.2

In my case I used wrong port for test request - Tomcat was started with several ones exposed (including one for monitoring /actuator).

In my case I changed the package of configuration file. Moved it back to the original com.example.demo package and things started working.

Another case might be that you accidentally put a Java class in a Kotlin sources directory as I did.
Wrong:
src/main
┕ kotlin ← this is wrong for Java
┕ com
┕ example
┕ web
┕ Controller.class
Correct:
src/main
┕ java ← changed 'kotlin' to 'java'
┕ com
┕ example
┕ web
┕ Controller.class
Because when in Kotlin sources directory, Java class won't get picked up.

All other packages should be an extension of parent package then only spring boot app will scan them by default.
Other option will be to use #ComponentScan(com.webservice)
package structure

👋 I set up 🍃Spring Boot Security in Maven deps. And it automatically deny access to unlogged users also for login page if you haven't change rules for it.
So I prefered my own security system and deleted this dependency.🤓
If you want to use Spring Security. You can wrote WebSecurityConfig like this:
#Configuration
#EnableWebSecurity
public class WebSecurityConfig extends WebSecurityConfigurerAdapter {
#Autowired
UserService userService;
#Bean
public BCryptPasswordEncoder bCryptPasswordEncoder() {
return new BCryptPasswordEncoder();
}
#Override
protected void configure(HttpSecurity httpSecurity) throws Exception {
httpSecurity
.csrf()
.disable()
.authorizeRequests()
//Доступ только для не зарегистрированных пользователей
.antMatchers("/registration").not().fullyAuthenticated()
//Доступ только для пользователей с ролью Администратор
.antMatchers("/admin/**").hasRole("ADMIN")
.antMatchers("/news").hasRole("USER")
//Доступ разрешен всем пользователей
.antMatchers("/", "/resources/**").permitAll()
//Все остальные страницы требуют аутентификации
.anyRequest().authenticated()
.and()
//Настройка для входа в систему
.formLogin()
.loginPage("/login")
//Перенарпавление на главную страницу после успешного входа
.defaultSuccessUrl("/")
.permitAll()
.and()
.logout()
.permitAll()
.logoutSuccessUrl("/");
}
#Autowired
protected void configureGlobal(AuthenticationManagerBuilder auth) throws Exception {
auth.userDetailsService(userService).passwordEncoder(bCryptPasswordEncoder());
}
}
from [https://habr.com/ru/post/482552/] (in russian)