We Keep Coding

Kafka transactional producer throws 'Timeout expired while initializing transactional state in 60000ms'

I have configured a Kafka ProducerFactory with a transactionIdPrefix, in order to enable transaction synchronization using #Transactional (see Spring documentation on producer-only transactions).
I'm running an EmbeddedKafka in my integration test, to see how it behaves.
The logs show the following :
DEBUG 8384 --- [ad | producer-1] o.a.k.clients.producer.internals.Sender :
[Producer clientId=producer-1, transactionalId=tx-0-0]
Sending transactional request (type=FindCoordinatorRequest, coordinatorKey=tx-0-0, coordinatorType=TRANSACTION) to node 127.0.0.1:61445 (id: -1 rack: null)
DEBUG 8384 --- [ad | producer-1] o.a.k.c.p.internals.TransactionManager :
[Producer clientId=producer-1, transactionalId=tx-0-0]
Enqueuing transactional request (type=FindCoordinatorRequest, coordinatorKey=tx-0-0, coordinatorType=TRANSACTION)
Timeout expired while initializing transactional state in 60000ms.
This is thrown when DefaultKafkaProducerFactory executes newProducer.initTransactions().
My configuration is the following :
IntegrationTest
#EmbeddedKafka(brokerProperties = { "transaction.state.log.replication.min.isr=1", "transaction.state.log.replication.factor=1" })
ProducerConfig
#Bean
public ProducerFactory<String, String> transactionalProducerFactory() {
Map<String, Object> configuration = new HashMap<>();
configuration.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, properties.getBootstrapServers());
configuration.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
configuration.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
String transactionIdPrefix = "tx-0-";
configuration.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
DefaultKafkaProducerFactory<String, String> factory = new DefaultKafkaProducerFactory<>(configuration);
factory.setTransactionIdPrefix(transactionIdPrefix);
return factory;
}
#Bean
public KafkaTemplate<String, String> transactionalKafka() {
return new KafkaTemplate<>(transactionalProducerFactory());
}
Spring-Kafka version : 2.2.7.RELEASE
I don't see how to move forward, I think that I followed every step from the documentation and the communication between the Kafka client and the broker should be fine during transaction initialization.
Could anyone please help me fix this?

I could solve the problem thanks to the embedded kafka server logs.
Property transaction.state.log.min.isr defaulted to 2, I had to overwrite it with transaction.state.log.min.isr = 1 to fix the server error. After that my integration test passed.

spring webclient load balance

Never used webclient with load balancing before and I fallowed https://spring.io/guides/gs/spring-cloud-loadbalancer/ and implemented webclient load balancer, now I am trying to use helthchecks and having problem.
#Bean
#Primary
ServiceInstanceListSupplier serviceInstanceListSupplier(ConfigurableApplicationContext ctx) {
return ServiceInstanceListSupplier
.builder()
.withRetryAwareness()
.withHealthChecks()
.withBase(new RestCaller("restCaller"))
.build(ctx);
}
and I got the error below
2021-06-27 17:32:01.562 WARN 12252 --- [ parallel-4] o.s.c.l.core.RoundRobinLoadBalancer : No servers available for service: httpbin.org
2021-06-27 17:32:01.564 WARN 12252 --- [ parallel-4] eactorLoadBalancerExchangeFilterFunction : LoadBalancer does not contain an instance for the service httpbin.org
2021-06-27 17:32:01.606 WARN 12252 --- [ parallel-4] o.s.c.l.core.RoundRobinLoadBalancer : No servers available for service: httpbin.org
2021-06-27 17:32:01.606 WARN 12252 --- [ parallel-4] eactorLoadBalancerExchangeFilterFunction : LoadBalancer does not contain an instance for the service httpbin.org
2021-06-27 17:32:01.607 WARN 12252 --- [ parallel-4] o.s.c.l.core.RoundRobinLoadBalancer : No servers available for service: httpbin.org
2021-06-27 17:32:01.607 WARN 12252 --- [ parallel-4] eactorLoadBalancerExchangeFilterFunction : LoadBalancer does not contain an instance for the service httpbin.org
2021-06-27 17:32:01.607 WARN 12252 --- [ parallel-4] o.s.c.l.core.RoundRobinLoadBalancer : No servers available for service: restCaller
2021-06-27 17:32:01.608 WARN 12252 --- [ parallel-4] eactorLoadBalancerExchangeFilterFunction : LoadBalancer does not contain an instance for the service restCaller
when I comment "withHealthChecks()" everything works as expected. My main target is to disable the "DefaultServiceInstance" in case it is failing (means http status 503 or 404 or any error).
I prepared a reproducer at https://github.com/ozkanpakdil/spring-examples/tree/master/web-client-loadbalancer just run "mvn test" you will see the error. you can see the configuration at fhttps://github.com/ozkanpakdil/spring-examples/tree/master/web-client-loadbalancer.

Thanks for providing the sample. Have gone through it. There are 2 issues:
The same #LoadBalanced WebClient.Builder instance is used both for handling the original request and sending health-check requests, so the calls coming out from HealthCheckServiceInstanceListSupplier are done with a load-balanced Webclient instead of a non-load-balanced one. Since at this stage the real hosts are being used, a non-load-balanced Webclient instance should be used for that. You can achieve it by instantiating 2 separate Webclient.Builder beans in your configuration and using qualifier to pass a non-loadbalanced one to the HealthCheckServiceInstanceListSupplier, like so:
#Configuration
#LoadBalancerClient(name = "restCaller", configuration = RestCallerConfiguration.class)
public class WebClientConfig {
#LoadBalanced
#Bean
#Qualifier("loadBalancedWebClientBuilder")
WebClient.Builder loadBalancedWebClientBuilder() {
return WebClient.builder();
}
#Bean
#Qualifier("webClientBuilder")
WebClient.Builder webClientBuilder() {
return WebClient.builder();
}
}
#Configuration
public class RestCallerConfiguration {
#Autowired
#Qualifier("webClientBuilder")
WebClient.Builder webClientBuilder;
#Bean
#Primary
ServiceInstanceListSupplier serviceInstanceListSupplier(ConfigurableApplicationContext ctx) {
return ServiceInstanceListSupplier
.builder()
.withRetryAwareness()
.withHealthChecks(webClientBuilder.build())
.withBase(new RestCaller("restCaller"))
.build(ctx);
}
The HealthCheckServiceInstanceListSupplier sends requests at a health-check URL to verify that the service instance is alive. By default, we assume that the collaborating services have spring-boot-starter-actuator in their dependencies and the request is being sent at th/actuator/health endpoint. Since this endpoint is not configured in httpbin, which the tests use, we get a 404. Changing the health-check path in properties will fix that:
spring.cloud.loadbalancer.health-check.path.default=/
I have pushed a branch with a fixed config here. If you run the test with this setup, it passes.

Testcontainers start two containers instead of one in Spring boot project

I'm using Testcontainers 1.15.3 with Spring Boot 2.4 and Junit5.
When I run my test, testcontainers starts the first container and execute flyway scripts and then stop the first container. Immediatly a second container is started (without launching flyway scripts).
My test fail because the second container does not contain data.
Abstract class:
#ExtendWith({RestDocumentationExtension.class, SpringExtension.class})
#TestPropertySource(locations = "classpath:application-test.properties")
#SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
#AutoConfigureTestDatabase(replace = AutoConfigureTestDatabase.Replace.NONE)
public abstract class AbstractIntegrationTest {
//...
}
Test class:
class ClassTest extends AbstractIntegrationTest{
#Test
void getById () throws Exception {
//...
}
}
Property file for test (jdbc url contains jdbc:tc to launch testcontainer):
spring.flyway.locations = classpath:database/structure,classpath:database/data
spring.datasource.url=jdbc:tc:postgresql:13.3:///databasename?TC_INITSCRIPT=file:src/test/resources/database/dataset/add_user.sql
Logs after launching test :
...
...
2021-06-21 12:56:52 [main] INFO 🐳 [postgres:13.3] - Creating container for image: postgres:13.3
2021-06-21 12:56:52 [main] INFO 🐳 [postgres:13.3] - Starting container with ID: 6a41054e8ec0f9045f8db9e945134234458a0e60b6157618f6f139cdf77d0cc4
2021-06-21 12:56:52 [main] INFO 🐳 [postgres:13.3] - Container postgres:13.3 is starting: 6a41054e8ec0f9045f8db9e945134234458a0e60b6157618f6f139cdf77d0cc4
...
...
2021-06-21 12:56:53 [main] INFO o.f.core.internal.command.DbMigrate - Migrating schema "public" to version "1.1.001 - init structure"
...
...
2021-06-21 12:56:55 [main] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting...
2021-06-21 12:56:55 [main] INFO 🐳 [postgres:13.3] - Creating container for image: postgres:13.3
2021-06-21 12:56:55 [main] INFO 🐳 [postgres:13.3] - Starting container with ID: f02fccb0706f047918d849f897ce52bf41870a53821663b21212760c779db05f
2021-06-21 12:56:55 [main] INFO 🐳 [postgres:13.3] - Container postgres:13.3 is starting: f02fccb0706f047918d849f897ce52bf41870a53821663b21212760c779db05f
As we see in the logs above, two containers are created.
Could you help me to solve this problem ?
Thank you.

The way I fixed it is by adding ?TC_DAEMON=true to the datasource url.
(in my case I used postgis, so just replace it with jdbc:tc:postgresql:13.3
spring:
datasource:
driver-class-name: org.testcontainers.jdbc.ContainerDatabaseDriver
url: jdbc:tc:postgis:9.6-2.5:///dbname?TC_DAEMON=true
username: xxx
password: xxx
flyway:
enabled: true
locations: 'classpath:db/migration'
url: ${spring.datasource.url}
user: ${spring.datasource.username}
password: ${spring.datasource.password}
validate-on-migrate: true

I found a solution for my case: remove flyway user and password properties to use only spring ones. The duplication of these properties caused the double launch of the datasourse.
Before
spring:
flyway:
locations: [ classpath:flyway-scripts ]
user: xxx
password: xxx
datasource:
url: jdbc:postgresql://localhost:5432/postgres
username: xxx
password: xxx
After
spring:
flyway:
locations: [ classpath:flyway-scripts ]
datasource:
url: jdbc:postgresql://localhost:5432/postgres
username: xxx
password: xxx

spring boot cold start taking too long on aws lambda and boot initializes twice

this is strange but my spring boot api taking much longer that expected when deployed on aws lambda.
in the cloudwatch log, i see spring boot is starting up twice first with default profile and second with a profile i set.
Why should it boot twice.. that is significantly costing time..
Source Code:
lambdahandler.java
public class LambdaHandler implements RequestStreamHandler {
private static SpringBootLambdaContainerHandler<AwsProxyRequest, AwsProxyResponse> handler;
static {
try {
handler = SpringBootLambdaContainerHandler.getAwsProxyHandler(Application.class);
handler.activateSpringProfiles("lambda");
} catch (ContainerInitializationException e) {
// Re-throw the exception to force another cold start
e.printStackTrace();
throw new RuntimeException("Could not initialize Spring Boot application", e);
}
}
application.java
#SpringBootApplication
public class Application extends SpringBootServletInitializer {
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
both these files are in the same package
config.java
#Configuration
#EnableWebMvc
#Profile("lambda")
public class Config {
/**
* Create required HandlerMapping, to avoid several default HandlerMapping instances being created
*/
#Bean
public HandlerMapping handlerMapping() {
return new RequestMappingHandlerMapping();
}
/**
* Create required HandlerAdapter, to avoid several default HandlerAdapter instances being created
*/
#Bean
public HandlerAdapter handlerAdapter() {
return new RequestMappingHandlerAdapter();
}
..
..
}
pom.xml
<dependency>
<groupId>com.amazonaws.serverless</groupId>
<artifactId>aws-serverless-java-container-spring</artifactId>
<version>[0.1,)</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-core</artifactId>
<version>1.2.1</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-events</artifactId>
<version>3.1.0</version>
</dependency>
cloudwatch log
07:16:51.546 [main] INFO com.amazonaws.serverless.proxy.internal.LambdaContainerHandler - Starting Lambda Container Handler
:: Spring Boot ::
2020-09-05 07:16:52.724 INFO 1 --- [ main] lambdainternal.LambdaRTEntry : Starting LambdaRTEntry on 169.254.184.173 with PID 1 (/var/runtime/lib/LambdaJavaRTEntry-1.0.jar started by sbx_user1051 in /)
2020-09-05 07:16:52.726 INFO 1 --- [ main] lambdainternal.LambdaRTEntry : No active profile set, falling back to default profiles: default
2020-09-05 07:16:52.906 INFO 1 --- [ main] ationConfigEmbeddedWebApplicationContext : Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext#1e81f4dc: startup date [Sat Sep 05 07:16:52 UTC 2020]; root of context hierarchy
..
..
2020-09-05 07:16:57.222 INFO 1 --- [ main] o.s.web.servlet.DispatcherServlet : FrameworkServlet 'dispatcherServlet': initialization completed in 40 ms
:: Spring Boot ::
2020-09-05 07:16:57.442 INFO 1 --- [ main] lambdainternal.LambdaRTEntry : Starting LambdaRTEntry on 169.254.184.173 with PID 1 (/var/runtime/lib/LambdaJavaRTEntry-1.0.jar started by sbx_user1051 in /)
2020-09-05 07:16:57.442 INFO 1 --- [ main] lambdainternal.LambdaRTEntry : The following profiles are active: lambda
2020-09-05 07:16:57.445 INFO 1 --- [ main] ationConfigEmbeddedWebApplicationContext : Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext#5ef60048: startup date [Sat Sep 05 07:16:57 UTC 2020]; root of context hierarchy

Why should it boot twice ?
I suspect your code change with activateSpringProfiles force reinitialisation.
handler.activateSpringProfiles("lambda");
https://github.com/awslabs/aws-serverless-java-container/blob/master/aws-serverless-java-container-spring/src/main/java/com/amazonaws/serverless/proxy/spring/SpringBootLambdaContainerHandler.java#L149
Try setting active profile with env variable SPRING_PROFILES_ACTIVE as part of lambda configuration file.
Java and serverless
If you use java for serverless application like AWS lambdas I would recommend for looking a framework which supports Ahead-of-Time compilation which will boost a lot your application start.
For instance have a look at Micronaut, Quarkus using with Graalvm.
Spring Boot is not the best option using with directly with AWS lambdas.

Flyway unable to set schema in java spring?

my problem is that I cannot perform a migration from flyway java spring, even though the migration files are detected, and the same migration files work from cmd.
I have already tried to set all possibly useful parameters I found on the internet to configure the schema, but it still sticks at "PUBLIC"
First of all the problem is as below: (logs from Java spring)
"2019-07-01 15:06:04.296 INFO 296 --- [ main] o.f.core.internal.command.DbMigrate : Current version of schema "PUBLIC": << Empty Schema >>
2019-07-01 15:06:04.297 INFO 296 --- [ main] o.f.core.internal.command.DbMigrate : Migrating schema "PUBLIC" to version 1 - Create person table
2019-07-01 15:06:04.324 INFO 296 --- [ main] o.f.core.internal.command.DbMigrate : Migrating schema "PUBLIC" to version 2 - Add people
2019-07-01 15:06:04.339 INFO 296 --- [ main] o.f.core.internal.command.DbMigrate : Migrating schema "PUBLIC" to version 3 - Add people2
2019-07-01 15:06:04.356 INFO 296 --- [ main] o.f.core.internal.command.DbMigrate : Successfully applied 3 migrations to schema "PUBLIC" (execution time 00:00.094s)"
The table is called public, and I also cannot see it from mysql workbench.
But if I do it from command line with flyway migrate, it alters the schema called td, which is my intention:
"Migrating schema `td` to version 1 - Create person table
Migrating schema `td` to version 2 - Add people
Successfully applied 2 migrations to schema `td` (execution time 00:00.207s)"
The flyway config for Java:
public static void main(String[] args) {
Flyway flyway = new Flyway();
flyway.setBaselineOnMigrate(true);
flyway.migrate();
SpringApplication.run(TimeReportApplication.class, args);
}
application.properties:
flyway.user=root
flyway.password=root
flyway.url=jdbc:mysql://localhost:3306/td
flyway.schemas=TD
The working flyway config for command line:
flyway.url=jdbc:mysql://localhost:3306/td
flyway.user=root
flyway.password=root
Do you have any suggestions what could go wrong?

So after a day of trying, I found a solution:
You have to add "Datasource" to your initialization file. This will be autoconfigured by Spring from your application.properties file, which you have to place in src/main/resources:
public class TimeReportApplication {
#Autowired
static DataSource dataSource;
public static void main(String[] args) {
PrintLog.print("Server started");
System.out.println("Server started");
Flyway flyway = new Flyway();
flyway.clean();
flyway.setDataSource(dataSource);
flyway.setSqlMigrationPrefix("V");
flyway.setBaselineOnMigrate(true);
flyway.migrate();
SpringApplication.run(TimeReportApplication.class, args);
}
}
In your application.properties file write before each parameter "spring", e.g.:
spring.flyway.user=root