We Keep Coding

Spring boot test using testcontainers postgresql

I have a Java Spring boot project, heavily using the database (Postgres) for it's repository/data. It is the basic MVC project, controllers are all REST controllers. The project works well (service is up, able to call service via REST clients and all).
Now, I am adding the unit tests to it. I am pretty new to Spring boot and mostly the unit test part. Owing to the CI/CD (build pipeline), I cannot use the persistent/external DB for tests. Hence I need to use in-memory DB.
The initial run (Main class) runs a bunch of DB queries to build up cache while project comes up. So I would need postgres DB for testing (lots of DB functions used).
Basically, I would need to use Testcontainers (postgresql). I am writing a very basic test first to get hold of it.
I have the schema.sql and data.sql stored (to be used only for testing).
src
|
main
test
|
resources
|
application-test.properties
schema.sql
data.sql
Relevant pom.xml
<properties>
<java.version>11</java.version>
<testcontainers.version>1.15.1</testcontainers.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-cache</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
</dependency>
<dependency>
<groupId>io.jsonwebtoken</groupId>
<artifactId>jjwt</artifactId>
<version>0.9.1</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-validation</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<scope>runtime</scope>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.junit.vintage</groupId>
<artifactId>junit-vintage-engine</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.11.2</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.11</version>
</dependency>
<dependency>
<groupId>commons-beanutils</groupId>
<artifactId>commons-beanutils</artifactId>
<version>1.9.4</version>
</dependency>
<dependency>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
<version>20201115</version>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>testcontainers</artifactId>
<version>${testcontainers.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>localstack</artifactId>
<version>${testcontainers.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>junit-jupiter</artifactId>
<version>${testcontainers.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>postgresql</artifactId>
<version>${testcontainers.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
My Test class:
#Testcontainers
#Sql(scripts = {"file:src/test/resources/schema.sql","file:src/test/resources/data.sql"})
class ApplicationTests {
#Container
static PostgreSQLContainer<?> postgreSQLContainer = new PostgreSQLContainer<>("postgres:12")
.withUsername("testcontainers")
.withPassword("testcontainers")
.withDatabaseName("tescontainers");
#Test
void testPostgreSQLModule() throws SQLException {
try (Connection connection = DriverManager
.getConnection(postgreSQLContainer.getJdbcUrl(), "testcontainers", "testcontainers");
PreparedStatement preparedStatement = connection.prepareStatement("SELECT * FROM table_from_schema")) {
try (ResultSet resultSet = preparedStatement.executeQuery()) {
while (resultSet.next()) {
System.out.println(resultSet.getString("column1"));
}
}
}
}
}
I am simply trying to test the DB.
However, when I run the test, it fails saying that
org.postgresql.util.PSQLException: ERROR: relation "table_from_schema" does not exist
I tried to debug it, i.e. stopped just inside my Test (testPostgreSQLModule). I can see the docker component with Postgres.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
fc3ff1e04ceb postgres:12 "docker-entrypoint.s…" 16 seconds ago Up 15 seconds
But when I login to it and run psql, I see the DB (tescontainers) is created, however it does not have any schema (tables/functions).
tescontainers=# \dt
Did not find any relations.
tescontainers=#
Basically, my files are not being run.
Does #SQL annotation on class level does not work with Testcontainers initialization as in my case?
What is needed here so both my initial scripts run?
I tried using .withInitScript, and it runs. However, I have lots of data to initialize and the file is too large (and would grow), so I separate DDL (schema) and Inserts (data). Now, my issue is how to run multiple init files (schema.sql, data.sql) using "withInitScript" ? So I tries #SQL annotation, but it does not seems to work.
---UPDATE/EDIT----
To give the context clear, I am looking for below. If anyone can guide please?
All profile (dev/ist/uat/prod) should be using their respective persistence DB (from their application.env.properties).
Only for the test, I need the in-memory DB, but cannot use H2 (and similar) as I have lots of DB related tests and need Postgres (functions , etc). So, trying out Testcontainers.
When the application boots up, it fetches some data from respective DB (based on env) to prepare the initial cache and other methods will use it while servicing any rest calls. Hence, for Test (only) I need a new in-memory DB with all schema/data (which I can provide via SQL files) , so that while test, the boot up should use that test DB and corresponding tests wil work based on that initial data.
So I need a way to bring up test DB (in-memory/testcontainers) whenever Test runs, and pass multiple SQL files for initialization of Test DB (before any test runs). Any idea on what is the best approach?

The error is telling you no such table exists
org.postgresql.util.PSQLException: ERROR: relation "table_from_schema" does not exist
It seems that the connection to the DB is established, but the query you're attempting is not working.
I had a quick look and from what I could see the table table_from_schema is not present in the standard postgres schema.
However select * from pg_tables; may work. Try accessing a different table and see if this resolves the issue for you temporarily, so you can confirm the issue is selecting from a table name that doesn't exist rather than something more obscure.
If you are able to, you can put a break point in after the docker container has spun up and connect using a database client. Once you're connected you can then try inspect the database manually, testing your SQL commands before you copy-paste them to your codebase.
If you want the SQL scripts to run in your docker container, you need to mount the test/resources dir to the container's /docker-entrypoint-initdb.d dir using a volume mapping for the container. Any *.sh and *.sql scripts will then be run. You may need to restructure your sql scripts so you have one root script that calls scripts in a directory lower down. This is especially true if there's any ordering requirement between the two.
See https://hub.docker.com/_/postgres "Initialization scripts"
Mounting into your container with testcontainers is as simple as:
String pathToFile = "<project-root-dir>/src/test/resources";
new GenericContainer(...)
.withFileSystemBind(pathToFile, "/docker-entrypoint-initdb.d", BindMode.READ_ONLY)
See https://www.testcontainers.org/features/files/

I guess you missed the following
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<scope>test</scope>
</dependency>
test-dependency. (Or remove the <scope/> at all.)
Table not found errors often indicate a mismatch of the schema and model and the available driver.
Also your test class seems to be spring-boot independent. How do you run it? You could benefit of auto-configuration, like described here: https://www.baeldung.com/spring-boot-testcontainers-integration-test
(but I guess you don't like that ;))

Spring Boot - Issue with Spring Boot Starter Actuator

So I am creating a new spring boot project and wanted to play around with spring-boot-starter-actuator. However I am facing issues when starting the application.
Pom Snippet:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-logging</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>
<!--<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>-->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-validation</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<spring-boot.version>2.2.0.RELEASE</spring-boot.version>
spring-boots on my classpath:
Error while starting the application:
***************************
APPLICATION FAILED TO START
***************************
Description:
An attempt was made to call a method that does not exist. The attempt was made from the following location:
org.springframework.boot.actuate.autoconfigure.metrics.orm.jpa.HibernateMetricsAutoConfiguration.bindEntityManagerFactoryToRegistry(HibernateMetricsAutoConfiguration.java:68)
The following method did not exist:
io.micrometer.core.instrument.binder.jpa.HibernateMetrics.<init>(Lorg/hibernate/SessionFactory;Ljava/lang/String;Ljava/lang/Iterable;)V
The method's class, io.micrometer.core.instrument.binder.jpa.HibernateMetrics, is available from the following locations:
jar:file:/C:/Users/rahul/.m2/repository/io/micrometer/micrometer-core/1.0.2/micrometer-core-1.0.2.jar!/io/micrometer/core/instrument/binder/jpa/HibernateMetrics.class
It was loaded from the following location:
file:/C:/Users/rahul/.m2/repository/io/micrometer/micrometer-core/1.0.2/micrometer-core-1.0.2.jar
Action:
Correct the classpath of your application so that it contains a single, compatible version of io.micrometer.core.instrument.binder.jpa.HibernateMetrics
At this point the exception happens:
However there is only one constructor of HibernateMetrics which looks like this:
public HibernateMetrics(EntityManagerFactory entityManagerFactory, String entityManagerFactoryName, Iterable<Tag> tags) {
this.tags = Tags.concat(tags, "entityManagerFactory", entityManagerFactoryName);
this.stats = hasStatisticsEnabled(entityManagerFactory) ? getStatistics(entityManagerFactory) : null;
}
From the dependency analyzer, one could see that there are not multiple versions of micrometer-core:
I also tried with spring-boot-starter-actuator version of 2.2.0.RELEASE but that has the same issue.
I am not sure what am I missing here, any help will be really appreciated.

Assuming that you'll connect a spring-boot-actuator application to a JMX console. ( "because it's not a web application")
I've used Spring Initializr based on your pom dependencies and a CommandLineRunner example. Github example: https://github.com/thiagochagas/actuator-example
Adjusts:
I've removed the "spring-boot-starter" dependency :
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-logging</artifactId>
</exclusion>
</exclusions>
</dependency>
I've used a Thread.sleep(30000L) in the DemoApplication class to simplify the example.
Install and Run the Application:
./mvnw clean install
java -jar target/demo-0.0.1-SNAPSHOT.jar
Open the jconsole:
$JAVA_HOME/bin/jconsole
While your application is running it should be on the jconsole.
Select your "demo-0.0.1-SNAPSHOT.jar" to analyze:
If this message is shown, select the option "Insecure Connection":
Analysis of the running application:

Connecting Squirrel Client to Embedded Teiid Server

I have a Spring Boot project, on which I have some csv files and have converted them into entities and querying them based on my requirement. For this approach, I am using Teiid Spring Boot Starter, which is starting a embedded server -
This is the console startup log -
Starting embedded database: url='jdbc:teiid:spring;PassthroughAuthentication=true;useCallingThread=true;autoFailover=true;waitForLoad=5000;autoCommitTxn=OFF;disableLocalTxn=true', username='null' ````
pom.xml
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.teiid</groupId>
<artifactId>teiid-spring-boot-starter</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.teiid</groupId>
<artifactId>teiid-12.1.1-jdbc</artifactId>
<version>12.2.1</version>
<scope>system</scope>
<systemPath>${project.basedir}/src/main/resources/teiid-12.2.1-jdbc.jar</systemPath>
</dependency>
</dependencies>
application.properties
spring.application.name=Teiid-spring-boot
spring.teiid.model.package=com.example.demo.model
spring.teiid.file.parent-directory=src/main/resources/csv
#######
logging.level.org.teiid.spring=TRACE
spring.main.allow-bean-definition-overriding=true
spring.jpa.show-sql=true
spring.jpa.properties.hibernate.format_sql=true
I have Squirrel SQL client setup, have added the driver for Teiid as well.
teiid-12.2.1-jdbc.jar
Squirrel SQL connection settings used -
name - Teiid
example url - jdbc:teiid:spring
website url - http://teiid.org
But while connecting, getting this following error -
teiid: JDBC Driver class not found
class java.lang.ClassNotFoundException: org.jboss.modules.ModuleLoadException

There are multiple issues that need to be fixed.
You do not need the teiid-12.1.1-jdbc dependency in pom.xml
Add teiid.jdbc-enable=true to your application.properties that will open a jdbc port 31000 for the application you built.
Run your application
Then add the Teiid JDBC driver to SquirreL (which you seemed to be already done)
Use the URL as jdbc:teiid:spring#mm://localhost:31000 where localhost is host where you are running your teiid-spring application.

Spring unknown property 'spring.cloud.config.server'

I am attempting to connect to a git config server in my spring project following the example below in an eclipse IDE
https://cloud.spring.io/spring-cloud-static/Camden.SR5/
However I am running into issues being able to resolve the properties needed as eclipse is not able to recognize the server property in my bootstrap.yml (unknown property 'spring.cloud.config.server')
spring:
cloud:
config:
*server*:
git:
uri: .....config_server.git
I have included all the dependencies listed in the example above (spring-cloud-dependencies, spring-cloud-starter-config, spring-boot-starter-test) however I still receive the error.
Is there limitations as to when this property can be used? Or is there an additional dependency that is needed?

This was added to the question by the OP and has been moved here.
Turns I just needed to add a spring-cloud-starter-config-server dependency. Not sure why it did not get pulled in by by the others but that did the trick.

here is a small example of how we use it using the spring cloud config:
bootstrap.yml
spring:
profiles:
# this will tell spring to pick the correct profile file
active: sheba
# u must specify your application name for it to be found in the git repository!
application:
name: OpscI2aClient
# now we telling our git client where to locate our config files
cloud:
config:
server:
bootstrap: true
git:
# this the full uri to our repository where the config file
# wich its name is ApplicationName-profileName.yml
# are found
uri: https://some.company.git.com/config/config-repo.git
username: someGitUserName
password: thePasswordOfTheGitUser
# clone the whole config repository on startup
the whole files in the repository are cloned!
clone-on-start: true
# this tell spring the name of the local repository directory
# which in this case it shall create a directory called git_local
relatively to where the application started
basedir:
git_local
so, assuming you have a git repo, it must contain a file OpscI2aClient-sheba.yml
this file is just a simple spring boot configuration file, however, should not contain active profile entry.
that is it.
Just to clarify:
Assuming the application named OpscI2aClient have two profiles, dev and prod,
your git configuration repositoty should actually contain, two files:
1. OpscI2aClient-dev.yml
2. OpscI2aClient-prod.yml
Hope this helps.
I am just adding the POM in which you might find some hints as well
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<project.scm.id>opsc-scm-server</project.scm.id>
<java.version>1.8</java.version>
<slf4j.version>1.7.2</slf4j.version>
<logback.version>1.1.3</logback.version>
<spring-cloud.version>Edgware.RELEASE</spring-cloud.version>
</properties>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.5.9.RELEASE</version>
<relativePath /> <!-- lookup parent from repository -->
</parent>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-config-server</artifactId>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-configuration-processor</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-core</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>de.codecentric</groupId>
<artifactId>spring-boot-admin-starter-client</artifactId>
<version>1.5.6</version>
</dependency>
<dependency>
<groupId>org.jolokia</groupId>
<artifactId>jolokia-core</artifactId>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>

First of all, is it just Eclipse doesn't see server and you can run it from command line with no errors?
In case if it's just Eclipse it could be just wrong indention or tabs. Eclipse is not so smart sometimes.
Try to set it all to plain application.properties file.
Try to run it from command line
Also, I think you have to add credentials section for git + encrypted or not.
Here is valid example of YML that works fine in STS Eclipse:
https://github.com/zobarov/spring-cloud/blob/master/edu-springcloud-configserver/src/main/resources/application.yml

Spring boot with Google Cloud SQL

im trying to deploy my spring boot app to Google App Engine Flex Environment and G Cloud MySQL DB.
I'm having issues connecting to the db.
Tried already some variants, but all unsuccessful.
spring-boot-with-google-cloud-datastore-api-fails-to-run
My properties:
spring:
profiles: googlecloud
jpa:
database: MYSQL
show-sql: false
hibernate:
ddl-auto: update
datasource:
url: jdbc:mysql://google/myproject?cloudSqlInstance=XXXX&user=xxx&password=xxx
My pom.xml (the db dependencies only):
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>6.0.6</version>
</dependency>
<dependency>
<groupId>com.google.cloud.sql</groupId>
<artifactId>mysql-socket-factory</artifactId>
<version>1.0.2</version>
</dependency>
<dependency>
<groupId>javax.persistence</groupId>
<artifactId>persistence-api</artifactId>
<version>1.0.2</version>
</dependency>
With this configuration, im getting:
Loading class com.mysql.jdbc.Driver'. This is deprecated. The new driver class iscom.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
Caused by: java.lang.ClassNotFoundException: com.mysql.jdbc.SocketFactory
...Error creating repository beans...
java.lang.NoClassDefFoundError: com/mysql/jdbc/SocketFactory

You can read more details about the change from 5x to 6x connector :
https://dev.mysql.com/doc/connector-j/6.0/en/connector-j-api-changes.html
For the NCDF exception, according to this issue :
https://github.com/GoogleCloudPlatform/cloud-sql-mysql-socket-factory/issues/18
I think you have to change your dependencies :
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>6.0.6</version>
</dependency>
<dependency>
<groupId>com.google.cloud.sql</groupId>
<artifactId>mysql-socket-factory-connector-j-6</artifactId>
<version>1.0.2</version>
</dependency>