I have a Java Spring boot project, heavily using the database (Postgres) for it's repository/data. It is the basic MVC project, controllers are all REST controllers. The project works well (service is up, able to call service via REST clients and all).
Now, I am adding the unit tests to it. I am pretty new to Spring boot and mostly the unit test part. Owing to the CI/CD (build pipeline), I cannot use the persistent/external DB for tests. Hence I need to use in-memory DB.
The initial run (Main class) runs a bunch of DB queries to build up cache while project comes up. So I would need postgres DB for testing (lots of DB functions used).
Basically, I would need to use Testcontainers (postgresql). I am writing a very basic test first to get hold of it.
I have the schema.sql and data.sql stored (to be used only for testing).
src
|
main
test
|
resources
|
application-test.properties
schema.sql
data.sql
Relevant pom.xml
<properties>
<java.version>11</java.version>
<testcontainers.version>1.15.1</testcontainers.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-cache</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
</dependency>
<dependency>
<groupId>io.jsonwebtoken</groupId>
<artifactId>jjwt</artifactId>
<version>0.9.1</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-validation</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<scope>runtime</scope>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.junit.vintage</groupId>
<artifactId>junit-vintage-engine</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.11.2</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.11</version>
</dependency>
<dependency>
<groupId>commons-beanutils</groupId>
<artifactId>commons-beanutils</artifactId>
<version>1.9.4</version>
</dependency>
<dependency>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
<version>20201115</version>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>testcontainers</artifactId>
<version>${testcontainers.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>localstack</artifactId>
<version>${testcontainers.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>junit-jupiter</artifactId>
<version>${testcontainers.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>postgresql</artifactId>
<version>${testcontainers.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
My Test class:
#Testcontainers
#Sql(scripts = {"file:src/test/resources/schema.sql","file:src/test/resources/data.sql"})
class ApplicationTests {
#Container
static PostgreSQLContainer<?> postgreSQLContainer = new PostgreSQLContainer<>("postgres:12")
.withUsername("testcontainers")
.withPassword("testcontainers")
.withDatabaseName("tescontainers");
#Test
void testPostgreSQLModule() throws SQLException {
try (Connection connection = DriverManager
.getConnection(postgreSQLContainer.getJdbcUrl(), "testcontainers", "testcontainers");
PreparedStatement preparedStatement = connection.prepareStatement("SELECT * FROM table_from_schema")) {
try (ResultSet resultSet = preparedStatement.executeQuery()) {
while (resultSet.next()) {
System.out.println(resultSet.getString("column1"));
}
}
}
}
}
I am simply trying to test the DB.
However, when I run the test, it fails saying that
org.postgresql.util.PSQLException: ERROR: relation "table_from_schema" does not exist
I tried to debug it, i.e. stopped just inside my Test (testPostgreSQLModule). I can see the docker component with Postgres.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
fc3ff1e04ceb postgres:12 "docker-entrypoint.s…" 16 seconds ago Up 15 seconds
But when I login to it and run psql, I see the DB (tescontainers) is created, however it does not have any schema (tables/functions).
tescontainers=# \dt
Did not find any relations.
tescontainers=#
Basically, my files are not being run.
Does #SQL annotation on class level does not work with Testcontainers initialization as in my case?
What is needed here so both my initial scripts run?
I tried using .withInitScript, and it runs. However, I have lots of data to initialize and the file is too large (and would grow), so I separate DDL (schema) and Inserts (data). Now, my issue is how to run multiple init files (schema.sql, data.sql) using "withInitScript" ? So I tries #SQL annotation, but it does not seems to work.
---UPDATE/EDIT----
To give the context clear, I am looking for below. If anyone can guide please?
All profile (dev/ist/uat/prod) should be using their respective persistence DB (from their application.env.properties).
Only for the test, I need the in-memory DB, but cannot use H2 (and similar) as I have lots of DB related tests and need Postgres (functions , etc). So, trying out Testcontainers.
When the application boots up, it fetches some data from respective DB (based on env) to prepare the initial cache and other methods will use it while servicing any rest calls. Hence, for Test (only) I need a new in-memory DB with all schema/data (which I can provide via SQL files) , so that while test, the boot up should use that test DB and corresponding tests wil work based on that initial data.
So I need a way to bring up test DB (in-memory/testcontainers) whenever Test runs, and pass multiple SQL files for initialization of Test DB (before any test runs). Any idea on what is the best approach?
The error is telling you no such table exists
org.postgresql.util.PSQLException: ERROR: relation "table_from_schema" does not exist
It seems that the connection to the DB is established, but the query you're attempting is not working.
I had a quick look and from what I could see the table table_from_schema is not present in the standard postgres schema.
However select * from pg_tables; may work. Try accessing a different table and see if this resolves the issue for you temporarily, so you can confirm the issue is selecting from a table name that doesn't exist rather than something more obscure.
If you are able to, you can put a break point in after the docker container has spun up and connect using a database client. Once you're connected you can then try inspect the database manually, testing your SQL commands before you copy-paste them to your codebase.
If you want the SQL scripts to run in your docker container, you need to mount the test/resources dir to the container's /docker-entrypoint-initdb.d dir using a volume mapping for the container. Any *.sh and *.sql scripts will then be run. You may need to restructure your sql scripts so you have one root script that calls scripts in a directory lower down. This is especially true if there's any ordering requirement between the two.
See https://hub.docker.com/_/postgres "Initialization scripts"
Mounting into your container with testcontainers is as simple as:
String pathToFile = "<project-root-dir>/src/test/resources";
new GenericContainer(...)
.withFileSystemBind(pathToFile, "/docker-entrypoint-initdb.d", BindMode.READ_ONLY)
See https://www.testcontainers.org/features/files/
I guess you missed the following
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<scope>test</scope>
</dependency>
test-dependency. (Or remove the <scope/> at all.)
Table not found errors often indicate a mismatch of the schema and model and the available driver.
Also your test class seems to be spring-boot independent. How do you run it? You could benefit of auto-configuration, like described here: https://www.baeldung.com/spring-boot-testcontainers-integration-test
(but I guess you don't like that ;))
Related
I'm making custom JDBC driver and I need to test it (with Spring-boot). I added a dependency (I use Maven) this driver JAR to another project and it's work fine in the main program. But when I'm trying to make integration tests (in unit tests the driver registered in DriverManager by itself) driver doesn't load so Spring-boot can't create bean with name 'dataSource'.
I also tried to write unit tests to use the driver directly, but the driver cannot load the properties file that is contained in the resources (file path is src/main/resources/driver.properties). In this case the driver registered in DriverManager by itself.
How can I make it so that:
The driver can see the file from resources during unit tests? FIXED
The driver start register itself during integration tests with Spring?
Note: JDBC use the service provider mechanism to load drivers, so I also have the java.sql.Driver file in resources.(src/main/resources/META-INF/services/java.sql.Driver).
Link for the JDBC driver code. Readme have also link to project, where I'm trying to test driver
Code of test:
#SpringBootTest
public class ApplicationSmokeTest {
#Test
public void contextLoad(){
assertTrue(true);
}
}
In main program driver is loaded and spring work with them fine.
Dependencies:
<dependencies>
<dependency>
<groupId>org.mock.jdbc</groupId>
<artifactId>HttpDriver</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-test</artifactId>
<version>5.3.3</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
</dependencies>
I’m following this tutorial: https://www.youtube.com/watch?v=vtPkZShrvXQ
… and I am having trouble with database migrations. I am using Spring Boot 2.2.7, and I have created a PostgreSQL database called “demodb”
When I run the program, the console gives the error:
org.postgresql.util.PSQLException: FATAL: database "demodb" does not exist
Here is my application.yml file, which contains the database info:
app:
datasource:
plaltform: postgres
jdbc-url: jdbc:postgresql://localhost:5432/demodb
username: postgres
password: password
pool-size: 30
Here are my dependencies in the pom.xml file:
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.junit.vintage</groupId>
<artifactId>junit-vintage-engine</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.flywaydb</groupId>
<artifactId>flyway-core</artifactId>
</dependency>
<dependency>
<groupId>com.zaxxer</groupId>
<artifactId>HikariCP</artifactId>
<version>3.2.0</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot</artifactId>
<version>2.2.7.RELEASE</version>
</dependency>
</dependencies>
I'm running my migrations in a separate fold, and like I said, the database "demodb" DOES exist (I created it from the terminal), so I’m not sure why I’m getting this error. Any ideas?
So I was following along the same tutorial. Then I switched to the same persons video on installing postgres: https://youtu.be/4smnWU0BhrA?t=811
At the time stamp I linked you end up starting postgres in the mac application. Make sure you stop it. The issue for me was that since I started that postgres before the one inside docker, that original postgres was listening on port 5432. And that original postgres did not have demodb.
If that is not the solution try to kill whatever is listening to port 5432, then try again or restart the docker instance.
lsof -i :5432
Get the PID, lets say its 1001, then do:
kill -9 1001
could be related to a typo in your application.yml file...
app:
datasource:
platform: postgres
(your wrote plaltform)
other than that I wonder why you don't use the default Spring Boot properties to configure your database connection:
spring:
datasource:
driver-class-name: com.mysql.jdbc.Driver
url: dbc:postgresql://localhost:5432/demodb
username: postgres
password: password
please also have a look at this reference of Spring Boot properties: https://docs.spring.io/spring-boot/docs/current/reference/html/appendix-application-properties.html#data-properties
I have a Spring Boot project, on which I have some csv files and have converted them into entities and querying them based on my requirement. For this approach, I am using Teiid Spring Boot Starter, which is starting a embedded server -
This is the console startup log -
Starting embedded database: url='jdbc:teiid:spring;PassthroughAuthentication=true;useCallingThread=true;autoFailover=true;waitForLoad=5000;autoCommitTxn=OFF;disableLocalTxn=true', username='null' ````
pom.xml
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.teiid</groupId>
<artifactId>teiid-spring-boot-starter</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.teiid</groupId>
<artifactId>teiid-12.1.1-jdbc</artifactId>
<version>12.2.1</version>
<scope>system</scope>
<systemPath>${project.basedir}/src/main/resources/teiid-12.2.1-jdbc.jar</systemPath>
</dependency>
</dependencies>
application.properties
spring.application.name=Teiid-spring-boot
spring.teiid.model.package=com.example.demo.model
spring.teiid.file.parent-directory=src/main/resources/csv
#######
logging.level.org.teiid.spring=TRACE
spring.main.allow-bean-definition-overriding=true
spring.jpa.show-sql=true
spring.jpa.properties.hibernate.format_sql=true
I have Squirrel SQL client setup, have added the driver for Teiid as well.
teiid-12.2.1-jdbc.jar
Squirrel SQL connection settings used -
name - Teiid
example url - jdbc:teiid:spring
website url - http://teiid.org
But while connecting, getting this following error -
teiid: JDBC Driver class not found
class java.lang.ClassNotFoundException: org.jboss.modules.ModuleLoadException
There are multiple issues that need to be fixed.
You do not need the teiid-12.1.1-jdbc dependency in pom.xml
Add teiid.jdbc-enable=true to your application.properties that will open a jdbc port 31000 for the application you built.
Run your application
Then add the Teiid JDBC driver to SquirreL (which you seemed to be already done)
Use the URL as jdbc:teiid:spring#mm://localhost:31000 where localhost is host where you are running your teiid-spring application.
I am trying to add JPA Maven dependency to already created spring-boot project but I am getting following error:
Error: Could not find or load main class com.kame.demo.DemoApplication
When I remove it the error is gone.
pom.xml
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<version>1.4.196</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.tomcat</groupId>
<artifactId>tomcat-jasper</artifactId>
<version>8.5.32</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
</dependencies>
application.properties
spring.mvc.view.prefix=/pages/
spring.mvc.view.suffix=.jsp
spring.h2.console.enabled=true
spring.datasource.platform=h2
spring.datasource.url=jdbc:h2:mem:navin
I tried to find answer online but none was solution for me.
Also tried to create > Spring starter project > an there immediately add JPA, Web and H2 but still same error.
I am using STS IDE, is something related to it maybe?
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
Is itself a demo project with an extending SpringBoot class as the main class of your project. But this dependency does the same :
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
So there is high chances that both will clash...
The solution is to import the correct dependency for jpa and not the spring boot starter jpa dependency.
EDIT
This one might do the trick instead :
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-jpa</artifactId>
</dependency>
But I recommend you read the official documents to get started properly : https://docs.spring.io/spring-data/jpa/docs/2.1.0.RC2/reference/html/
I faced the same issue while I tried to add JPA dependency manually in POM.xml after creating Spring boot with Web only.
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<scope>runtime</scope>
</dependency>
Root cause of the problem:- JPA dependency jar was missing. verify:-
File->Project structure->Modules->Dependencies and you will not find any jar org.springframework.boot:spring-boot-starter-data-jpa:2.4.3 (I am using spring boot 2.4.3 but in your case version can be different)
Solution:-
Step 1
File -> Invalidate Caches / Restart... and choosing the Invalidate and Restart option fixed the issue. can be a stale cache issue
Step 2
Under IntelliJ Idea open POM.xml right click-> Maven-> Reload project.
Maven will download the JPA dependency jar and will add into the project. Verify if JPA jar is added or not.
Step 3
Selecting File -> Invalidate Caches / Restart... and choosing the Invalidate and Restart option fixed the issue.
NB:- If you are not using ItellIj id then please force Maven to download dependency (for step2)
I was trying to deploy my own custom sink of spring cloud data flow onto cloud foundry.
My Dependency are below :
<dependencies>
<dependency>
<groupId>org.springframework.cloud.stream.app</groupId>
<artifactId>spring-cloud-starter-stream-sink-log</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-rabbit</artifactId>
<version>1.2.0.RC1</version>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud.stream.app</groupId>
<artifactId>app-starters-core-dependencies</artifactId>
<version>1.2.0.RELEASE</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>org.springframework.cloud.stream.app</groupId>
<artifactId>log-app-dependencies</artifactId>
<version>1.2.0.RELEASE</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
And the custom code is very basic as below :
#EnableBinding(Sink.class)
public class SinkConfiguration {
private static final Log logger = LogFactory.getLog(SinkConfiguration.class);
#ServiceActivator(inputChannel=Sink.INPUT)
public void loggerSink(String payload) {
logger.info("Hello, Rahul. The time is: " + payload);
}
}
All I see when i deploy this application is that the Error channel subscriber is created , but no Input subscriber was created. Due to this no messages are being received on to this app. Source for this app is a custom source with rest controller. The default out of box streamer app -- LogSink works successfully. But i need to create a customsink to build things on top. Do anyone see an issue what I am missing here?
If your goal is to create an entirely new custom sink, you would develop that as a standalone application by following the Spring Initializr procedure. It is simple this way and you don't need to use the existing log-sink starter in this case.
If you are trying to patch any of the OOTB application; in this case, the log-sink, then follow the patching procedure. Pay attention to importing the configuration class. Unless you do that, the associated app-starter's behavior won't kick in.
Also, it seems you're using an old release for rabbit-binder. It is better to rely on the Spring Initializr generated artifact as opposed to handcrafting dependency versions.