We Keep Coding

How do I replace the values of a YAML file with definitions in the same file? [duplicate]

We have a spring boot application with configuration being driven from application.yml file. In this configuration file we use the feature of defining a property by referring to another property inside the same application.yml file:
my-games-app:
base-property: foo
games:
- game-one:
game-name: ${my-games-app.base-property}one
game-location: ${my-games-app.base-property}/one
- game-two:
game-name: ${my-games-app.base-property}two
game-location: ${my-games-app.base-property}/two
And we have a #ConfigurationProperties bean loading games configuration:
#Configuration
#ConfigurationProperties(prefix = "my-games-app.games")
public class GamesConfig {
private Map<String, Game> games;
...
}
Useless to say the above is just an example, in reality it is a very complex setup with GamesConfig bean being used as a constructor argument for many other beans inside our application:
#Component
public class GamesRunner {
private final GamesConfig gamesConfig;
...
}
Everything works as expected. The problem we have is related to testing the beans where GamesConfig is injected; in the above example GamesRunner. At the moment we use #SpringBootTest to get hold of the beans we want to test. This again, works OK but the main inconvenient is that the whole application needs to be started in order to access the GamesConfig bean. This means setting up a lot of infrastructure such as a Database a JMS message broker and a Kafka broker. This takes time and makes our CI builds longer to run which started to become a bit of an inconvenient. Because the beans we want to test don't need any other setup than having the GamesConfig constructor argument provided we would prefer to have unit tests in place rather than integration tests as they are much faster to run.
In other words, we want to be able to recreate GamesConfig by hand by parsing our application.yml with a test helper method. To do this we use snakeyaml library:
public final class TestHelper {
public static GamesConfig getGamesConfig() {
var yaml = new Yaml();
var applicationYaml = (Map<String, Object>) yaml.load(readResourceAsString("application.yml");
return createGamesConfig(applicationYaml.get("games");
}
private static GamesConfig createGamesConfig(Object config) {
// The config Object passed here is a `Map<String, Map<String, String>>`
// as defeined in our `games` entry in our `application.yml`.
// The issue is that game name and game locations are loaded exactly like
// configured without property place holders being resolved
return gamesConfig;
}
}
We resolved the issue by manually parsing the property placeholders and looking up their values in the application.yml file. Even if our own property placeholder implementation is quite generic, my feeling is that this extra work is not needed as it should be a basic expectation the library would have some specific set up to do this out of the box. Being very new to snakeyaml I hope someone else hit the same problem and knows how to do it.
We use snakeyaml because it just happened to be in the class path as a transitive dependency, we are open to any suggestions that would achieve the same thing.
Thank you in advance.

To my knowledge, SnakeYAML only supports substitution of environment variables, which is why what you want is not possible as far as I know. What you can do instead, of course, is simply use Spring's classes without setting up a full ApplicationContext.
For example, assuming your game config from above, you could use:
final var loader = new YamlPropertySourceLoader();
final var sources = loader.load(
"games-config.yml",
new ClassPathResource("games-config.yml")
);
final var mutablePropertySources = new MutablePropertySources();
sources.forEach(mutablePropertySources::addFirst);
final var resolver = new PropertySourcesPropertyResolver(mutablePropertySources);
resolver.setIgnoreUnresolvableNestedPlaceholders(true);
System.out.println(resolver.getProperty("my-games-app.games[0].game-one.game-name"));
System.out.println(resolver.getProperty("my-games-app.games[0].game-one.game-location"));
System.out.println(resolver.getProperty("my-games-app.games[1].game-two.game-name"));
System.out.println(resolver.getProperty("my-games-app.games[1].game-two.game-location"));
which outputs:
fooone
foo/one
footwo
foo/two
If you are actually interested in how Spring does it, a good starting point is the source code of the PropertySourcesPlaceholderConfigurer class.

Reading custom objects from additional YAML in Spring Boot

I have a task of reading settings from a YAML file in a Spring Boot application. The requirement is that these settings are stored in a specific file, separate from application.yml. The file is called applicationFeatureToggles.yml and is supposed to have contents like these:
features:
- key: feature1
isEnabled: false
description: First feature
- key: feature2
isEnabled: true
description: Second feature
...
What I need to implement right now is to check in my components' code if a specific feature is enabled. To do this, I created a class for a single feature:
#NoArgsConstructor
#Getter
#Setter
public class Feature {
private String key;
private boolean isEnabled;
private String description;
}
then a configuration properties class to store all settings:
#Component
#ConfigurationProperties
#PropertySource(value = "classpath:applicationFeatureToggles.yml", factory = YamlPropertySourceFactory.class)
public class FeatureProperties {
private List<Feature> features;
// Constructor, getters and setters
}
and a service that uses it to check if a feature is enabled:
#Service
#EnableConfigurationProperties(FeatureProperties.class)
public class FeatureService {
#Autowired
private FeatureProperties featureProperties;
// logic that reads required info from featureProperties
}
The class YamlPropertySourceFactory used in FeatureProperties looks like this:
public class YamlPropertySourceFactory implements PropertySourceFactory {
#Override
public PropertySource<?> createPropertySource(String name, EncodedResource resource) {
YamlPropertiesFactoryBean factory = new YamlPropertiesFactoryBean();
factory.setResources(resource.getResource());
Properties properties = factory.getObject();
return new PropertiesPropertySource(resource.getResource().getFilename(), properties);
}
}
As I understand, this should result in FeatureService having access to FeatureProperties populated with data from applicationFeatureToggles.yml, but the data is missing. At the same time, I checked with a breakpoint that YamlPropertySourceFactory is invoked and reads the data, it's present in the properties object before exiting createPropertySource(). So all seems fine with reading properties from the file, but they don't get into the FeatureProperties object.
What else can my code need to populate FeatureProperties?
If it's not possible at all or can cause some other issues, I'd also be thankful for details, as it may help convince the architect to change the approach.
Spring Boot version used: 2.5.6

I have found the cause. One other thing I did in my code was calling FeatureService in an SpEL expression in a #Conditional for another bean. So it seems FeatureService tried to initialize before FeatureProperties. Without this condition, FeatureService gets proper FeatureProperties.
I think I will eventually find a workaround now, but would be thankful if someone knows how to conditionally initialize a bean depending on some ConfigurationProperties if it also involves some logic to process the configs.

Java Spring MongoDB Repository Interface

So I have an AppUser class:
#Data
#Builder
#Document(collection = "app_users")
#Component
#AllArgsConstructor
#NoArgsConstructor
#Import(AppConfig.class)
public class AppUser {
#Id
#NotBlank(message = ErrorConstants.ANDROID_USER_ACCOUNT_MANAGER_ID_IS_NULL)
private String androidUserAccountManagerId;
#NotBlank(message = ErrorConstants.NULL_NAME)
private String name;
private Friend bestFriend;
#Setter(AccessLevel.NONE)
private FriendList friendList;
private boolean manualBestFriendOverride;
public Optional<Friend> getFriend(String friendName) {
return friendList.getFriend(friendName);
}
public void calculateBestFriend() {
if (!manualBestFriendOverride) {
bestFriend = friendList.calculateAndReturnBestFriend();
}
}
}
I have created an AppUserRepository interface that extends MongoRepository:
#Repository
public interface AppUserRepository extends MongoRepository<AppUser, String> {}
I have a WebController class that interacts with the interface. The AppUserRepository field in this class is #Autowired. This all seems to work but I have a few questions regarding how, and how I go forward and write integration tests for this:
How do I configure this AppUserRepository that has been created? Can I run it on a specific port etc?
Why has the Autowiring worked as I have not created this AppUserRepository Bean in an AppConfig like I have other Beans that are Autowired in my application.
If I was to create a Bean, wouldn't I also have to implement the class and return the instantiation? I started doing this but I had to implement all of the MongoRepository classes methods which I wasn't sure seemed quite right.
How do I write integration tests with an AppUserRepository? I need an AppUserRepository for my requests to interact with, but I do not want this to be the same DB as the real-time application DB when the service is up and running. Can I #Autowire the database into the integration test class and then close the DB after the integration tests run? If this is how I go forward, I think I then need to do point 3 above.
Thanks for your help in advance, I have tried reading some documentation but I think I am missing some key knowledge that means it is all quite overwhelming and confusing.
Thanks!

That's actually quite a big story to tell. This topic is called Spring Data JPA, Hibernate. You might wanna do a research on that, and watch some tutorials and so on.
Briefly, that MongoRepository just gives you a lot of methods which you can use. You can also define your own methods, add queries and etc.
Your starting points: https://www.baeldung.com/spring-boot-hibernate
https://www.baeldung.com/the-persistence-layer-with-spring-data-jpa
https://www.baeldung.com/spring-data-jpa-query
Of course you can set a port number (and some other properties) via application.properties file. This is a list of most common properties, you can find properties for mongodb on it:
https://docs.spring.io/spring-boot/docs/current/reference/html/appendix-application-properties.html
Now about bean. You basically created one with #Repository annotation actually. So Spring Context loads it on the start of application. You can autowire it.

Spring MVC: issue between xml and annotation configurations

I have created a simple controller
#GetMapping("/playerAccount")
public Iterable<PlayerAccount> getPlayerAccounts(com.querydsl.core.types.Predicate predicate) {
return repository.findAll(predicate);
}
When I call the GET /playerAccount API, I get the exception IllegalStateException "No primary or default constructor found for interface com.querydsl.core.types.Predicate" (thrown by org.springframework.web.method.annotation.ModelAttributeMethodProcessor#createAttribute).
After some (deep!) digging, I found out that if I delete the following line in my spring.xml file:
<mvc:annotation-driven />
And if I add the following line in my Spring.java file:
#EnableWebMvc
then the problem disappears.
I really don't understand why. What could be the cause of that ? I thought that these were really equivalent (one being a xml based configuration, the other being java/annotation based).
I read this documentation on combining Java and Xml configuration, but I didn't see anything relevant there.
edit:
from the (few) comments/answers that I got so far, I understand that maybe using a Predicate in my API is not the best choice.
Although I would really like to understand the nature of the bug, I first want to address the initial issue I'm trying to solve:
Let's say I have a MyEntity entity that is composed of 10 different fields (with different names and types). I would like to search on it easily. If I create the following (empty) interface:
public interface MyEntityRepository extends JpaRepository<MyEntity, Long>, QuerydslPredicateExecutor<MyEntity> {
}
then without any other code (apart from the xml configuration ), I am able to easily search a myEntity entity in the database.
Now I just want to expose that functionality to a Rest endpoint. And ideally, if I add a new field to my MyEntity, I want that API to automatically work with that new field, just like the MyEntityRepository does, without modifying the controller.
I thought this was the purpose of Spring Data and a good approach, but please tell me if there's a better / more common way of creating a search API to a given Entity.

I didn't see that it returned an exception, that's why I thought it was a dependency problem.
Try to make your code look like this, and it will do it.
#RestController
public class MyClass {
#Autowired
private final MyRepository repository;
#GetMapping("/playerAccount")
public Iterable<PlayerAccount> getPlayerAccounts() {
return repository.findAll();
}
If you have a parameter in your request you add #RequestParam.
Code time (yaaaaaay) :
#RestController
public class MyClass {
#Autowired
private final MyRepository repository;
#GetMapping("/playerAccount")
public Iterable<PlayerAccount> getPlayerAccounts(#RequestParam(required = false) Long id) {
return repository.findById(id);
}
Ps: the request should keep the same variable name e.g
.../playerAccount?id=6

How to populate #Query value for native queries in Spring boot Repository when Queries are stored as external files

Similar question has been asked and answered back in 2015 here.
However , rather than storing queries in XML files , I would like to store them in external .sql files and read query from there.
To achieve this in our spring boot application - 2.1.5.RELEASE with Java 8 , we are storing queries in resources folder as below
src/main/resources
- queries
- A.sql
- B.sql
To read the above queries , I'm reading them in QueryReader class as below
#Component
public class QueryReader {
public String getQueryFromFile(String fileName) {
System.out.println("getQueryFromFile() : " + fileName);
return this.convertInputStreamToString(this.getTemplateAsStream(fileName));
}
private InputStream getTemplateAsStream(String queryFileName){
ClassLoader classLoader = getClass().getClassLoader();
InputStream iStream =classLoader.getResourceAsStream("queries/" + queryFileName);
return iStream;
}
}
And to use it anywhere in the code , i'm having below class , so that i can call it's methods
#Component
public class MyQueries {
#Autowired
private QueryReader qfr;
public String queryErroredRecords() {
return qfr.getQueryFromFile("A.sql");
}
While using it with JDBCTemplate , this works as expected but when I'm trying to use this method from #Query annotation from Repository , i'm not able to do so with errors as below.
Repository Code
#Repository
public interface AdminRepository extends JpaRepository<OrderBackupRecordEO, BigDecimal>{
#Autowired
public MyQueries queries;
#Query(value = queries.queryErroredRecords() , nativeQuery= true)
List<String> findX();
}
Above repository is giving errors as below :
1.The value for annotation attribute Query.value must be a constant expression (For #Query annotation)
2.The blank final field queries may not have been initialized (For #Autowired annotation)
How can I make it work?

I would just add a note that may be helpful for you.
Your solution cannot work just because in java in an interface all the fields (variables) are by default public, static and final and that why you cannot #Autowire any dependencies inside the interface.
That's why it works for your case for JDBCTemplate (which is a class) and doesn't work for spring data repository (which is an interface).

Spring already has a solution "from the box", you don't need to make all these manipulations.
#Repository
public interface AdminRepository extends JpaRepository<OrderBackupRecordEO, BigDecimal> {
List<String> findX();
}
File src/main/resources/META-INF/jpa-named-queries.properties:
OrderBackupRecordEO.findX=\
SELECT record FROM OrderBackupRecordEO record WHERE ...
That's all! No any loading, manipulations, manual handling - an implementation is hidden in Spring - very simple and reliable.