We Keep Coding

How do I replace the values of a YAML file with definitions in the same file? [duplicate]

We have a spring boot application with configuration being driven from application.yml file. In this configuration file we use the feature of defining a property by referring to another property inside the same application.yml file:
my-games-app:
base-property: foo
games:
- game-one:
game-name: ${my-games-app.base-property}one
game-location: ${my-games-app.base-property}/one
- game-two:
game-name: ${my-games-app.base-property}two
game-location: ${my-games-app.base-property}/two
And we have a #ConfigurationProperties bean loading games configuration:
#Configuration
#ConfigurationProperties(prefix = "my-games-app.games")
public class GamesConfig {
private Map<String, Game> games;
...
}
Useless to say the above is just an example, in reality it is a very complex setup with GamesConfig bean being used as a constructor argument for many other beans inside our application:
#Component
public class GamesRunner {
private final GamesConfig gamesConfig;
...
}
Everything works as expected. The problem we have is related to testing the beans where GamesConfig is injected; in the above example GamesRunner. At the moment we use #SpringBootTest to get hold of the beans we want to test. This again, works OK but the main inconvenient is that the whole application needs to be started in order to access the GamesConfig bean. This means setting up a lot of infrastructure such as a Database a JMS message broker and a Kafka broker. This takes time and makes our CI builds longer to run which started to become a bit of an inconvenient. Because the beans we want to test don't need any other setup than having the GamesConfig constructor argument provided we would prefer to have unit tests in place rather than integration tests as they are much faster to run.
In other words, we want to be able to recreate GamesConfig by hand by parsing our application.yml with a test helper method. To do this we use snakeyaml library:
public final class TestHelper {
public static GamesConfig getGamesConfig() {
var yaml = new Yaml();
var applicationYaml = (Map<String, Object>) yaml.load(readResourceAsString("application.yml");
return createGamesConfig(applicationYaml.get("games");
}
private static GamesConfig createGamesConfig(Object config) {
// The config Object passed here is a `Map<String, Map<String, String>>`
// as defeined in our `games` entry in our `application.yml`.
// The issue is that game name and game locations are loaded exactly like
// configured without property place holders being resolved
return gamesConfig;
}
}
We resolved the issue by manually parsing the property placeholders and looking up their values in the application.yml file. Even if our own property placeholder implementation is quite generic, my feeling is that this extra work is not needed as it should be a basic expectation the library would have some specific set up to do this out of the box. Being very new to snakeyaml I hope someone else hit the same problem and knows how to do it.
We use snakeyaml because it just happened to be in the class path as a transitive dependency, we are open to any suggestions that would achieve the same thing.
Thank you in advance.

To my knowledge, SnakeYAML only supports substitution of environment variables, which is why what you want is not possible as far as I know. What you can do instead, of course, is simply use Spring's classes without setting up a full ApplicationContext.
For example, assuming your game config from above, you could use:
final var loader = new YamlPropertySourceLoader();
final var sources = loader.load(
"games-config.yml",
new ClassPathResource("games-config.yml")
);
final var mutablePropertySources = new MutablePropertySources();
sources.forEach(mutablePropertySources::addFirst);
final var resolver = new PropertySourcesPropertyResolver(mutablePropertySources);
resolver.setIgnoreUnresolvableNestedPlaceholders(true);
System.out.println(resolver.getProperty("my-games-app.games[0].game-one.game-name"));
System.out.println(resolver.getProperty("my-games-app.games[0].game-one.game-location"));
System.out.println(resolver.getProperty("my-games-app.games[1].game-two.game-name"));
System.out.println(resolver.getProperty("my-games-app.games[1].game-two.game-location"));
which outputs:
fooone
foo/one
footwo
foo/two
If you are actually interested in how Spring does it, a good starting point is the source code of the PropertySourcesPlaceholderConfigurer class.

Accessing CDI from simple objects?

Assume I have a configuration class accessible via the stock CDI that defines some application-wide parameters:
#ApplicationScoped
class AppConfig {
public double getMaxAllowedBrightness() { ... }
};
And I have a simple class for my data objects:
class LightSource {
double brightness;
...
boolean isValid() {
double maxAllowedBrightness = ...; // Somehow use AppConfig#getMaxAllowedBrightness() here
return brightness <= maxAllowedBrightness;
}
}
How can my data object access the single AppConfig instance?
Somehow I hate the idea of autowiring AppConfig into every single data object (there are lots of them). Is there any other way to get access to AppConfig in the above example from my data object?
What's the best pattern to use here?

The simplest example is a runtime lookup akin to:
import jakarta.enterprise.inject.spi.CDI;
CDI.current().select(cls).get();
With cls being the class that you're looking up. (Note the package name, this is the latest version of CDI 2.x in the new jakarta namespace, the original is in javax.)
It gets more detailed from there, but that's the gist of it.
Note, that semantically there's little difference between autowiring something and doing a runtime lookup, especially for something mostly static at the instance level. It's still a dependency. You still have to touch the code of the classes to pull it off.
A nice thing of relying on the autowiring is that you can disable it situationally, and the class reverts to a simple bean, that you can do with what you will. Coding in the lookup, it's a little bit more than that.
Dynamic lookup is more for special circumstances.

On my current project, our team has been doing this using the #Value annotation. In our case, we have all the properties in a properties bean, which I'll call mainAppConfiguration. The bean is populated from a properties file like main-app-config.properties (which was read into the bean with a Properties prop = new Properties().load(mainAppConfigFilePath) method.
Assuming you have something like that set up, then we inject the properties into the classes that need them using a little SpEL magic something like:
private Integer refreshRateSeconds;
#Value("#{ mainAppConfiguration.getProperties()['funny-property-base-name.refreshRateSeconds'] }")
public void setRefreshRateSeconds(Integer refreshRateSeconds) {
if (refreshRateSeconds == null) {
throw new IllegalArgumentException("Required config property 'funny-property-base-name.refreshRateSeconds' was not found"));
}
this.refreshRateSeconds = refreshRateSeconds;
}
Baeldung has examples (without defaults) and more with defaults.

Java Properties Automatically Use Test Resources

I have a few classes that need some environment specific configuration. I turned to using properties files, which are loaded in the constructor of the class.
public class MyClass {
public MyClass() {
try {
ValidatedEnvironmentProperties props = new ValidatedEnvironmentProperties();
props.load(MyClass.class.getResourceAsStream("/myclass.properties"));
ValidatedEnvironmentProperties extends Properties. Basically, it uses a Java System Property to set a key prefix. I set the System property to, say, production, and in the properties files, I have staging.url=... and production.url=.... This allows me build/runtime selection of which configuration is used while not needing to change property file name.
myclass.properties is stored in src/main/resources.
That works fine, and I rather like the how it works. My problem is that I'm sort of stuck with TestNG. I want to test a bunch of other properties in TestNG unit tests. This led me to create src/test/resources/myclass.properties. Instead of "environment" keys, I use test names like bad_url_test.url=this_ain't_a_url.
I was thinking that src/test/resources would get a higher priority in the classpath/classloader (terminology?), causing the test-specific properties to load. Then, for my various tests, I just set the Java System property to bad_url_test, instantiate MyClass, test my assertions, and then set the System property to a new test, instantiate a new object, and repeat.
I believe the source of the problem is this line in MyClass's constructor:
props.load(MyClass.class.getResourceAsStream("/myclass.properties"));
In MyClassTest, I put these lines to try to understand what's happening:
File f = new File(MyClass.class.getProtectionDomain().getCodeSource().getLocation().getPath());
System.out.println("MyClass classpath: " + f.toString());
f = new File(MyClassTest.class.getProtectionDomain().getCodeSource().getLocation().getPath());
System.out.println("MyClassTest classpath: " + f.toString());
Output is:
MyClass classpath: /home/fandingo/code/project/build/classes/main
MyClassTest classpath: /home/fandingo/code/project/build/classes/test
MyClassTest is correct, but I need something accessible within MyClass's constructor that will automatically prefer /src/test/resources/ when running tests but /src/main/resources/ when running normally.

You need inversion of control. i.e. MyClass needs to be told where to get its properties from and not decide on its own.
There are lots of ways to do this but they all come down to the same idea: MyClass should not know at compile time where its properties come from.
e.g.
In MyClass.java
ValidatedEnvironmentProperties props = new ValidatedEnvironmentProperties();
String resourceSupplierClassName = System.getProperty("resource-supplier-class-name",
MyClass.class.getName());
Class<?> resourceSupplierClass = Class.forName(resourceSupplierClassName);
props.load(resourceSupplierClass.getResourceAsStream("/myclass.properties"));
In MyClassTest.java
System.setProperty("resource-supplier-class-name", MyClassTest.class.getName());
Again, there are many ways to do this. Instead of passing around system properties, etc. you can also use dependency injection (a form of inversion of control). You can change the MyClass constructor to take your props as an argument and then put the onus on the instantiators of MyClass to provide the props or you can use a dependency injection framework such as Spring or Guice to manage creating the appropriate props instance for main/test execution and provide it to objects that need it as needed.

Spring properties that depend on other properties

I would like to have properties, that I can reference via #Value in spring beans, that can only be created dependend on other properties.
In particular I am having a property, that describes the file system location of a directory.
myDir=/path/to/mydir
And by convention, there is a file in that directory, that is always called myfile.txt.
Now i want to have access to both, the directory and the file, via #Value annotations inside my beans. And sometimes I want to access them as Strings, sometimes as java.io.Files and sometimes as org.springframework.core.io.FileSystemResource (which by the way works very well out of the box!). But because of that concatenating Strings on demand is not an option.
So what I of course could do is just declare both, but I would end up with
myDir=/path/to/mydir
myFile/path/to/mydir/myfile.txt
and I would like to avoid that.
So I came up with an #Configuration class, that takes the property and adds it as new PropertySource:
#Autowired
private ConfigurableEnvironment environment;
#Value("${myDir}")
private void addCompleteFilenameAsProperty(Path myDir) {
Path absoluteFilePath = myDir.resolve("myfile.txt");
Map<String, Object> props = new HashMap<>();
props.put("myFile, absoluteFilePath.toString());
environment.getPropertySources().addFirst(new MapPropertySource("additional", props));
}
As you can see, in my context I even created a PropertyEditor, that can convert to java.nio.file.Paths.
Now the problem is, that for some reason, this "works on my machine" (in my IDE), but does not run on the intended target environment. There I get
java.lang.IllegalArgumentException: Could not resolve placeholder 'myFile' in string value "${myFile}"

Spring can combine properties
myDir=/path/to/mydir
myFile=${myDir}/myfile.txt
You can also use a default value without defining your myFile in the properties at first:
Properties file
myDir=/path/to/mydir
In class:
#Value("#{myFile:${myDir}/myfile.txt}")
private String myFileName;

Spring expressions can be used to refer the properties.
In my example it was
query-parm=QueryParam1=
query-value=MyParamaterValue
Now while binding them in Spring Bean.
#Configuration
public class MyConfig {
#Value("${query-param}${query-value}")
private String queryString;
}
Above code will inject QueryParam1=MyParamaterValue to the variable queryString.

How to load all key/value pairs in properties file

I'm trying to load all key/value pairs in my properties file.
One approach is that I load all the properties using #Value manually, but for that I should know all the keys.
I cannot do this, since property file may be changed in future to include more set of key/value pairs and I may need to modify code again to accommodate them.
Second approach is that I should some how load the properties file and Iterate over it to load all the key/value pairs without knowing the keys.
Say I have following properties file sample.properties
property_set.name="Database MySQL"
db.name=
db.url=
db.user=
db.passwd=
property_set.name="Database Oracle"
db.name=
db.url=
db.user=
db.passwd=
Here is what I'm trying to do
#Configuration
#PropertySource(value="classpath:sample.properties")
public class AppConfig {
#Autowired
Environment env;
#Bean
public void loadConfig(){
//Can I some how iterate over the loaded sampe.properties and load all
//key/value pair in Map<String,Map<String, String>>
// say Map<"Database MySQL", Map<key,vale>>
// I cannot get individual properties like env.getProperty("key");
// since I may not know all the keys
}
}

Spring stores all properties in Environment.
Environment contains collection of PropertySource. Every PropertySource contains properties from specific source. There are system properties, and java environment properties and many other. Properties from you file will be there as well.
Any source has own name. In your case automatically generated name will be look like "class path resource [sample.properties]". As you see, the name is not so convenient. So lets set more convenient name:
#PropertySource(value="classpath:sample.properties", name="sample.props")
Now you can get source by this name:
AbstractEnvironment ae = (AbstractEnvironment)env;
org.springframework.core.env.PropertySource source =
ae.getPropertySources().get("sample.props");
Properties props = (Properties)source.getSource();
Note that I specified full name of PropertySource class, to avoid conflict with #PropertySource annotation class. After that, you can work with properties. For example output them to console:
for(Object key : props.keySet()){
System.out.println(props.get(key));
}

You can autowire in an EnumerablePropertySource which contains the method getPropertyNames()

you can look up the class Properties in the jdk api and use the method load(InputStream inStream)
InputStream in = new FileInputStream("your properties location");
Properties prop = new Properties();
prop.load(in);
ps: prop is subClass of HashTable and don't forget to close stream