We Keep Coding

How can I set up a redundant MongoDB template?

I have a MongoDB with two redundant MongoS router hosts. When using org.springframework.data.mongo to create a MongoTemplate and MongoClient, I can only add a single host. In the event that the host in use falls over, there is no failover to the alternate router host.
I initially referenced https://dzone.com/articles/multiple-mongodb-connectors-with-spring-boot , but the use case there is for two entirely different repositories, where as my case is a single database with dual routers.
In the code below, we would like to add a redundant second host in case the first host fails during runtime.
public class MongoConfiguration extends AbstractMongoConfiguration {
#Value("${mongo.database}")
private String databaseName;
#Value("${mongo.host}")
private String host;
#Value("${mongo.readFromSecondary}")
private String readFromSecondary;
#Value("${mongo.port}")
private int port;
#VaultKey("vault.mongo_username")
private String username;
#VaultKey("vault.mongo_password")
private String password;
#Override
protected String getDatabaseName() {
return databaseName;
}
#Override
#Primary
public MongoClient mongoClient() {
final ServerAddress serverAddress = new ServerAddress(host, port);
final MongoCredential credential = MongoCredential.createCredential(username,
getDatabaseName(), password.toCharArray());
return new MongoClient(serverAddress, credential,
MongoClientOptions.builder().build());
}
#Override
#Primary
#Bean(name = "mongoTemplate")
public MongoTemplate mongoTemplate() throws Exception {
final MongoTemplate template = super.mongoTemplate();
if (this.readFromSecondary != null && Boolean.valueOf(this.readFromSecondary)) {
template.setReadPreference(ReadPreference.secondary());
}
return template;
}
}
Currently at startup a connection to the host in the config file will be loaded with out error, we would like to rotate in a backup host.

You can achieve this by two ways:
1. Multiple Mongo Client or multiple Server Address(hosts):
A MongoDB client with internal connection pooling. For most applications, you should have one MongoClient instance for the entire JVM.
The following are equivalent, and all connect to the local database running on the default port:
MongoClient mongoClient1 = new MongoClient();
MongoClient mongoClient1 = new MongoClient("localhost");
MongoClient mongoClient2 = new MongoClient("localhost", 27017);
MongoClient mongoClient4 = new MongoClient(new ServerAddress("localhost"));
MongoClient mongoClient5 = new MongoClient(new ServerAddress("localhost"),
new MongoClientOptions.Builder().build());
You can connect to a replica set using the Java driver by passing a ServerAddress list to the MongoClient constructor. For example:
MongoClient mongoClient = new MongoClient(Arrays.asList(
new ServerAddress("localhost", 27017),
new ServerAddress("localhost", 27018),
new ServerAddress("localhost", 27019)));
You can connect to a sharded cluster using the same constructor. MongoClient will auto-detect whether the servers are a list of replica set members or a list of mongos servers.
By default, all read and write operations will be made on the primary, but it's possible to read from secondaries by changing the read preference:
mongoClient.setReadPreference(ReadPreference.secondaryPreferred());
By default, all write operations will wait for acknowledgment by the server, as the default write concern is WriteConcern.ACKNOWLEDGED
2. Using Multiple Mongo Connectors and Multiple Mongo Templates:
First of all create the following #ConfigurationProperties class.
#ConfigurationProperties(prefix = "mongodb")
public class MultipleMongoProperties {
private MongoProperties primary = new MongoProperties();
private MongoProperties secondary = new MongoProperties();
}
And then add the following properties in the application.yml
mongodb:
primary:
host: localhost
port: 27017
database: second
secondary:
host: localhost
port: 27017
database: second
Now it’s necessary to create the MongoTemplates to bind the given configuration in the previous step.
#EnableConfigurationProperties(MultipleMongoProperties.class)
public class MultipleMongoConfig {
private final MultipleMongoProperties mongoProperties;
#Primary
#Bean(name = "primaryMongoTemplate")
public MongoTemplate primaryMongoTemplate() throws Exception {
return new MongoTemplate(primaryFactory(this.mongoProperties.getPrimary()));
}
#Bean(name = "secondaryMongoTemplate")
public MongoTemplate secondaryMongoTemplate() throws Exception {
return new MongoTemplate(secondaryFactory(this.mongoProperties.getSecondary()));
}
#Bean
#Primary
public MongoDbFactory primaryFactory(final MongoProperties mongo) throws Exception {
return new SimpleMongoDbFactory(new MongoClient(mongo.getHost(), mongo.getPort()),
mongo.getDatabase());
}
#Bean
public MongoDbFactory secondaryFactory(final MongoProperties mongo) throws Exception {
return new SimpleMongoDbFactory(new MongoClient(mongo.getHost(), mongo.getPort()),
mongo.getDatabase());
}
}
With the configuration above you’ll be able to have two different MongoTemplates based in the custom configuration properties that we provided previously in this guide.
In the previous step we created two MongoTemplates, primaryMongoTemplate and secondaryMongoTemplate
More details: https://blog.marcosbarbero.com/multiple-mongodb-connectors-in-spring-boot/

Connection pooling in Spring Boot and mongo db

I am going through spring boot application and mongoDb connection POC.
I have added following dependency:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>
Also I have gone through mongoB properties with properties: https://docs.spring.io/spring-boot/docs/current/reference/html/common-application-properties.html
Can you please how do we define connection pooling mechanism here?

You cannot do this out of the box with application properties. You need to make use of MongoClientOptions to configure various aspects of connection pool.
Have a look at the documentation for various options available.
Here is a simple example.
#Bean(name="mongoTempl")
public MongoTemplate mongoTempl() throws Exception {
return new MongoTemplate(createMongoClient(new ServerAddress(host, port))
,dbName);
}
Mongo createMongoClient(ServerAddress serverAddress) {
final MongoClientOptions options = MongoClientOptions.builder()
.threadsAllowedToBlockForConnectionMultiplier(...)
.connectionsPerHost(...)
.connectTimeout(...)
.maxWaitTime(...)
.socketKeepAlive(...)
.socketTimeout(...)
.heartbeatConnectTimeout(...)
.minHeartbeatFrequency(...)
.build();
return new MongoClient(serverAddress, options);
}

You can use also MongoClientSettingsBuilderCustomizer like in this spring sample
#Bean
public MongoClientSettingsBuilderCustomizer customizer() {
return (builder) -> builder.applyToConnectionPoolSettings(
(connectionPool) -> {
connectionPool.maxSize(10);
connectionPool.minSize(2);
connectionPool.maxConnectionIdleTime(5, TimeUnit.MINUTES);
connectionPool.maxWaitTime(2, TimeUnit.MINUTES);
connectionPool.maxConnectionLifeTime(30, TimeUnit.MINUTES);
connectionPool.addConnectionPoolListener();
});
}

Spring Data, MySQL, connection dies after 8 hours of inactivity

I am facing an issue where my datasource bean is going down after a period of inactivity. My question is how could I re instantiate the datasource bean that gets hit on application startup.
Here is how we setup the bean on startup.
#ConfigurationProperties(prefix = "spring.datasource")
#Bean
public DataSource dataSource(){
byte[] encryptedFile = fileRetriever.getFile(bucket, key);
String unencryptedJson = fileDecrypter.decryptFile(encryptedFile);
JSONParser parser = new JSONParser();
JSONObject jsonObject = null;
try{
jsonObject = (JSONObject) parser.parse(unencryptedJson);
}catch (Exception ex){
log.error(ex.getMessage());
}
String password = (String)jsonObject.get("password");
DataSource ds = DataSourceBuilder
.create()
.url(url)
.username(userName)
.password(password)
.build();
return ds;
}
This class also has a #Configuration annotation on it.
We have other applications that do not have this issue where the service needs to be bounced after inactivity, but they are also not setting up the data source in this manner and have all the details specified in the application.property file
I have added a custom health check that uses a repository that hits every 30 seconds so that should keep the data source bean alive but incase it does go down I would need a way to recreate it.
Thanks in advance

I assume that boot is configuring the DataSource for you. In this case, and since you are using MySQL, you can add the following to your application.properties up to 1.3
spring.datasource.test-on-borrow=true
spring.datasource.validationQuery=SELECT 1

Might considered a pooled datasource connector. Look at apache dbcb2.
Here is a sample i have that keeps a minimum of 10 idle and increases as needed from the pool.
private static DataSource createDataSource(String url, String userName, String passwrd) throws Exception {
Class.forName(DRIVER).newInstance();
Properties props = new Properties();
props.setProperty("user", userName);
props.setProperty("password", passwrd);
//Create a connection factory that the pool will use to create connections
ConnectionFactory cf = new DriverManagerConnectionFactory(url, props);
//Create the poolable connection factory
PoolableConnectionFactory pcf = new PoolableConnectionFactory(cf, null);
pcf.setValidationQuery("SELECT 1");
pcf.setDefaultAutoCommit(true);
GenericObjectPoolConfig poolConfig = new GenericObjectPoolConfig();
poolConfig.setMinIdle(10);
poolConfig.setMaxTotal(100);
AbandonedConfig abandonConfig = new AbandonedConfig();
abandonConfig.setRemoveAbandonedTimeout(60);
abandonConfig.setLogAbandoned(false);
abandonConfig.setRemoveAbandonedOnBorrow(true);
abandonConfig.setRemoveAbandonedOnMaintenance(true);
//Create the pool of connections
GenericObjectPool<PoolableConnection> connectionPool = new GenericObjectPool<>(pcf, poolConfig);
connectionPool.setTestOnBorrow(true);
connectionPool.setTestWhileIdle(true);
connectionPool.setTimeBetweenEvictionRunsMillis(10000);
connectionPool.setMinEvictableIdleTimeMillis(1000);
connectionPool.setAbandonedConfig(abandonConfig);
pcf.setPool(connectionPool);
//Pooling data source itself
PoolingDataSource<PoolableConnection> dataSource = new PoolingDataSource<>(connectionPool);
return dataSource;
}
Maven dependencies for apache dbcb2
<!-- Database connection pools -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-dbcp2</artifactId>
<version>2.1.1</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-pool2</artifactId>
<version>2.4.2</version>
</dependency>

Using MongoTemplate with MongoClient and UserCredentials

One of the application uses MongoClient as core for interacting with MongoDB in which authentication has been enabled recently. In this mongoClient is initialize as:
mongoClient = new MongoClient(serverAddress, Arrays.asList(MongoCredential.createCredential(userName, dbName, password.toCharArray())));
However at many places app uses mongoTemplate to query the data. Now if MongoTemplate is created as :
new MongoTemplate(mongoClient, dbName);
It leads to authentication failure.
The only way seems to pass user credentials to MongoTemplate is via using UserCredentials class by
However if we pass UserCredentials as :
public MongoTemplate(Mongo mongo, String databaseName, UserCredentials userCredentials) {
Which results to :
Usage of 'UserCredentials' with 'MongoClient' is no longer supported. Please use 'MongoCredential' for 'MongoClient' or just 'Mongo'.
It seems like two different API exists in parallel. What's the best way so that both of them can live together.
This app uses mongodata version as '1.10.6.RELEASE'

Try this:
MongoCredential mongoCredential = MongoCredential.createCredential("user", "database","password".toCharArray());
ServerAddress address = new ServerAddress("mymongo.mycompany.com", 62797);
MongoClient mongoClient = new MongoClient(address, Arrays.asList(mongoCredential));
MongoTemplate mongoTemplate = new MongoTemplate(mongoClient, "database");

Try this configuration:
#Configuration
public class MongoConfiguration {
#Bean
public MongoDbFactory mongoDbFactory() throws Exception {
UserCredentials userCredentials = new UserCredentials("YOUR_USER_NAME", "YOUR_PASSWORD");
return new SimpleMongoDbFactory(new Mongo(), "YOUR_DATABASE", userCredentials);
}
#Bean
public MongoTemplate mongoTemplate() throws Exception {
return new MongoTemplate(mongoDbFactory());
}
}
And to create database repositories just use MongoRepository like this:
public interface UserRepository extends MongoRepository<User,Serializable>{
User findById(String id);
}

In this case seems like it was problem of applying user authentication at mongod server end.
Needed authentication was applied to mongo and has been validated by
db.auth('user','pass');
command which results to 1. However database at that time doesn't exists. Afterwards database was created first by inserted a dummy record and permissions were assigned.
App uses all together a different DB for unit test cases for which it looks like configuration was not correctly applied where this issue was arriving.
Once corrected applied
new MongoClient(serverAddress, Arrays.asList(MongoCredential.createCredential(userName, dbName, password.toCharArray())));
seems to work fine. However at same time mongo driver errors are a bit cryptic without much explanation leading to making debugging time consuming.

MongoDB and Java

I'm trying to use MongoDB in a Java Web Service.
As suggested in Mongo tutorial I should have a MongoClient, let it be dbInstance, connection pool and call dbinstance to get a connection to the database, which is in localhost.
So this is what I have:
private static MongoClient dbInstance = null;
public static DB getDBInstance() {
if (dbInstance == null) {
try {
dbInstance = new MongoClient();
registerShutdownHook();
}
catch (Exception exc) {
System.out.println("Exception");
}
}
return dbInstance.getDB("SAED");
}
What I don't understand is how I can understand if I'm connected to the DB, because, also il mongo isn't working (by starting mongod service) it doesn't throw exceptions.
And another question, I have multiple thread calling Class.getDBInstance, should I synchronize it, and if yes, how can I do that?

You will be thrown an exception when the mongo is not running while you try to connect.
When you do the MongoClient(), it will always look for in the localhost for port 27017 to connect. You can also parameterize this to connect to a different machine and/or port.
You can read more in depth details about this at Mongo Documentation.
MongoClient mongoClient = new MongoClient();
// or
MongoClient mongoClient = new MongoClient( "localhost" );
// or
MongoClient mongoClient = new MongoClient( "localhost" , 27017 );
Regarding your synchronization question,
Yes, you can synchronize at a block level to make it better instead of at the method level.