I am going through spring boot application and mongoDb connection POC.
I have added following dependency:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>
Also I have gone through mongoB properties with properties: https://docs.spring.io/spring-boot/docs/current/reference/html/common-application-properties.html
Can you please how do we define connection pooling mechanism here?
You cannot do this out of the box with application properties. You need to make use of MongoClientOptions to configure various aspects of connection pool.
Have a look at the documentation for various options available.
Here is a simple example.
#Bean(name="mongoTempl")
public MongoTemplate mongoTempl() throws Exception {
return new MongoTemplate(createMongoClient(new ServerAddress(host, port))
,dbName);
}
Mongo createMongoClient(ServerAddress serverAddress) {
final MongoClientOptions options = MongoClientOptions.builder()
.threadsAllowedToBlockForConnectionMultiplier(...)
.connectionsPerHost(...)
.connectTimeout(...)
.maxWaitTime(...)
.socketKeepAlive(...)
.socketTimeout(...)
.heartbeatConnectTimeout(...)
.minHeartbeatFrequency(...)
.build();
return new MongoClient(serverAddress, options);
}
You can use also MongoClientSettingsBuilderCustomizer like in this spring sample
#Bean
public MongoClientSettingsBuilderCustomizer customizer() {
return (builder) -> builder.applyToConnectionPoolSettings(
(connectionPool) -> {
connectionPool.maxSize(10);
connectionPool.minSize(2);
connectionPool.maxConnectionIdleTime(5, TimeUnit.MINUTES);
connectionPool.maxWaitTime(2, TimeUnit.MINUTES);
connectionPool.maxConnectionLifeTime(30, TimeUnit.MINUTES);
connectionPool.addConnectionPoolListener();
});
}
Related
I am facing an issue where my datasource bean is going down after a period of inactivity. My question is how could I re instantiate the datasource bean that gets hit on application startup.
Here is how we setup the bean on startup.
#ConfigurationProperties(prefix = "spring.datasource")
#Bean
public DataSource dataSource(){
byte[] encryptedFile = fileRetriever.getFile(bucket, key);
String unencryptedJson = fileDecrypter.decryptFile(encryptedFile);
JSONParser parser = new JSONParser();
JSONObject jsonObject = null;
try{
jsonObject = (JSONObject) parser.parse(unencryptedJson);
}catch (Exception ex){
log.error(ex.getMessage());
}
String password = (String)jsonObject.get("password");
DataSource ds = DataSourceBuilder
.create()
.url(url)
.username(userName)
.password(password)
.build();
return ds;
}
This class also has a #Configuration annotation on it.
We have other applications that do not have this issue where the service needs to be bounced after inactivity, but they are also not setting up the data source in this manner and have all the details specified in the application.property file
I have added a custom health check that uses a repository that hits every 30 seconds so that should keep the data source bean alive but incase it does go down I would need a way to recreate it.
Thanks in advance
I assume that boot is configuring the DataSource for you. In this case, and since you are using MySQL, you can add the following to your application.properties up to 1.3
spring.datasource.test-on-borrow=true
spring.datasource.validationQuery=SELECT 1
Might considered a pooled datasource connector. Look at apache dbcb2.
Here is a sample i have that keeps a minimum of 10 idle and increases as needed from the pool.
private static DataSource createDataSource(String url, String userName, String passwrd) throws Exception {
Class.forName(DRIVER).newInstance();
Properties props = new Properties();
props.setProperty("user", userName);
props.setProperty("password", passwrd);
//Create a connection factory that the pool will use to create connections
ConnectionFactory cf = new DriverManagerConnectionFactory(url, props);
//Create the poolable connection factory
PoolableConnectionFactory pcf = new PoolableConnectionFactory(cf, null);
pcf.setValidationQuery("SELECT 1");
pcf.setDefaultAutoCommit(true);
GenericObjectPoolConfig poolConfig = new GenericObjectPoolConfig();
poolConfig.setMinIdle(10);
poolConfig.setMaxTotal(100);
AbandonedConfig abandonConfig = new AbandonedConfig();
abandonConfig.setRemoveAbandonedTimeout(60);
abandonConfig.setLogAbandoned(false);
abandonConfig.setRemoveAbandonedOnBorrow(true);
abandonConfig.setRemoveAbandonedOnMaintenance(true);
//Create the pool of connections
GenericObjectPool<PoolableConnection> connectionPool = new GenericObjectPool<>(pcf, poolConfig);
connectionPool.setTestOnBorrow(true);
connectionPool.setTestWhileIdle(true);
connectionPool.setTimeBetweenEvictionRunsMillis(10000);
connectionPool.setMinEvictableIdleTimeMillis(1000);
connectionPool.setAbandonedConfig(abandonConfig);
pcf.setPool(connectionPool);
//Pooling data source itself
PoolingDataSource<PoolableConnection> dataSource = new PoolingDataSource<>(connectionPool);
return dataSource;
}
Maven dependencies for apache dbcb2
<!-- Database connection pools -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-dbcp2</artifactId>
<version>2.1.1</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-pool2</artifactId>
<version>2.4.2</version>
</dependency>
I want to change the default size of connection pool provided by java mongodb driver which is 100 according to mongo docs.
Below is the mongo client bean which I used to customize the connection pool size (refered this question). I set both min and max connectionPerHost attributes to 1 and ran 10 parallel worker threads which interact with the DB to make sure that my change is applied.
#Bean
public Mongo mongo() throws Exception {
MongoClientOptions.Builder clientOptions = new MongoClientOptions.Builder();
clientOptions.minConnectionsPerHost(1);
clientOptions.connectionsPerHost(1);
MongoClient mongoClient = new MongoClient(new MongoClientURI(env.getProperty("mongodbhost"), clientOptions));
return mongoClient;
}
Then I calculated the starting and ending time spots of each worker thread. So that I know for sure the threads are working parallely and my connection pool size haven't changed by these configuration.
Could anyone help me to get through this please? any help would be highly appreciated!
You can configure connection parameters by uri.
spring.data.mongodb.uri=mongodb://localhost:27017/?connectTimeoutMS=300000&minPoolSize=0&maxPoolSize=10&maxIdleTimeMS=900000
Please see the following documentation for other parameters.
https://docs.mongodb.com/manual/reference/connection-string/#connections-connection-options
With updated Spring boot(2.0.0 +) and Mongo DB java(3.9 +) driver versions following code can be used for creating configurable mongo template in spring boot.
Most of the configurations that were earlier part of MongoClientOptions are moved to MongoClientSettings.
import com.mongodb.*;
import com.mongodb.client.MongoClient;
import com.mongodb.client.MongoClients;
import com.mongodb.connection.*;
import org.springframework.data.mongodb.core.MongoTemplate;
#Configuration
public class MongoConfig {
//-- variables
#Bean(name = "mongoTemplate")
public MongoTemplate getMongoTemplate(){
MongoTemplate mongoTemplate = new MongoTemplate(getMongoClient(), mongoDatabaseName);
return mongoTemplate;
}
private MongoClient getMongoClient(){
List<ServerAddress> serverAddressList = new ArrayList<>();
String[] hostPortList = mongoHostPortList.split(",");
for (String serverAddress : hostPortList) {
String[] hostPortArr = serverAddress.split(":");
serverAddressList.add(new ServerAddress(hostPortArr[0], Integer.parseInt(hostPortArr[1])));
}
MongoClientSettings mongoSettingsProperties = getMongoClientSettings();
MongoClient mongoClient = MongoClients.create(mongoSettingsProperties);
return mongoClient;
}
private MongoClientSettings getMongoClientSettings() {
return MongoClientSettings.builder()
.applicationName(appName)
.applyToSslSettings(sslBuilder ->
SslSettings.builder().
enabled(sslEnabled).
invalidHostNameAllowed(false).build())
.applyToConnectionPoolSettings(connPoolBuilder ->
ConnectionPoolSettings.builder().
maxWaitTime(maxWaitTime, MILLISECONDS).
maxSize(connectionPoolMinSize).
maxSize(connectionPoolMaxSize).build())
.applyToSocketSettings(socketBuilder ->
SocketSettings.builder().
connectTimeout(connectionTimeout,MILLISECONDS).build())
.readPreference(ReadPreference.secondaryPreferred())
.build();
}
}
Above code is verified with dependencies -
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-mongodb</artifactId>
<version>2.3.1.RELEASE</version>
</dependency>
<dependency>
<groupId>org.mongodb</groupId>
<artifactId>mongo-java-driver</artifactId>
<version>3.11.2</version>
</dependency>
You can configure connection pool size via MongoDb uri parameters. Details - https://stackoverflow.com/a/50407284/6629515
One of the application uses MongoClient as core for interacting with MongoDB in which authentication has been enabled recently. In this mongoClient is initialize as:
mongoClient = new MongoClient(serverAddress, Arrays.asList(MongoCredential.createCredential(userName, dbName, password.toCharArray())));
However at many places app uses mongoTemplate to query the data. Now if MongoTemplate is created as :
new MongoTemplate(mongoClient, dbName);
It leads to authentication failure.
The only way seems to pass user credentials to MongoTemplate is via using UserCredentials class by
However if we pass UserCredentials as :
public MongoTemplate(Mongo mongo, String databaseName, UserCredentials userCredentials) {
Which results to :
Usage of 'UserCredentials' with 'MongoClient' is no longer supported. Please use 'MongoCredential' for 'MongoClient' or just 'Mongo'.
It seems like two different API exists in parallel. What's the best way so that both of them can live together.
This app uses mongodata version as '1.10.6.RELEASE'
Try this:
MongoCredential mongoCredential = MongoCredential.createCredential("user", "database","password".toCharArray());
ServerAddress address = new ServerAddress("mymongo.mycompany.com", 62797);
MongoClient mongoClient = new MongoClient(address, Arrays.asList(mongoCredential));
MongoTemplate mongoTemplate = new MongoTemplate(mongoClient, "database");
Try this configuration:
#Configuration
public class MongoConfiguration {
#Bean
public MongoDbFactory mongoDbFactory() throws Exception {
UserCredentials userCredentials = new UserCredentials("YOUR_USER_NAME", "YOUR_PASSWORD");
return new SimpleMongoDbFactory(new Mongo(), "YOUR_DATABASE", userCredentials);
}
#Bean
public MongoTemplate mongoTemplate() throws Exception {
return new MongoTemplate(mongoDbFactory());
}
}
And to create database repositories just use MongoRepository like this:
public interface UserRepository extends MongoRepository<User,Serializable>{
User findById(String id);
}
In this case seems like it was problem of applying user authentication at mongod server end.
Needed authentication was applied to mongo and has been validated by
db.auth('user','pass');
command which results to 1. However database at that time doesn't exists. Afterwards database was created first by inserted a dummy record and permissions were assigned.
App uses all together a different DB for unit test cases for which it looks like configuration was not correctly applied where this issue was arriving.
Once corrected applied
new MongoClient(serverAddress, Arrays.asList(MongoCredential.createCredential(userName, dbName, password.toCharArray())));
seems to work fine. However at same time mongo driver errors are a bit cryptic without much explanation leading to making debugging time consuming.
I am developing a Spring Boot [web] REST-style application with a ServletInitializer (since it needs to be deployed to an existing Tomcat server). It has a #RestController with a method that, when invoked, needs to write to a Redis pub-sub channel. I have the Redis server running on localhost (default port, no password). The relevant part of the POM file has the required starter dependency:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
When I deploy the WAR and hit the endpoint http://localhost:8080/springBootApp/health, I get this response:
{
"status": "DOWN",
"diskSpace": {
"status": "UP",
"total": 999324516352,
"free": 691261681664,
"threshold": 10485760
},
"redis": {
"status": "DOWN",
"error": "org.springframework.data.redis.RedisConnectionFailureException: java.net.SocketTimeoutException: Read timed out; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out"
}
}
I added the following to my Spring Boot application class:
#Bean
JedisConnectionFactory jedisConnectionFactory() {
return new JedisConnectionFactory();
}
#Bean
public RedisTemplate<String, Object> redisTemplate() {
RedisTemplate<String, Object> template = new RedisTemplate<String, Object>();
template.setConnectionFactory(jedisConnectionFactory());
return template;
}
I also tried adding the following to my #RestController before executing some test Redis code, but I get the same error as above in the stack trace:
#Autowired
private RedisTemplate<String, String> redisTemplate;
Edit (2017-05-09)
My understanding is that Spring Boot Redis starter assumes the default values of spring.redis.host=localhost and spring.redis.port=6379, I still added the two to application.properties, but that did not fill the gap.
Update (2017-05-10)
I added an answer to this thread.
I done a simple example with redis and spring boot
First I installed redis on docker:
$ docker run --name some-redis -d redis redis-server --appendonly yes
Then I Used this code for receiver :
import java.util.concurrent.CountDownLatch;
public class Receiver {
private static final Logger LOGGER = LoggerFactory.getLogger(Receiver.class);
private CountDownLatch latch;
#Autowired
public Receiver(CountDownLatch latch) {
this.latch = latch;
}
public void receiveMessage(String message) {
LOGGER.info("Received <" + message + ">");
latch.countDown();
}
}
And this is my spring boot app and my listener:
#SpringBootApplication
// after add security library then it is need to use security configuration.
#ComponentScan("omid.spring.example.springexample.security")
public class RunSpring {
private static final Logger LOGGER = LoggerFactory.getLogger(RunSpring.class);
public static void main(String[] args) throws InterruptedException {
ConfigurableApplicationContext contex = SpringApplication.run(RunSpring.class, args);
}
#Autowired
private ApplicationContext context;
#RestController
public class SimpleController{
#RequestMapping("/test")
public String getHelloWorld(){
StringRedisTemplate template = context.getBean(StringRedisTemplate.class);
CountDownLatch latch = context.getBean(CountDownLatch.class);
LOGGER.info("Sending message...");
Thread t = new Thread(new Runnable() {
#Override
public void run() {
for (int i = 0 ; i < 100 ; i++) {
template.convertAndSend("chat", i + " => Hello from Redis!");
try {
Thread.sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
});
t.start();
try {
latch.await();
} catch (InterruptedException e) {
e.printStackTrace();
}
return "hello world 1";
}
}
///////////////////////////////////////////////////////////////
#Bean
RedisMessageListenerContainer container(RedisConnectionFactory connectionFactory,
MessageListenerAdapter listenerAdapter) {
RedisMessageListenerContainer container = new RedisMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.addMessageListener(listenerAdapter, new PatternTopic("chat"));
return container;
}
#Bean
MessageListenerAdapter listenerAdapter(Receiver receiver) {
return new MessageListenerAdapter(receiver, "receiveMessage");
}
#Bean
Receiver receiver(CountDownLatch latch) {
return new Receiver(latch);
}
#Bean
CountDownLatch latch() {
return new CountDownLatch(1);
}
#Bean
StringRedisTemplate template(RedisConnectionFactory connectionFactory) {
return new StringRedisTemplate(connectionFactory);
}
}
The important point is the redis IP. if you installed it on docker like me then
you should set ip address in application.properties like this:
spring.redis.host=172.17.0.4
I put all my spring examples on github here
In addition I used redis stat to monitor redis. it is simple monitoring.
Spring data redis properties are updated, e.g. spring.redis.host is now spring.data.redis.host.
You need to configure your redis server information using the application.properties:
# REDIS (RedisProperties)
spring.redis.cluster.nodes= # Comma-separated list of "host:port"
spring.redis.database=0 # Database index
spring.redis.url= # Connection URL,
spring.redis.host=localhost # Redis server host.
spring.redis.password= # Login password of the redis server.
spring.redis.ssl=false # Enable SSL support.
spring.redis.port=6379 # Redis server port.
Spring data docs: https://docs.spring.io/spring-boot/docs/current/reference/html/common-application-properties.html#REDIS
This was a proxy related problem, where even access to localhost was somehow being curtailed. Once I disabled the proxy settings, Redis health was UP! So the problem is solved. I did not have to add any property to application.properties and neither did I have to explicitly configure anything in the Spring Boot application class, because Spring Boot and the Redis Starter auto-configures based on Redis defaults (as applicable in my development environment). I just added the following to the pom.xml:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
and the following to the #RestController annotated class, and Spring Boot auto-wired as needed (awesome!).
#Autowired
private RedisTemplate<String, String> redisTemplate;
To publish a simple message to a channel, this single line of code was sufficient for validating the setup:
this.redisTemplate.convertAndSend(channelName, "hello world");
I appreciate all the comments, which were helpful in backing up my checks.
I'm creating two springboot server & client applications communicating using JMS, and everything is working fine with the release 5.12.1 for activemq, but as soon as I update to the 5.12.3 version, I'm getting the following error :
org.springframework.jms.support.converter.MessageConversionException: Could not convert JMS message; nested exception is javax.jms.JMSException: Failed to build body from content. Serializable class not available to broker. Reason: java.lang.ClassNotFoundException: Forbidden class MyClass! This class is not trusted to be serialized as ObjectMessage payload. Please take a look at http://activemq.apache.org/objectmessage.html for more information on how to configure trusted classes.
I went on the URL that is provided and I figured out that my issue is related to the new security implemented in the 5.12.2 release of ActiveMQ, and I understand that I could fix it by defining the trusted packages, but I have no idea on where to put such a configuration in my SpringBoot project.
The only reference I'm making to the JMS queue in my client and my server is setting up it's URI in application.properties and enabling JMS on my "main" class with #EnableJms, and here's my configuration on the separate broker :
#Configuration
#ConfigurationProperties(prefix = "activemq")
public class BrokerConfiguration {
/**
* Defaults to TCP 10000
*/
private String connectorURI = "tcp://0.0.0.0:10000";
private String kahaDBDataDir = "../../data/activemq";
public String getConnectorURI() {
return connectorURI;
}
public void setConnectorURI(String connectorURI) {
this.connectorURI = connectorURI;
}
public String getKahaDBDataDir() {
return kahaDBDataDir;
}
public void setKahaDBDataDir(String kahaDBDataDir) {
this.kahaDBDataDir = kahaDBDataDir;
}
#Bean(initMethod = "start", destroyMethod = "stop")
public BrokerService broker() throws Exception {
KahaDBPersistenceAdapter persistenceAdapter = new KahaDBPersistenceAdapter();
persistenceAdapter.setDirectory(new File(kahaDBDataDir));
final BrokerService broker = new BrokerService();
broker.addConnector(getConnectorURI());
broker.setPersistent(true);
broker.setPersistenceAdapter(persistenceAdapter);
broker.setShutdownHooks(Collections.<Runnable> singletonList(new SpringContextHook()));
broker.setUseJmx(false);
final ManagementContext managementContext = new ManagementContext();
managementContext.setCreateConnector(true);
broker.setManagementContext(managementContext);
return broker;
}
}
So I'd like to know where I'm supposed to specify the trusted packages.
Thanks :)
You can just set one of the below spring boot properties in application.properties to set trusted packages.
spring.activemq.packages.trust-all=true
or
spring.activemq.packages.trusted=<package1>,<package2>,<package3>
Add the following bean:
#Bean
public ActiveMQConnectionFactory activeMQConnectionFactory() {
ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory("your broker URL");
factory.setTrustedPackages(Arrays.asList("com.my.package"));
return factory;
}
The ability to do this via a configuration property has been added for the next release:
https://github.com/spring-projects/spring-boot/issues/5631
Method: public void setTrustedPackages(List<String> trustedPackages)
Description: add all packages which is used in send and receive Message object.
Code : connectionFactory.setTrustedPackages(Arrays.asList("org.api","java.util"))
Implementated Code:
private static final String DEFAULT_BROKER_URL = "tcp://localhost:61616";
private static final String RESPONSE_QUEUE = "api-response";
#Bean
public ActiveMQConnectionFactory connectionFactory(){
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory();
connectionFactory.setBrokerURL(DEFAULT_BROKER_URL);
connectionFactory.setTrustedPackages(Arrays.asList("org.api","java.util"));
return connectionFactory;
}
#Bean
public JmsTemplate jmsTemplate(){
JmsTemplate template = new JmsTemplate();
template.setConnectionFactory(connectionFactory());
template.setDefaultDestinationName(RESPONSE_QUEUE);
return template;
}
If any one still looking for an answer, below snippet worked for me
#Bean
public ActiveMQConnectionFactory connectionFactory() {
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory();
connectionFactory.setBrokerURL(BROKER_URL);
connectionFactory.setPassword(BROKER_USERNAME);
connectionFactory.setUserName(BROKER_PASSWORD);
connectionFactory.setTrustAllPackages(true); // all packages are considered as trusted
//connectionFactory.setTrustedPackages(Arrays.asList("com.my.package")); // selected packages
return connectionFactory;
}
I am setting Java_opts something like below and passing to java command and its working for me.
JAVA_OPTS=-Xmx256M -Xms16M -Dorg.apache.activemq.SERIALIZABLE_PACKAGES=*
java $JAVA_OPTS -Dapp.config.location=/data/config -jar <your_jar>.jar --spring.config.location=file:/data/config/<your config file path>.yml
Yes I found it's config in the new version
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.4.0.RELEASE</version>
</parent>
spring:
profiles:
active: #profileActive#
cache:
ehcache:
config: ehcache.xml
activemq:
packages:
trusted: com.stylrplus.api.model