We Keep Coding

Configure SQL datasource through properties in Camel Blueprint (within Karaf)

Given a very simple Camel bundle for Karaf, generated with the camel-archetype-blueprint, I want to add a datasource that is configured via properties and not within the blueprint.xml.
I tried configuring a PropertiesComponent and accessing the property within the property value for the MySQL datasource in various ways, but none seems to work. When logging a message though, the properties are accessible.
How can a datasource be configured using parameter values from a properties file?
I especially need this to use the same datasource configuration for multiple bundles and distinguish between production/test environments. I thought about writing the properties with Maven during the build, depending on the target environment. Are there any other best practices on how to solve this datasource issue?
<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
http://camel.apache.org/schema/blueprint http://camel.apache.org/schema/blueprint/camel-blueprint.xsd">
<bean id="properties" class="org.apache.camel.component.properties.PropertiesComponent">
<property name="location" value="classpath:config.properties" />
</bean>
<bean id="dataSourceMySQL" class="com.mysql.jdbc.jdbc2.optional.MysqlDataSource">
<property name="url" value="jdbc:mysql://127.0.0.1/test_database" />
<!-- This causes an error, as it tries to connect with
`${mysqlUser}`#`localhost` without any evaluation -->
<property name="user" value="${mysqlUser}" />
<property name="password" value="${mysqlPassword}" />
</bean>
<service interface="javax.sql.DataSource" ref="dataSourceMySQL">
<service-properties>
<entry key="osgi.jndi.service.name" value="jdbc/mysqlDatasource" />
</service-properties>
</service>
<bean id="sql" class="org.apache.camel.component.sql.SqlComponent">
<property name="dataSource" ref="dataSourceMySQL" />
</bean>
<camelContext xmlns="http://camel.apache.org/schema/blueprint">
<route id="messageQuery">
<from uri="sql:SELECT * FROM messages" />
<log message="The user property is: {{mysqlUser}}, the query result is: ${body}" />
</route>
</camelContext>
</blueprint>
Just for an overview, the project layout looks like this:

a) Datasource in bundle with xml and bundle properties
You could use bundle properties. The example below optionally uses a bundle configuration in etc/org.camel.demo.cfg:
<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
http://camel.apache.org/schema/blueprint http://camel.apache.org/schema/blueprint/camel-blueprint.xsd">
<!-- etc/org.camel.demo.cfg -->
<cm:property-placeholder persistent-id="org.camel.demo" xmlns:cm="http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.0.0">
<cm:default-properties>
<cm:property name="mysqlUser" value="root"/>
<cm:property name="mysqlPassword" value=""/>
</cm:default-properties>
</cm:property-placeholder>
<bean id="properties" class="org.apache.camel.component.properties.PropertiesComponent">
<property name="location" value="classpath:config.properties" />
</bean>
<bean id="dataSourceMySQL" class="com.mysql.jdbc.jdbc2.optional.MysqlDataSource">
<property name="url" value="jdbc:mysql://127.0.0.1/test_database" />
<property name="user" value="${mysqlUser}" />
<property name="password" value="${mysqlPassword}" />
</bean>
</blueprint>
b) Shared datasource
Another option would be to use a shared datasource. Just deploy a blueprint file containing only the datasource (example below uses postgres).
You can combine this also with config properties as shown above.
Providing the OSGi service
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0">
<!-- use config properties if needed -->
<bean id="dataSource" class="org.postgresql.ds.PGPoolingDataSource" destroy-method="close">
<property name="serverName" value=":"/>
<property name="user" value="postgres"/>
<property name="password" value="postgres"/>
<property name="dataSourceName" value="demo"/>
<property name="initialConnections" value="2"/>
<property name="maxConnections" value="4" />
</bean>
<service interface="javax.sql.DataSource" ref="dataSource">
<service-properties>
<entry key="osgi.jndi.service.name" value="jdbc/demo"/>
</service-properties>
</service>
</blueprint>
Referencing from another bundle
In your bundle you can then lookup the datasource osgi:service/javax.sql.DataSource/(osgi.jndi.service.name=jdbc/demo)
To use this datasource in a SqlComponent, a reference must be created like this:
<reference id="dataSource" interface="javax.sql.DataSource"
filter="(osgi.jndi.service.name=jdbc/mysql)">
</reference>
<bean id="sql" class="org.apache.camel.component.sql.SqlComponent">
<property name="dataSource" ref="dataSource" />
</bean>
Using the persistence.xml
Make sure you import org.demo.osgi.datasource.**. Here is a usage example with persistence.xml:
<persistence version="2.0"
xmlns="http://java.sun.com/xml/ns/persistence" >
<persistence-unit name="demo" transaction-type="JTA">
<provider>org.hibernate.ejb.HibernatePersistence</provider>
<jta-data-source>osgi:service/javax.sql.DataSource/(osgi.jndi.service.name=jdbc/demo)</jta-data-source>
<mapping-file>META-INF/foo.xml</mapping-file>
</persistence-unit>
</persistence>
c) Use jdbc-feature (optional)
The xml file from above can be created and managed using the jdbc feature. It depends a bit on your version if it is available or not:
JBossFuse:admin#545074693af1> features:install jdbc hibernate jndi
JBossFuse:admin#545074693af1> install mvn:org.postgresql/postgresql/9.4.1208
Bundle ID: 292
JBossFuse:admin#545074693af1> resolve 292
JBossFuse:admin#545074693af1> jdbc:create -t postgres -u postgres -p postgres -url ${postgres.addr}:${postgres.port} demo
P.S.: If you want to remove the cleartext password from the config file use something like jasypt.

Hello I can see two things. The first one is that you have not said in which folder in the class path the config file reside. Unless the OSGI-INF folder is not already scanned you should have :
The second thing is about how you reference the properties. Unless otherwise defined the property prefixToken and suffixToken are by default "{{" and "}}"
http://camel.apache.org/properties.html
Which means that you need to replace your ${mysqlUser} with {{mysqlUser}}

Setup database connection with tomcat 7

I have a working programm with MySQL, Spring and a tomcat server, but the database connection is in spring-database.xml:
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.0.xsd">
<bean id="baseController" class="com.afterguard.sailplanner.controller.BaseController">
<property name="dao" ref="daoImpl" />
</bean>
<bean id="daoImpl" class="com.afterguard.sailplanner.dao.DaoImpl">
<property name="dataSource" ref="dataSource" />
</bean>
<bean id="dataSource"
class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="com.mysql.jdbc.Driver" />
<property name="url" value="jdbc:mysql://localhost:3306/sailplanner" />
<property name="username" value="sailplanner" />
<property name="password" value="sailplanner2" />
</bean>
</beans>
In order to deploy this program on a server now, I would need bring this database connection to the server, e.g. the context.xml oder server.xml.
How do I do I setup my tomcat 7 server to connect to the database and tell my application, that the database connection is setup in the tomcat?

You have to create a dataSource with JNDI :
First, in your spring bean configuration (spring-database.xml) add this :
<beans:bean id="dbDataSource" class="org.springframework.jndi.JndiObjectFactoryBean">
<beans:property name="jndiName" value="java:comp/env/jdbc/myDB"/>
</beans:bean>
Second, create the jndi resource in the file apache-tomcat/conf/server.xml :
<Resource name="jdbc/globalDB"
global="jdbc/globalDB"
auth="Container"
type="javax.sql.DataSource"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/sailplanner"
username="sailplanner"
password="sailplanner2"/>
Finally, create the resource link in the file apache-tomcat/conf/context.xml :
<ResourceLink name="jdbc/myDB"
global="jdbc/globalDB"
auth="Container"
type="javax.sql.DataSource" />

For Sql Server I do the following configuration:
You must be using Tomcat to manage connections. To do so we will be adding data sources to the TOMCAT_HOME/conf/context.xml file
<Context>
<Resource
name="jdbc/Oltp1"
auth="Container"
type="javax.sql.DataSource"
driverClassName="net.sourceforge.jtds.jdbc.Driver"
url="jdbc:jtds:sqlserver://localhost:3306/sailplanner"
username=user
password=password
testOnBorrow="true"
testOnConnect="true"
validationQuery="SELECT 1"
removeAbandoned="true"
removeAbandonedTimeout="30"
/>
</Context>
SQL Server JDBC driver (jtds-version.jar) must be copied to the TOMCAT_HOME/lib folder (if it is not there already)
If there are multiple data sources (OLTP1, OLTP2, etc) we will be adding multiple <Resource/> tags
You will be using the following JNDI names to name our data sources:
jdbc/Oltp1 – main OLTP database
jdbc/Oltp2 – secondary database
You will be configuring data sources so connection pool will know how to test connections and what to do with closed connections:
testOnBorrow="true"
testOnConnect="true"
validationQuery="SELECT 1"
removeAbandoned="true"
removeAbandonedTimeout="30"
Tomcat must be restarted when new resource is added or existing resource is modified.
application.properties file
Important! This file is deployed as a part of the classpath (src/main/resources/application.properties), not the external application.properties with externalized properties deployed to the TOMCAT_HOME/conf
You will be added the following property to this file:
spring.datasource.jndi-name=java:comp/env/jdbc/Oltp1
You need to remove all other spring.datasource properties from the application.properties if any exist
Important! You had to remove all spring.datasource properties from the external configuration file (application.properties deployed to the COMCAT_HOME/conf)

the database connection properties you should put in a property file and and depending upon your requirement you can change values in property file
<bean id="dataSource" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClass" value="${jdbc.driverClassName}" />
<property name="jdbcUrl" value="${jdbc.databaseurl}" />
<property name="user" value="${jdbc.username}" />
<property name="password" value="${jdbc.password}" />
</bean>
the property file
jdbc.driverClassName=com.mysql.jdbc.Driver
jdbc.databaseurl=jdbc:mysql://localhost:3306/sailplanner
jdbc.username=root
jdbc.password=root
and to include this file in xml use
<bean id="propertyConfigurer"
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="ignoreUnresolvablePlaceholders" value="true" />
<property name="locations">
<list>
<value>classpath:applications.properties</value>
</list>
</property>
</bean>

Does Application connection pooling override Tomcat connection pooling

I want to maintain connection pooling into my java/spring application. For this i used spring configuration file. Which looks like this
<bean id="dataSource" destroy-method="close"
class="org.apache.commons.dbcp.BasicDataSource">
<property name="driverClassName" value="${db.driver}"/>
<property name="url" value="${db.jdbcurl}"/>
<property name="username" value="${db.username}"/>
<property name="password" value="${db.password}"/>
<property name="initialSize" value="3"/> T
<property name="maxActive" value="100"/>
</bean>
This application war is hosted into tomcat. When i saw tomcat context.xml file which is set by some network team. Context.xml file also contain some configuration regarding the connection pooling.It looks like this
<Context>
<Resource name="jdbc/test"
auth="Container"
type="javax.sql.DataSource"
username="naveen"
password="12345"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql:localhost:3306/cyl"
maxActive="15"
maxIdle="3"/>
</Context>
Now i have two question to asked
1) Which connection pool configuration will apply when application runs.
2) If we have multiple war a.war(with connection pool config), b.war(with out CPC), c.war(without CPC) then, will tomcat connection pool setting will apply to all application.
3) what can be max value for maxActive attribute.
Thanks in advance

Loading Bean property values from context.xml

Currently we are loading our JDBC source values from properties file as per following:
<context:property-placeholder location="classpath:master.properties" ignore-unresolvable="true" />
<bean id="mainDataSource" class="com.jolbox.bonecp.BoneCPDataSource"
destroy-method="close">
<property name="driverClass" value="${database.driver}" />
<property name="jdbcUrl" value="${database.url}" />
<property name="username" value="${database.user}" />
<property name="password" value="${database.password}" />
<property name="idleConnectionTestPeriod" value="60" />
<property name="idleMaxAge" value="240" />
<property name="maxConnectionsPerPartition" value="2" />
<property name="minConnectionsPerPartition" value="2" />
<property name="partitionCount" value="3" />
<property name="acquireIncrement" value="10" />
<property name="statementsCacheSize" value="50" />
<property name="releaseHelperThreads" value="3" />
</bean>
<bean id="dataSource"
class="org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy"
scope="singleton">
<property name="targetDataSource">
<ref local="mainDataSource" />
</property>
</bean>
This seg. works fine with classpath based app.properties file and get rid of app.properties.
We would like to load this values from context.xml (either placed in META-INF or $CATALINA_HOME/conf/context.xml). It will help us load proper values at prod/staging servers.
Will appreciate any help or alternate method/suggestions. (If similar question is already answered, please share the link)
Thanks!

As Alan Hay mentioned you could externalize the datasource configuration into Tomcat's own context.xml and then have Spring do a JNDI lookup to retrieve it. This is an approach that I've commonly used on some of the projects I've worked on.
The pieces you need to put in place to achieve would be:
1. Add the datasource configuration to $CATALINA_HOME/conf/context.xml
<GlobalNamingResources>
<Resource type="javax.sql.DataSource"
name="dsName"
factory="com.jolbox.bonecp.BoneCPDataSource"
driverClassName="your.driver.classname"
jdbcUrl="your:driver:url"
username="username"
password="password"
idleMaxAge="240"
idleConnectionTestPeriod="60"
partitionCount="3"
acquireIncrement="10"
maxConnectionsPerPartition="2"
minConnectionsPerPartition="2"
statementsCacheSize="50"
releaseHelperThreads="3" />
</GlobalNamingResources>
2. Add a resource link in the application's META-INF/context.xml
<Context path="/YourApp">
<ResourceLink description="Datasource for YourApp"
global="jdbc/dsName"
name="jdbc/dsName"
type="javax.sql.DataSource" />
</Context>
3. Modify the Spring config to lookup the datasource in JNDI
<beans xmlns:jee="http://www.springframework.org/schema/jee"
xsi:schemaLocation="http://www.springframework.org/schema/jee classpath:/org/springframework/ejb/config/spring-jee-3.0.xsd">
<jee:jndi-lookup id="dataSource"
jndi-name="java:comp/env/jdbc/dsName" />
4. Move the driver and datasource jars
Since the datasource configuration is now container managed, you should place the database driver and datasource jars into $CATALINA_HOME/lib so they are available to Tomcat when it creates the datasource. These jars should no longer need to reside in the WEB-INF/lib of your application.

The way I do it, I have default properties file in my classpath, and I have one properties file that I use to override those default (development) properties:
in my spring file:
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="ignoreResourceNotFound" value="true"/>
<property name="locations">
<list>
<value>classpath:jdbc.properties</value>
<value>classpath:camel.properties</value>
<value>classpath:email.properties</value>
<value>${external_config}</value>
</list>
</property>
</bean>
In development, I use the properties file in the classpath.
On the different envionment, staging, integration, production we start the application with the following parameter:
-Dexternal_config=file:c:/staging.properties
That way I have the same build for all environments and can choose the location of my overriding properties file.
So, ${external_config} is a place holder for a configuration file that depends on the environment.
Spring looks for an environment variable named external_config, that variable should point to a configuration file. If the file is not found, Spring ignores it with ignoreResourceNotFound = true;

Spring 3.1 introduced support for environment specific profiles. Define some profiles, associate them with some properties files e.g. test.properties, live.properties and set an environment variable on server start -Dspring.profiles.active=Test. Clean and simple.
http://spring.io/blog/2011/02/11/spring-framework-3-1-m1-released/
Other option as hinted at above is to shift to a container managed Connection Pool looked up via JNDI i.e define a JNDI datasource named xyz the same on each server and then:

How can I use a datasource defined in Websphere Liberty Profile 8.5?

Im using WebSphere 8.5.5 Liberty profile to deploy an application that is using a datasource defined and exposed thru jndi. But im unable to use the datasource from my application. My server.xml looks like this:
<server description="new server">
<!-- Enable features -->
<featureManager>
<feature>jsp-2.2</feature>
<feature>jndi-1.0</feature>
<feature>ejbLite-3.1</feature>
<feature>jdbc-4.0</feature>
</featureManager>
<dataSource id="mssqlserver" jndiName="jdbc/sqlserver_prod" type="javax.sql.DataSource">
<jdbcDriver libraryRef="MSJDBCLib"/>
<connectionManager numConnectionsPerThreadLocal="10" id="ConnectionManager" minPoolSize="1"/>
<properties.microsoft.sqlserver username="sa" password="" databaseName="PROD"
serverName="10.211.55.4" portNumber="1433"/>
</dataSource>
<library id="MSJDBCLib">
<fileset dir="/Users/alter/Devel/sqlserver" includes="sqljdbc4.jar"/>
</library>
<httpEndpoint id="defaultHttpEndpoint"
host="0.0.0.0"
httpPort="9080"
httpsPort="9443" />
<application id="ee1" location="/Users/alter/Devel/xxxx/src/ear/target/ee1-ear.ear" name="ear_ear_exploded" type="ear" >
<classloader delegation="parentLast" commonLibraryRef="MSJDBCLib" />
</application>
<application id="ee1-web" location="/Users/alter/Devel/xxxx/src/web/target/ee1-web" name="web_exploded" type="war" />
</server>
Im using this spring configuration file to inject the datasource:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
<!-- START CONFIG DS -->
<bean id="dataSourcePROD" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="jdbc/sqlserver_prod"/>
</bean>
<bean id="sqlSessionFactory" class="org.mybatis.spring.SqlSessionFactoryBean">
<property name="dataSource" ref="dataSourcePROD"/>
<property name="mapperLocations" value="classpath*:mappers/**/*.xml"/>
</bean>
<bean id="sqlSessionTemplate" class="org.mybatis.spring.SqlSessionTemplate">
<constructor-arg index="0" ref="sqlSessionFactory"/>
</bean>
<bean id="genericMapper" class="org.mybatis.spring.mapper.MapperFactoryBean" abstract="true">
<property name="sqlSessionTemplate" ref="sqlSessionTemplate"/>
<property name="addToConfig" value="true"/>
</bean>
</beans>
Spring is able to found my "datasource" object from JNDI, but it can't cast the delivered object to javax.sql.Datasource.
The actual exception is :
org.springframework.beans.ConversionNotSupportedException: Failed to convert property value of type 'com.ibm.ws.jdbc.DataSourceService' to required type 'javax.sql.DataSource' for property 'dataSource';
Note1: I have no reference to the jdbc drivers in the deployable artifacts (ear, wars, etc...)
Im missing some configuration parameters in server.xml for the datasource definition??.

you may want to have a look at the Spring helper class WebSphereDataSourceAdapter. IBM has a knack of 'wrapping' services in services which requires a little 'unwrapping' to use.
have a look at this in the Spring API docs as a possible hint on what to try

I have similar code working but I've added the spring jee:jndi-lookup tag for lookup of the datasource inside spring.
<jee:jndi-lookup id="dataSourcePROD" jndi-name="jdbc/sqlserver_prodS" resource-ref="false" expected-type="javax.sql.DataSource"/>

username is wrong attribute in properties.microsoft.sqlserver element. You should use user instead:
<properties.microsoft.sqlserver user="sa" password="" databaseName="PROD"
serverName="10.211.55.4" portNumber="1433"/>