I have the following issue that I have to solve in my business. I am using Spring for project development I have 8 dataSource to connect.
A request will be made informing the contract number, through this contact number I will have to select one of the 8 dataSource and make the client consultation.
Example:
I have the base Brazil = 1,Spain = 2 and Germany = 3
If the contract was id = 1, then you should get the customer data from Brazil base.
If the contract was id = 2, then you should get the customer data
from Spain base.
If the contract was id = 3, then you should fetch customer data from the Germany base.
I don't know how to solve this problem if I use multitenancy or AbstractRouting. And I don't know how to start the code for this.
Would anyone have any solution and an example?
Spring has way to determine datasource dynamically using AbstractRoutingDataSource. But need to use ThreadLocal to bind context to current thread.
This may complicate things if you are spawning multiple threads or using async in your app.
You can refer simple exmaple here
If you really need to create database connection as per different clients then hibernate provides a way to manage Multi-tenancy, Hibernate Guide. As explained general approach would be to use connection pool per-tenant or single connection pool per all tenants sharing same database but different schema.
Below are the two implementation used to switch to multiple tenants -
public class TestDataSourceBasedMultiTenantConnectionProviderImpl
extends AbstractDataSourceBasedMultiTenantConnectionProviderImpl {
private static final long serialVersionUID = 14535345L;
#Autowired
private DataSource defaultDataSource;
#Autowired
private TestDataSourceLookup dataSourceLookup;
/**
*
* Select datasources in situations where not tenantId is used (e.g. startup
* processing).
*
*/
#Override
protected DataSource selectAnyDataSource() {
//logger.trace("Select any dataSource: " + defaultDataSource);
return defaultDataSource;
}
/**
*
* Obtains a DataSource based on tenantId
*
*/
#Override
protected DataSource selectDataSource(String tenantIdentifier) {
DataSource ds = dataSourceLookup.getDataSource(tenantIdentifier);
// logger.trace("Select dataSource from " + tenantIdentifier + ": " + ds);
return ds;
}
import org.hibernate.context.spi.CurrentTenantIdentifierResolver;
import org.springframework.beans.factory.annotation.Autowired;
public class TestCurrentTenantIdentifierResolverImpl implements CurrentTenantIdentifierResolver {
#Autowired
private RequestContext context;
#Override
public String resolveCurrentTenantIdentifier() {
// TODO Auto-generated method stub
return context.getTenantID();
}
#Override
public boolean validateExistingCurrentSessions() {
// TODO Auto-generated method stub
return true;
}
}
AbstractRouting would be a way to do it, indeed. You can mix that approach with headers on the request to easily implement multi-tenancy.
The link provided in another post on Baeldung does indeed provide a solution using such construct.
Sharding and deploying a separate micro-service for each tenant, with a routing service in front, would be another one and I think it offers several benefits both from coding (no need for thread-locals or other similarly difficult construct) and maintenance perspective (you can take down your service for maintenance/customisation at tenant level), so it would be my solution of choice in your specific case, assuming that business requirements allow for it.
Still, again, the best solution depends on your specific case.
The easiest solution matching the title of the question - for readers that are not necessarily concerned with multi-tenancy - would be to create several dataSource instances in your Spring's #Configuration beans, put them in a Map<Integer, DataSource> and then return that Map<Integer, DataSource> object as a Bean. When you have to access them, you access the Map bean you created and then create your NamedParameterJdbcTemplate or similar resource passing the specific DataSource in the constructor.
Pseudo-code example:
#Configuration
class DataSourceConfig {
public final static int SPAIN = 2;
public final static int BRAZIL = 1;
#Bean
#Qualifier("dataSources")
public Map<Integer, DataSource> dataSources() {
Map<Integer, DataSource> ds = new HashMap<>();
ds.put(SPAIN, buildSpainDataSource());
ds.put(BRAZIL, buildBrazilDataSource());
return ds;
}
private DataSource buildSpainDataSource() {
...
}
private DataSource buildBrazilDataSource() {
...
}
}
#Service
class MyService {
#Autowired
#Qualifier("dataSources")
Map<Integer, DataSource> dataSources;
Map<String, Object> getObjectForCountry(int country) {
NamedParameterJdbcTemplate t = new NamedParameterJdbcTemplate(dataSources.get(country));
return t.queryForMap("select value from mytable", new HashMap<>());
}
}
Related
I’m trying to implement a multi tenant micro service using Spring Boot. I already implemented the web layer and the persistence layer. On web layer, I’ve implement a filter which sets the tenant id in a prototype bean (using ThreadLocalTargetSource), on persistence layer I’ve used Hibernate multi tenancy configuration (schema per tenant), they work fine, data is persisted in the appropriate schema. Currently I am implementing the same behaviour on messaging layer, using spring-kaka library, so far ir works the way I expected, but I’d like to know if there is a better way to do it.
Here is my code:
This si the class that manage a KafkaMessageListenerContainer:
#Component
public class MessagingListenerContainer {
private final MessagingProperties messagingProperties;
private KafkaMessageListenerContainer<String, String> container;
#PostConstruct
public void init() {
ContainerProperties containerProps = new ContainerProperties(
messagingProperties.getConsumer().getTopicsAsList());
containerProps.setMessageListener(buildCustomMessageListener());
container = createContainer(containerProps);
container.start();
}
#Bean
public MessageListener<String, String> buildCustomMessageListener() {
return new CustomMessageListener();
}
private KafkaMessageListenerContainer<String, String> createContainer(
ContainerProperties containerProps) {
Map<String, Object> props = consumerProps();
…
return container;
}
private Map<String, Object> consumerProps() {
Map<String, Object> props = new HashMap<>();
…
return props;
}
#PreDestroy
public void finish() {
container.stop();
}
}
This is the CustomMessageListener:
#Slf4j
public class CustomMessageListener implements MessageListener<String, String> {
#Autowired
private TenantStore tenantStore; // Prototype Bean
#Autowired
private List<ServiceListener> services;
#Override
public void onMessage(ConsumerRecord<String, String> record) {
log.info(“Tenant {} | Payload: {} | Record: {}", record.key(),
record.value(), record.toString());
tenantStore.setTenantId(record.key()); // Currently tenant is been setting as key
services.stream().forEach(sl -> sl.onMessage(record.value()));
}
}
This is a test service which would use the message data and tenant:
#Slf4j
#Service
public class ConsumerService implements ServiceListener {
private final MessagesRepository messages;
private final TenantStore tenantStore;
#Override
public void onMessage(String message) {
log.info("ConsumerService {}, tenant {}", message, tenantStore.getTenantId());
messages.save(new Message(message));
}
}
Thanks for your time!
Just to be clear ( correct me if I'm wrong ): you are using the same topic(s) for all your tenants. The way that you distinguish the message according to each tenant is by using the message key which in your case is the tenant id.
A slight improvement can be done by using message headers to store the tenant id instead of the key. By doing this then you will not be limited to partitioning messages based on tenants.
Although the model described by you works it has a major security issue. If someone gets access to your topic then you will be leaking data of all your tenants.
A more secure approach is using topic naming conventions and ACL's ( access control lists ). You can find a short explanation here. In a nutshell, you can include the name of your tenant in the topic's name by either using a suffix or a prefix.
e.g: orders_tenantA, orders_tenantB or tenantA_orders, tenantB_orders
Then, using ACL's you can restrict which applications can connect to those specific topics. This scenario is also helpful if one of your tenants need to connect one of their applications directly to your Kafka cluster.
So far, the only way I know to set the name of a database, to use with Spring Data ArangoDB, is by hardcoding it in a database() method while extending AbstractArangoConfiguration, like so:
#Configuration
#EnableArangoRepositories(basePackages = { "com.company.mypackage" })
public class MyConfiguration extends AbstractArangoConfiguration {
#Override
public ArangoDB.Builder arango() {
return new ArangoDB.Builder();
}
#Override
public String database() {
// Name of the database to be used
return "example-database";
}
}
What if I'd like to implement multi-tenancy, where each tenant has data in a separate database and use e.g. a subdomain to determine which database name should be used?
Can the database used by Spring Data ArangoDB be determined at runtime, dynamically?
This question is related to the discussion here: Manage multi-tenancy ArangoDB connection - but is Spring Data ArangoDB specific.
Turns out this is delightfully simple: Just change the ArangoConfiguration database() method #Override to return a Spring Expression (SpEL):
#Override
public String database() {
return "#{tenantProvider.getDatabaseName()}";
}
which in this example references a TenantProvider #Component which can be implemented like so:
#Component
public class TenantProvider {
private final ThreadLocal<String> databaseName;
public TenantProvider() {
super();
databaseName = new ThreadLocal<>();
}
public String getDatabaseName() {
return databaseName.get();
}
public void setDatabaseName(final String databaseName) {
this.databaseName.set(databaseName);
}
}
This component can then be #Autowired wherever in your code to set the database name, such as in a servlet filter, or in my case in an Apache Camel route Processor and in database service methods.
P.s. I became aware of this possibility by reading the ArangoTemplate code and a Spring Expression support documentation section
(via), and one merged pull request.
I have started converting an existing Spring Boot(1.5.4.RELEASE) application to support multi-tenant capabilities. So i am using MySQL as the database and Spring Data JPA as the data access mechanism. i am using the schema based multi-tenant approach. As Hibernate document suggests below
https://docs.jboss.org/hibernate/orm/4.2/devguide/en-US/html/ch16.html
I have implemented MultiTenantConnectionProvider and CurrentTenantIdentifierResolver interfaces and I am using ThreadLocal variable to maintain the current tenant for the incoming request.
public class TenantContext {
final public static String DEFAULT_TENANT = "master";
private static ThreadLocal<Tenant> tenantConfig = new ThreadLocal<Tenant>() {
#Override
protected Tenant initialValue() {
Tenant tenant = new Tenant();
tenant.setSchemaName(DEFAULT_TENANT);
return tenant;
}
};
public static Tenant getTenant() {
return tenantConfig.get();
}
public static void setTenant(Tenant tenant) {
tenantConfig.set(tenant);
}
public static String getTenantSchema() {
return tenantConfig.get().getSchemaName();
}
public static void clear() {
tenantConfig.remove();
}
}
Then i have implemented a filter and there i set the tenant dynamically looking at a request header as below
String targetTenantName = request.getHeader(TENANT_HTTP_HEADER);
Tenant tenant = new Tenant();
tenant.setSchemaName(targetTenantName);
TenantContext.setTenant(tenant);
This works fine and now my application points to different schema based on the request header value.
However there is a master schema where i store the some global settings and i need to access that schema while in a middle of a request for a tenant. Therefore i tried to hard code the Threadlocal variable just before that database call in the code as below.
Tenant tenant = new Tenant();
tenant.setSchemaName("master");
TenantContext.setTenant(tenant);
However this does not point to the master schema and instead it tries to access the original schema set during the filter. What is the reason for this?
As per my understanding Hibernate invokes openSession() during the first database call to a tenant and after i try to invoke another database call for "master" it still use the previous tenant as CurrentTenantIdentifierResolver invokes only during the openSession(). However these different database calls does not invoke within a transaction.
Can you please help me to understand the issue with my approach and any suggestions to fix the issue
Thanks
Keth
#JonathanJohx actually i am trying to override the TenantContext set by the filter in one of the controllers. First i am loging in a tenant where TenantContext is set to that particular tenant. While the request is in that tenant i am requesting data from master. In order to do that i am simply hard code the tenant as below
#RestController
#RequestMapping("/jobTemplates")
public class JobTemplateController {
#Autowired
JobTemplateService jobTemplateService;
#GetMapping
public JobTemplateList list(Pageable pageable){
Tenant tenant = new Tenant();
tenant.setSchemaName(multitenantMasterDb);
TenantContext.setTenant(tenant);
return jobTemplateService.list(pageable);
}
I work for a company that has multiple brands, therefore we have a couple MongoDB instances on some different hosts holding the same document model for our Customer on each of these brands. (Same structure, not same data)
For the sake of simplicity let's say we have an Orange brand, with database instance serving on port 27017 and Banana brand with database instance serving on port 27018
Currently I'm developing a fraud detection service which is required to connect to all databases and analyze all the customers' behavior together regardless of the brand.
So my "model" has a shared entity for Customer, annotated with #Document (org.springframework.data.mongodb.core.mapping.Document)
Next thing I have is two MongoRepositories such as:
public interface BananaRepository extends MongoRepository<Customer, String>
List<Customer> findAllByEmail(String email);
public interface OrangeRepository extends MongoRepository<Customer, String>
List<Customer> findAllByEmail(String email);
With some stub method for finding customers by Id, Email, and so on. Spring is responsible for generating all implementation classes for such interfaces (pretty standard spring stuff)
In order to hint each of these repositories to connect to the right mongodb instance, I need two Mongo Config such as:
#Configuration
#EnableMongoRepositories(basePackageClasses = {Customer.class})
public class BananaConfig extends AbstractMongoConfiguration {
#Value("${database.mongodb.banana.username:}")
private String username;
#Value("${database.mongodb.banana.database}")
private String database;
#Value("${database.mongodb.banana.password:}")
private String password;
#Value("${database.mongodb.banana.uri}")
private String mongoUri;
#Override
protected Collection<String> getMappingBasePackages() {
return Collections.singletonList("com.acme.model");
}
#Override
protected String getDatabaseName() {
return this.database;
}
#Override
#Bean(name="bananaClient")
public MongoClient mongoClient() {
final String authString;
//todo: Use MongoCredential
//todo: Use ServerAddress
//(See https://docs.spring.io/spring-data/mongodb/docs/current/reference/html/#repositories) 10.3.4
if ( valueIsPresent(username) ||valueIsPresent(password)) {
authString = String.format("%s:%s#", username, password);
} else {
authString = "";
}
String conecctionString = "mongodb://" + authString + mongoUri + "/" + database;
System.out.println("Going to connect to: " + conecctionString);
return new MongoClient(new MongoClientURI(conecctionString, builder()
.connectTimeout(5000)
.socketTimeout(8000)
.readPreference(ReadPreference.secondaryPreferred())
.writeConcern(ACKNOWLEDGED)));
}
#Bean(name = "bananaTemplate")
public MongoTemplate mongoTemplate(#Qualifier("bananaFactory") MongoDbFactory mongoFactory) {
return new MongoTemplate(mongoFactory);
}
#Bean(name = "bananaFactory")
public MongoDbFactory mongoFactory() {
return new SimpleMongoDbFactory(mongoClient(),
getDatabaseName());
}
private static int sizeOfValue(String value){
if (value == null) return 0;
return value.length();
}
private static boolean valueIsMissing(String value){
return sizeOfValue(value) == 0;
}
private static boolean valueIsPresent(String value){
return ! valueIsMissing(value);
}
}
I also have similar config for Orange which points to the proper mongo instance.
Then I have my service like this:
public List<? extends Customer> findAllByEmail(String email) {
return Stream.concat(
bananaRepository.findAllByEmail(email).stream(),
orangeRepository.findAllByEmail(email).stream())
.collect(Collectors.toList());
}
Notice that I'm calling both repositories and then collecting back the results into one single list
What I would expect to happen is that each repository would connect to its corresponding mongo instance and query for the customer by its email.
But this don't happened. I always got the query executed against the same mongo instance.
But in the database log I can see both connections being made by spring.
It just uses one connection to run the queries for both repositories.
This is not surprising as both Mongo Config points to the same model package here. Right. But I also tried other approaches such as creating a BananaCustomer extends Customer, into its own model.banana package, and another OrangeCustomer extends Customer into its model.orange package, along with specifying the proper basePackageClasses into each config. But that neither worked, I've ended up getting both queries run against the same database.
:(
After scavenging official Spring-data-mongodb documentation for hours, and looking throughout thousands of lines of code here and there, I've run out of options: seems like nobody have done what I'm trying to accomplish before.
Except for this guy here that had to do the same thing but using JPA instead of mongodb: Link to article
Well, while it's still spring-data it't not for mongodb.
So here is my question:
¿How can I explicitly tell each repository to use a specific mongo config?
Magical autowiring rules, except when it doesn't work and nobody understands the magic.
Thanks in advance.
Well, I had a very detailed answer but StackOverflow complained about looking like spam and didn't allow me to post
The full answer is still available as a Gist file here
The bottom line is that both MongoRepository (interface) and the model object must be placed in the same package.
I am trying to implement a multi-threaded solution so I can parallelize my business logic that includes reading and writing to a database.
Technology stack: Spring 4.0.2, Hibernate 4.3.8
Here is some code to discuss on:
Configuration
#Configuration
public class PartitionersConfig {
#Bean
public ForkJoinPoolFactoryBean forkJoinPoolFactoryBean() {
final ForkJoinPoolFactoryBean poolFactory = new ForkJoinPoolFactoryBean();
return poolFactory;
}
}
Service
#Service
#Transactional
public class MyService {
#Autowired
private OtherService otherService;
#Autowired
private ForkJoinPool forkJoinPool;
#Autowired
private MyDao myDao;
public void performPartitionedActionOnIds() {
final ArrayList<UUID> ids = otherService.getIds();
MyIdPartitioner task = new MyIdsPartitioner(ids, myDao, 0, ids.size() - 1);
forkJoinPool.invoke(task);
}
}
Repository / DAO
#Repository
#Transactional(propagation = Propagation.MANDATORY)
public class IdsDao {
public MyData getData(List<UUID> list) {
// ...
}
}
RecursiveAction
public class MyIdsPartitioner extends RecursiveAction {
private static final long serialVersionUID = 1L;
private static final int THRESHOLD = 100;
private ArrayList<UUID> ids;
private int fromIndex;
private int toIndex;
private MyDao myDao;
public MyIdsPartitioner(ArrayList<UUID> ids, MyDao myDao, int fromIndex, int toIndex) {
this.ids = ids;
this.fromIndex = fromIndex;
this.toIndex = toIndex;
this.myDao = myDao;
}
#Override
protected void compute() {
if (computationSetIsSamllEnough()) {
computeDirectly();
} else {
int leftToIndex = fromIndex + (toIndex - fromIndex) / 2;
MyIdsPartitioner leftPartitioner = new MyIdsPartitioner(ids, myDao, fromIndex, leftToIndex);
MyIdsPartitioner rightPartitioner = new MyIdsPartitioner(ids, myDao, leftToIndex + 1, toIndex);
invokeAll(leftPartitioner, rightPartitioner);
}
}
private boolean computationSetIsSamllEnough() {
return (toIndex - fromIndex) < THRESHOLD;
}
private void computeDirectly() {
final List<UUID> subList = ids.subList(fromIndex, toIndex);
final MyData myData = myDao.getData(sublist);
modifyTheData(myData);
}
private void modifyTheData(MyData myData) {
// ...
// write to DB
}
}
After executing this I get:
No existing transaction found for transaction marked with propagation 'mandatory'
I understood that this is perfectly normal since the transaction doesn't propagate through different threads. So one solution is to create a transaction manually in every thread as proposed in another similar question. But this was not satisfying enough for me so I kept searching.
In Spring's forum I found a discussion on the topic. One paragraph I find very interesting:
"I can imagine one could manually propagate the transaction context to another thread, but I don't think you should really try it. Transactions are bound to single threads with a reason - the basic underlying resource - jdbc connection - is not threadsafe. Using one single connection in multiple threads would break fundamental jdbc request/response contracts and it would be a small wonder if it would work in more then trivial examples."
So the first question arise: Is it worth it to pararellize the reading/writing to the database and can this really hurt the DB consistency?
If the quote above is not true, which I doubt, is there a way to achieve the following:
MyIdPartitioner to be Spring managed - with #Scope("prototype") - and pass the needed arguments for the recursive calls to it and that way leave the transaction management to Spring?
After further readings I managed to solve my problem. Kind of (as I see it now there wasn't a problem at the first place).
Since the reading I do from the DB is in chunks and I am sure that the results won't get edited during that time I can do it outside transaction.
The writing is also safe in my case since all values I write are unique and no constraint violations can occur. So I removed the transaction from there too.
What I mean by saying "I removed the transaction" just override the method's Propagation mode in my DAO like:
#Repository
#Transactional(propagation = Propagation.MANDATORY)
public class IdsDao {
#Transactional(propagation = Propagation.SUPPORTS)
public MyData getData(List<UUID> list) {
// ...
}
}
Or if you decide you need the transaction for some reason then you can still leave the transaction management to Spring by setting the propagation to REQUIRED.
So the solution turns out to be much much simpler than I thought.
And to answer my other questions:
Is it worth it to pararellize the reading/writing to the database and can this really hurt the DB consistency?
Yes, it's worth it. And as long as you have transaction per thread you are cool.
Is there a way to achieve the following: MyIdPartitioner to be Spring managed - with #Scope("prototype") - and pass the needed arguments for the recursive calls to it and that way leave the transaction management to Spring?
Yes there is a way by using pool (another stackoverflow question). Or you can define your bean as #Scope(value = "prototype", proxyMode = ScopedProxyMode.TARGET_CLASS) but then it won't work if you need to set parameters to it since every usage of the instance will give you a new instance. Ex.
#Autowire
MyIdsPartitioner partitioner;
public void someMethod() {
...
partitioner.setIds(someIds);
partitioner.setFromIndex(fromIndex);
partitioner.setToIndex(toIndex);
...
}
This will create 3 instances and you won't be able to use the object beneficial since the fields won't be set.
So in short - there is a way but I didn't need to go for it at first place.
This should be possible with atomikos (http://www.atomikos.com) and optionally with nested transactions.
If you do this, then take care to avoid deadlocks if multiple threads of a same root transaction write to the same tables in the database.