I’m trying to implement a multi tenant micro service using Spring Boot. I already implemented the web layer and the persistence layer. On web layer, I’ve implement a filter which sets the tenant id in a prototype bean (using ThreadLocalTargetSource), on persistence layer I’ve used Hibernate multi tenancy configuration (schema per tenant), they work fine, data is persisted in the appropriate schema. Currently I am implementing the same behaviour on messaging layer, using spring-kaka library, so far ir works the way I expected, but I’d like to know if there is a better way to do it.
Here is my code:
This si the class that manage a KafkaMessageListenerContainer:
#Component
public class MessagingListenerContainer {
private final MessagingProperties messagingProperties;
private KafkaMessageListenerContainer<String, String> container;
#PostConstruct
public void init() {
ContainerProperties containerProps = new ContainerProperties(
messagingProperties.getConsumer().getTopicsAsList());
containerProps.setMessageListener(buildCustomMessageListener());
container = createContainer(containerProps);
container.start();
}
#Bean
public MessageListener<String, String> buildCustomMessageListener() {
return new CustomMessageListener();
}
private KafkaMessageListenerContainer<String, String> createContainer(
ContainerProperties containerProps) {
Map<String, Object> props = consumerProps();
…
return container;
}
private Map<String, Object> consumerProps() {
Map<String, Object> props = new HashMap<>();
…
return props;
}
#PreDestroy
public void finish() {
container.stop();
}
}
This is the CustomMessageListener:
#Slf4j
public class CustomMessageListener implements MessageListener<String, String> {
#Autowired
private TenantStore tenantStore; // Prototype Bean
#Autowired
private List<ServiceListener> services;
#Override
public void onMessage(ConsumerRecord<String, String> record) {
log.info(“Tenant {} | Payload: {} | Record: {}", record.key(),
record.value(), record.toString());
tenantStore.setTenantId(record.key()); // Currently tenant is been setting as key
services.stream().forEach(sl -> sl.onMessage(record.value()));
}
}
This is a test service which would use the message data and tenant:
#Slf4j
#Service
public class ConsumerService implements ServiceListener {
private final MessagesRepository messages;
private final TenantStore tenantStore;
#Override
public void onMessage(String message) {
log.info("ConsumerService {}, tenant {}", message, tenantStore.getTenantId());
messages.save(new Message(message));
}
}
Thanks for your time!
Just to be clear ( correct me if I'm wrong ): you are using the same topic(s) for all your tenants. The way that you distinguish the message according to each tenant is by using the message key which in your case is the tenant id.
A slight improvement can be done by using message headers to store the tenant id instead of the key. By doing this then you will not be limited to partitioning messages based on tenants.
Although the model described by you works it has a major security issue. If someone gets access to your topic then you will be leaking data of all your tenants.
A more secure approach is using topic naming conventions and ACL's ( access control lists ). You can find a short explanation here. In a nutshell, you can include the name of your tenant in the topic's name by either using a suffix or a prefix.
e.g: orders_tenantA, orders_tenantB or tenantA_orders, tenantB_orders
Then, using ACL's you can restrict which applications can connect to those specific topics. This scenario is also helpful if one of your tenants need to connect one of their applications directly to your Kafka cluster.
Related
I'm new with Kafka and want to persist data from kafka topics to database tables (each topic flow to a specific table). I know Kafka connect exists and can be used to achieve this but there are reasons why this approach is preferred.
Unfortunately only one topic is writing the database. Kafka seems to not process() all processors concurrently. Either MyFirstData is writing to database or MySecondData but never but at the same time.
According the my readings the is the option overriding init() from of kafka stream Processor interface which offers context.forward() not sure if this will help and how to use it in my used case.
I use Spring Cloud Stream (but got the same behaviour with Kafka DSL and Processor API implementations)
My code snippet:
Configuring the consumers:
#Configuration
#RequiredArgsConstructor
public class DatabaseProcessorConfiguration {
private final MyFirstDao myFirstDao;
private final MySecondDao mySecondDao;
#Bean
public Consumer<KStream<GenericData.Record, GenericData.Record>> myFirstDbProcessor() {
return stream -> stream.process(() -> {
return new MyFirstDbProcessor(myFirstDao);
});
}
#Bean
public Consumer<KStream<GenericRecord, GenericRecord>> mySecondDbProcessor() {
return stream -> stream.process(() -> new MySecondDbProcessor(mySecondDao));
}
}
This MyFirstDbProcessor and MySecondDbProcessor is analog to this.
#Slf4j
#RequiredArgsConstructor
public class MyFirstDbProcessor implements Processor<GenericData.Record, GenericData.Record, Void, Void> {
private final MyFirstDao myFirstDao;
#Override
public void process(Record<GenericData.Record, GenericData.Record> record) {
CdcRecordAdapter adapter = new CdcRecordAdapter(record.key(), record.value());
MyFirstTopicKey myFirstTopicKey = adapter.getKeyAs(MyFirstTopicKey.class);
MyFirstTopicValue myFirstTopicValue = adapter.getValueAs(MyFirstTopicValue.class);
MyFirstData data = PersistenceMapper.map(myFirstTopicKey, myFirstTopicValue);
switch (myFirstTopicValue.getCrudOperation()) {
case UPDATE, INSERT -> myFirstDao.persist(data);
case DELETE -> myFirstDao.delete(data);
default -> System.err.println("unimplemented CDC operation streamed by kafka");
}
}
}
My Dao implementations: I try an implementation of MyFirstRepository with JPARepository and ReactiveCrudRepository but same behaviour. MySecondRepository is implemented analog to MyFirstRepository.
#Component
#RequiredArgsConstructor
public class MyFirstDaoImpl implements MyFirstDao {
private final MyFirstRepository myFirstRepository;
#Override
public MyFirstData persist(MyFirstData myFirstData) {
Optional<MyFirstData> dataOptional = MyFirstRepository.findById(myFirstData.getId());
if (dataOptional.isPresent()){
var data = dataOptional.get();
myFirstData.setCreatedDate(data.getCreatedDate());
}
return myFirstRepository.save(myFirstData);
}
#Override
public void delete(MyFirstData myFirstData) {
System.out.println("delete() from transaction detail dao called");
MyFirstRepository.delete(myFirstData);
}
}
My task is to get the Kafka message before the method with the #KafkaListner annotation, check the correlationId and requestId headers in it. If they're present, flush them to MDC or generate them otherwise.
And my question is how to get Kafka message with headers before method with the #KafkaListner?
You can try to write your own ConsumerInterceptor following instructions from here.
Apache Kafka provides a mechanism to add interceptors to producers and consumers. These objects are managed by Kafka, not Spring, and so normal Spring dependency injection won’t work for wiring in dependent Spring Beans. However, you can manually wire in those dependencies using the interceptor config() method. The following Spring Boot application shows how to do this by overriding boot’s default factories to add some dependent bean into the configuration properties.
ConsumerFactory definition:
#Bean
public ConsumerFactory<?, ?> kafkaConsumerFactory(SomeBean someBean) {
Map<String, Object> consumerProperties = new HashMap<>();
// consumerProperties.put(..., ...)
// ...
consumerProperties.put(ConsumerConfig.INTERCEPTOR_CLASSES_CONFIG, MyConsumerInterceptor.class.getName());
consumerProperties.put("some.bean", someBean);
return new DefaultKafkaConsumerFactory<>(consumerProperties);
}
Interceptor definition:
public class MyConsumerInterceptor implements ConsumerInterceptor<String, String> {
private SomeBean bean;
#Override
public void configure(Map<String, ?> configs) {
this.bean = (SomeBean) configs.get("some.bean");
}
#Override
public ConsumerRecords<String, String> onConsume(ConsumerRecords<String, String> records) {
this.bean.someMethod("consumer interceptor");
return records;
}
#Override
public void onCommit(Map<TopicPartition, OffsetAndMetadata> offsets) {
}
#Override
public void close() {
}
}
Add a RecordInterceptor to the listener container (or factory that creates it).
Background:
I am working on a java Spring REST microservice that needs to work with multiple identical back-end systems and multiple identical databases depending on the request parameters.
Basically I have 3 "brands". For each brand there is a set of downstream services and a database. I have no control over those.
My spring service will receive brand as a part of request and will need to call the right downstream services and use the correct database.
Previously I would deal with this by having a separate instance of the spring service for each of the brands. There would be a single property file for each brand and spring would use it to wire up beans. I would have separate URL's for each brand and there was no problem.
Some of my beans need to know about "brand" during creation as they are wrappers around connections downstream services. I.e. once the bean is created there won't be a way to switch it to be a "different brand".
Problem:
I would like to change this so that a single instance of my service can handle requests for any brand.
Requirements:
I was thinking about the following solution:
Have a general property file for non-branded stuff. Spring would wire any non-branded beans and keep them as singleton beans.
Have a property file with brand specific urls etc for each of the brands
Spring would create set of singleton beans for each of the brand using appropriate property file.
Next when the request comes in spring would read the request params and use bean specific for that brand.
Performance is important to me so I would like to reuse the beans as much as possible.
I would like to make this thing as transparent as possible so that people creating new beans don't have to worry about doing anything outside standard configuration/context class.
Does anyone know what would be the best solution to achieve this?
I think you can solve the problem injecting the service in every request with the right set of configurations and beans; possibly already existing in your Application Context.
Given:
$ curl http://localhost:8080/greetings/rodo && echo
Hi from brand1, rodo
$ curl -H "x-brand-name: brand1" http://localhost:8080/greetings/rodo
Hi from brand1, rodo
$ curl -H "x-brand-name: brand2" http://localhost:8080/greetings/rodo && echo
Hi from brand2, rodo
The following code would work:
-- application.yml --
brand1:
greetingPrefix: Hi from brand1,
brand2:
greetingPrefix: Hi from brand2,
-- DemoApplication.java --
#SpringBootApplication
public class DemoApplication {
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
#Configuration
class ServiceConfig {
#Bean
public GreetingService greetingServiceBrand1(Brand1Config config) {
return new GreetingService(config);
}
#Bean
public GreetingService greetingServiceBrand2(Brand2Config config) {
return new GreetingService(config);
}
}
#Configuration
class WebConfig implements WebMvcConfigurer {
#Autowired
private ApplicationContext applicationContext;
#Override
public void addArgumentResolvers(List<HandlerMethodArgumentResolver> resolvers) {
resolvers.add(greetingServiceResolver());
}
private GreetingServiceResolver greetingServiceResolver() {
GreetingService greetingServiceBrand1 = applicationContext.getBean("greetingServiceBrand1", GreetingService.class);
GreetingService greetingServiceBrand2 = applicationContext.getBean("greetingServiceBrand2", GreetingService.class);
return new GreetingServiceResolver(greetingServiceBrand1, greetingServiceBrand2);
}
}
}
#RestController
#RequestMapping("/greetings")
class GreetingController {
#GetMapping("/{name}")
public String get(GreetingService greetingService, #PathVariable String name) {
return greetingService.sayHi(name);
}
}
class GreetingServiceResolver implements HandlerMethodArgumentResolver {
private final GreetingService greetingServiceBrand1;
private final GreetingService greetingServiceBrand2;
public GreetingServiceResolver(GreetingService greetingServiceBrand1, GreetingService greetingServiceBrand2) {
this.greetingServiceBrand1 = greetingServiceBrand1;
this.greetingServiceBrand2 = greetingServiceBrand2;
}
#Override
public boolean supportsParameter(MethodParameter parameter) {
return parameter.getParameterType().equals(GreetingService.class);
}
#Override
public Object resolveArgument(
MethodParameter methodParameter,
ModelAndViewContainer modelAndViewContainer,
NativeWebRequest nativeWebRequest,
WebDataBinderFactory webDataBinderFactory
) throws Exception {
String brand = nativeWebRequest.getHeader("x-brand-name");
return resolveGreetingService(brand);
}
private GreetingService resolveGreetingService(String brand) {
if ("brand2".equals(brand)) {
return greetingServiceBrand2;
}
return greetingServiceBrand1; // default
}
}
class GreetingService {
private BaseConfig config;
public GreetingService(BaseConfig config) {
this.config = config;
}
public String sayHi(String name) {
return config.getGreetingPrefix() + " " + name;
}
}
abstract class BaseConfig {
private String greetingPrefix;
public String getGreetingPrefix() {
return greetingPrefix;
}
public void setGreetingPrefix(String greetingPrefix) {
this.greetingPrefix = greetingPrefix;
}
}
#Configuration
#ConfigurationProperties("brand1")
class Brand1Config extends BaseConfig {
}
#Configuration
#ConfigurationProperties("brand2")
class Brand2Config extends BaseConfig {
}
As you can see, it's fundamental to pass the service to each controller method, write a resolver and inject the right set of dependencies depending on a parameter passed to the request, in this case via header.
Since your property files need to be declared statically anyway, you can just write all your different brand stuff in the same property file, like in a key-value format, that Spring can pick up as a list of configurations.
brandConfigs:
- brand: foo
property: foos
- brand2: bar
porperty: bars
Load all your connection beans to your downstream services on startup and just route to them according to your request param. Imo this seems to be the most straight forward and performant way. If some of these downstreams are used very rarely you can lazy load the beans on-demand, but probably this wouldn't make a sense unless you have thousands of different downstream routes.
I have the following issue that I have to solve in my business. I am using Spring for project development I have 8 dataSource to connect.
A request will be made informing the contract number, through this contact number I will have to select one of the 8 dataSource and make the client consultation.
Example:
I have the base Brazil = 1,Spain = 2 and Germany = 3
If the contract was id = 1, then you should get the customer data from Brazil base.
If the contract was id = 2, then you should get the customer data
from Spain base.
If the contract was id = 3, then you should fetch customer data from the Germany base.
I don't know how to solve this problem if I use multitenancy or AbstractRouting. And I don't know how to start the code for this.
Would anyone have any solution and an example?
Spring has way to determine datasource dynamically using AbstractRoutingDataSource. But need to use ThreadLocal to bind context to current thread.
This may complicate things if you are spawning multiple threads or using async in your app.
You can refer simple exmaple here
If you really need to create database connection as per different clients then hibernate provides a way to manage Multi-tenancy, Hibernate Guide. As explained general approach would be to use connection pool per-tenant or single connection pool per all tenants sharing same database but different schema.
Below are the two implementation used to switch to multiple tenants -
public class TestDataSourceBasedMultiTenantConnectionProviderImpl
extends AbstractDataSourceBasedMultiTenantConnectionProviderImpl {
private static final long serialVersionUID = 14535345L;
#Autowired
private DataSource defaultDataSource;
#Autowired
private TestDataSourceLookup dataSourceLookup;
/**
*
* Select datasources in situations where not tenantId is used (e.g. startup
* processing).
*
*/
#Override
protected DataSource selectAnyDataSource() {
//logger.trace("Select any dataSource: " + defaultDataSource);
return defaultDataSource;
}
/**
*
* Obtains a DataSource based on tenantId
*
*/
#Override
protected DataSource selectDataSource(String tenantIdentifier) {
DataSource ds = dataSourceLookup.getDataSource(tenantIdentifier);
// logger.trace("Select dataSource from " + tenantIdentifier + ": " + ds);
return ds;
}
import org.hibernate.context.spi.CurrentTenantIdentifierResolver;
import org.springframework.beans.factory.annotation.Autowired;
public class TestCurrentTenantIdentifierResolverImpl implements CurrentTenantIdentifierResolver {
#Autowired
private RequestContext context;
#Override
public String resolveCurrentTenantIdentifier() {
// TODO Auto-generated method stub
return context.getTenantID();
}
#Override
public boolean validateExistingCurrentSessions() {
// TODO Auto-generated method stub
return true;
}
}
AbstractRouting would be a way to do it, indeed. You can mix that approach with headers on the request to easily implement multi-tenancy.
The link provided in another post on Baeldung does indeed provide a solution using such construct.
Sharding and deploying a separate micro-service for each tenant, with a routing service in front, would be another one and I think it offers several benefits both from coding (no need for thread-locals or other similarly difficult construct) and maintenance perspective (you can take down your service for maintenance/customisation at tenant level), so it would be my solution of choice in your specific case, assuming that business requirements allow for it.
Still, again, the best solution depends on your specific case.
The easiest solution matching the title of the question - for readers that are not necessarily concerned with multi-tenancy - would be to create several dataSource instances in your Spring's #Configuration beans, put them in a Map<Integer, DataSource> and then return that Map<Integer, DataSource> object as a Bean. When you have to access them, you access the Map bean you created and then create your NamedParameterJdbcTemplate or similar resource passing the specific DataSource in the constructor.
Pseudo-code example:
#Configuration
class DataSourceConfig {
public final static int SPAIN = 2;
public final static int BRAZIL = 1;
#Bean
#Qualifier("dataSources")
public Map<Integer, DataSource> dataSources() {
Map<Integer, DataSource> ds = new HashMap<>();
ds.put(SPAIN, buildSpainDataSource());
ds.put(BRAZIL, buildBrazilDataSource());
return ds;
}
private DataSource buildSpainDataSource() {
...
}
private DataSource buildBrazilDataSource() {
...
}
}
#Service
class MyService {
#Autowired
#Qualifier("dataSources")
Map<Integer, DataSource> dataSources;
Map<String, Object> getObjectForCountry(int country) {
NamedParameterJdbcTemplate t = new NamedParameterJdbcTemplate(dataSources.get(country));
return t.queryForMap("select value from mytable", new HashMap<>());
}
}
I have started converting an existing Spring Boot(1.5.4.RELEASE) application to support multi-tenant capabilities. So i am using MySQL as the database and Spring Data JPA as the data access mechanism. i am using the schema based multi-tenant approach. As Hibernate document suggests below
https://docs.jboss.org/hibernate/orm/4.2/devguide/en-US/html/ch16.html
I have implemented MultiTenantConnectionProvider and CurrentTenantIdentifierResolver interfaces and I am using ThreadLocal variable to maintain the current tenant for the incoming request.
public class TenantContext {
final public static String DEFAULT_TENANT = "master";
private static ThreadLocal<Tenant> tenantConfig = new ThreadLocal<Tenant>() {
#Override
protected Tenant initialValue() {
Tenant tenant = new Tenant();
tenant.setSchemaName(DEFAULT_TENANT);
return tenant;
}
};
public static Tenant getTenant() {
return tenantConfig.get();
}
public static void setTenant(Tenant tenant) {
tenantConfig.set(tenant);
}
public static String getTenantSchema() {
return tenantConfig.get().getSchemaName();
}
public static void clear() {
tenantConfig.remove();
}
}
Then i have implemented a filter and there i set the tenant dynamically looking at a request header as below
String targetTenantName = request.getHeader(TENANT_HTTP_HEADER);
Tenant tenant = new Tenant();
tenant.setSchemaName(targetTenantName);
TenantContext.setTenant(tenant);
This works fine and now my application points to different schema based on the request header value.
However there is a master schema where i store the some global settings and i need to access that schema while in a middle of a request for a tenant. Therefore i tried to hard code the Threadlocal variable just before that database call in the code as below.
Tenant tenant = new Tenant();
tenant.setSchemaName("master");
TenantContext.setTenant(tenant);
However this does not point to the master schema and instead it tries to access the original schema set during the filter. What is the reason for this?
As per my understanding Hibernate invokes openSession() during the first database call to a tenant and after i try to invoke another database call for "master" it still use the previous tenant as CurrentTenantIdentifierResolver invokes only during the openSession(). However these different database calls does not invoke within a transaction.
Can you please help me to understand the issue with my approach and any suggestions to fix the issue
Thanks
Keth
#JonathanJohx actually i am trying to override the TenantContext set by the filter in one of the controllers. First i am loging in a tenant where TenantContext is set to that particular tenant. While the request is in that tenant i am requesting data from master. In order to do that i am simply hard code the tenant as below
#RestController
#RequestMapping("/jobTemplates")
public class JobTemplateController {
#Autowired
JobTemplateService jobTemplateService;
#GetMapping
public JobTemplateList list(Pageable pageable){
Tenant tenant = new Tenant();
tenant.setSchemaName(multitenantMasterDb);
TenantContext.setTenant(tenant);
return jobTemplateService.list(pageable);
}