I have made a custom Partitioner class that extends default Partitioner.
Problem : I want to add this custom Partitioner in KafkaSender.send method()
KafkaSender.send method() code :
sender.send(Flux.just(SenderRecord.create(new ProducerRecord<>(topic, partition, key, record, recordHeaders), 1))))
The partitioner here is an integer
Custom Partitioner Code:
public class CustomPartitioner extends DefaultPartitioner {
private final static String CHAR_FORMAT = "UTF-8";
#Override
public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
// my logic
try {
return super.partition(topic, key, iocKey.toString().getBytes(CHAR_FORMAT), value, valueBytes, cluster);
} catch (UnsupportedEncodingException e) {
//error message
}
}
}
Note : I tried to hard code it using this below code
Properties properties = new Properties();
properties.put("partitioner.class", "CustomPartitioner ");
How can we force KafkaSender.send method() to use our custom partitioner?
You have to pass the properties map to KafkaTemplate bean as part of your producer configuration.
#Bean
public KafkaTemplate<String, String> kafkaTemplate() {
Map<String, Object> configProps = new HashMap<>();
configProps.put("partitioner.class", "<packagename>.CustomPartitioner");
return new KafkaTemplate<>(configProps );
}
Related
I'm trying to develop two producer configuration classes with different Serialization types(JSONNode, Avro) but, at runtime I'm able to instantiate only one , the other one is not working.
First class :
#Configuration
#EnableKafka
public class KafkaProducerConfig extends SomeClassConfig{
#Autowired
private SomeClassProps someClassProps ;
#Bean
public ProducerFactory<JsonNode, JsonNode> eventProducerFactory() throws UnknownHostException{
return new DefaultKafkaProducerFactory<JsonNode, JsonNode>(producerConfigs(someClassProps ));
}
#Bean
public Map<String, Object> producerConfigs(SomeClassProps someClassProps ) throws UnknownHostException{
Properties props = this.initProps(someClassProps );
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, ***.getBootstrapServers());
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.connect.json.JsonSerializer");
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,"org.apache.kafka.connect.json.JsonSerializer");
Map<String, Object> map = new HashMap<>();
for (final String name: props.stringPropertyNames()) {
map.put(name, props.getProperty(name));
}
return map;
}
#Bean(name="eventProducerKafkaTemplate")
public KafkaTemplate<JsonNode,JsonNode> eventProducerKafkaTemplate() throws UnknownHostException{
return new KafkaTemplate<JsonNode,JsonNode>(eventProducerFactory());
}
}
Second class:
#Configuration("avroKafkaProducerConfig")
#EnableKafka
public class AvroKafkaProducerConfig extends SomeClassConfig{
//private static final org.slf4j.Logger LOG = LoggerFactory.getLogger(KafkaProducerConfig.class);
#Autowired
private SomeClassProps someClassProps ;
#Bean(name = KafkaStreamsDefaultConfiguration.DEFAULT_STREAMS_CONFIG_BEAN_NAME)
public StreamsConfig kafkaStreamsConfig(SomeClassProps someClassProps ) throws UnknownHostException {
Map<String, Object> props = producerConfigs(someClassProps );
return new StreamsConfig(props);
}
#Bean
public ProducerFactory<SpecificRecord, SpecificRecord> eventProducerFactory() throws UnknownHostException{
return new DefaultKafkaProducerFactory<SpecificRecord, SpecificRecord>(producerConfigs(someClassProps ));
}
#Bean
public Map<String, Object> producerConfigs(SomeClassProps someClassProps ) throws UnknownHostException{
Properties props = this.initProps(someClassProps );
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, ***.getBootstrapServers());
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, io.confluent.kafka.serializers.KafkaAvroSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,io.confluent.kafka.serializers.KafkaAvroSerializer.class);
Map<String, Object> map = (Map) props;
System.out.println("Avro"+map.values());
return map;
}
#Bean(name="eventAvroProducerKafkaTemplate")
public KafkaTemplate<SpecificRecord,SpecificRecord> eventProducerKafkaTemplate() throws UnknownHostException{
return new KafkaTemplate<SpecificRecord,SpecificRecord>(eventProducerFactory());
}
}
when I'm trying to use these kafka templates in my controller class it is only intializing the Json Serializer not the avro one.
THis is how I'm using these kafka templates in other class
#Autowired
#Qualifier("eventProducerKafkaTemplate")
private KafkaTemplate<JsonNode, JsonNode> eventProducerKafkaTemplate;
#Autowired
#Qualifier("eventAvroProducerKafkaTemplate")
private KafkaTemplate<SpecificRecord,SpecificRecord> eventAvroProducerKafkaTemplate;
public ReturnTYpe methodName() {
eventProducerKafkaTemplate.send(****, ****, ****);
eventAvroProducerKafkaTemplate.send(****, ****, ****);
}
You must give the 2 producerConfigs and 2 eventProducerFactory beans different method (bean) names. Otherwise one overrides the other.
Is there a way to bind a java Map<String, Object> to a varchar in the the JDBI #BindBean annotation.
So for example I have a class Something.class and I create a
#SqlBatch("INSERT INTO Something (name, payload) VALUES(:name, :payload)").
Now in my java class the name is of type String and payload is of type Map<String, Object> and I want in the DB table the types are varchar(...). Now I want the Map object to be inserted in the column as a JSON object, is that somehow achievable wihtout creating my own complex Binder as defined in http://jdbi.org/sql_object_api_argument_binding/ ? and other than making my payload be of type String in java.
public class MapArgument implements Argument {
private Map<String, Object> payload;
private ObjectMapper objectMapper;
public MapArgument(ObjectMapper objectMapper, Map<String, Object> payload) {
this.objectMapper = objectMapper;
this.payload = payload;
}
#Override
public void apply(int i, PreparedStatement statement, StatementContext statementContext) throws SQLException {
try {
statement.setString(i, objectMapper.writeValueAsString(payload));
} catch (JsonProcessingException e) {
log.info("Failed to serialize payload ", e);
}
}
}
public class MapArgumentFactory implements ArgumentFactory<Map<String, Object>> {
private ObjectMapper mapper;
public MapArgumentFactory(ObjectMapper mapper) {
this.mapper = mapper;
}
#Override
public boolean accepts(Class<?> type, Object value, StatementContext statementContext) {
return value instanceof Map;
}
#Override
public Argument build(Class<?> type, Map<String, Object> map, StatementContext statementContext) {
return new MapArgument(mapper, map);
}
}
Fixed my problem with creating an ArgumentFactory binder suggested in this post.
So what I needed was to create a class that just contained one field of type Map<String, Object> implemented the Arugment interface from org.skife.jdbi.v2.tweak so I ended up with the following
public class NotificationPayloadArgument implements Argument {
private NotificationPayload payload;
NotificationPayloadArgument(NotificationPayload payload) {
this.payload = payload;
}
#Override
public void apply(int i, PreparedStatement preparedStatement, StatementContext statementContext)
throws SQLException {
preparedStatement.setString(i, toString());
}
#Override
public String toString() {
return new JSONObject(payload).toString();
}
}
To make this work I of course needed to implement a Factory class which implements the org.skife.jdbi.v2.tweak.ArgumentFactory<T> interface with my newly created type, so the factory ended up as such:
public class NotificationPayloadFactory implements ArgumentFactory<NotificationPayload> {
#Override
public boolean accepts(Class<?> expectedType, Object value, StatementContext ctx) {
return value instanceof NotificationPayload;
}
#Override
public Argument build(Class<?> expectedType, NotificationPayload value, StatementContext ctx) {
return value;
}
}
and of course lastly also as mentioned in Does JDBI accept UUID parameters? I had to register my factory:
jdbi.registerArgumentFactory(new NotificationPayloadFactory());
I tried to do this with just a factory for a Map<String, Object> but could not make it work for that, and there was a risk for NPE.
EDIT
The reason I am Overriding the toString() in NotificationPayload is because I need the payload in json format some places where I need a String object. But else it can be removed and then just use the new JSONObject(payload).toString() in the preparedStatement.setString() where toString() is called.
In kotlin-way using JDBI to insert a data class as JSONB; just create one ArgumentFactory for that datatype and register that factory with JDBI. This is as simple as these 3 lines needed -
internal class CvMetadataArgumentFactory : AbstractArgumentFactory<CvMetadata?>(Types.OTHER) {
override fun build(value: CvMetadata?, config: ConfigRegistry): Argument {
return Argument { position, statement, _ -> statement.setString(position, value?.toString()?:"") }
}
}
Later register this factory -
jdbi.registerArgument(CvMetadataArgumentFactory())
i am playing with Kafka and streams technology; i have created a custom serializer and deserializer for the KStream that i will use to receive messages from a given topic.
Now, the problem is that i am creating a serde in this way:
JsonSerializer<EventMessage> serializer = new JsonSerializer<>();
JsonDeserializer<EventMessage> deserializer = new JsonDeserializer<>(EventMessage.class);
Serde<EventMessage> messageSerde = Serdes.serdeFrom(serializer, deserializer);
Serializer implementation:
public class JsonSerializer<T> implements Serializer<T> {
private Gson gson = new Gson();
public void configure(Map<String, ?> map, boolean b) {
}
#Override
public byte[] serialize(String topic, T data) {
return gson.toJson(data).getBytes(Charset.forName("UTF-8"));
}
#Override
public void close() {
}
}
Deserializer implementation:
public class JsonDeserializer<T> implements Deserializer<T> {
private Gson gson = new Gson();
private Class<T> deserializedClass;
public JsonDeserializer() {
}
public JsonDeserializer(Class<T> deserializedClass) {
this.deserializedClass = deserializedClass;
}
#Override
#SuppressWarnings("unchecked")
public void configure(Map<String, ?> map, boolean b) {
if(deserializedClass == null) {
deserializedClass = (Class<T>) map.get("serializedClass");
}
}
#Override
public T deserialize(String topic, byte[] data) {
System.out.print(data);
if(data == null){
return null;
}
return gson.fromJson(new String(data),deserializedClass);
}
#Override
public void close() {
}
}
When i try to execute the code, i receive the following error:
Caused by: org.apache.kafka.common.KafkaException: Could not instantiate class org.apache.kafka.common.serialization.Serdes$WrapperSerde Does it have a public no-argument constructor?
Full dump here: https://pastebin.com/WwpuXuxB
This is the way i am trying to use serde:
KStreamBuilder builder = new KStreamBuilder();
KStream<String, EventMessage> eventsStream = builder.stream(stringSerde, messageSerde, topic);
KStream<String, EventMessage> outStream = eventsStream
.mapValues(value -> EventMessage.build(value.type, value.timestamp));
outStream.to("output");
Also, i am not totally sure i am setting up correctly the properties to setup up serializer and deserializer globally:
streamsConfiguration.put(StreamsConfig.KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
streamsConfiguration.put(StreamsConfig.VALUE_SERDE_CLASS_CONFIG, messageSerde.getClass());
To complete the Matthias answer I've just coded a simple example of how to create a custom Serde (Serializer / Deserializer) within a Kafka Stream App. It's is available to clone and try in: https://github.com/Davidcorral94/Kafka-Streams-Custom-Seder
First I create two classes, one for the Serializer and another for the Deserializer. In this case I use Gson library to perform the serialization/deserialization.
Serializer
public class PersonSerializer implements Closeable, AutoCloseable, Serializer<Person> {
private static final Charset CHARSET = Charset.forName("UTF-8");
static private Gson gson = new Gson();
#Override
public void configure(Map<String, ?> map, boolean b) {
}
#Override
public byte[] serialize(String s, Person person) {
// Transform the Person object to String
String line = gson.toJson(person);
// Return the bytes from the String 'line'
return line.getBytes(CHARSET);
}
#Override
public void close() {
}
}
Deserializer
public class PersonDeserializer implements Closeable, AutoCloseable, Deserializer<Person> {
private static final Charset CHARSET = Charset.forName("UTF-8");
static private Gson gson = new Gson();
#Override
public void configure(Map<String, ?> map, boolean b) {
}
#Override
public Person deserialize(String topic, byte[] bytes) {
try {
// Transform the bytes to String
String person = new String(bytes, CHARSET);
// Return the Person object created from the String 'person'
return gson.fromJson(person, Person.class);
} catch (Exception e) {
throw new IllegalArgumentException("Error reading bytes", e);
}
}
#Override
public void close() {
}
}
Then, I wrap both of them into a Serde to be able to use it into my Kafka Stream App.
Serde
public class PersonSerde implements Serde<Person> {
private PersonSerializer serializer = new PersonSerializer();
private PersonDeserializer deserializer = new PersonDeserializer();
#Override
public void configure(Map<String, ?> configs, boolean isKey) {
serializer.configure(configs, isKey);
deserializer.configure(configs, isKey);
}
#Override
public void close() {
serializer.close();
deserializer.close();
}
#Override
public Serializer<Person> serializer() {
return serializer;
}
#Override
public Deserializer<Person> deserializer() {
return deserializer;
}
}
Finally, you are able to use this Serde class into your Kafka Stream App with the next line:
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, PersonSerde.class);
This is actually working with the latest Kafka version available at this moment which is 1.0.0!
If you call Serdes.serdeFrom(...) you get a WrappedSerde type back that is for internal usage (and WrappedSerde does not have an non-argument constructor). There is currently no API you can call to get a custom Serde. Instead, you need to implement you own Serde class and wrap you serializer and deserializer "manually".
public class EventMessageSerde implements Serde<EventMessage> {
final private JsonSerializer<EventMessage> serializer;
final private JsonDeserializer<EventMessage> deserializer;
#Override
public void configure(Map<String, ?> configs, boolean isKey) {
serializer.configure(configs, isKey);
deserializer.configure(configs, isKey);
}
#Override
public void close() {
serializer.close();
deserializer.close();
}
#Override
public Serializer<EventMessage> serializer() {
return serializer;
}
#Override
public Deserializer<EventMessage> deserializer() {
return deserializer;
}
}
In your Properties you can set:
streamsConfiguration.put(StreamsConfig.VALUE_SERDE_CLASS_CONFIG, EventMessageSerde.class);
Another way is using StreamsBuilder instead of KStreamBuilder. KStreamBuilder is deprecated in 1.0.0. You can directly pass serde object using Consumed.with while creating stream. You need not to create custom Serde class in this scenario.
Serde<EventMessage> messageSerde = Serdes.serdeFrom(serializer, deserializer);
StreamsBuilder builder = new StreamsBuilder();
KStream<String, EventMessage> eventsStream = builder.stream(topic, Consumed.with(Serdes.String(), messageSerde));
You can keep StringSerde in below code instead of using messageSerde.getClass() which is failing because messageSerde is just a WrappedSerde that does not have non-argument constructor.
streamsConfiguration.put(StreamsConfig.VALUE_SERDE_CLASS_CONFIG, StringSerde.class.getName());
I am working on web application, which is deployed on apache tomcat web server. I want to associate data with current thread of http request. For this purpose I have created following class:
public class ThreadData {
private static final Map<Long, Map<String, String>> data = new HashMap<>();
public static Map<String, String> getDataMap(long threadId) {
if (data.get(threadId) == null) {
data.put(threadId, new HashMap<String, String>());
}
return data.get(threadId);
}
public static void put(String key, String value) {
long threadId = Thread.currentThread().getId();
getDataMap(threadId).put(key, value);
}
public static String get(String key) {
long threadId = Thread.currentThread().getId();
return getDataMap(threadId).get(key);
}
}
I want to know if such kind of solution is right and safe for this problem .
What you are trying to achieve is covered by ThreadLocal class, in your case it would be
ThreadLocal<Map<String, String>> data = new ThreadLocal<Map<String, String>>() {
#Override
protected Map<String, String> initialValue() {
return new HashMap<>();
}
};
If you still want to use put or get, you may define them in such a way:
public static void put(String key, String value) {
data.get().put(key, value);
}
public static String get(String key) {
return data.get().get(key);
}
Hi I need to do multiple insertions of the form
SADD key value
I have the key value pair and needed to know how to perform mass insertions using JAVA . I have written a file in the Redis Protocol. How to proceed further
If you have inputs written to Redis protocol format then why don't just use pipe mode of redis-cli or nc? It's explained from http://redis.io/topics/mass-insert.
If you have mass (key, value) inputs then you can use Jedis to perform sadd with pipelining to get higher performance.
Below example assumes that iter (Iterator) has elements each item is key"\t"value form.
try (Jedis jedis = new Jedis(host, port)) {
Pipeline pipeline = jedis.pipelined();
while (iter.hasNext()) {
String[] keyValue = iter.next().split("\t");
pipeline.sadd(keyValue[0], keyValue[1]);
// you can call pipeline.sync() and start new pipeline here if you think there're so much operations in one pipeline
}
pipeline.sync();
}
If you are doing the actual read/write operations through Spring CacheManager with RedisTemplate configured to use Redis as the cache, you can also use the executePipelined method of RedisTemplate which takes a callback as an argument. The callback needs to define the doInRedis method which does the work (read/write operations) in Redis that you want to do in a batch.
Following code shows inserting a List of objects wrapped in a CacheableObject interface that has a getKey() and getValue() by calling redisTemplate.opsForHash().put().
#Component
public class RedisClient {
#Autowired
RedisTemplate redisTemplate;
//batch-insert using Redis pipeline, a list of objects into the cache specified by cacheName
public void put(String cacheName, List<CacheableObject> objects) {
try {
this.redisTemplate.executePipelined(new RedisCallback<Object>() {
#Override
public Object doInRedis(RedisConnection connection) throws DataAccessException {
for(CacheableObject object: objects) {
redisTemplate.opsForHash().put(cacheName, object.getKey(), object.getValue());
}
return null;
}
});
}
catch(Exception e) {
log.error("Error inserting objects into Redis cache: {}", e.getMessage());
}
}
RedisTemplate itself is configured using a configuration class such as the following:
#Configuration
#EnableCaching
public class RedisCacheConfig extends CachingConfigurerSupport implements
CachingConfigurer {
#Value("${redis.hostname}")
private String redisHost;
#Value("${redis.port}")
private int redisPort;
#Value("${redis.timeout.secs:1}")
private int redisTimeoutInSecs;
#Value("${redis.socket.timeout.secs:1}")
private int redisSocketTimeoutInSecs;
#Value("${redis.ttl.hours:1}")
private int redisDataTTL;
#Bean
JedisConnectionFactory jedisConnectionFactory() {
RedisStandaloneConfiguration redisStandaloneConfiguration = new RedisStandaloneConfiguration(redisHost, redisPort);
return new JedisConnectionFactory(redisStandaloneConfiguration);
}
#Bean
public RedisTemplate<Object, Object> redisTemplate() {
RedisTemplate<Object, Object> redisTemplate = new RedisTemplate<Object, Object>();
redisTemplate.setConnectionFactory(jedisConnectionFactory());
return redisTemplate;
}
#Bean
public RedisCacheManager redisCacheManager (JedisConnectionFactory jedisConnectionFactory) {
RedisCacheConfiguration redisCacheConfiguration =
RedisCacheConfiguration.defaultCacheConfig().disableCachingNullValues()
.entryTtl(Duration.ofHours(redisDataTTL)) .serializeValuesWith(RedisSerializationContext.SerializationPair.fromSerializer(RedisSerializer.java()));
redisCacheConfiguration.usePrefix();
RedisCacheManager redisCacheManager =
RedisCacheManager.RedisCacheManagerBuilder.
fromConnectionFactory(jedisConnectionFactory)
.cacheDefaults(redisCacheConfiguration).build();
redisCacheManager.setTransactionAware(true);
return redisCacheManager;
}
#Bean
public JedisPoolConfig poolConfig() {
final JedisPoolConfig jedisPoolConfig = new JedisPoolConfig();
jedisPoolConfig.setTestOnBorrow(true);
jedisPoolConfig.setMaxTotal(100);
jedisPoolConfig.setMaxIdle(100);
jedisPoolConfig.setMinIdle(10);
jedisPoolConfig.setTestOnReturn(true);
jedisPoolConfig.setTestWhileIdle(true);
return jedisPoolConfig;
}
#Override
public CacheErrorHandler errorHandler() {
return new RedisCacheErrorHandler();
}
}