Spring Redis: Parsing Object from Redis Stream throws ConversionFailedException - java

I'm trying to parse objects from a Redis stream via Spring Boot Reactive Redis that are added by an external service. I'm using the following the tutorial to retrieve the elements from the stream via a StreamListener<String, ObjectRecord<String, TestDTO>>.
The object in the Redis stream consists of an id, a number and a Protobuf byte array (which is produced from a Python service via SerializeToString())
The Redis data retrieved via the redis-cli looks like this:
1) "1234567891011-0"
2) 1) "id"
2) "f63c2bcd...."
3) "number"
4) "5"
5) "raw_data"
6) "\b\x01\x12...
I've created the following DTO to match the objects in the Redis stream:
#Data
#NoArgsConstructor
public class TestDTO {
private UUID id;
private long number;
private byte[] raw_data;
}
However this throws the following error:
org.springframework.core.convert.ConversionFailedException: Failed to convert from type [org.springframework.data.redis.connection.stream.StreamRecords$ByteMapBackedRecord] to type [com.test.test.TestDTO] for value 'MapBackedRecord{recordId=1647417370847-0, kvMap={[B#2beed3c=[B#523baefb, [B#76cea664=[B#62358d82, [B#7ad95089=[B#35d4c48e}}'; nested exception is java.lang.IllegalArgumentException: Value must not be null!
Reading it a as generic MapRecord<String, String, String> works without any problem, but converting it directly to an Object would make for cleaner code. I have the feeling that I need to specify a deserializer, but I haven't found out yet, how to do that. Any recommendations on how to tackle this issue would be more than welcome!

You need to specify a RedisTemplate bean, where you can specify the Key/Value serialization/deserialization. In your case probably you should use GenericJackson2JsonRedisSerializer.
Example using StringRedisSerializer:
#Bean
public RedisTemplate<String, String> redisTemplate(RedisConnectionFactory connectionFactory) {
RedisTemplate<String, String> template = new RedisTemplate<>();
template.setConnectionFactory(connectionFactory);
template.setKeySerializer(new StringRedisSerializer());
template.setValueSerializer(new StringRedisSerializer());
template.afterPropertiesSet();
return template;
}
RedisTemplate javadoc: https://docs.spring.io/spring-data/redis/docs/current/api/org/springframework/data/redis/core/RedisTemplate.html
Spring Data Redis documentation: https://docs.spring.io/spring-data/redis/docs/current/reference/html/
Available serializers: https://docs.spring.io/spring-data/redis/docs/current/reference/html/#redis:serializer
This question can also help you: RedisTemplate hashvalue serializer to use for nested object with multiple types

It seems:
StreamListener<String, ObjectRecord<String, SomeDTO>>
is not available in spring-boot 3.
In my case I was using spring-boot version 3.0.0 and even upgraded to 3.0.1.
I noticed this same issue in these spring-boot versions. I fixed it by adding a spring-data-redis dependency and specifying the version as any of the older 2.x.x versions
Previous POM.xml
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
New POM.xml
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-redis</artifactId>
<version>2.7.7</version>
</dependency>

Related

How to use Redis Cache to store the data in java spring boot application?

I have already a running instance of Redis Cache in AWS Account.
How can I use the redis instance using the redis instance Endpoint in my java code to store the data.
I don't have any idea how to start with Redis Cache in java.
Please help me out to resolve this.
You can use spring-data-redis by including following dependency.
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
<version>2.2.6.RELEASE</version>
</dependency>
Then specify properties as below:
spring.redis.database=0
spring.redis.host="Specify URL"
spring.redis.port=6379
spring.redis.password=mypass
spring.redis.timeout=60000
Then using RedisTemplate
#Autowired
private RedisTemplate<Long, Book> redisTemplate;
public void save(Book book) {
redisTemplate.opsForValue().set(book.getId(), book);
}
public Book findById(Long id) {
return redisTemplate.opsForValue().get(id);
}

Springfox-boot-starter swagger Instant handling

I have a problem with swagger documentation using SpringBoot with Springfox-boot-starter.
I use java.time.Instant wrapped in java.util.Optional in my REST API which works fine:
#GetMapping("/{subscriptionId}/{variableAlias}")
public PaginatedResultDTO<MonitoredVariableDTO> getReportedVariables(
#PathVariable String subscriptionId,
#PathVariable String variableAlias,
Optional<Instant> from,
Optional<Instant> to) { ... }
But for some reason, Swagger documentation cannot handle the Optional type correctly and seems to handle it through reflection as EpochSeconds and Nano attributes instead of one field:
I would like to make swagger expect from and to instants in ISO format, just like Spring does and how I use it in Insomnia:
When I tried to remove the Optional wrapper, it seems to work
Is there a way to make this work with the Optional? Thanks for any advice!
Spring boot version:
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.3.4.RELEASE</version>
<relativePath />
</parent>
Springfox-boot-starter version
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-boot-starter</artifactId>
<version>3.0.0</version>
</dependency>
We had exactly the same problem that you.
We solved it with this SpringFox configuration:
#Configuration
#EnableSwagger2
public class SpringfoxConfiguration {
#Value("${api-doc.version}")
private String apiInfoVersion;
#Autowired
private TypeResolver typeResolver;
#Bean
public Docket customDocket(){
return new Docket(DocumentationType.SWAGGER_2)
.groupName("xxx")
//Some other code unrelated to this problem
.alternateTypeRules(
// Rule to correctly process Optional<Instant> variables
// and generate "type: string, format: date-time", as for Instant variables,
// instead of "$ref" : "#/definitions/Instant"
AlternateTypeRules.newRule(
typeResolver.resolve(Optional.class, Instant.class),
typeResolver.resolve(Date.class),
Ordered.HIGHEST_PRECEDENCE
))
.genericModelSubstitutes(Optional.class)
.select()
//Some more code unrelated to this problem
.build();
}
}
With spring fox the problem is it doesn't use the custom ObjectMapper which you have defined as a Bean.
Springfox creates own ObjectMapper using new keyword. Hence, any module you register with your custom ObjectMapper is pointless for SpringFox. However, Springfox provides an interface to register modules with it's own ObjectMapper.
Create a configuration bean like below in your project and it should work.
#Configuration
public class ObjectMapperModuleRegistrar implements JacksonModuleRegistrar {
#Override
public void maybeRegisterModule(ObjectMapper objectMapper) {
objectMapper.registerModule(new ParameterNamesModule())
.registerModule(new Jdk8Module())
.registerModule(new JavaTimeModule())
.findAndRegisterModules();
}
}

Is there any option to register Serializer/Deserializer only once for java.time.* packages using Jackson in Spring Boot?

Hello Team,
I am working on a Spring Boot Project (Version 2.3.4 Release) in which I am using java.time.LocalDate, java.time.LocalDateTime and java.time.LocalTime datatypes for some of the properties in several beans.
However, these fields are not getting deserialized automatically unless I am explicitly providing following annotations to them along with my custom deserializer.
#JsonFormat(pattern="dd-MMM-yyyy")
#JsonDeserialize(using= LocalDateWithStringsDeserializer.class)
private LocalDate date_of_joining;
It is really a time consuming job to add these annotations to all the beans as there are more than 400-500 beans in this project.
If these annotations were not provided, I get below error.
com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `java.time.LocalDate` (no Creators, like default construct, exist): no String-argument constructor/factory method to deserialize from String value ('27-May-1999')
I have also tried defining a explicit Bean for overriding Spring Boot objectmapper with my custom deserializers but that even didn't helped me. The code snippet of the same is mentioned below.
#Bean
#Primary
public ObjectMapper objectMapper(Jackson2ObjectMapperBuilder builder) {
ObjectMapper objectMapper = builder.build();
objectMapper.findAndRegisterModules();
objectMapper.configure(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS, false);
objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
objectMapper.setDateFormat(dateformat);
SimpleModule module = new SimpleModule();
module.addDeserializer(LocalDateTime.class, new MillisOrLocalDateTimeDeserializer());
module.addDeserializer(LocalDate.class, new LocalDateWithStringsDeserializer());
module.addDeserializer(LocalTime.class, new LocalTimeWithStringDeserializer());
objectMapper.registerModule(module);
return objectMapper;
}
I have even added following dependencies to pom.xml file in my project.
<dependency>
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-jsr310</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
<artifactId>jackson-module-parameter-names</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-jdk8</artifactId>
</dependency>
<dependency>
<groupId>javax.xml.bind</groupId>
<artifactId>jaxb-api</artifactId>
</dependency>
Please suggest a solution.
So I figured out the solution. There were some other object mappers previously configured which were using Joda Time in a project which was added to the build path of my project as a required library.
Modifying the existing object mappers and registering Java Time module and custom deserializers for Date & Time with them in that project like I have mentioned in this question resolved this issue.

Kafka Avro serialize/deserialize into concrete type using schema registry failing

I can't seem to be able to consume messages in their concrete avro implementation, I get the following exception:
class org.apache.avro.generic.GenericData$Record cannot be cast to class my.package.MyConcreteClass
Here is the code (I use Spring Boot)
MyProducer.java
private final KafkaTemplate<String, MyConcreteClass> kafkaTemplate;
public PositionProducer(KafkaTemplate<String, MyConcreteClass> kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
public void sendMessage(MyConcreteClass myConcreteClass) {
this.kafkaTemplate.send(topic, myConcreteClass);
}
MyConsumer.java
#KafkaListener(topics = "#{'${consumer.topic.name}'}", groupId = "#{'${spring.kafka.consumer.group-id}'}")
public void listen(MyConcreteClass incomingMsg) {
//handle
}
Note that if I change everything to GenericRecord, the deserialization works properly, so I know all config (not pasted) is configured correctly.
Also maybe important to note that I didn't register the schema myself, and instead let my client code do it for me.
Any ideas?
EDIT:
Config:
#Bean
public ConsumerFactory<String, MyConcreteClass> consumerFactory() {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
props.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "io.confluent.kafka.serializers.KafkaAvroDeserializer");
props.put(KafkaAvroDeserializerConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl);
return new DefaultKafkaConsumerFactory<>(props);
}
MyConcreteClass needs to extend SpecificRecord
You can use the Avro maven plugin to generate it from a schema
Then you must configure the serializer to know you want to use specific records
props.put(KafkaAvroDeserializerConfig.SPECIFIC_AVRO_READER_CONFIG, "true") ;
In addition to OneCricketeer's answer, I encountered another java.lang.ClassCastException after setting the specific avro reader config. It was nested exception is java.lang.ClassCastException: class my.package.Envelope cannot be cast to class my.package.Envelope (my.package.Envelope is in unnamed module of loader 'app'; my.package.Envelope is in unnamed module of loader org.springframework.boot.devtools.restart.classloader.RestartClassLoader #3be312bd); It seems like spring boot devtools wrapped the class in it's reloader module causing jvm thought that's a different class.
I removed the spring boot devtools in pom and it finally worked as expected now.
<!-- Remove this from pom.xml -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<scope>runtime</scope>
<optional>true</optional>
</dependency>
I was getting blocked by the same issue as you. Thing is that the KafkaAvroDeserializer was deserializing the message as GenericData$Record so then spring kafka is searching the class annotated with #KafkaListener to have #KafkaHandler methods with a parameter of this type.
You'll need to add this property to your spring kafka configuration so the deserializer can return directly the SpecificRecord classes that you previously need to generate with the avro plugin:
spring:
kafka:
properties:
specific.avro.reader: true
Then your consumer may be like this
#KafkaListener(...)
public void consumeCreation(MyAvroGeneratedClass specificRecord) {
log.info("Consuming record: {}", specificRecord);
}
You need to customize your consumer configuration.
The ContentDeserializer needs to be an KafkaAvroDeserializer with a reference to your schema registry.

java.lang.IllegalArgumentException: No converter found for return value of type

With this code
#RequestMapping(value = "/bar/foo", method = RequestMethod.GET)
public ResponseEntity<foo> foo() {
Foo model;
...
return ResponseEntity.ok(model);
}
}
I get the following exception
java.lang.IllegalArgumentException: No converter found for return value of type
My guess is that the object cannot be converted to JSON because Jackson is missing. I don't understand why because I thought that Jackson was built in with spring boot.
Then I have tried to add Jackson to the pom.xml but I still have the same error
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.4.3</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.4.3</version>
</dependency>
Do I have to change any spring boot properties to make this work?
The problem was that one of the nested objects in Foo didn't have any getter/setter
Add the below dependency to your pom.xml:
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.10.0.pr3</version>
</dependency>
Add the getter/setter missing inside the bean mentioned in the error message.
Use #ResponseBody and getter/setter. Hope it will solve your issue.
#RequestMapping(value = "/bar/foo", method = RequestMethod.GET)
#ResponseBody
public ResponseEntity<foo> foo() {
and update your mvc-dispatcher-servlet.xml:
<mvc:annotation-driven>
<mvc:message-converters>
<bean class="org.springframework.http.converter.StringHttpMessageConverter"/>
<bean class="org.springframework.http.converter.json.MappingJackson2HttpMessageConverter"/>
</mvc:message-converters>
</mvc:annotation-driven>
The answer written by #Marc is also valid. But the concrete answer is the Getter method is required. You don't even need a Setter.
The issue occurred in my case because spring framework couldn't fetch the properties of nested objects. Getters/Setters is one way of solving. Making the properties public is another quick and dirty solution to validate if this is indeed the problem.
#EnableWebMvc annotation on config class resolved my problem. (Spring 5, no web.xml, initialized by AbstractAnnotationConfigDispatcherServletInitializer)
I had the very same problem, and unfortunately it could not be solved by adding getter methods, or adding jackson dependencies.
I then looked at Official Spring Guide, and followed their example as given here - https://spring.io/guides/gs/actuator-service/ - where the example also shows the conversion of returned object to JSON format.
I then again made my own project, with the difference that this time I also added the dependencies and build plugins that's present in the pom.xml file of the Official Spring Guide example I mentioned above.
The modified dependencies and build part of XML file looks like this!
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
You can see the same in the mentioned link above.
And magically, atleast for me, it works. So, if you have already exhausted your other options, you might want to try this out, as was the case with me.
Just a side note, it didn't work for me when I added the dependencies in my previous project and did Maven install and update project stuff. So, I had to again make my project from scratch. I didn't bother much about it as mine is an example project, but you might want to look for that too!
I was getting the same error for a while.I had verify getter methods were available for all properties.Still was getting the same error.
To resolve an issue Configure MVC xml(configuration) with
<mvc:annotation-driven/>
.This is required for Spring to detect the presence of jackson and setup the corresponding converters.
While using Spring Boot 2.2 I run into a similiar error message and while googling my error message
No converter for [class java.util.ArrayList] with preset Content-Type 'null'
this question here is on top, but all answers here did not work for me, so I think it's a good idea to add the answer I found myself:
I had to add the following dependencies to the pom.xml:
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-oxm</artifactId>
</dependency>
<dependency>
<groupId>com.thoughtworks.xstream</groupId>
<artifactId>xstream</artifactId>
<version>1.4.11.1</version>
</dependency>
After this I need to add the following to the WebApplication class:
#SpringBootApplication
public class WebApplication
{
// ...
#Bean
public HttpMessageConverter<Object> createXmlHttpMessageConverter()
{
final MarshallingHttpMessageConverter xmlConverter = new MarshallingHttpMessageConverter();
final XStreamMarshaller xstreamMarshaller = new XStreamMarshaller();
xstreamMarshaller.setAutodetectAnnotations(true);
xmlConverter.setMarshaller(xstreamMarshaller);
xmlConverter.setUnmarshaller(xstreamMarshaller);
return xmlConverter;
}
}
Last but not least within my #Controller I used:
#GetMapping(produces = {MediaType.APPLICATION_XML_VALUE, MediaType. APPLICATION_JSON_VALUE})
#ResponseBody
public List<MeterTypeEntity> listXmlJson(final Model model)
{
return this.service.list();
}
So now I got JSON and XML return values depending on the requests Accept header.
To make the XML output more readable (remove the complete package name from tag names) you could also add #XStreamAlias the following to your entity class:
#Table("ExampleTypes")
#XStreamAlias("ExampleType")
public class ExampleTypeEntity
{
// ...
}
Hopefully this will help others with the same problem.
In my case i'm using spring boot , and i have encountered a similar error :
No converter for [class java.util.ArrayList] with preset Content-Type 'null'
turns out that i have a controller with
#GetMapping(produces = { "application/xml", "application/json" })
and shamefully i wasn't adding the Accept header to my requests
you didn't have any getter/setter methods.
In my case, I was returning Boolean in Response Entity
and had :
produces = MediaType.TEXT_PLAIN_VALUE,
When i changed it to below
produces = MediaType.APPLICATION_JSON_VALUE
It worked!
Example of what i had.
#PostMapping(value = "/xxx-xxxx",
produces = MediaType.APPLICATION_JSON_VALUE,
consumes = MediaType.APPLICATION_JSON_VALUE)
public ResponseEntity<Boolean> yyyy(
I was facing same issue for long time then comes to know have to convert object into JSON using Object Mapper and pass it as JSON Object
#RequestMapping(value = "/getTags", method = RequestMethod.GET)
public #ResponseBody String getTags(#RequestParam String tagName) throws
JsonGenerationException, JsonMappingException, IOException {
List<Tag> result = new ArrayList<Tag>();
for (Tag tag : data) {
if (tag.getTagName().contains(tagName)) {
result.add(tag);
}
}
ObjectMapper objectMapper = new ObjectMapper();
String json = objectMapper.writeValueAsString(result);
return json;
}
I also experienced such error when by accident put two #JsonProperty("some_value") identical lines on different properties inside the class
In my case, I forgot to add library jackson-core.jar, I only added jackson-annotations.jar and jackson-databind.jar. When I added jackson-core.jar, it fixed the problem.
I saw the same error when the scope of the jackson-databind dependency had been set to test:
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.9.9</version>
<scope>test</scope>
</dependency>
Removing the <scope> line fixed the issue.
Faced same error recently - the pojo had getters/setters and all jackson dependencies were imported in pom correctly but some how "< scope > " was "provided" for jackson dependency and this caused the issue. Removing " < Scope > " from jackson dependency fixed the issue
I faced the same problem but I was using Lombok and my UploadFileResponse pojo was a builder.
public ResponseEntity<UploadFileResponse>
To solve I added #Getter annotation:
#Builder
#NoArgsConstructor
#AllArgsConstructor
#Getter
public class UploadFileResponse
Add below dependency in pom.xml:
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.10.1</version>
</dependency>
Was facing the same issue as the return type cannot be bind with the MediaType of Class Foo. After adding the dependency it worked.
This might also happen due low Jackson version; e.g. Spring Boot 2.4 default Jackson version is too low when using Java records; you need at least 2.5 to serialize them properly.
I also encountered the same error on a Spring 5 project (not Spring Boot), by running a SpringMVC JUnit test-case on a method that returns ResponseEntity<List<MyPojo>>
Error: No converter found for return value of type: class java.util.ArrayList
I thought I had all the correct Jackson artifacts in my pom, but later realized that I had the legacy versions. The Maven groupId changed on the Jackson jars from org.codehaus.jacksonto com.fasterxml.jackson.core. After switching to the new jars the error went away.
Updated maven pom.xml:
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.9.7</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.9.7</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.9.7</version>
</dependency>
You are missing an Annotation #ResponseBody

Categories