I am using JsonDeserializer to deserialize my custom Object, but in my method annotated with #KafkaListener get the object with Map field as null.
public ConsumerFactory<String, BizWebKafkaTopicMessage> consumerFactory(String groupId) {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
props.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
return new DefaultKafkaConsumerFactory<>(props, new StringDeserializer(), new JsonDeserializer<>(BizWebKafkaTopicMessage.class));
}
and my BizWebKafkaTopicMessage is
#Data
public class BizWebKafkaTopicMessage {
// Elastic Search Index Name
private String indexName;
// ElasticSearch Index's type name
private String indexType;
// Source document to be used
private Map<String, Object> source; <=== This is being delivered as null.
// ElasticSearch document primary id
private Long id;
}
and the listener method listenToKafkaMessages
#KafkaListener(topics = "${biz-web.kafka.message.topic.name}", groupId = "${biz-web.kafka.message.group.id}")
public void listenToKafkaMessages(BizWebKafkaTopicMessage message) {
............................................
............................................
// Here message.source is null
............................................
............................................
}
Inside listenToKafkaMessages method, message argument looks like this
message.indexName = 'neeraj';
message.indexType = 'jain';
message.id = 123;
message.source = null;
My strong suspicion would be that it is the polymorphic nature of your value rather than Maps per-se.
Spring is using Jackson underneath the hood for it's serialisation/deserialisation - and by default Jackson (in serialisation) when handling Object instances does not encode what class it is serialising.
Why? Well it makes for bad compatibility issues e.g. you serialised an Object (Really MyPojoV1.class) into the database 1 year ago, and then later read it out - but your code has no MyPojoV1.class anymore because things have moved on... It can even cause issues if you move MyPojoV1 to a different package anywhere within the lifetime of your application!
So when it comes to deserialising Jackson doesn't know what class to deserialise the Object into.
A hacky idea would be to run the following somewhere:
ObjectMapper.enableDefaultTyping(ObjectMapper.DefaultTyping.NON_FINAL);
Or the nicer/more spring way would be to:
#Configuration
public class JacksonConfiguration {
#Bean
public ObjectMapper objectMapper() {
ObjectMapper mapper = new ObjectMapper();
#Your Configuration Here for example mapper.configure(DeserializationFeature.something, true);
return mapper;
}
}
Finally it's worth adding that deserialising classes arbitrarily is generally a big security risk. There exists classes in Java which execute command line or even reflection based logic based on the value in their fields (which Jackson will happily fill for you). Hence someone can craft JSON such that you deserialise into a class that basically executes whatever command is in the value={} field.
You can read more about exploits here - although I recognise it may not concern you since your Kafka cluster and it's producers may inherently be within your 'trusted boundaries':
https://www.nccgroup.trust/globalassets/our-research/us/whitepapers/2018/jackson_deserialization.pdf
Related
I'm trying to fix a bug in our company's homemade framework. Basically, we should be able to inject EclipseLink properties into the EntityManager through the following class which is part of our framework:
#ConfigurationProperties(prefix = "our.framework.eclipselink")
public class CustomEclipseLinkProperties {
private Map<String, Object> properties;
public Map<String, Object> getProperties() {
return properties;
}
public void setProperties(Map<String, Object> properties) {
this.properties = properties;
}
public String getBatchSize() {
return (String) properties.get(PersistenceUnitProperties.BATCH_WRITING_SIZE);
}
}
Our service built on top of that framework has the following properties file (application.properties):
our.framework.eclipselink.properties.eclipselink.logging.level=FINEST
our.framework.eclipselink.properties.eclipselink.logging.level.cache=FINEST
our.framework.eclipselink.properties.eclipselink.logging.level.sql=FINEST
our.framework.eclipselink.properties.eclipselink.logging.parameters=true
our.framework.eclipselink.properties.eclipselink.jdbc.batch-writing.size=1000
our.framework.eclipselink.properties.eclipselink.jdbc.bind-parameters=true
our.framework.eclipselink.properties.eclipselink.jdbc.batch-writing=JDBC
our.framework.eclipselink.properties.eclipselink.jpa.uppercase-column-names=false
our.framework.eclipselink.properties.eclipselink.jpa.uppercase-columns=false
When I put a breakpoint after the CustomEclipseLinkProperties has been initialized, I can see that getBatchSize() returns null. If I look into getProperties(), I do see the values were detected, but they were inserted as a LinkedHashMap.
The expected behavior would be that we would obtain a Map that would use the entire suffix as the String key, rather than getting this LinkedHashMap that has essentially called String#split() on the properties list. This would mean that the call to getBatchSize() would return 1000.
I've seen a few answers such as this one but they don't seem generic enough to my liking. Is there not a way to simply get the entire suffix as the key when injected by #ConfigurationProperties? Else, it seems like it would require intervention whenever we would want to support a different type of property.
Turns out "suffix as key" is the default behavior if I swap from Map<String, Object> to Map<String, String>.
The Object value isn't actually useful in our case, so that solves this problem.
I hope you're doing great.
I'm facing design problem in spring batch.
Let me explain:
I have a modular spring batch job architecture,
each job has its own config file et context.
I am designing a master Job to launch the subjobs (50+ types of subjobs).
X obj has among other name, state and blob which contains the csv file attached to it.
X obj will be updated after being processed.
I follow the first approach of fetching all X obj and then looping (in java stream) to call the appropriate job.
But this approach has a lot of limitations.
So I design a masterJob with reader processor and writer.
MasterJob should read X obj and call the appropriate subJob and the update the state of X obj.
masterJobReader which call a custom service to get a list of let's say X obj.
I started by trying to launch subjob from within the masterJob processor but It did not work.
I did some research and I find that JobStep could be more adequate for this scenario.
But I'm stuck with how to pass the item read by masterJobReader to JobStep has parameter.
I did saw DefaultJobParameterExtractor and I try to set the Item read into the stepExecutionContext but It's not working.
My question how to pass parameter from MasterJob to SubJob using
JobStep approach?
If there is better way to deal with this then I'm all yours!
I'm using Java Config and spring batch 4.3.
Edit to provide sample code:
#Configuration
public class MasterJob {
#Value("${defaultCompletionPolicy}")
private Integer defaultCompletionPolicy;
#Autowired
protected StepBuilderFactory masterStepBuilderFactory;
private Logger logger = LoggerFactory.getLogger(MasterJob.class);
#Autowired
protected JobRepository jobRepo;
#Autowired
protected PlatformTransactionManager transactionManager;
#Autowired
#Qualifier("JOB_NAME1")
private Job JOB_NAME1; // this should change to be dynamic as there are around 50 types of job
#Bean(name = "masterJob")
protected Job masterBatchJob() throws ApiException {
return new JobBuilderFactory(jobRepo).get("masterJob")
.incrementer(new RunIdIncrementer())
.start(masterJobStep(masterJobReader(), masterJobWriter()))
.next(jobStepJobStep1(null))
.next(masterUpdateStep()) // update the state of objX
.build();
}
#Bean(name = "masterJobStep")
protected Step masterJobStep(#Qualifier("masterJobReader") MasterJobReader masterReader,
#Qualifier("masterJobWriter") MasterJobWriter masterWriter) throws ApiException {
logger.debug("inside masterJobStep");
return this.masterStepBuilderFactory.get("masterJobStep")
.<Customer, Customer>chunk(defaultCompletionPolicy)
.reader(masterJobReader())
.processor(masterJobProcessor())
.writer(masterJobWriter())
.transactionManager(transactionManager)
.listener(new MasterJobWriter()) // I set the parameter inside this.
.listener(masterPromotionListener())
.build();
}
#Bean(name = "masterJobWriter", destroyMethod = "")
#StepScope
protected MasterJobWriter masterJobWriter() {
return new MasterJobWriter();
}
#Bean(name = "masterJobReader", destroyMethod = "")
#StepScope
protected MasterJobReader masterJobReader() throws ApiException {
return new MasterJobReader();
}
protected FieldSetMapper<Customer> mapper() {
return new CustomerMapper();
}
#Bean(name="masterPromotionListener")
public ExecutionContextPromotionListener masterPromotionListener() {
ExecutionContextPromotionListener listener = new ExecutionContextPromotionListener();
listener.setKeys(
new String[]
{
"inputFile",
"outputFile",
"customerId",
"comments",
"customer"
});
//listener.setStrict(true);
return listener;
}
#Bean(name = "masterUpdateStep")
public Step masterUpdateStep() {
return this.masterStepBuilderFactory.get("masterCleanStep").tasklet(new MasterUpdateTasklet()).build();
}
#Bean(name = "masterJobProcessor", destroyMethod = "")
#StepScope
protected MasterJobProcessor masterJobProcessor() {
return new MasterJobProcessor();
}
#Bean
public Step jobStepJobStep1(JobLauncher jobLauncher) {
return this.masterStepBuilderFactory.get("jobStepJobStep1")
.job(JOB_NAME1)
.launcher(jobLauncher)
.parametersExtractor(jobParametersExtractor())
.build();
}
#Bean
public DefaultJobParametersExtractor jobParametersExtractor() {
DefaultJobParametersExtractor extractor = new DefaultJobParametersExtractor();
extractor.setKeys(
new String[] { "inputFile", "outputFile", , "customerId", "comments", "customer" });
return extractor;
}
}
This is how I set parameter from within the MasterJobWriter:
String inputFile = fetchInputFile(customer);
String outputFile = buildOutputFileName(customer);
Comments comments = "comments"; // from business logic
ExecutionContext stepContext = this.stepExecution.getExecutionContext();
stepContext.put("inputFile", inputFile);
stepContext.put("outputFile", outputFile);
stepContext.put("customerId", customer.getCustomerId());
stepContext.put("comments", new CustomJobParameter<Comments>(comments));
stepContext.put("customer", new CustomJobParameter<Customer>(customer));
I follow this section of the documentation of spring batch
My question how to pass parameter from MasterJob to SubJob using JobStep approach?
The JobParametersExtractor is what you are looking for. It allows you to extract parameters from the main job and pass them to the subjob. You can find an example here.
EDIT: Adding suggestions based on comments
I have a list of X obj in the DB. X obj has among other fields, id, type(of work), name, state and blob which contains the csv file attached to it. The blob field containing the csv file depends on the type field so it's not one pattern csv file. I need to process each X obj and save the content of the csv file in the DB and generate a csv result file containing the original data plus a comment field in the result csv file and update X obj state with the result csv field attached to X obj and other fields.
As you can see, the process is already complex for a single X object. So trying to process all X objects in the same job of jobs is too complex IMHO. So much complexity in software comes from trying to make one thing do two things..
If there is better way to deal with this then I'm all yours!
Since you are open for suggestions, I will recommend two options:
Option 1:
If it were up to me, I would create a job instance per X obj. This way, I can 1) parallelize things and 2) in case of failure, restart only the failed job. These two characteristics (Scalability and Restartability) are almost impossible with the job of jobs approach. Even if you have a lot of X objects, this is not a problem. You can use one of the scaling techniques provided by Spring Batch to process things in parallel.
Option 2:
If you really can't or don't want to use different job instances, you can use a single job with a chunk-oriented step that iterates over X objects list. The processing logic seems independent from one record to another, so this step should be easily scalable with multiple threads.
I have facade interface where users can ask for information about lets say Engineers. That information should be transferred as JSON of which we made a DTO for. Now keep in mind that I have multiple datasources that can provide an item to this list of DTO.
So I believe right now that I can use Decorative Pattern by adding handler of the datasource to the myEngineerListDTO of type List<EngineerDTO>. So by that I mean all the datasources have the same DTO.
This picture below shows that VerticalScrollbar and HorizontalScrollBar have different behaviours added. Which means they add behaviour to the WindowDecorator interface.
My question, does my situation fit the decorator pattern? Do I specifically need to add a behaviour to use this pattern? And is there another pattern that does fit my situation? I have already considered Chain of Responsibility pattern, but because I don't need to terminate my chain on any given moment, i thought maybe Decorator pattern would be better.
Edit:
My end result should be: List<EngineersDTO> from all datasources. The reason I want to add this pattern is so that I can easily add another datasource behind the rest of the "pipeline". This datasource, just like the others, will have addEngineersDTOToList method.
To further illustrate on how you can Chain-of-responsibility pattern I put together a small example. I believe you should be able to adapt this solution to suit the needs of your real world problem.
Problem Space
We have an unknown set of user requests which contain the name of properties to be retrieved. There are multiple datasources which each have varying amounts of properties. We want to search through all possible data sources until all of the properties from the request have been discovered. Some data types and data sources might look like bellow (note I am using Lombok for brevity):
#lombok.Data
class FooBarData {
private final String foo;
private final String bar;
}
#lombok.Data
class FizzBuzzData {
private final String fizz;
private final String buzz;
}
class FooBarService {
public FooBarData invoke() {
System.out.println("This is an expensive FooBar call");
return new FooBarData("FOO", "BAR");
}
}
class FizzBuzzService {
public FizzBuzzData invoke() {
System.out.println("This is an expensive FizzBuzz call");
return new FizzBuzzData("FIZZ", "BUZZ");
}
}
Our end user might require multiple ways to resolve the data. The following could be a valid user input and expected response:
// Input
"foobar", "foo", "fizz"
// Output
{
"foobar" : {
"foo" : "FOO",
"bar" : "BAR"
},
"foo" : "FOO",
"fizz" : "FIZZ"
}
A basic interface and simple concrete implementation for our property resolver might look like bellow:
interface PropertyResolver {
Map<String, Object> resolve(List<String> properties);
}
class UnknownResolver implements PropertyResolver {
#Override
public Map<String, Object> resolve(List<String> properties) {
Map<String, Object> result = new HashMap<>();
for (String property : properties) {
result.put(property, "Unknown");
}
return result;
}
}
Solution Space
Rather than using a normal "Decorator pattern", a better solution may be a "Chain-of-responsibility pattern". This pattern is similar to the decorator pattern, however, each link in the chain is allowed to either work on the item, ignore the item, or end the execution. This is helpful for deciding if a call needs to be made, or terminating the chain if the work is complete for the request. Another difference from the decorator pattern is that resolve will not be overriden by each of the concrete classes; our abstract class can call out to the sub class when required using abstract methods.
Back to the problem at hand... For each resolver we need two components. A way to fetch data from our remote service, and a way to extract all the required properties from the data retrieved. For fetching the data we can provide an abstract method. For extracting a property from the fetched data we can make a small interface and maintain a list of these extractors seeing as multiple properties can be pulled from a single piece of data:
interface PropertyExtractor<Data> {
Object extract(Data data);
}
abstract class PropertyResolverChain<Data> implements PropertyResolver {
private final Map<String, PropertyExtractor<Data>> extractors = new HashMap<>();
private final PropertyResolver successor;
protected PropertyResolverChain(PropertyResolver successor) {
this.successor = successor;
}
protected abstract Data getData();
protected final void setBinding(String property, PropertyExtractor<Data> extractor) {
extractors.put(property, extractor);
}
#Override
public Map<String, Object> resolve(List<String> properties) {
...
}
}
The basic idea for the resolve method is to first evaluate which properties can be fulfilled by this PropertyResolver instance. If there are eligible properties then we will fetch the data using getData. For each eligible property we extract the property value and add it to a result map. Each property which cannot be resolved, the successor will be requested to be resolve that property. If all properties are resolved the chain of execution will end.
#Override
public Map<String, Object> resolve(List<String> properties) {
Map<String, Object> result = new HashMap<>();
List<String> eligibleProperties = new ArrayList<>(properties);
eligibleProperties.retainAll(extractors.keySet());
if (!eligibleProperties.isEmpty()) {
Data data = getData();
for (String property : eligibleProperties) {
result.put(property, extractors.get(property).extract(data));
}
}
List<String> remainingProperties = new ArrayList<>(properties);
remainingProperties.removeAll(eligibleProperties);
if (!remainingProperties.isEmpty()) {
result.putAll(successor.resolve(remainingProperties));
}
return result;
}
Implementing Resolvers
When we go to implement a concrete class for PropertyResolverChain we will need to implement the getData method and also bind PropertyExtractor instances. These bindings can act as an adapter for the data returned by each service. This data can follow the same structure as the data returned by the service, or have a custom schema. Using the FooBarService from earlier as an example, our class could be implemented like bellow (note that we can have many bindings which result in the same data being returned).
class FooBarResolver extends PropertyResolverChain<FooBarData> {
private final FooBarService remoteService;
FooBarResolver(PropertyResolver successor, FooBarService remoteService) {
super(successor);
this.remoteService = remoteService;
// return the whole object
setBinding("foobar", data -> data);
// accept different spellings
setBinding("foo", data -> data.getFoo());
setBinding("bar", data -> data.getBar());
setBinding("FOO", data -> data.getFoo());
setBinding("__bar", data -> data.getBar());
// create new properties all together!!
setBinding("barfoo", data -> data.getBar() + data.getFoo());
}
#Override
protected FooBarData getData() {
return remoteService.invoke();
}
}
Example Usage
Putting it all together, we can invoke the Resolver chain as shown bellow. We can observe that the expensive getData method call is only performed once per Resolver only if the property is bound to the resolver, and that the user gets only the exact fields which they require:
PropertyResolver resolver =
new FizzBuzzResolver(
new FooBarResolver(
new UnknownResolver(),
new FooBarService()),
new FizzBuzzService());
Map<String, Object> result = resolver.resolve(Arrays.asList(
"foobar", "foo", "__bar", "barfoo", "invalid", "fizz"));
ObjectMapper mapper = new ObjectMapper();
mapper.enable(SerializationFeature.INDENT_OUTPUT);
System.out.println(mapper
.writerWithDefaultPrettyPrinter()
.writeValueAsString(result));
Output
This is an expensive FizzBuzz call
This is an expensive FooBar call
{
"foobar" : {
"foo" : "FOO",
"bar" : "BAR"
},
"__bar" : "BAR",
"barfoo" : "BARFOO",
"foo" : "FOO",
"invalid" : "Unknown",
"fizz" : "FIZZ"
}
public class SimpleDTO{
private String firstElement;
private String lastElement;
}
public class ComplexSource{
private List<String> elementList;
}
I tried to map it usingmap().setFirstElement(source.getElementList().get(0)) but I get an error stating "1) Invalid source method java.util.List.get(). Ensure that method has zero parameters and does not return void."
How do I map an element in a list to a field in a Pojo using ModelMapper or any other alternative?
In this case is you can't use a PropertyMap. If you want map it using ModelMapper you must use a Converter instead of PropertyMap as you have done.
First your Converter would be as next where the source is ComplexSource and SimpleDTO is the destination:
Converter<ComplexSource, SimpleDTO> converter = new AbstractConverter<ComplexSource, SimpleDTO>() {
#Override
protected SimpleDTO convert(ComplexSource source) {
SimpleDTO destination = new SimpleDTO();
List<String> sourceList = source.getElementList();
if(null != sourceList && !sourceList.isEmpty()){
int sizeList = sourceList.size();
destination.setFirstElement(sourceList.get(0));
destination.setLastElement(sourceList.get(sizeList - 1));
}
return destination;
}
};
Then you just need to add the converter to your ModelMapper instance:
ModelMapper mapper = new ModelMapper();
mapper.addConverter(converter);
If you try the map, it works perfectly:
ComplexSource complexSource = new ComplexSource();
complexSource.setElementList(Arrays.asList("firstElement", "lastElement"));
SimpleDTO simpleDto = mapper.map(complexSource, SimpleDTO.class);
System.out.println(simpleDto);
Output
SimpleDTO [firstElement=firstElement, lastElement=lastElement]
Respect your comment, you need to check nulls if it is need in your source instance (in this case it is possible a null pointer if the list is null). But it inits for you the destination instance, even you can configure the destination instance how you want with a Provider (Providers documentation).
In cases of special use cases like this, you need to worry about null checks and exceptions handling because a Converter I would say is the way of modelmapper to map manually pojos.
The advantadges of use ModelMapper are explained in its web:
If you configure it correctly in some cases it is no need to do the map manually.
It centralizes the mapping.
It provides a mapping API for handling special use cases. (This is your case)
And so on (take a look its web)
Let's say I have a POJO with quite a few fields. I also have a map with a bunch of properties that would map nicely to fields in the POJO. Now I want to apply the properties in the map to my POJO. How can I do this?
Jackson provides method new ObjectMapper().convertValue(), but that creates a fresh instance of the POJO. Do I really have to do something like this?
om = new ObjectMapper();
pojoMap = om.convertValue(pojo, Map.class);
pojoMap.putAll(properties);
pojo = om.convertValue(pojoMap, Pojo.class);
Isn't there an easier way?
As I have no experience with GSON and we also have it lying around here, how would I do that with GSON?
Yes, you can create an ObjectReader that will update an existing instance from the root JSON object rather than instantiating a new one, using the readerForUpdating method of ObjectMapper:
#Test
public void apply_json_to_existing_object() throws Exception {
ExampleRecord record = new ExampleRecord();
ObjectReader reader = mapper.readerForUpdating(record)
.with(JsonParser.Feature.ALLOW_SINGLE_QUOTES)
.with(JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES);
reader.readValue("{ firstProperty: 'foo' }");
reader.readValue("{ secondProperty: 'bar' }");
assertThat(record.firstProperty, equalTo("foo"));
assertThat(record.secondProperty, equalTo("bar"));
}
public static class ExampleRecord {
public String firstProperty;
public String secondProperty;
}
You can also create a value-updating reader from an existing ObjectReader. The following declaration seems equivalent:
ObjectReader reader = mapper.reader(ExampleRecord.class)
.withValueToUpdate(record)
.with(/* features etc */);
Addition
The above didn't actually answer your question, though.
Since you don't have the changes you want to make to the record as JSON, but rather as a map, you have to finagle things so that Jackson will read your Map. Which you can't do directly, but you can write the "JSON" out to a token buffer and then read it back:
#Test
public void apply_map_to_existing_object_via_json() throws Exception {
ExampleRecord record = new ExampleRecord();
Map<String, Object> properties = ImmutableMap.of("firstProperty", "foo", "secondProperty", "bar");
TokenBuffer buffer = new TokenBuffer(mapper, false);
mapper.writeValue(buffer, properties);
mapper.readerForUpdating(record).readValue(buffer.asParser());
assertThat(record.firstProperty, equalTo("foo"));
assertThat(record.secondProperty, equalTo("bar"));
}
(btw if this seems laborious, serializing to a token buffer and deserializing again is in fact how ObjectMapper.convertValue is implemented, so it's not a big change in functionality)