getting error on desirializing java object from Kafka - java

I started to learn Kafka, and now,
I'm on sending/receiving serialized/desirialised java class.
My question is about: what have I missed in my config, so I can't deserialize the object from Kafka
here is my class:
public class Foo {
private String item;
private int quantity;
private Double price;
public Foo(String item, int quantity, final double price) {
this.item = item;
this.quantity = quantity;
this.price = price;
}
public String getItem() { return item; }
public int getQuantity() { return quantity; }
public Double getPrice() { return price; }
public void setQuantity(int quantity) { this.quantity = quantity; }
public void setPrice(double price) { this.price = price; }
#Override
public String toString() {
return "item=" + item + ", quantity=" + quantity + ", price=" + price;
}
}
my Properties in main class:
producerPropsObject.put(ProducerConfig.CLIENT_ID_CONFIG,
AppConfigs.applicationProducerSerializedObject);
producerPropsObject.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
AppConfigs.bootstrapServers);
producerPropsObject.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
StringSerializer.class.getName());
producerPropsObject.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
FooSerializer.class.getName());
producerPropsObject.put("topic", AppConfigs.topicNameForSerializedObject);
consumerPropsObject.put(ConsumerConfig.GROUP_ID_CONFIG, AppConfigs.applicationProducerSerializedObject);
consumerPropsObject.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, AppConfigs.bootstrapServers);
consumerPropsObject.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
consumerPropsObject.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,FooDeserializer.class.getName());
consumerPropsObject.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, 300000);
consumerPropsObject.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, true);
consumerPropsObject.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
consumerPropsObject.put("topic", AppConfigs.topicNameForSerializedObject);
following are serializer/deserializer implementations:
public class FooSerializer implements org.apache.kafka.common.serialization.Serializer {
public void configure(Map map, boolean b) { }
public byte[] serialize(String s, Object o) {
try {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(o);
oos.close();
byte[] b = baos.toByteArray();
return b;
} catch (IOException e) { return new byte[0]; }
}
public void close() { }
}
public class FooDeserializer implements org.apache.kafka.common.serialization.Deserializer {
#Override
public void close() { }
#Override
public Foo deserialize(String arg0, byte[] arg1) {
//Option #1:
//ObjectMapper mapper = new ObjectMapper();
//Option #2:
JsonFactory factory = new JsonFactory();
factory.enable(JsonParser.Feature.ALLOW_SINGLE_QUOTES);
ObjectMapper mapper = new ObjectMapper(factory);
Foo fooObj = null;
try {
//Option #1:
//fooObj = mapper.readValue(arg1, Foo.class); // BREAKS HERE!!!
//Option #2:
fooObj = mapper.reader().forType(Foo.class).readValue(arg1); // BREAKS HERE!!!
}
catch (Exception e) { e.printStackTrace(); }
return fooObj;
}
}
and finally the way I'm trying to produce and consume my Foo from main:
seems, like it works fine, cause I see in kafka-topic my Key && Value later on
public void produceObjectToKafka(final Properties producerProps) {
final String[] ar = new String[]{"Matrix", "Naked Gun", "5th Element", "Die Hard", "Gone with a wind"};
KafkaProducer<String, byte[]> producer = new KafkaProducer<>(producerProps);
final Foo j = new Foo(ar[getAnInt(4)], getAnInt(10), getAnDouble());
producer.send(new ProducerRecord<>(producerProps.getProperty("topic"), j.getItem(), j.toString().getBytes()));
producer.flush();
producer.close();
}
however, while my Consumer is catching the output:
public void consumeFooFromKafka(final Properties consumerProps) {
final Consumer<String, Foo> myConsumer = new KafkaConsumer<>(consumerProps);
final Thread separateThread = new Thread(() -> {
try {
myConsumer.subscribe(Collections.singletonList(consumerProps.getProperty("topic")));
while (continueToRunFlag) {
final StringBuilder sb = new StringBuilder();
final ConsumerRecords<String, Foo> consumerRecords = myConsumer.poll(Duration.ofMillis(10));
if (consumerRecords.count() > 0) {
for (ConsumerRecord<String, Foo> cRec : consumerRecords) {
sb.append( cRec.key() ).append("<<").append(cRec.value().getItem() + ",").append(cRec.value().getQuantity() + ",").append(cRec.value().getPrice()).append("|");
}
}
if (sb.length() > 0) { System.out.println(sb.toString()); }
}
}
finally {
myConsumer.close();
}
});
separateThread.start();
}
=======================================
so, actually by running "consumeFooFromKafka" , when it trigger "FooDeserializer" ...... there, I always have same error(regardless of Option #1, or Option #2):
exception:
Method threw 'com.fasterxml.jackson.core.JsonParseException' exception.
detailedMessage:
Unexpected character ('¬' (code 172)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or
'false')
will be very appresiated for help.......
Thank you in advance,
Steve

If you want to deserialize from json, than u need to serialize it to json, use jackson in ur serializer also, and everything should be fine
public class FooSerializer implements org.apache.kafka.common.serialization.Serializer {
public void configure(Map map, boolean b) { }
public byte[] serialize(String s, Object o) {
try {
ObjectMapper om = new ObjectMapper();//objectmapper from jackson
byte[] b = om.writeValueAsString(o).getBytes();
return b;
} catch (IOException e) { return new byte[0]; }
}
public void close() { }
}

I don't know why you're using a bytearray outputstream, but trying to read JSON in the deserializer, but that explains the error. You could even test that without using Kafka at all by invoking the serialize/deserialize methods directly
In the link provided, the serializer uses objectMapper.writeValueAsString, which returns JSON text, and not the Java specific outputstream. If you wanted to consume and produce data between different programming languages (as is often the case in most companies), you'd want to avoid such specific serialization formats
Note: Confluent provides Avro, Protobuf, and JSON serializers for Kafka, so you shouldn't need to write your own if you want to use one of those formats

Related

GraphQL Error: Expected a user-defined GraphQL scalar type with name 'FileUpload' but found none

for several days I have been trying to implement the upload file in Java-GraphQL. I found this topic: How to upload files with graphql-java? I implemented second solutions.
public class FileUpload {
private String contentType;
private byte[] content;
public FileUpload(String contentType, byte[] content) {
this.contentType = contentType;
this.content = content;
}
public String getContentType() {
return contentType;
}
public byte[] getContent() {
return content;
}
}
public class MyScalars {
public static final GraphQLScalarType FileUpload = new GraphQLScalarType(
"FileUpload",
"A file part in a multipart request",
new Coercing<FileUpload, Void>() {
#Override
public Void serialize(Object dataFetcherResult) {
throw new CoercingSerializeException("Upload is an input-only type");
}
#Override
public FileUpload parseValue(Object input) {
if (input instanceof Part) {
Part part = (Part) input;
try {
String contentType = part.getContentType();
byte[] content = new byte[part.getInputStream().available()];
part.delete();
return new FileUpload(contentType, content);
} catch (IOException e) {
throw new CoercingParseValueException("Couldn't read content of the uploaded file");
}
} else if (null == input) {
return null;
} else {
throw new CoercingParseValueException(
"Expected type " + Part.class.getName() + " but was " + input.getClass().getName());
}
}
#Override
public FileUpload parseLiteral(Object input) {
throw new CoercingParseLiteralException(
"Must use variables to specify Upload values");
}
});
}
public class FileUploadResolver implements GraphQLMutationResolver {
public Boolean uploadFile(FileUpload fileUpload) {
String fileContentType = fileUpload.getContentType();
byte[] fileContent = fileUpload.getContent();
// Do something in order to persist the file :)
return true;
}
}
scalar FileUpload
type Mutation {
uploadFile(fileUpload: FileUpload): Boolean
}
I get this error during compilation:
Caused by: com.coxautodev.graphql.tools.SchemaClassScannerError: Expected a user-defined GraphQL scalar type with name 'FileUpload' but found none!
Have you registered it via RuntimeWiring?
Take a look here: Custom Scalar in Graphql-java
You have to extend GraphQLScalarType in your MyScalars class

How to define array of objects in a properties file and read from Java program

I have a properties file like this.
property[0].name=A
property[0].value=1
property[1].name=B
property[1].value=2
property[2].name=C
property[2].value=3
How to read this file as a list of objects of a class {name, value} in plain java program using ResourceBundle or Properties?
Here is the class.
public class XYZ {
private String name;
private String value;
// Getters & Setters
}
I need to get like this.
ArrayList<XYZ> propertiesList = SomeUtility.getProperties("property", XYZ.class);
Utility class might be like this.
public class SomeUtility {
public static ArrayList getProperties(String key, Class cls) {
//logic
}
}
I might not understand exactly what you want so feel free to correct me and give me more constraints to work with but here is a simple way to read a Properties file located somewhere in your project:
private static void readPropertiesFile(String path) throws IOException {
java.util.Map<String, String> map = new java.util.LinkedHashMap<>();
Properties properties = new Properties();
InputStream inputStream = new FileInputStream(path);
properties.load(inputStream);
for (String name : properties.stringPropertyNames()) {
map.put(name, properties.getProperty(name));
}
for (java.util.Map.Entry<String, String> entry : map.entrySet()) {
System.out.printf("Property Key: %s, Property Value: %s%n", entry.getKey(), entry.getValue());
}
}
Output
Property Key: property[0].name, Property Value: A
Property Key: property[1].name, Property Value: B
Property Key: property[0].value, Property Value: 1
Property Key: property[1].value, Property Value: 2
Property Key: property[2].name, Property Value: C
Property Key: property[2].value, Property Value: 3
This is the solution I wrote, but it involves Reflect and Gson. Is there any better way to do this? Anything already available which is fine tuned like Apache's.
import com.google.gson.Gson;
import com.google.gson.JsonObject;
import java.lang.reflect.Field;
import java.util.*;
public class ListResourceBundle {
public static final Gson gson = new Gson();
private final ResourceBundle bundle;
public ListResourceBundle(ResourceBundle bundle) {
this.bundle = bundle;
}
public List<?> getProperties(String key, Class<?> cls) {
final int maxArraySize = getMaxArraySize(key, getMatchingKeys(key));
final List<String> fields = getFields(cls);
final List<Object> result = new ArrayList<>();
for (int i = 0; i < maxArraySize; i++) {
JsonObject jsonObject = new JsonObject();
for (String field : fields) {
jsonObject.addProperty(field, getStringOrNull(key + "[" + i + "]." + field));
}
result.add(gson.fromJson(jsonObject, cls));
}
System.out.println("result.toString() = " + result.toString());
return result;
}
public List<String> getMatchingKeys(String key) {
Enumeration<String> keys = bundle.getKeys();
List<String> matchingKeys = new ArrayList<>();
while(keys.hasMoreElements()) {
String k = keys.nextElement();
if(k.startsWith(key)) {
matchingKeys.add(k);
}
}
Collections.sort(matchingKeys);
return matchingKeys;
}
public int getMaxArraySize(String key, List<String> matchingKeys) {
int maxArraySize = 0;
for (int i = 0; ; i++) {
boolean indexAvailable = false;
for (String matchingKey : matchingKeys) {
if(matchingKey.startsWith(key + "[" + i + "]")) {
indexAvailable = true;
break;
}
}
if(indexAvailable) {
maxArraySize++;
} else {
break;
}
}
return maxArraySize;
}
public String getStringOrNull(String key) {
try {
return bundle.getString(key);
} catch (MissingResourceException e) {
return null;
}
}
public List<String> getFields(Class<?> cls) {
final List<String> fields = new ArrayList<>();
for (Field field : cls.getDeclaredFields()) {
fields.add(field.getName());
}
return fields;
}
public static void main(String[] args) {
ResourceBundle bundle = ResourceBundle.getBundle("com.example.application.resources.Resource");
ListResourceBundle applicationResourceBundle = new ListResourceBundle(bundle);
applicationResourceBundle.getProperties("property", ReportParam.class);
}
}
Resource:
property[0].name=A
property[0].value=1
property[1].name=B
property[1].value=2
property[2].name=C
property[2].value=3
Output:
result.toString() = [
ReportParam{name='A', value='1'},
ReportParam{name='B', value='2'},
ReportParam{name='C', value='3'}]
Process finished with exit code 0
I know it's bit late of an answer, but if I understand your problem statement correctly, you can use :
#ConfigurationProperties
to get your job done.
Here is my spring-boot example with a YAML file for the sake of convenience (same can be achieved through properties file as well).
application.yaml:
xyz:
xyzprops :
-
name: cbc
value: 441
-
name: obc
value: 443
XYZ class:
#Component
#ConfigurationProperties(prefix = "xyz")
public class XYZ{
private List<XYZProps> xyzprops;
public List<XYZProps> getXyzprops() {
return xyzprops;
}
public void setXyzprops(List<XYZProps> xyzprops) {
this.xyzprops = xyzprops;
}
public class XYZProps{
String name;
String value;
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public String getValue() {
return value;
}
public void setValue(String value) {
this.value = value;
}
}
}
And then #Autowire XYZ where you want to use it.
I would use JSON:
in your file:
property=[{"name":"A","value":"1"},{"name":"B","value":"2"},{"name":"C","value":"3"}]
and then deserialize it using com.google.gson.gson (or any other) library:
ArrayList<XYZ> propertiesList;
propertiesList = new gsonbuilder().create().fromjson(property, propertiesList.class);
NOTE: I haven't tested this code, and i'm not very familiar with java so i am sure there is a better,cleaner way to implement this.

Java Object Serialization: serialize a Hashtable, then deserialize it in a HashMap

Java object serialization: I serialized this type of Hashtable for example of (Integer,Employee) and when I deserialize this, I would like to put the output in a HashMap. Is this possible? Because I get the java.lang.ClassCastException
public class Employee implements Serializable {
private static final long serialVersionUID = -7260877684654746408L;
private String name;
private int age;
Employee(String n, int a) {
name=n;
age=a;
}
public String toString() {
return "Name: "+name+". "+"Age: "+age+".";
}
}
public class Test {
public static void main(String[] args) {
Hashtable<Integer, Employee> ht = new Hashtable<Integer, Employee>() {
{
put(1, new Employee("John", 37));
put(2, new Employee("Julia", 36));
}
};
//HashMap<Integer,Employee> hm = new HashMap<Integer,Employee>();
try {
FileOutputStream outSer = new FileOutputStream("outSer.ser");
ObjectOutputStream os = new ObjectOutputStream(outSer);
os.writeObject(ht);
os.close();
FileInputStream input = new FileInputStream("outSer.ser");
ObjectInputStream ois = new ObjectInputStream(input);
HashMap<Integer,Employee> hm= (HashMap<Integer, Employee>)ois.readObject();
ois.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
Is this possible?
Yes. But not the way you coded it. You would have to create your own HashMap and load it from the HashTable:
HashMap<Integer,Employee> hm= new HashMap<>((HashTable<Integer, Employee>)ois.readObject());

Jackson JSON array accessing

I have a Json file like this:
[{
dname: "xxxx",
dage: "24"
}, {
dname: "yyyy",
dage: "26"
}]
Target:
I want to access them as an array
Search through the names in the JSON file to look for a particular name
Same for age.
What I did:
file name : DtExtract.java
public class DtExtract{
public static ObjectMapper mapper = new ObjectMapper();
private Dtmain[] dtmain =mapper.readValue(new File("file location"), TypeFactory.defaultInstance().constructArrayType(D tmain.class));
public DtExtract() throws IOException{}
public String getname(int i) throws IOException { String strname = dtmain[i].getjname(); return strname;}
public String getage(int i) throws IOException { String strage = dtmain[i].getjage(); return strage;}
}
class Dtmain {
private String dname;
private String dage;
public Dtmain(){}
public String getjname(){return dname;}
public String getjage(){return dage;}
public void setjname(String dname){ this.dname=dname;}
public void setjage(String dage){ this.dage=dage;}
public String toString(){ return "Student [ name" + dname +", age " +dage +"]";
}
============================
file name: Myclass.java
public class Myclass{
public static void main(String args[]) throws IOException {
DtExtract dtextract= new DtExtract();
for(int i=0; i< 2; i++)
{
if (dtextract.getname(i).equals("xxxx")) {System.out.print("Name matches");}
if (dtextract.getage(i).equals("24")) {System.out.print("Age matches");}
}
}
}
=============================
This is the abstract of a code that I have, but my question is:
Does this for loop is really accessing the JSON array elements?
Is there any other faster way to do this JSON parsing and comparison?
You can do it as follows. You need to call both methods from main method
Please note that I have added two methods to process the array. processJsonArrayJava8 will be faster compared to processJsonArrayJava7.
public List<Dtmain> readFromJson()
throws JsonParseException, JsonMappingException, IOException {
ObjectMapper mapper = new ObjectMapper();
TypeReference<List<Dtmain>> mapType = new TypeReference<List<Dtmain>>() {};
List<Dtmain> jsonList = mapper.readValue(
"[{\"dname\": \"xxxx\",\"dage\": \"24\" },{\"dname\": \"yyyy\",\"dage\": \"26\" }]",
mapType);
return jsonList;
}
public void processJsonArrayJava7(List<Dtmain> jsonList) {
for(Dtmain obj : jsonList) {
// do what ever you want with obj
}
}
public void processJsonArrayJava8(List<Dtmain> jsonList) {
jsonList.parallelStream().forEach(obj->{
//do what ever you want with obj
});
}
Please also give better names to the methods.
public class SayHi {
public static void main(String args[]) throws IOException {
try {
ObjectMapper mapper = new ObjectMapper();
TypeReference<List<Person>> maptype = new TypeReference<List<Person>>() {};
List<Person> jsonTopersonList=mapper.readValue(new File("JSON_file_location"), maptype);
for(int i=0; i<jsonTopersonList.size(); i++){
System.out.println("Name"+i+":"+jsonTopersonList.get(i).dname);
}
for(int i=0; i<jsonTopersonList.size(); i++){
System.out.println("Age"+i+":"+jsonTopersonList.get(i).dage);
}
} catch (JsonGenerationException e) {
e.printStackTrace();
} catch (JsonMappingException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}}
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
public class Person {
public String dname;
public int dage;
public Person() {
}
public Person(String dname,
int dage) {
this.dname = dname;
this.dage = dage;
}
public String toString() {
return "[" + dname + " " + dage +"]";
}
}
I came up with this one to deserialize that json data.

equivalent to python's shelve module in Java

Is there any module in Java equivalent to python's shelve module? I need this to achieve dictionary like taxonomic data access. Dictionary-like taxonomic data access is a powerful way to save Python objects in a persistently easy access database format. I need something for the same purpose but in Java.
I also needed this, so I wrote one. A bit late, but maybe it'll help.
It doesn't implement the close() method, but just use sync() since it only hold the file open when actually writing it.
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.util.HashMap;
public class Shelf extends HashMap<String, Object> {
private static final long serialVersionUID = 7127639025670585367L;
private final File file;
public static Shelf open(File file) {
Shelf shelf = null;
try {
if (file.exists()) {
final FileInputStream fis = new FileInputStream(file);
ObjectInputStream ois = new ObjectInputStream(fis);
shelf = (Shelf) ois.readObject();
ois.close();
fis.close();
} else {
shelf = new Shelf(file);
shelf.sync();
}
} catch (Exception e) {
// TODO: handle errors
}
return shelf;
}
// Shelf objects can only be created or opened by the Shelf.open method
private Shelf(File file) {
this.file = file;
sync();
}
public void sync() {
try {
final FileOutputStream fos = new FileOutputStream(file);
ObjectOutputStream oos = new ObjectOutputStream(fos);
oos.writeObject(this);
oos.close();
fos.close();
} catch (Exception e) {
// TODO: handle errors
}
}
// Simple Test Case
public static void main(String[] args) {
Shelf shelf = Shelf.open(new File("test.obj"));
if (shelf.containsKey("test")) {
System.out.println(shelf.get("test"));
} else {
System.out.println("Creating test string. Run the program again.");
shelf.put("test", "Hello Shelf!");
shelf.sync();
}
}
}
You could use a serialisation library like Jackson which serialises POJOs to JSON.
An example from the tutorial:
Jackson's org.codehaus.jackson.map.ObjectMapper "just works" for
mapping JSON data into plain old Java objects ("POJOs"). For example,
given JSON data
{
"name" : { "first" : "Joe", "last" : "Sixpack" },
"gender" : "MALE",
"verified" : false,
"userImage" : "Rm9vYmFyIQ=="
}
It takes two lines of Java to turn it into a User instance:
ObjectMapper mapper = new ObjectMapper(); // can reuse, share globally
User user = mapper.readValue(new File("user.json"), User.class);
Where the User class looks something like this (from an entry on Tatu's blog):
public class User {
public enum Gender { MALE, FEMALE };
public static class Name {
private String _first, _last;
public String getFirst() { return _first; }
public String getLast() { return _last; }
public void setFirst(String s) { _first = s; }
public void setLast(String s) { _last = s; }
}
private Gender _gender;
private Name _name;
private boolean _isVerified;
private byte[] _userImage;
public Name getName() { return _name; }
public boolean isVerified() { return _isVerified; }
public Gender getGender() { return _gender; }
public byte[] getUserImage() { return _userImage; }
public void setName(Name n) { _name = n; }
public void setVerified(boolean b) { _isVerified = b; }
public void setGender(Gender g) { _gender = g; }
public void setUserImage(byte[] b) { _userImage = b; }
}

Categories