java.util.set fields serialization/deSerialization using Apache Avro - java

I am trying to serialize custom object which has java.util.set fields using Apache Avro using below code:
final Schema schemaItemImportSchema = ReflectData.get().getSchema(clazz);
final DatumWriter<T> writer = new ReflectDatumWriter<>(clazz);
byte[] data = new byte[0];
final ByteArrayOutputStream stream = new ByteArrayOutputStream();
try {
final Encoder encoder = EncoderFactory.get().jsonEncoder(schema, stream);
datumWriter.write(data, encoder);
encoder.flush();
data = stream.toByteArray();
} catch (final Exception excp) {
log.error(excp);
}
And deSerialization using below code,
final Schema schemaItemImportSchema = ReflectData.get().getSchema(clazz);
final DatumReader<T> reader = new ReflectDatumReader<>(clazz);
Object dataActual = new Object();
try {
final Decoder decoder = DecoderFactory.get().jsonDecoder(schema, new String(data));
dataActual = reader.read(null, decoder);
} catch (final IOException excp) {
log.error(excp);
}
Using above code I am able to serialize successfully with set fields but during de-serialization getting below error,
java.lang.RuntimeException: java.lang.NoSuchMethodException: java.util.Set.<init>()
If I use #AvroIgnore for set fields, both serialization and de-serialization works perfectly.
How can I serialize and deserialize java.util.set fields?

Resolved this issue by changing type to HashSet instead of set.
Referred https://blog.51cto.com/shadowisper/1947979

Related

How to write serializable object to String without writing to file?

I want to write a class object to the string and then again create an object from it.
I searched on the net but all I found is to write an object to file however I want to write in the string, not on file.
Below is the example of writing to file similarly I want to write in String or similar Object and not in the file.
some_class implements serializable {
...
}
FileOutputStream f = new FileOutputStream(new File("myObjects.txt"));
ObjectOutputStream o = new ObjectOutputStream(f);
// Write objects to file
o.writeObject(object1);
o.close();
f.close();
FileInputStream fi = new FileInputStream(new File("myObjects.txt"));
ObjectInputStream oi = new ObjectInputStream(fi);
// Read objects
some_class object2 = (some_class) oi.readObject();
oi.close();
fi.close();
Please help with the same.
This would be one way:
try
{
// To String
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutputStream os = new ObjectOutputStream(bos);
os.writeObject(object1);
String serializedObject1 = bos.toString();
os.close();
// To Object
ByteArrayInputStream bis = new ByteArrayInputStream(serializedObject1.getBytes());
ObjectInputStream oInputStream = new ObjectInputStream(bis);
YourObject restoredObject1 = (YourObject) oInputStream.readObject();
oInputStream.close();
} catch(Exception ex) {
ex.printStackTrace();
}
I would prefer the Base64 way though.
This would be an example of encoding:
private static String serializableToString( Serializable o ) throws IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(o);
oos.close();
return Base64.getEncoder().encodeToString(baos.toByteArray());
}
And this is an example of decoding:
private static Object objectFromString(String s) throws IOException, ClassNotFoundException
{
byte [] data = Base64.getDecoder().decode(s);
ObjectInputStream ois = new ObjectInputStream(
new ByteArrayInputStream(data));
Object o = ois.readObject();
ois.close();
return o;
}
the best way to serialize an object to String and vice versa you should convert the object into JSON String and encode into base64. and to get object decode base64 and convert to object using GSON (opensource google provide java library)
class foo{ String name, email;
//setter getter
}
convert Object to base64 JSON
public static String convertToJson(Object o){
String result=new Gson().toJson(o);
return Base64.getEncoder().encodeToString(result);
}
//read base64
public static <T> T convertJsonToObject(String base64Object,Class<T> classOfT){
Gson gson = new Gson();
return gson.fromJson(new InputStreamReader(new ByteArrayInputStream(Base64.getDecoder().decode(base64Object))),classOfT);
}
public static void main(String[] args) {
foo obj=new foo("jhon","jhon#gamil.com");
String json=convertToJson(foo);
System.out.println(json);
foo obj_fromJson=convertJsonToObject(json,foo.class);
System.out.println(obj_fromJson.getName());
}

ClassNotFoundException when trying to decode object from String

Alright so I've got this Android Studio app with user class
public class User{
int age;
String name;
int yearOfBirth;
}
And then I've got this two methods
public static Object fromString(String s) throws IOException,ClassNotFoundException {
byte [] data = Base64.getDecoder().decode(s);
ObjectInputStream ois = new ObjectInputStream(new ByteArrayInputStream(data));
Object o = ois.readObject();
ois.close();
return o;
}
public static String toString(Serializable o) throws IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(o);
oos.close();
return Base64.getEncoder().encodeToString(baos.toByteArray());
}
So I create User object, convert it to String, save it to database on other device where I need to convert the object back to User object to change some stuff, convert to String again save to database and then be able to decode the User object in my app again.
Problem is when trying to decode the String on the server side I'm getting this exception
Exception in thread "main" java.lang.ClassNotFoundException: com.example.mikithenics.User
What could be the possible solution ? Any help is appreciated.
You can serialize this using Google Gson lib like this:
Gson gson = new Gson();
String jsonString = gson.toJson(userObject);
And deserialize like this:
Gson gson = new Gson();
User userObject = gson.fromJson(userJsonString, User.class);
To use Gson lib, put this in your app build.gradle file, inside dependencies:
implementation 'com.google.code.gson:gson:2.8.6'

Issue in deserialize protostuff object in Kafka Consumer

Getting following exception while deserializing byte[] into protostuff object in Kafka Consumer
java.lang.NegativeArraySizeException
at com.dyuproject.protostuff.GraphIOUtil.mergeDelimitedFrom(GraphIOUtil.java:209)
at com.gme.protocols.protostuff.GmeTrade.readExternal(GmeTrade.java:2772)
at java.io.ObjectInputStream.readExternalData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
Converted protostuff object to byte[] using following code.
public static byte[] toBytes(Object o)
{
try
{
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(o);
oos.close();
byte[] b = baos.toByteArray();
return b;
}
catch (IOException e)
{
return new byte[0];
}
}
Sent that byte[] using Kafka producer with topic 'XX', where byte[] length is just 240.
Received that record using Kafka consumer. record.value().length (byte[]) length is same 240 what I sent from producer side.
Deserialized that byte[] to object using following code.
public static Object fromBytes(byte[] bytes)
{
try
{
return new ObjectInputStream(new ByteArrayInputStream(bytes)).readObject();
}
catch (Exception e)
{
e.printStackTrace();
return null;
}
}
Getting above mentioned exception. What I am doing wrong ?
Using kafka_2.11-0.9.0.0 for your reference. Is there any other things needed?
I found the solution. Issue is there in toBytes and fromBytes.
Need to convert it into byte[] using ProtostuffIOUtil.toByteArray method.
Serialization
public static byte[] toBytes(Foo o)
{
LinkedBuffer BUFFER = LinkedBuffer.allocate(1024*1024);
Schema<Foo> SCHEMA = Foo.getSchema();
return ProtostuffIOUtil.toByteArray(o, SCHEMA, BUFFER);
}
Again need to convert byte[] to object using ProtostuffIOUtil.mergeFrom method.
Deserialization
public static Foo fromBytes(byte[] bytes)
{
Foo tmp = Foo.getSchema().newMessage();
ProtostuffIOUtil.mergeFrom(bytes, tmp, Foo.getSchema());
return tmp;
}
Note : Serialization/Deserialization with ObjectOutputStream/ObjectInputStream will not work for protostuff objects.

how to convert an input stream to a java object

ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(reg_be);
oos.flush();
oos.close();
InputStream is = new ByteArrayInputStream(baos.toByteArray());
This code convert Java Object to InputStream and how can I convert InputStream to an Object? I need to convert my Object to an InputStream then I pass it and I want to get my Object back.
In try block you should write:
ObjectInputStream ois = new ObjectInputStream(is);
Object object = ois.readObject();
ObjectInputStream is initialized with another stream, e.g. BufferedInputStream or your input stream is.
ObjectInputStream ois = new ObjectInputStream(is);
Object object = ois.readObject();
Try the following
ObjectInputStream ois = new ObjectInputStream(is);
Object obj = ois .readObject();
ObjectInputStream ois = new ObjectInputStream(is);
Object object = ois.readObject();
As mentioned by #darijan is working fine.
But again we need to do try, catch block for that code, & for blank input stream it will give EOF (End Of File) related error.
So, I am converting it to a string. Then if the string is not empty or null, then only I am converting it to Object using ObjectMapper
Although it's not an efficient approach, I don't need to worry about try-catch, null handling also is done in a string instead of the input stream
String responseStr = IOUtils.toString(is, StandardCharsets.UTF_8.name());
Object object = null;
// is not null or whitespace consisted string
if (StringUtils.isNotBlank(response)) {
object = getJsonFromString(response);
}
// below codes are already used in project (Util classes)
private Object getJsonFromString(String jsonStr) {
if (StringUtils.isEmpty(jsonStr)) {
return new LinkedHashMap<>();
}
ObjectMapper objectMapper = getObjectMapper();
Map<Object, Object> obj = null;
try {
obj = objectMapper.readValue(jsonStr, new TypeReference<Map<Object, Object>>() {
});
} catch (IOException e) {
LOGGER.error("Unable to parse JSON : {}",e)
}
return obj;
}
private ObjectMapper getObjectMapper() {
ObjectMapper objectMapper = new ObjectMapper();
objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
objectMapper.enable(DeserializationFeature.ACCEPT_EMPTY_STRING_AS_NULL_OBJECT);
return objectMapper;
}

Read objects in a file with Camel?

I have a file containing java objects, wrote with this code:
from(somewhere).process(new Processor() {
#Override
public void process(final Exchange exchange) {
...
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutput out = new ObjectOutputStream(bos);
out.writeObject(myObject);
exchange.getOut().setBody(bos.toByteArray());
}
}).to("file://pathFile");
And now, I want read them fastly. I don't know how can I do that, something like the following code I gess.
from("file://pathFile").convertBodyTo(String.class)
.split(body().tokenize("???")) // How can I tokenize my file ?
.streaming().threads(2)
.process(new Processor() {
#Override
public void process(final Exchange exchange) {
String filePath = (String) exchange.getIn().getHeader(Exchange.FILE_PATH);
File file = new File(filePath);
MyObject myObject = null;
try {
FileInputStream fis = new FileInputStream(file);
InputStream buffer = new BufferedInputStream(fis);
ObjectInput input = new ObjectInputStream(buffer);
Object obj = null;
while ((obj = input.readObject()) != null) {
// Do something
myObject = obj;
}
} catch (Exception e) {
...
} finally {
...
}
exchange.getIn().setBody(myObject);
}
}).to(somewhere);
EDIT: I edit my way to read object. There is still a problem with that code, we can't append to an ObjectOutputStream. That will corrupt the stream. There is a solution [here] for this problem. We can only write the stream header one time.
But If I do that, I wont be able to split and read my file with multiple threads. So can I split or access my file on ObjectOutputStream header ?
you just converted it to a String using convertBodyTo(String.class), therefore you have a String in the body rather than an InputStream....

Categories