HashMap serialization and deserialization changes - java

We are working with an in memory data grid (IMDG) and we have a migration tool. In order to verify that all the objects are migrated successfully, we calculate the chucksum of the objects from its serialized version.
We are seeing some problems with HashMap, where we serialize it, but when we deserialize it the checksum changes. Here is a simple test case:
#Test
public void testMapSerialization() throws IOException, ClassNotFoundException {
TestClass tc1 = new TestClass();
tc1.init();
String checksum1 = SpaceObjectUtils.calculateChecksum(tc1);
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutput out = null;
byte[] objBytes = null;
out = new ObjectOutputStream(bos);
out.writeObject(tc1);
objBytes = bos.toByteArray();
out.close();
ByteArrayInputStream bis = new ByteArrayInputStream(objBytes);
ObjectInputStream in = new ObjectInputStream(bis);
TestClass tc2 = (TestClass) in.readObject();
String checksum2 = SpaceObjectUtils.calculateChecksum(tc2);
assertEquals(checksum1, checksum2);
}
The TestClass looks like this:
class TestClass implements Serializable {
private static final long serialVersionUID = 5528034467300853270L;
private Map<String, Object> map;
public TestClass() {
}
public Map<String, Object> getMap() {
return map;
}
public void setMap(Map<String, Object> map) {
this.map = map;
}
public void init() {
map = new HashMap<String, Object>();
map.put("name", Integer.valueOf(4));
map.put("type", Integer.valueOf(4));
map.put("emails", new BigDecimal("43.3"));
map.put("theme", "sdfsd");
map.put("notes", Integer.valueOf(4));
map.put("addresses", Integer.valueOf(4));
map.put("additionalInformation", new BigDecimal("43.3"));
map.put("accessKey", "sdfsd");
map.put("accountId", Integer.valueOf(4));
map.put("password", Integer.valueOf(4));
map.put("domain", new BigDecimal("43.3"));
}
}
And this is the method to calculate the checksum:
public static String calculateChecksum(Serializable obj) {
if (obj == null) {
throw new IllegalArgumentException("The object cannot be null");
}
MessageDigest digest = null;
try {
digest = MessageDigest.getInstance("MD5");
} catch (java.security.NoSuchAlgorithmException nsae) {
throw new IllegalStateException("Algorithm MD5 is not present", nsae);
}
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutput out = null;
byte[] objBytes = null;
try {
out = new ObjectOutputStream(bos);
out.writeObject(obj);
objBytes = bos.toByteArray();
out.close();
} catch (IOException e) {
throw new IllegalStateException(
"There was a problem trying to get the byte stream of this object: " + obj.toString());
}
digest.update(objBytes);
byte[] hash = digest.digest();
StringBuilder hexString = new StringBuilder();
for (int i = 0; i < hash.length; i++) {
String hex = Integer.toHexString(0xFF & hash[i]);
if (hex.length() == 1) {
hexString.append('0');
}
hexString.append(hex);
}
return hexString.toString();
}
If you print the maps of tc1 and tc2, you can see that the elements are not in the same place:
{accessKey=sdfsd, accountId=4, theme=sdfsd, name=4, domain=43.3, additionalInformation=43.3, emails=43.3, addresses=4, notes=4, type=4, password=4}
{accessKey=sdfsd, accountId=4, name=4, theme=sdfsd, domain=43.3, emails=43.3, additionalInformation=43.3, type=4, notes=4, addresses=4, password=4}
I would like to be able to serialize the HashMap and get the same checksum when I deserialize it. Do you know if there is a solution or if I'm doing something wrong?
Thanks!
Diego

You are doing nothing wrong, it just can't be done with a HashMap. In a HashMap, order is not guaranteed. Use a TreeMap instead.
Hash table based implementation of the
Map interface. This implementation
provides all of the optional map
operations, and permits null values
and the null key. (The HashMap class
is roughly equivalent to Hashtable,
except that it is unsynchronized and
permits nulls.) This class makes no
guarantees as to the order of the map;
in particular, it does not guarantee
that the order will remain constant
over time.
Source: Hashmap

Your check sum cannot depend on the order of entries as HashMap is not ordered. An alternative to using TreeMap is LinkedHashMap (which retains an order), but the real solution is to use a hashCode which doesn't depending on the order of the entries.

Use LinkedHashMap which is order one.
TreeMap is not ordered. TreeMap is sorted map.
TreeMap sorts elements irrespective of insertion order.

Related

How to read all objects from ObjectInputStream

I have a file with some info how can I read all info?
Name names;
try (FileInputStream fileInputStream = new FileInputStream(file)) {
ObjectInputStream objectInputStream = new ObjectInputStream(fileInputStream);
names = (Name) objectInputStream.readObject();
} catch (IOException | ClassNotFoundException e) {
e.printStackTrace();
}
You have several solution, all depending on the input:
You can iterate until the stream is fully consumed: I think that is the worse solution out of those I provide you. It is worse because you are checking if EOF was reached, whilst you should know when you're done (eg: your file format is wrong).
Set<Name> result = new HashSet<>();
try {
for (;;) {
result.add((Name)objectInputStream.readObject());
}
} catch (EOFException e) {
// End of stream
}
return result;
When producing the input, serialize a collection and invoke readObject() on it. Serialization should be able to read the collection, as long as each object implements Serializable.
static void write(Path path, Set<Name> names) throws IOException {
try (OutputStream os = Files.newOutputStream(path);
ObjectOutputStream oos = new ObjectOutputStream(os)) {
oos.writeObject(names);
}
}
static Set<Name> read(Path path) throws IOException {
try (InputStream is = Files.newInputStream(path);
ObjectInputStream ois = new ObjectInputStream(is)) {
// WARN Files.newInputStream is not buffered; ObjectInputStream might
// be buffered (I don't remember).
return (Set<Name>) ois.readObject();
}
}
When producing the input, you can add a int indicating the number of object to read, and iterate over it: this is useful in case where you don't really care of the collection (HashSet). The resulting file will be smaller (because you won't have the HashSet metadata).
int result = objectInputStream.readInt();
Name[] names = new Name[result]; // do some check on result!
for (int i = 0; i < result; ++i) {
names[i] = (Name) objectInputStream.readObject();
}
Also, Set are good, but since they remove duplicate using hashCode()/equals() you may get less object if your definition of equals/hashCode changed after the fact (example: your Name was case sensitive and now it is not, eg: new Name("AA").equals(new Name("aa"))).

How to deepclone Map with List as value

Is this the best way to deepclone this data structure example: Map<String, List<Object>>?
Map<String, List<Object>> mapB = new LinkedHashMap<String, List<Object>>();
for(String key: mapA.keySet()){
mapB.put(key, new ArrayList<Object>());
mapB.get(key).addAll(mapA.get(key));
}
Thanks for your time.
Is this the best way to deepclone this data structure
More or less, yes. You can make it a bit shorter using the ArrayList constructor that takes a source Collection as argument, and a bit more efficient (but more wordy) by iterating key-value pairs instead of looking up each key again, but it amounts to the same thing.
Map<String, List<Object>> mapB = new LinkedHashMap<>();
for (Map.Entry<String, List<Object>> entry : mapA.entrySet()) {
mapB.put(entry.getKey(), new ArrayList<>(entry.getValue()));
}
It's fine if you don't want deep copy of contaning objects.
An alternative method of deep cloning is to use serialization. The following method shows how this is done:
public Object deepClone(Object obj) {
try {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(obj);
ByteArrayInputStream bais = new ByteArrayInputStream(baos.toByteArray());
ObjectInputStream ois = new ObjectInputStream(bais);
return ois.readObject();
} catch (IOException e) {
return null;
} catch (ClassNotFoundException e) {
return null;
}
}
The apache commons lang SerializationUtils offers a generic method to do this.
Both ways assume that all objects in your object graph implement serializable.

Java - Error deserializing HashTable containing primitive type

I have serialized a HashTable<String,Object> object using an ObjectOutputStream. When serializing the object, I get no exception, but upon deserialization, the following exception occurs:
Exception in thread "main" java.io.InvalidClassException: java.lang.Long; local class
incompatible: stream classdesc serialVersionUID = 4290774032661291999, local class
serialVersionUID = 4290774380558885855
I no longer get the error when I remove all of the keys in the HashTable that have a value that is not a String (all of the key / value pairs I removed had a primitive type as their value).
What could be causing this error?
UPDATE - Here's the code
public static String serialize(Quiz quiz) throws IOException{
HashMap<String,Object> quizData = new HashMap<String,Object>();
quizData.put("version", 0); //int
quizData.put("name", quiz.getName()); //String
quizData.put("desc", quiz.getDesc()); //String
quizData.put("timelimitType", quiz.getTimelimitType()); //String
quizData.put("timelimit", quiz.getTimelimit()); //long
ArrayList<String> serializedQuestionsData = new ArrayList<String>();
for (Question question : quiz.getQuestions())
serializedQuestionsData.add(Question.serialize(question));
quizData.put("questions", serializedQuestionsData.toArray(new String[0])); //String[]
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos;
try { oos = new ObjectOutputStream(baos); } catch (IOException error){ throw error; }
try { oos.writeObject(quizData); } catch (IOException error){ throw error; }
return baos.toString();
}
#SuppressWarnings("unchecked")
public static Quiz deserialize(String serializedQuizData) throws IOException, ClassNotFoundException{
ByteArrayInputStream bais = new ByteArrayInputStream(serializedQuizData.getBytes());
ObjectInputStream ois;
try { ois = new ObjectInputStream(bais); } catch (IOException error){ throw error; }
HashMap<String,Object> quizData;
// Exception occurs on the following line!!
try { quizData = (HashMap<String,Object>) ois.readObject(); } catch (ClassNotFoundException error){ throw error; }
Quiz quiz;
if ((int) quizData.get("version") == 0){
quiz = new Quiz((String) quizData.get("name"),
(String) quizData.get("desc"),
(String) quizData.get("timelimitType"),
(long) quizData.get("timelimit"));
for (String serializedQuestionData : (String[]) quizData.get("questions"))
quiz.addQuestion(Question.deserialize(serializedQuestionData));
} else {
throw new UnsupportedOperationException("Unsupported version: \"" + quizData.get("version") + "\"");
}
return quiz;
}
The problem is that you're transforming a byte array output stream to a String using toString(). The toString() method simply uses the platform default encoding to transform the bytes (which do not represent characters at all but are purely binary data) into a String. This is thus a lossy operation, because your platform default encoding doesn't have a valid character for every possible byte.
You shouldn't use String to hold binary data. A String contains characters. If you really need a String, then encode the byte array using a Hexadecimal or Base64 encoder. Otherwise, simply use a byte array to hold your binary data:
public static byte[] serialize(Quiz quiz) throws IOException{
...
ByteArrayOutputStream baos = new ByteArrayOutputStream();
...
return baos.toByteArray();
}
#SuppressWarnings("unchecked")
public static Quiz deserialize(byte[] serializedQuizData) throws IOException, ClassNotFoundException{
ByteArrayInputStream bais = new ByteArrayInputStream(serializedQuizData);
...
return quiz;
}
The only explanation I can think of is that is that something is corrupting your object stream between you reading it and writing it. The serialVersionID in "the local class) (4290774380558885855) is standard across all Java implementations that try to be compatible with Java (tm). The source code for java.lang.Long says that that serial version id has not changed since Java 1.0.2.
If you need further help, you will need to provide an SSCCE that covers both creation and reading of the serialized object.

Convert HashMap.toString() back to HashMap in Java

I put a key-value pair in a Java HashMap and converted it to a String using the toString() method.
Is it possible to convert this String representation back to a HashMap object and retrieve the value with its corresponding key?
Thanks
It will work if toString() contains all data needed to restore the object. For example it will work for map of strings (where string is used as key and value):
// create map
Map<String, String> map = new HashMap<String, String>();
// populate the map
// create string representation
String str = map.toString();
// use properties to restore the map
Properties props = new Properties();
props.load(new StringReader(str.substring(1, str.length() - 1).replace(", ", "\n")));
Map<String, String> map2 = new HashMap<String, String>();
for (Map.Entry<Object, Object> e : props.entrySet()) {
map2.put((String)e.getKey(), (String)e.getValue());
}
This works although I really do not understand why do you need this.
toString() approach relies on implementation of toString() and it can be lossy in most of the cases.
There cannot be non lossy solution here. but a better one would be to use Object serialization
serialize Object to String
private static String serialize(Serializable o) throws IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(o);
oos.close();
return Base64.getEncoder().encodeToString(baos.toByteArray());
}
deserialize String back to Object
private static Object deserialize(String s) throws IOException,
ClassNotFoundException {
byte[] data = Base64.getDecoder().decode(s);
ObjectInputStream ois = new ObjectInputStream(
new ByteArrayInputStream(data));
Object o = ois.readObject();
ois.close();
return o;
}
Here if the user object has fields which are transient, they will be lost in the process.
old answer
Once you convert HashMap to String using toString(); It's not that you can convert back it to Hashmap from that String, Its just its String representation.
You can either pass the reference to HashMap to method or you can serialize it
Here is the description for toString() toString()
Here is the sample code with explanation for Serialization.
and to pass hashMap to method as arg.
public void sayHello(Map m){
}
//calling block
Map hm = new HashMap();
sayHello(hm);
you cannot do this directly but i did this in a crazy way as below...
The basic idea is that, 1st you need to convert HashMap String into Json then you can deserialize Json using Gson/Genson etc into HashMap again.
#SuppressWarnings("unchecked")
private HashMap<String, Object> toHashMap(String s) {
HashMap<String, Object> map = null;
try {
map = new Genson().deserialize(toJson(s), HashMap.class);
} catch (TransformationException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return map;
}
private String toJson(String s) {
s = s.substring(0, s.length()).replace("{", "{\"");
s = s.substring(0, s.length()).replace("}", "\"}");
s = s.substring(0, s.length()).replace(", ", "\", \"");
s = s.substring(0, s.length()).replace("=", "\":\"");
s = s.substring(0, s.length()).replace("\"[", "[");
s = s.substring(0, s.length()).replace("]\"", "]");
s = s.substring(0, s.length()).replace("}\", \"{", "}, {");
return s;
}
implementation...
HashMap<String, Object> map = new HashMap<String, Object>();
map.put("Name", "Suleman");
map.put("Country", "Pakistan");
String s = map.toString();
HashMap<String, Object> newMap = toHashMap(s);
System.out.println(newMap);
i converted HashMap into an String
using toString() method and pass to
the another method that take an String
and convert this String into HashMap
object
This is a very, very bad way to pass around a HashMap.
It can theoretically work, but there's just way too much that can go wrong (and it will perform very badly). Obviously, in your case something does go wrong. We can't say what without seeing your code.
But a much better solution would be to change that "another method" so that it just takes a HashMap as parameter rather than a String representation of one.
You can make use of Google's "GSON" open-source Java library for this,
Example input (Map.toString) : {name=Bane, id=20}
To Insert again in to HashMap you can use below code:
yourMap = new Gson().fromJson(yourString, HashMap.class);
That's it Enjoy.
(In Jackson Library mapper It will produce exception "expecting double-quote to start field name")
What did you try?
objectOutputStream.writeObject(hashMap);
should work just fine, providing that all the objects in the hashMap implement Serializable.
You cannot revert back from string to an Object. So you will need to do this:
HashMap<K, V> map = new HashMap<K, V>();
//Write:
OutputStream os = new FileOutputStream(fileName.ser);
ObjectOutput oo = new ObjectOutputStream(os);
oo.writeObject(map);
oo.close();
//Read:
InputStream is = new FileInputStream(fileName.ser);
ObjectInput oi = new ObjectInputStream(is);
HashMap<K, V> newMap = oi.readObject();
oi.close();
Are you restricted to use only HashMap ??
Why can't it be so much flexible JSONObject you can do a lot with it.
You can convert String jsonString to JSONObject jsonObj
JSONObject jsonObj = new JSONObject(jsonString);
Iterator it = jsonObj.keys();
while(it.hasNext())
{
String key = it.next().toString();
String value = jsonObj.get(key).toString();
}
Using ByteStream can convert the String but it can encounter OutOfMemory exception in case of large Strings. Baeldung provides some nice solutions in his pot here : https://www.baeldung.com/java-map-to-string-conversion
Using StringBuilder :
public String convertWithIteration(Map<Integer, ?> map) {
StringBuilder mapAsString = new StringBuilder("{");
for (Integer key : map.keySet()) {
mapAsString.append(key + "=" + map.get(key) + ", ");
}
mapAsString.delete(mapAsString.length()-2, mapAsString.length()).append("}");
return mapAsString.toString(); }
Please note that lambdas are only available at language level 8 and above
Using Stream :
public String convertWithStream(Map<Integer, ?> map) {
String mapAsString = map.keySet().stream()
.map(key -> key + "=" + map.get(key))
.collect(Collectors.joining(", ", "{", "}"));
return mapAsString; }
Converting String Back to Map using Stream :
public Map<String, String> convertWithStream(String mapAsString) {
Map<String, String> map = Arrays.stream(mapAsString.split(","))
.map(entry -> entry.split("="))
.collect(Collectors.toMap(entry -> entry[0], entry -> entry[1]));
return map; }
I hope you actually need to get the value from string by passing the hashmap key. If that is the case, then we don't have to convert it back to Hashmap. Use following method and you will be able to get the value as if it was retrieved from Hashmap itself.
String string = hash.toString();
String result = getValueFromStringOfHashMap(string, "my_key");
/**
* To get a value from string of hashmap by passing key that existed in Hashmap before converting to String.
* Sample string: {fld_category=Principal category, test=test 1, fld_categoryID=1}
*
* #param string
* #param key
* #return value
*/
public static String getValueFromStringOfHashMap(String string, String key) {
int start_index = string.indexOf(key) + key.length() + 1;
int end_index = string.indexOf(",", start_index);
if (end_index == -1) { // because last key value pair doesn't have trailing comma (,)
end_index = string.indexOf("}");
}
String value = string.substring(start_index, end_index);
return value;
}
Does the job for me.
It is possible to rebuild a collection out of its string presentation but it will not work if the elements of the collection don't override their own toString method.
Therefore it's much safer and easier to use third party library like XStream which streams objects in human readable XML.
This may be inefficient and indirect. But
String mapString = "someMap.toString()";
new HashMap<>(net.sf.json.JSONObject.fromObject(mapString));
should work !!!

How to encode a Map<String,String> as Base64 string?

i like to encode a java map of strings as a single base 64 encoded string. The encoded string will be transmitted to a remote endpoint and maybe manipulated by a not nice person. So the worst thing that should happen are invaild key,value-tuples, but should not bring any other security risks aside.
Example:
Map<String,String> map = ...
String encoded = Base64.encode(map);
// somewhere else
Map<String,String> map = Base64.decode(encoded);
Yes, must be Base64. Not like that or that or any other of these. Is there an existing lightweight solution (Single Utils-Class prefered) out there? Or do i have to create my own?
Anything better than this?
// marshalling
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(map);
oos.close();
String encoded = new String(Base64.encodeBase64(baos.toByteArray()));
// unmarshalling
byte[] decoded = Base64.decodeBase64(encoded.getBytes());
ByteArrayInputStream bais = new ByteArrayInputStream(decoded);
ObjectInputStream ois = new ObjectInputStream(bais);
map = (Map<String,String>) ois.readObject();
ois.close();
Thanks,
my primary requirements are: encoded string should be as short as possible and contain only latin characters or characters from the base64 alphabet (not my call). there are no other reqs.
Use Google Gson to convert Map to JSON. Use GZIPOutputStream to compress the JSON string. Use Apache Commons Codec Base64 or Base64OutputStream to encode the compressed bytes to a Base64 string.
Kickoff example:
public static void main(String[] args) throws IOException {
Map<String, String> map = new HashMap<String, String>();
map.put("key1", "value1");
map.put("key2", "value2");
map.put("key3", "value3");
String serialized = serialize(map);
Map<String, String> deserialized = deserialize(serialized, new TypeToken<Map<String, String>>() {}.getType());
System.out.println(deserialized);
}
public static String serialize(Object object) throws IOException {
ByteArrayOutputStream byteaOut = new ByteArrayOutputStream();
GZIPOutputStream gzipOut = null;
try {
gzipOut = new GZIPOutputStream(new Base64OutputStream(byteaOut));
gzipOut.write(new Gson().toJson(object).getBytes("UTF-8"));
} finally {
if (gzipOut != null) try { gzipOut.close(); } catch (IOException logOrIgnore) {}
}
return new String(byteaOut.toByteArray());
}
public static <T> T deserialize(String string, Type type) throws IOException {
ByteArrayOutputStream byteaOut = new ByteArrayOutputStream();
GZIPInputStream gzipIn = null;
try {
gzipIn = new GZIPInputStream(new Base64InputStream(new ByteArrayInputStream(string.getBytes("UTF-8"))));
for (int data; (data = gzipIn.read()) > -1;) {
byteaOut.write(data);
}
} finally {
if (gzipIn != null) try { gzipIn.close(); } catch (IOException logOrIgnore) {}
}
return new Gson().fromJson(new String(byteaOut.toByteArray()), type);
}
Another possible way would be using JSON which is a very ligthweight lib.
The the encoding then would look like this:
JSONObject jso = new JSONObject( map );
String encoded = new String(Base64.encodeBase64( jso.toString( 4 ).toByteArray()));
Your solution works. The only other approach would be to serialize the map yourself (iterate over the keys and values). That would mean you'd have to make sure you handle all the cases correctly (for example, if you transmit the values as key=value, you must find a way to allow = in the key/value and you must separate the pairs somehow which means you must also allow this separation character in the name, etc).
All in all, it's hard to get right, easy to get wrong and would take a whole lot more code and headache. Plus don't forget that you'd have to write a lot of error handling code in the parser (receiver side).

Categories