i like to encode a java map of strings as a single base 64 encoded string. The encoded string will be transmitted to a remote endpoint and maybe manipulated by a not nice person. So the worst thing that should happen are invaild key,value-tuples, but should not bring any other security risks aside.
Example:
Map<String,String> map = ...
String encoded = Base64.encode(map);
// somewhere else
Map<String,String> map = Base64.decode(encoded);
Yes, must be Base64. Not like that or that or any other of these. Is there an existing lightweight solution (Single Utils-Class prefered) out there? Or do i have to create my own?
Anything better than this?
// marshalling
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(map);
oos.close();
String encoded = new String(Base64.encodeBase64(baos.toByteArray()));
// unmarshalling
byte[] decoded = Base64.decodeBase64(encoded.getBytes());
ByteArrayInputStream bais = new ByteArrayInputStream(decoded);
ObjectInputStream ois = new ObjectInputStream(bais);
map = (Map<String,String>) ois.readObject();
ois.close();
Thanks,
my primary requirements are: encoded string should be as short as possible and contain only latin characters or characters from the base64 alphabet (not my call). there are no other reqs.
Use Google Gson to convert Map to JSON. Use GZIPOutputStream to compress the JSON string. Use Apache Commons Codec Base64 or Base64OutputStream to encode the compressed bytes to a Base64 string.
Kickoff example:
public static void main(String[] args) throws IOException {
Map<String, String> map = new HashMap<String, String>();
map.put("key1", "value1");
map.put("key2", "value2");
map.put("key3", "value3");
String serialized = serialize(map);
Map<String, String> deserialized = deserialize(serialized, new TypeToken<Map<String, String>>() {}.getType());
System.out.println(deserialized);
}
public static String serialize(Object object) throws IOException {
ByteArrayOutputStream byteaOut = new ByteArrayOutputStream();
GZIPOutputStream gzipOut = null;
try {
gzipOut = new GZIPOutputStream(new Base64OutputStream(byteaOut));
gzipOut.write(new Gson().toJson(object).getBytes("UTF-8"));
} finally {
if (gzipOut != null) try { gzipOut.close(); } catch (IOException logOrIgnore) {}
}
return new String(byteaOut.toByteArray());
}
public static <T> T deserialize(String string, Type type) throws IOException {
ByteArrayOutputStream byteaOut = new ByteArrayOutputStream();
GZIPInputStream gzipIn = null;
try {
gzipIn = new GZIPInputStream(new Base64InputStream(new ByteArrayInputStream(string.getBytes("UTF-8"))));
for (int data; (data = gzipIn.read()) > -1;) {
byteaOut.write(data);
}
} finally {
if (gzipIn != null) try { gzipIn.close(); } catch (IOException logOrIgnore) {}
}
return new Gson().fromJson(new String(byteaOut.toByteArray()), type);
}
Another possible way would be using JSON which is a very ligthweight lib.
The the encoding then would look like this:
JSONObject jso = new JSONObject( map );
String encoded = new String(Base64.encodeBase64( jso.toString( 4 ).toByteArray()));
Your solution works. The only other approach would be to serialize the map yourself (iterate over the keys and values). That would mean you'd have to make sure you handle all the cases correctly (for example, if you transmit the values as key=value, you must find a way to allow = in the key/value and you must separate the pairs somehow which means you must also allow this separation character in the name, etc).
All in all, it's hard to get right, easy to get wrong and would take a whole lot more code and headache. Plus don't forget that you'd have to write a lot of error handling code in the parser (receiver side).
Related
I'm converting Json to avro. I have json data in JSONArray. So while converting it into byte array i'm facing the problem.
below is my code:
static byte [] fromJsonToAvro(JSONArray json, String schemastr) throws Exception {
ExcelToJson ejj = new ExcelToJson();
List<String> list = new ArrayList<String>();
if (json != null) {
int len = json.length();
for (int i=0;i<len;i++){
list.add(json.get(i).toString());
}
}
InputStream input = new ByteArrayInputStream(list.getBytes()); //json.toString().getBytes()
DataInputStream din = new DataInputStream(input);
.
.
.//rest of the logic
So how can i do it? How to convert JsonArray object to bytes(i.e., how to use getBytes() method for JsonArray objects). The above code giving an error at list.getBytes() and saying getBytes() is undifined for list.
Avro works at the record level, bound to a schema. I don't think there's such a concept as "convert this JSON fragment to bytes for an Avro field independent of any schema or record".
Assuming the array is part of a larger JSON record, if you're starting with a string of the record, you could do
public static byte[] jsonToAvro(String json, String schemaStr) throws IOException {
InputStream input = null;
DataFileWriter<GenericRecord> writer = null;
Encoder encoder = null;
ByteArrayOutputStream output = null;
try {
Schema schema = new Schema.Parser().parse(schemaStr);
DatumReader<GenericRecord> reader = new GenericDatumReader<GenericRecord>(schema);
input = new ByteArrayInputStream(json.getBytes());
output = new ByteArrayOutputStream();
DataInputStream din = new DataInputStream(input);
writer = new DataFileWriter<GenericRecord>(new GenericDatumWriter<GenericRecord>());
writer.create(schema, output);
Decoder decoder = DecoderFactory.get().jsonDecoder(schema, din);
GenericRecord datum;
while (true) {
try {
datum = reader.read(null, decoder);
} catch (EOFException eofe) {
break;
}
writer.append(datum);
}
writer.flush();
return output.toByteArray();
} finally {
try { input.close(); } catch (Exception e) { }
}
}
For an on-line json to avro converter check the following URL
http://avro4s-ui.landoop.com
It is using the library avro4s that offers a lot of conversions including json=>avro
This discussion is likely useful:
http://mail-archives.apache.org/mod_mbox/avro-user/201209.mbox/%3CCALEq1Z8s1sfaAVB7YE2rpZ=v3q1V_h7Vm39h0HsOzxJ+qfQRSg#mail.gmail.com%3E
The gist is that there is a special Json schema and you can use JsonReader/Writer to get to and from that. The Json schema you should use is defined here:
https://github.com/apache/avro/blob/trunk/share/schemas/org/apache/avro/data/Json.avsc
Is this the best way to deepclone this data structure example: Map<String, List<Object>>?
Map<String, List<Object>> mapB = new LinkedHashMap<String, List<Object>>();
for(String key: mapA.keySet()){
mapB.put(key, new ArrayList<Object>());
mapB.get(key).addAll(mapA.get(key));
}
Thanks for your time.
Is this the best way to deepclone this data structure
More or less, yes. You can make it a bit shorter using the ArrayList constructor that takes a source Collection as argument, and a bit more efficient (but more wordy) by iterating key-value pairs instead of looking up each key again, but it amounts to the same thing.
Map<String, List<Object>> mapB = new LinkedHashMap<>();
for (Map.Entry<String, List<Object>> entry : mapA.entrySet()) {
mapB.put(entry.getKey(), new ArrayList<>(entry.getValue()));
}
It's fine if you don't want deep copy of contaning objects.
An alternative method of deep cloning is to use serialization. The following method shows how this is done:
public Object deepClone(Object obj) {
try {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(obj);
ByteArrayInputStream bais = new ByteArrayInputStream(baos.toByteArray());
ObjectInputStream ois = new ObjectInputStream(bais);
return ois.readObject();
} catch (IOException e) {
return null;
} catch (ClassNotFoundException e) {
return null;
}
}
The apache commons lang SerializationUtils offers a generic method to do this.
Both ways assume that all objects in your object graph implement serializable.
I have serialized a HashTable<String,Object> object using an ObjectOutputStream. When serializing the object, I get no exception, but upon deserialization, the following exception occurs:
Exception in thread "main" java.io.InvalidClassException: java.lang.Long; local class
incompatible: stream classdesc serialVersionUID = 4290774032661291999, local class
serialVersionUID = 4290774380558885855
I no longer get the error when I remove all of the keys in the HashTable that have a value that is not a String (all of the key / value pairs I removed had a primitive type as their value).
What could be causing this error?
UPDATE - Here's the code
public static String serialize(Quiz quiz) throws IOException{
HashMap<String,Object> quizData = new HashMap<String,Object>();
quizData.put("version", 0); //int
quizData.put("name", quiz.getName()); //String
quizData.put("desc", quiz.getDesc()); //String
quizData.put("timelimitType", quiz.getTimelimitType()); //String
quizData.put("timelimit", quiz.getTimelimit()); //long
ArrayList<String> serializedQuestionsData = new ArrayList<String>();
for (Question question : quiz.getQuestions())
serializedQuestionsData.add(Question.serialize(question));
quizData.put("questions", serializedQuestionsData.toArray(new String[0])); //String[]
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos;
try { oos = new ObjectOutputStream(baos); } catch (IOException error){ throw error; }
try { oos.writeObject(quizData); } catch (IOException error){ throw error; }
return baos.toString();
}
#SuppressWarnings("unchecked")
public static Quiz deserialize(String serializedQuizData) throws IOException, ClassNotFoundException{
ByteArrayInputStream bais = new ByteArrayInputStream(serializedQuizData.getBytes());
ObjectInputStream ois;
try { ois = new ObjectInputStream(bais); } catch (IOException error){ throw error; }
HashMap<String,Object> quizData;
// Exception occurs on the following line!!
try { quizData = (HashMap<String,Object>) ois.readObject(); } catch (ClassNotFoundException error){ throw error; }
Quiz quiz;
if ((int) quizData.get("version") == 0){
quiz = new Quiz((String) quizData.get("name"),
(String) quizData.get("desc"),
(String) quizData.get("timelimitType"),
(long) quizData.get("timelimit"));
for (String serializedQuestionData : (String[]) quizData.get("questions"))
quiz.addQuestion(Question.deserialize(serializedQuestionData));
} else {
throw new UnsupportedOperationException("Unsupported version: \"" + quizData.get("version") + "\"");
}
return quiz;
}
The problem is that you're transforming a byte array output stream to a String using toString(). The toString() method simply uses the platform default encoding to transform the bytes (which do not represent characters at all but are purely binary data) into a String. This is thus a lossy operation, because your platform default encoding doesn't have a valid character for every possible byte.
You shouldn't use String to hold binary data. A String contains characters. If you really need a String, then encode the byte array using a Hexadecimal or Base64 encoder. Otherwise, simply use a byte array to hold your binary data:
public static byte[] serialize(Quiz quiz) throws IOException{
...
ByteArrayOutputStream baos = new ByteArrayOutputStream();
...
return baos.toByteArray();
}
#SuppressWarnings("unchecked")
public static Quiz deserialize(byte[] serializedQuizData) throws IOException, ClassNotFoundException{
ByteArrayInputStream bais = new ByteArrayInputStream(serializedQuizData);
...
return quiz;
}
The only explanation I can think of is that is that something is corrupting your object stream between you reading it and writing it. The serialVersionID in "the local class) (4290774380558885855) is standard across all Java implementations that try to be compatible with Java (tm). The source code for java.lang.Long says that that serial version id has not changed since Java 1.0.2.
If you need further help, you will need to provide an SSCCE that covers both creation and reading of the serialized object.
We are working with an in memory data grid (IMDG) and we have a migration tool. In order to verify that all the objects are migrated successfully, we calculate the chucksum of the objects from its serialized version.
We are seeing some problems with HashMap, where we serialize it, but when we deserialize it the checksum changes. Here is a simple test case:
#Test
public void testMapSerialization() throws IOException, ClassNotFoundException {
TestClass tc1 = new TestClass();
tc1.init();
String checksum1 = SpaceObjectUtils.calculateChecksum(tc1);
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutput out = null;
byte[] objBytes = null;
out = new ObjectOutputStream(bos);
out.writeObject(tc1);
objBytes = bos.toByteArray();
out.close();
ByteArrayInputStream bis = new ByteArrayInputStream(objBytes);
ObjectInputStream in = new ObjectInputStream(bis);
TestClass tc2 = (TestClass) in.readObject();
String checksum2 = SpaceObjectUtils.calculateChecksum(tc2);
assertEquals(checksum1, checksum2);
}
The TestClass looks like this:
class TestClass implements Serializable {
private static final long serialVersionUID = 5528034467300853270L;
private Map<String, Object> map;
public TestClass() {
}
public Map<String, Object> getMap() {
return map;
}
public void setMap(Map<String, Object> map) {
this.map = map;
}
public void init() {
map = new HashMap<String, Object>();
map.put("name", Integer.valueOf(4));
map.put("type", Integer.valueOf(4));
map.put("emails", new BigDecimal("43.3"));
map.put("theme", "sdfsd");
map.put("notes", Integer.valueOf(4));
map.put("addresses", Integer.valueOf(4));
map.put("additionalInformation", new BigDecimal("43.3"));
map.put("accessKey", "sdfsd");
map.put("accountId", Integer.valueOf(4));
map.put("password", Integer.valueOf(4));
map.put("domain", new BigDecimal("43.3"));
}
}
And this is the method to calculate the checksum:
public static String calculateChecksum(Serializable obj) {
if (obj == null) {
throw new IllegalArgumentException("The object cannot be null");
}
MessageDigest digest = null;
try {
digest = MessageDigest.getInstance("MD5");
} catch (java.security.NoSuchAlgorithmException nsae) {
throw new IllegalStateException("Algorithm MD5 is not present", nsae);
}
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutput out = null;
byte[] objBytes = null;
try {
out = new ObjectOutputStream(bos);
out.writeObject(obj);
objBytes = bos.toByteArray();
out.close();
} catch (IOException e) {
throw new IllegalStateException(
"There was a problem trying to get the byte stream of this object: " + obj.toString());
}
digest.update(objBytes);
byte[] hash = digest.digest();
StringBuilder hexString = new StringBuilder();
for (int i = 0; i < hash.length; i++) {
String hex = Integer.toHexString(0xFF & hash[i]);
if (hex.length() == 1) {
hexString.append('0');
}
hexString.append(hex);
}
return hexString.toString();
}
If you print the maps of tc1 and tc2, you can see that the elements are not in the same place:
{accessKey=sdfsd, accountId=4, theme=sdfsd, name=4, domain=43.3, additionalInformation=43.3, emails=43.3, addresses=4, notes=4, type=4, password=4}
{accessKey=sdfsd, accountId=4, name=4, theme=sdfsd, domain=43.3, emails=43.3, additionalInformation=43.3, type=4, notes=4, addresses=4, password=4}
I would like to be able to serialize the HashMap and get the same checksum when I deserialize it. Do you know if there is a solution or if I'm doing something wrong?
Thanks!
Diego
You are doing nothing wrong, it just can't be done with a HashMap. In a HashMap, order is not guaranteed. Use a TreeMap instead.
Hash table based implementation of the
Map interface. This implementation
provides all of the optional map
operations, and permits null values
and the null key. (The HashMap class
is roughly equivalent to Hashtable,
except that it is unsynchronized and
permits nulls.) This class makes no
guarantees as to the order of the map;
in particular, it does not guarantee
that the order will remain constant
over time.
Source: Hashmap
Your check sum cannot depend on the order of entries as HashMap is not ordered. An alternative to using TreeMap is LinkedHashMap (which retains an order), but the real solution is to use a hashCode which doesn't depending on the order of the entries.
Use LinkedHashMap which is order one.
TreeMap is not ordered. TreeMap is sorted map.
TreeMap sorts elements irrespective of insertion order.
We have a Client/Server application which communicates over RMI. The server sends HashMaps to the client. All works well, however when sending large HashMaps, transfer times can be slow.
Is there any way to compress the HashMaps before sending, then decompress on the client? I do not want to create any files on disk whatsoever (All must be in RAM)
Thanks
You can use DeflatorOutputStream to a ByteArrayOutputStream, however you will end up with a byte[] so your RMI call should return a byte[].
Small serializable obejct won't compress well, however if you have many Serializable objects it can compress very well. So can large amounts of text.
The simplest thing to do is to try it. If there are repeated strings or even portions of strings, this will help compression.
public static void main(String... args) throws IOException {
Map<String, String> map = new HashMap<String, String>();
for(int i=0;i<1000;i++)
map.put(""+Math.random(), ""+Math.random());
byte[] bytes1 = toBytes(map);
byte[] bytes2 = toCompressedBytes(map);
System.out.println("HashMap with "+map.size()+" entries, Uncompressed length="+bytes1.length+", compressed length="+bytes2.length);
}
public static byte[] toCompressedBytes(Object o) throws IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(new DeflaterOutputStream(baos));
oos.writeObject(o);
oos.close();
return baos.toByteArray();
}
public static byte[] toBytes(Object o) throws IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(o);
oos.close();
return baos.toByteArray();
}
public static Object fromCompressedBytes(byte[] bytes) throws IOException, ClassNotFoundException {
ObjectInputStream ois = new ObjectInputStream(new InflaterInputStream(new ByteArrayInputStream(bytes)));
return ois.readObject();
}
Prints
HashMap with 1000 entries, Uncompressed length=42596, compressed length=19479
Don't do anything to the hashmap. Instead, Write a custom socket factory that compresses the data using a DeflaterOutputStream.
Many years ago I used to serialize objects into byte array and then zip it. Zip is still supported by Java :) so try this method.
You may try a custom serialization mechanism for the elements inside the hashmap.
What kind of information are you sending? how do the object inside look like?
Even using the default mechanism, and marking all the unneeded attributes as transient will help.
Additionally you may attempt to sending the data your self serializing it before to a ZipOutputStream but I would let that as a last resource, for the binary content won't compress too much.
EDIT
Since your using only strings, you can create an wrapper whose custom serialization is a compressed array ( pretty much as Peter Lawrey answer ) but, using a custom serialization would let you encapsulate the serialization process and have it working some how "transparently" for RMI ( RMI serialization would never know you're using a compressed version )
Here's a demo:
import java.io.*;
import java.util.*;
import java.util.zip.*;
public class MapDemo implements Serializable {
private Map<String,String> map = new HashMap<String,String>();
// only for demo/comparison purposes, default would use compressoin always
private boolean useCompression;
public MapDemo( Map<String,String> map , boolean compressed ) {
this.map = map;
this.useCompression = compressed;
}
// This is the custom serialization using compression
private void writeObject(ObjectOutputStream out) throws IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
OutputStream os = useCompression ? new DeflaterOutputStream( baos ) : baos;
ObjectOutputStream oos = new ObjectOutputStream( os );
oos.writeObject( this.map );
oos.close();
out.write( baos.toByteArray() );
}
}
class Main {
public static void main( String [] args ) throws IOException {
Map<String,String> regular = new HashMap<String,String>();
Map<String,String> compressed = new HashMap<String,String>();
Random r = new Random();
for( int i = 0 ; i < 100000 ; i++ ) {
String key = ""+r.nextInt(1000000);
String value = ""+r.nextInt(1000000) ;
// put the same info
compressed.put( key , value );
regular.put( key , value );
}
save( new MapDemo( compressed, true ) , "map.compressed");
save( new MapDemo( regular, false ) , "map.regular");
}
private static void save( Object o, String toFile ) throws IOException {
// This is similar to what RMI serialization would do behind scenes
ObjectOutputStream oos = new ObjectOutputStream( new FileOutputStream(toFile));
oos.writeObject( o );
oos.close();
}
}