Java Serialization - Automatically handling changed fields? - java

UPDATE MAY 20: I should have mentioned that the objects in question do have "serialVersionUID" set (same value in both old & new), but serialization fails before readObject() is called with the following exception:
Exception in thread "main" java.io.InvalidClassException: TestData; incompatible types for field number
I'm also now including an example down below.
I'm working with a large application that sends serialized objects (implements Serializable, not Exernalizable) from client to server. Unfortunately, we now have a situation where an existing field has changed type altogether, which breaks the serialization.
The plan is to upgrade the server side first. From there, I have control over the new versions of the serialized objects, as well as the ObjectInputStream to read the (at first old) objects coming from the clients.
I had at first thought of implementing readObject() in the new version; however, after trying it out, I discovered that validation fails (due to incompatible types) prior to the method ever being called.
If I subclass ObjectInputStream, can I accomplish what I want?
Even better, are there any 3rd-party libraries that do any sort of serialization 'magic'? It would be really interesting if there were any tools/libraries that could convert a serialized object stream into something like an array of HashMaps...without needing to load the objects themselves. I'm not sure if it is possible to do that (convert a serialized object to a HashMap without loading the object definition itself), but if it is possible, I could imagine a tool that could convert a serialized object (or stream of objects) to a new version, using, say, a set of Properties for conversion hints/rules, etc...
Thanks for any suggestions.
Update May 20 - example source below - The field 'number' in TestData changes from an 'int' in the old version to a 'Long' in the new version. Note readObject() in new version of TestData is not called, because an exception is thrown before it ever gets there:
Exception in thread "main" java.io.InvalidClassException: TestData; incompatible types for field number
Below is the source. Save it to a folder, then create sub-folders 'old' and 'new'. Put the 'old' version of the TestData class in the "old" folder, and the new version in "new". Put the "WriteIt" and "ReadIt" classes in the main (parent of old/new) folder. 'cd' to the 'old' folder, and compile with: javac -g -classpath . TestData.java
...then do the same thing in the 'new' folder. 'cd' back to the parent folder, and compile WriteIt/ReadIt:
javac -g -classpath .;old WriteIt.java
javac -g -classpath .;new ReadIt.java
Then run with:
java -classpath .;old WriteIt //Serialize old version of TestData to TestData_old.ser
java -classpath .;new ReadIt //Attempt to read back the old object using reference to new version
[OLD version]TestData.java
import java.io.*;
public class TestData
implements java.io.Serializable
{
private static final long serialVersionUID = 2L;
public int number;
}
[NEW version] TestData.java
import java.io.*;
public class TestData
implements java.io.Serializable
{
private static final long serialVersionUID = 2L;
public Long number; //Changed from int to Long
private void readObject(final ObjectInputStream ois)
throws IOException, ClassNotFoundException
{
System.out.println("[TestData NEW] readObject() called...");
/* This is where I would, in theory, figure out how to handle
* both the old and new type. But a serialization exception is
* thrown before this method is ever called.
*/
}
}
WriteIt.java - Serialize old version of TestData to 'TestData_old.ser'
import java.io.*;
public class WriteIt
{
public static void main(String args[])
throws Exception
{
TestData d = new TestData();
d.number = 2013;
ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream("TestData_old.ser"));
oos.writeObject(d);
oos.close();
System.out.println("Done!");
}
}
ReadIt.java - Attempt to de-serialized old object into new version of TestData. readObject() in new version is not called, due to exception beforehand.
import java.io.*;
public class ReadIt
{
public static void main(String args[])
throws Exception
{
ObjectInputStream ois = new ObjectInputStream(new FileInputStream("TestData_old.ser"));
TestData d = (TestData)ois.readObject();
ois.close();
System.out.println("Number = " + d.number);
}
}

Your immediate problem seems to be that you have not defined fixed the serialVersionUID strings for your classes. If you don't do that, the object serialization and deserialization code generates the UIDs based on the representational structure of the types being sent and received. If you change the type at one end and not the other, the UIDs don't match. If you fix the UIDs, then the reader code will get past the UID check, and your readObject method will have a chance to "work around" the differences in serialized data.
Apart from that, my suggestion would be:
Try to change all of the clients and servers at the same time so that you don't need to use readObject / writeObject hacks
If that's not practical, try to move away from using Java serialized objects in anything where there is a possibility of version matches as your system evolves. Use XML, JSON, or something else that is less sensitive to "protocol" or "schema" changes.
If that's not practical, version your client/server protocol.
I doubt that you will get any traction by subclassing the ObjectStream classes. (Take a look at the code ... in the OpenJDK codebase.)

You can continue to use serialization even when changes to classes will make the old serialized data incompatible with the new class, if you implement Externalizable and write an extra field indicating the version before writing the class data. This allows readExternal to handle old versions in the way that you specify. I know of no way to automatically do this, but using this manual method may work for you.
Here are the modified classes that will compile and will run without throwing an exception.
old/TestData.java
import java.io.*;
public class TestData
implements Externalizable
{
private static final long serialVersionUID = 1L;
private static final long version = 1L;
public int number;
public void readExternal(ObjectInput in)
throws IOException, ClassNotFoundException {
long version = (long) in.readLong();
if (version == 1) {
number = in.readInt();
}
}
public void writeExternal(ObjectOutput out)
throws IOException {
out.writeLong(version); // always write current version as first field
out.writeInt(number);
}
}
new/TestData.java
import java.io.*;
public class TestData
implements Externalizable
{
private static final long serialVersionUID = 1L;
private static final long version = 2L; // Changed
public Long number; //Changed from int to Long
public void readExternal(ObjectInput in)
throws IOException, ClassNotFoundException {
long version = (long) in.readLong();
if (version == 1) {
number = new Long(in.readInt());
}
if (version == 2) {
number = (Long) in.readObject();
}
}
public void writeExternal(ObjectOutput out)
throws IOException {
out.writeLong(version); // always write current version as first field
out.writeObject(number);
}
}
These classes can be run with the following statements
$ javac -g -classpath .:old ReadIt.java
$ javac -g -classpath .:old WriteIt.java
$ java -classpath .:old WriteIt
Done!
$ java -classpath .:old ReadIt
Number = 2013
$ javac -g -classpath .:new ReadIt.java
$ java -classpath .:new ReadIt
Number = 2013
Changing a class to implement Exernalizable instead of Serializable will result in the following exception.
Exception in thread "main" java.io.InvalidClassException: TestData; Serializable incompatible with Externalizable
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:634)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at ReadIt.main(ReadIt.java:10)
In your case, you'd have to change the old version to implement Externalizable first, restart the server and clients at the same time, then you could upgrade the server before the clients with incompatible changes.

after trying it out, I discovered that validation fails (due to incompatible types) prior to the method ever being called
That's because you haven't defined a serialVersionUID. That's easy to fix. Just run the serialver tool on the old version of the classes, then paste the output into the new source code. You may then be able to write your readObject() method in a compatible way.
In addition, you could look into writing readResolve() and writeReplace() methods, or defining serialPersistentFields in a compatible way, if that's possible. See the Object Serialization Specification.
See also the valuable suggestions by #StephenC.
Post your edit I suggest you change the name of the variable along with its type. That counts as a deletion plus an insertion, which is serialization-compatible, and will work find as long as you don't want the old int number read into the new Long number (why Long instead of long?). If you do need that, I suggest leaving int number there and adding a new Long/long field with a different name, and modifying readObject() to set the new field to 'number' if it isn't already set by defaultReadObject().

If you change the type of a field, you should change it's name. Otherwise you will get the error you see. If you want to set the new field name from the old data (or compute it some other way during object read), you can do something like this:
public class TestData
implements java.io.Serializable
{
private static final long serialVersionUID = 2L;
public Long numberL; //Changed from "int number;"
private void readObject(final ObjectInputStream ois)
throws IOException, ClassNotFoundException
{
System.out.println("[TestData NEW] readObject() called...");
ois.defaultReadObject();
if (numberL == null) {
// deal with old structure
GetField fields = ois.readFields();
numberL = fields.get("number", 0L); // autoboxes old value
}
}
}
Do not change the value of serialVersionUID and the above should be able to read both new and old versions of TestData.

Related

How to get the MethodInfo of a Java 14 method reference?

I'm essentially asking the same as this old question, but for Java 14 instead of Java 8. To spare answerers the trouble of navigating to the old question, I'll rephrase it here.
I want to get the name of a function from a referenced method. The following Java code should give you the idea:
public class Main
{
public static void main(String[] args)
{
printMethodName(Main::main);
}
private static void printMethodName(Consumer<String[]> theFunc)
{
String funcName = // somehow get name from theFunc
System.out.println(funcName)
}
}
The equivalent in C# would be:
public class Main
{
public static void Main()
{
var method = Main.Main;
PrintMethodName(method)
}
private static void PrintMethodName(Action action)
{
Console.WriteLine(action.GetMethodInfo().Name);
}
}
According to the accepted answer of the old question, this was not possible in Java 8 without considerable work, such as this solution. Is there a more elegant solution in Java 14?
Getting a method info from a method reference never was a goal on the JDK developer’s side, so no effort was made to change the situation.
However, the approach shown in your link can be simplified. Instead of serializing the information, patching the serialized data, and restoring the information using a replacement object, you can simply intercept the original SerializedLambda object while serializing.
E.g.
public class GetSerializedLambda extends ObjectOutputStream {
public static void main(String[] args) { // example case
var lambda = (Consumer<String[]>&Serializable)GetSerializedLambda::main;
SerializedLambda sl = GetSerializedLambda.get(lambda);
System.out.println(sl.getImplClass() + " " + sl.getImplMethodName());
}
private SerializedLambda info;
GetSerializedLambda() throws IOException {
super(OutputStream.nullOutputStream());
super.enableReplaceObject(true);
}
#Override protected Object replaceObject(Object obj) throws IOException {
if(obj instanceof SerializedLambda) {
info = (SerializedLambda)obj;
obj = null;
}
return obj;
}
public static SerializedLambda get(Object obj) {
try {
GetSerializedLambda getter = new GetSerializedLambda();
getter.writeObject(obj);
return getter.info;
} catch(IOException ex) {
throw new IllegalArgumentException("not a serializable lambda", ex);
}
}
}
which will print GetSerializedLambda main. The only newer feature used here, is the OutputStream.nullOutputStream() to drop the written information immediately. Prior to JDK 11, you could write into a ByteArrayOutputStream and drop the information after the operation which is only slightly less efficient. The example also using var, but this is irrelevant to the actual operation of getting the method information.
The limitations are the same as in JDK 8. It requires a serializable method reference. Further, there is no guaranty that the implementation will map to a method directly. E.g., if you change the example’s declaration to public static void main(String... args), it will print something like lambda$1 when being compiled with Eclipse. When also changing the next line to var lambda = (Consumer<String>&Serializable)GetSerializedLambda::main;, the code will always print a synthetic method name, as using a helper method is unavoidable. But in case of javac, the name is rather something like lambda$main$f23f6912$1 instead of Eclipse’s lambda$1.
In other words, you can expect encountering surprising implementation details. Do not write applications relying on the availability of such information.

Efficiently get caller in byte buddy's MethodDelegation

I am trying to build a call tree in my java agent with byte buddy library. To add elements to the tree I want to use method delegation. However, to make sure who is the parent of any leaf, I need to know who called the method.
I don't want to use:
sun.reflect.Reflection#getCallerClass(int)
Because it's deprecated and unavailable in Java 8+.
Also, I tried:
public class ThreadUtil {
public static StackTraceElement getCaller() {
Instant now = Instant.now();
StackTraceElement ste = Thread.currentThread().getStackTrace()[3];
String callerClass = ste.getClassName();
String callerMethod = ste.getMethodName();
Instant now2= Instant.now();
System.out.println(Duration.between(now, now2));
return ste;
}
}
But, it's extremely slow(~1 ms - too much if I have thousands of calls).
Is there a way to get caller efficiently at this point (maybe some byte buddy's tricks)?
P.S.
My agent:
private static void instrument(String agentOps, Instrumentation inst) {
System.out.println("Agent");
new AgentBuilder.Default().with(new Eager())
.ignore(ElementMatchers.nameContains("com.dvelopp.agenttest"))
.type((ElementMatchers.any()))
.transform((builder, typeDescription, classLoader, module) -> builder.method(ElementMatchers.any())
.intercept(MethodDelegation.to(Interceptor.class))).installOn(inst);
}
public static class Interceptor {
#RuntimeType
public static Object intercept(#SuperCall Callable<?> zuper, #Origin Method method,
#AllArguments Object[] args, #This(optional = true) Object me) throws Exception {
System.out.println("CURRENT: " + method.getDeclaringClass().getName());
System.out.println("CALLER: " + ThreadUtil.getCaller().getClassName());
return zuper.call();
}
}
ENV: Java 8
Byte code instrumentation only allows you to generate code that you could also write yourself. For your case, you would need to create a fairly intrusive instrumentation that I would not recommend:
Instrument your target method to accept a new parameter of type Class.
Instrument every caller to supply their type as an additional argument.
The better solution is surely what Holger suggested in the comments. Use StackWalker and if not available, fall back to sun.reflect.Reflection (which is present in all JVMs I know of).

using BerkeleyDB to replace java.util.List

does someone maybe have sample code how to replace a java List (LinkedList or ArrayList) with something similar in BerkeleyDB? My problem is that I have to replace Lists to scale beyond main memory limits. Some simple sample code would be really nice.
I've now used a simple TupleBinding for Integers (keys) and a SerialBinding for the Diff-class (data values).
Now I'm receiving the Error:
14:03:29.287 [pool-5-thread-1] ERROR o.t.g.view.model.TraverseCompareTree - org.treetank.diff.Diff; local class incompatible: stream classdesc serialVersionUID = 8484615870884317488, local class serialVersionUID = -8805161170968505227
java.io.InvalidClassException: org.treetank.diff.Diff; local class incompatible: stream classdesc serialVersionUID = 8484615870884317488, local class serialVersionUID = -8805161170968505227
The listener and TransactionRunner classes which I'm using are:
/** {#inheritDoc} */
#Override
public void diffListener(final EDiff paramDiff, final IStructuralItem paramNewNode,
final IStructuralItem paramOldNode, final DiffDepth paramDepth) {
try {
mRunner.run(new PopulateDatabase(mDiffDatabase, mKey++, new Diff(paramDiff, paramNewNode.getNodeKey(), paramOldNode.getNodeKey(), paramDepth)));
} catch (final Exception e) {
LOGWRAPPER.error(e.getMessage(), e);
}
}
private static class PopulateDatabase implements TransactionWorker {
private StoredMap<Integer, Diff> mMap;
private int mKey;
private Diff mValue;
public PopulateDatabase(final DiffDatabase paramDatabase, final int paramKey, final Diff paramValue) {
Objects.requireNonNull(paramDatabase);
Objects.requireNonNull(paramValue);
mMap = paramDatabase.getMap();
mKey = paramKey;
mValue = paramValue;
}
#Override
public void doWork() throws DatabaseException {
mMap.put(mKey, mValue);
}
}
I don't know why it doesn't work :-/
Edit: Sorry, I just had to delete the generated environment/database and create a new one.
I'm afraid, it wont be that simple. A first step, you might want to take is to refactor your code to move all accesses to the list into a separate class (call it a DAO, if you like). Then it will be a lot easier to move to a database instead of the list.
Berkeley DB is severe over-kill for this type of task. It's a fair beast to configure and set up, plus I believe the license is now commercial. You'll be much better off using a disk-backed list or map. As an example of the latter take a look at Kyoto Cabinet. It's extremely fast, implements the standard Java Collections interface and is as easy to use as a List or Map. See my other answer for example code.

Serialize an Object Array to sent over Sockets

I have an array that I have created from a database ResultSet. I am trying to Serialize it so that I can send it over a socket stream. At the moment I am getting an error telling me that the array is not Serializable. The code I have is down below, the first part is the class to create an object for the array:
class ProteinData
{
private int ProteinKey;
public ProteinData(Integer ProteinKey)
{
this.ProteinKey = ProteinKey;
}
public Integer getProteinKey() {
return this.ProteinKey;
}
public void setProteinKey(Integer ProteinKey) {
this.ProteinKey = ProteinKey;
}
}
The code to populate the array:
public List<ProteinData> readJavaObject(String query, Connection con) throws Exception
{
PreparedStatement stmt = con.prepareStatement(query);
query_results = stmt.executeQuery();
while (query_results.next())
{
ProteinData pro = new ProteinData();
pro.setProteinKey(query_results.getInt("ProteinKey"));
tableData.add(pro);
}
query_results.close();
stmt.close();
return tableData;
}
And the code to call this is:
List dataList = (List) this.readJavaObject(query, con);
ObjectOutputStream output_stream = new ObjectOutputStream(socket.getOutputStream());
output_stream.writeObject(dataList);
And the code recieving this is:
List dataList = (List) input_stream.readObject();
Can someone help me serailize this array. All I can find in forums is simple arrays(EG. int[]).
I tried to add the serializable to the class and the UID number but got java.lang.ClassNotFoundException: socketserver.ProteinData error message. Does anyone now why?
Thanks for any help.
Basically you need that the classes you want to serialize are implementing Serializable. And if you want to avoid the warning related to the serial you should have also a long serialVersionUIDfor each one, that is a code used to distinguish your specific version of the class. Read a tutorial like this one to get additional info, serialization is not so hard to handle..
However remember that serialization is faulty when used between two different versions of the JVM (and it has some flaws in general).
Just a side note: the interface Serializabledoesn't actually give any required feature to the class itself (it's not a typical interface) and it is used just to distinguish between classes that are supposed to be sent over streams and all the others. Of course, if a class is Serializable, all the component it uses (instance variables) must be serializable too to be able to send the whole object.
Change your class declaration to:
class ProteinData implements Serializable {
...
}
I would have thought as a minimum that you would need
class ProteinData implements Serializable
and a
private static final long serialVersionUID = 1234556L;
(Eclipse will generate the magic number for you).
in the class.

Make Java runtime ignore serialVersionUIDs?

I have to work with a large number of compiled Java classes which didn't explicitly specify a serialVersionUID. Because their UIDs were arbitrarily generated by the compiler, many of the classes which need to be serialized and deserialized end up causing exceptions, even though the actual class definitions match up. (This is all expected behavior, of course.)
It is impractical for me to go back and fix all of this 3rd-party code.
Therefore, my question is: Is there any way to make the Java runtime ignore differences in serialVersionUIDs, and only fail to deserialize when there are actual differences in structure?
If you have access to the code base, you could use the SerialVer task for Ant to insert and to modify the serialVersionUID in the source code of a serializable class and fix the problem once for all.
If you can't, or if this is not an option (e.g. if you have already serialized some objects that you need to deserialize), one solution would be to extend ObjectInputStream. Augment its behavior to compare the serialVersionUID of the stream descriptor with the serialVersionUID of the class in the local JVM that this descriptor represents and to use the local class descriptor in case of mismatch. Then, just use this custom class for the deserialization. Something like this (credits to this message):
import java.io.IOException;
import java.io.InputStream;
import java.io.InvalidClassException;
import java.io.ObjectInputStream;
import java.io.ObjectStreamClass;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class DecompressibleInputStream extends ObjectInputStream {
private static Logger logger = LoggerFactory.getLogger(DecompressibleInputStream.class);
public DecompressibleInputStream(InputStream in) throws IOException {
super(in);
}
#Override
protected ObjectStreamClass readClassDescriptor() throws IOException, ClassNotFoundException {
ObjectStreamClass resultClassDescriptor = super.readClassDescriptor(); // initially streams descriptor
Class localClass; // the class in the local JVM that this descriptor represents.
try {
localClass = Class.forName(resultClassDescriptor.getName());
} catch (ClassNotFoundException e) {
logger.error("No local class for " + resultClassDescriptor.getName(), e);
return resultClassDescriptor;
}
ObjectStreamClass localClassDescriptor = ObjectStreamClass.lookup(localClass);
if (localClassDescriptor != null) { // only if class implements serializable
final long localSUID = localClassDescriptor.getSerialVersionUID();
final long streamSUID = resultClassDescriptor.getSerialVersionUID();
if (streamSUID != localSUID) { // check for serialVersionUID mismatch.
final StringBuffer s = new StringBuffer("Overriding serialized class version mismatch: ");
s.append("local serialVersionUID = ").append(localSUID);
s.append(" stream serialVersionUID = ").append(streamSUID);
Exception e = new InvalidClassException(s.toString());
logger.error("Potentially Fatal Deserialization Operation.", e);
resultClassDescriptor = localClassDescriptor; // Use local class descriptor for deserialization
}
}
return resultClassDescriptor;
}
}
Use CGLIB to insert them into the binary classes?
How impractical is this to fix ? If you have the source and can rebuild, can you not just run a script over the entire codebase to insert a
private long serialVersionUID = 1L;
everywhere ?
The serialization errors at runtime tell you explicitly what the ID is expected to be. Just change your classes to declare these as the ID and everything will be OK. This does involve you making changes but I don't believe that this can be avoided
You could possibly use Aspectj to 'introduce' the field into each serializable class as it is loaded. I would first introduce a marker interface into each class using the package and then introduce the field using a Hash of the class file for the serialVersionUID
public aspect SerializationIntroducerAspect {
// introduce marker into each class in the org.simple package
declare parents: (org.simple.*) implements SerialIdIntroduced;
public interface SerialIdIntroduced{}
// add the field to each class marked with the interface above.
private long SerialIdIntroduced.serialVersionUID = createIdFromHash();
private long SerialIdIntroduced.createIdFromHash()
{
if(serialVersionUID == 0)
{
serialVersionUID = getClass().hashCode();
}
return serialVersionUID;
}
}
You will need to add the aspectj load time weaver agent to the VM so it can weave the advice into your existing 3rd party classes. Its funny though once you get around to setting Aspectj up, its remarkable the number of uses that you will put to.
HTH
ste

Categories