Context
I'm doing my student project and building a testing tool for regression testing.
Main idea: capture all constructors/methods/functions invocations using AOP during runtime and record all data into a database. Later retrieve the data, run constructors/methods/functions in the same order, and compare return values.
Problem
I'm trying to serialize objects (and arrays of objects) into a byte array, record it into PostgreSQL as a blob, and later (in another runtime) retrieve that blob and deserialize it back to object. But when I deserialize data in another runtime it changes and, for example, instead of boolean, I retrieve int. If I do exactly the same operations in the same runtime (serialize - insert into the database - SELECT from the database - deserialize) everything seems to work correctly.
Here is how I record data:
private void writeInvocationRecords(InvocationData invocationData, boolean isConstructor) {
final List<InvocationData> invocationRecords = isConstructor ? constructorInvocationRecords : methodInvocationRecords;
final String recordsFileName = isConstructor ? "constructor_invocation_records.json" : "method_invocation_records.json";
byte[] inputArgsBytes = null;
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutputStream out = null;
try {
out = new ObjectOutputStream(bos);
out.writeObject(invocationData.inputArgs);
out.flush();
inputArgsBytes = bos.toByteArray();
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
bos.close();
} catch (IOException ex) {
// ignore close exception
}
}
byte[] returnValueBytes = null;
ByteArrayOutputStream rvBos = new ByteArrayOutputStream();
ObjectOutputStream rvOut = null;
try {
rvOut = new ObjectOutputStream(rvBos);
rvOut.writeObject(invocationData.returnValue);
rvOut.flush();
returnValueBytes = rvBos.toByteArray();
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
rvBos.close();
} catch (IOException ex) {
// ignore close exception
}
}
invocationRecords.add(invocationData);
if (invocationRecords.size() >= (isConstructor ? CONSTRUCTORS_CACHE_SIZE : METHODS_CACHE_SIZE)) {
List<InvocationData> tempRecords = new ArrayList<InvocationData>(invocationRecords);
invocationRecords.clear();
try {
for (InvocationData record : tempRecords) {
SerialBlob blob = new javax.sql.rowset.serial.SerialBlob(inputArgsBytes);
SerialBlob rvBlob = new javax.sql.rowset.serial.SerialBlob(returnValueBytes);
psInsert.setString(1, record.className);
psInsert.setString(2, record.methodName);
psInsert.setArray(3, conn.createArrayOf("text", record.inputArgsTypes));
psInsert.setBinaryStream(4, blob.getBinaryStream());
psInsert.setString(5, record.returnValueType);
psInsert.setBinaryStream(6, rvBlob.getBinaryStream());
psInsert.setLong(7, record.invocationTimeStamp);
psInsert.setLong(8, record.invocationTime);
psInsert.setLong(9, record.orderId);
psInsert.setLong(10, record.threadId);
psInsert.setString(11, record.threadName);
psInsert.setInt(12, record.objectHashCode);
psInsert.setBoolean(13, isConstructor);
psInsert.executeUpdate();
}
conn.commit();
} catch (Exception e) {
e.printStackTrace();
}
}
}
Here is how I retrieve data:
List<InvocationData> constructorsData = new LinkedList<InvocationData>();
List<InvocationData> methodsData = new LinkedList<InvocationData>();
Statement st = conn.createStatement();
ResultSet rs = st.executeQuery(SQL_SELECT);
while (rs.next()) {
Object returnValue = new Object();
byte[] returnValueByteArray = new byte[rs.getBinaryStream(7).available()];
returnValueByteArray = rs.getBytes(7);
final String returnType = rs.getString(6);
ByteArrayInputStream rvBis = new ByteArrayInputStream(returnValueByteArray);
ObjectInputStream rvIn = null;
try {
rvIn = new ObjectInputStream(rvBis);
switch (returnType) {
case "boolean":
returnValue = rvIn.readBoolean();
break;
case "double":
returnValue = rvIn.readDouble();
break;
case "int":
returnValue = rvIn.readInt();
break;
case "long":
returnValue = rvIn.readLong();
break;
case "char":
returnValue = rvIn.readChar();
break;
case "float":
returnValue = rvIn.readFloat();
break;
case "short":
returnValue = rvIn.readShort();
break;
default:
returnValue = rvIn.readObject();
break;
}
rvIn.close();
rvBis.close();
} catch (IOException e) {
e.printStackTrace();
} catch (ClassNotFoundException e) {
e.printStackTrace();
} finally {
try {
if (rvIn != null) {
rvIn.close();
}
} catch (IOException ex) {
// ignore close exception
}
}
Object[] inputArguments = new Object[0];
byte[] inputArgsByteArray = new byte[rs.getBinaryStream(5).available()];
rs.getBinaryStream(5).read(inputArgsByteArray);
ByteArrayInputStream bis = new ByteArrayInputStream(inputArgsByteArray);
ObjectInput in = null;
try {
in = new ObjectInputStream(bis);
inputArguments = (Object[])in.readObject();
} catch (IOException e) {
e.printStackTrace();
} catch (ClassNotFoundException e) {
e.printStackTrace();
} finally {
try {
if (in != null) {
in.close();
}
} catch (IOException ex) {
// ignore close exception
}
}
InvocationData invocationData = new InvocationData(
rs.getString(2),
rs.getString(3),
(String[])rs.getArray(4).getArray(),
inputArguments,
rs.getString(6),
returnValue,
rs.getLong(8),
rs.getLong(9),
rs.getLong(10),
rs.getLong(11),
rs.getString(12),
rs.getInt(13)
);
if (rs.getBoolean(14)) {
constructorsData.add(invocationData);
} else {
methodsData.add(invocationData);
}
}
st.close();
rs.close();
conn.close();
An explosion of errors and misguided ideas inherent in this question:
Your read and write code is broken.
available() doesn't work. Well, it does what the javadoc says it does, and if you read the javadoc, and read it very carefully, you should come to the correct conclusion that what that is, is utterly useless. If you ever call available(), you've messed up. You're doing so here. More generally your read and write code doesn't work. For example, .read(byteArr) also doesn't do what you think it does. See below.
The entire principle behind what you're attempting to do, doesn't work
You can't 'save the state' of arbitrary objects, and if you want to push the idea, then if you can, then certainly not in the way you're doing it, and in general this is advanced java that involves hacking the JDK itself to get at it: Think of an InputStream that represents data flowing over a network connection. What do you imagine the 'serialization' of this InputStream object should look like? If you consider serialization as 'just represent the underlying data in memory', then what you'd get is a number that represents the OS 'pipe handle', and possibly some IP, port, and sequence numbers. This is a tiny amount of data, and all this data is completely useless - it doesn't say anything meaningful about that connection and this data cannot be used to reconstitute it, at all. Even within the 'scope' of a single session (i.e. where you serialize, and then deserialize almost immediately afterwards), as networks are a stream and once you grab a byte (or send a byte), it's gone. The only useful, especially for the notion of 'lets replay everything that happened as a test', serialization strategy involves actually 'recording' all the bytes that were picked up, as it happens, on the fly. This is not a thing that you can do as a 'moment in time' concept, it's continuous. You need a system that is recording all the things (it needs to be recording every inputstream, every outputstream, every time System.currentTimeMillis() in invoked, every time a random number is generated, etc), and then needs to use the results of recording it all when your API is asked to 'save' an arbitrary state.
Serialization instead is a thing that objects need to opt into, and where they may have to write custom code to properly deal with it. Not all objects can even be serialized (an InputStream representing a network pipe, as above, is one example of an object that cannot be serialized), and for some, serializing them requires some fancy footwork, and the only hope you have is that the authors of the code that powers this object put in that effort. If they didn't, there is nothing you can do.
The serialization framework of java awkwardly captures both of these notions. It does mean that your code, even if you fix the bugs in it, will fail on most objects that can exist in a JVM. Your testing tool can only be used to test the most simplistic code.
If you're okay with that, read on. But if not, you need to completely rethink what you're going to do with this.
ObjectOutputStream sucks
This is not just my opinion, the openjdk team itself is broadly in agreement (they probably wouldn't quite put it like that, of course). The data emitted by OOS is a weird, inefficient, and underspecced binary blob. You can't analyse this data in any feasible way other than spending a few years reverse engineering the protocol, or just deserializing it (which requires having all the classes, and a JVM - this can be an acceptable burden, depends on your use case).
Contrast to e.g. Jackson which serializes data into JSON, which you can parse with your eyeballs, or in any language, and even without the relevant class files. You can construct 'serialized JSON' yourself without the benefit of first having an object (for testing purposes this sounds like a good idea, no? You need to test this testing framework too!).
How do I fix this code?
If you understand all the caveats above and somehow still conclude that this project, as written and continuing to use the ObjectOutputStream API is still what you want to do (I really, really doubt that's the right call):
Use the newer APIs. available() does not return the size of that blob. read(someByteArray) is not guaranteed to fill the entire byte array. Just read the javadoc, it spells it out.
There is no way to determine the size of an inputstream by asking that inputstream. You may be able to ask the DB itself (usually, LENGTH(theBlobColumn) works great in a SELECT query.
If you somehow (e.g. using LENGTH(tbc)) know the full size, you can use InputStream's readFully method, which will actually read all bytes, vs. read, which reads at least 1, but is not guaranteed to read all of it. The idea is: It'll read the smallest chunk that is available. Imagine a network pipe where bytes are dribbling into the network card's buffer, one byte a second. If so far 250 bytes have dribbled in and you call .read(some500SizeByteArr), then you get 250 bytes (250 of the 500 bytes are filled in, and 250 is returned). If you call .readFully(some500SizeByteArr), then the code will wait about 250 seconds, and then returns 500, and fills in all 500 bytes. That's the difference, and that explains why read works the way it does. Said differently: If you do not check what read() is returning, your code is definitely broken.
If you do not know how much data there is, your only option involves a while loop, or to call a helper method that does that. You need to make a temporary byte array, then in a loop keep calling read until it returns -1. For every loop, take the bytes in that array from 0 to (whatever the read call returned), and send these bytes someplace else. For example, a ByteArrayOutputStream.
Class matching
when I deserialize data in another runtime it changes and, for example, instead of boolean, I retrieve int
The java serialization system isn't magically changing your stuff on you. Well, put a pin that. Most likely the class file available in the first run (where you saved the blob in the db) was different vs what it looked like in your second run. Voila, problem.
More generally this is a problem in serialization. If you serialize, say, class Person {Date dob; String name;}, and then in a later version of the software you realize that using a j.u.Date to store a date of birth is a very silly idea, as Date is an unfortunately named class (it represents an instant in time and not a date at all), so you replace it with a LocalDate instead, thus ending up with class Person{LocalDate dob; String name;}, then how do you deal with the problem that you now want to deserialize a BLOB that was made back when the Person.class file still had the broken Date dob; field?
The answer is: You can't. Java's baked in serialization mechanism will flat out throw an exception here, it will not try to do this. This is the serialVersionUID system: Classes have an ID and changing anything about them (such as that field) changes this ID; the ID is stored in the serialized data. If the IDs don't match, deserialization cannot be done. You can force the ID (make a field called serialVersionUID - you can search the web for how to do that), but then you'd still get an error, java's deserializer will attempt to deserialize a Date object into a LocalDate dob; field and will of course fail.
Classes can write their own code to solve this problem. This is non-trivial and is irrelevant to you, as you're building a framework and presumably can't pop in and write code for your testing framework's userbase's custom class files.
I told you to put a pin in 'the serialization mechanism isnt going to magically change types on you'. Put in sufficient effort with overriding serialVersionUID and such and you can end up there. But that'd be because you wrote code that confuses types, e.g. in your readObject implementation (again, search the web for java's serialization mechanism, readObject/writeObject - or just start reading the javadoc of java.io.Serializable, that's a good starting-off point).
Style issues
You create objects for no purpose, you seem to have some trouble with the distinction between a variable/reference and an object. You aren't using try-with-resources. The way your SELECT calls are made suggests you have an SQL injection security issue. e.printStackTrace() as line line in a catch block is always incorrect.
I have two classes that interact:
Shelf, that stores CDs, DVDs, PaperMedias:
public class Shelf {
private ArrayList<CD> cds;
private ArrayList<DVD> dvds;
private ArrayList<PaperMedia> paperMedias;
...etc
And Customer :
public class Customer implements Serializable {
private Map<PhysicalMedia, Calendar> mediaOwned;
private Map<PhysicalMedia,Calendar> mediaReturned;
private Map<PhysicalMedia,CalendarPeriod> mediaOnHold;
...etc
Physical media is a parent of CD,DVD,PaperMedia.
First I will initialize shelf with some items, and customer to have a few of these items borrowed. Then I save these objects to ShelfObjects.txt and CustomerObjects.txt.
When I read these objects once again from these files, it seems like link between these two is lost , specifically between PhysicalMedia of customer and PhysicalMedia of shelf. These should definitely have the same reference for example Metallica CD should have same reference in Shelf as in Customer's account.
So when i change say a status of Metallica CD, it wont update it in the other source!
Is there a was to preserve this reference ?
I load and save media in the following way in CustomerDatabase class:
public class CustomersDatabase implements Serializable {
private ArrayList<Customer> customers = new ArrayList<Customer>();
//etc..
public void load() {
try {
ObjectInputStream in = new ObjectInputStream(new FileInputStream("CustomersObject.txt"));
try {
customers.clear();
while(true) {
customers.add((Customer)in.readObject());
}
} catch (ClassNotFoundException e) {
System.out.println("Customer class in file wasn't found");
}
in.close();
} catch (FileNotFoundException e) {
System.out.println("File not found");
e.printStackTrace();
} catch (IOException e) {
System.out.println("\n^ Customer load successful\n");
}
public void save() {
try {
ObjectOutputStream out = new ObjectOutputStream(new FileOutputStream("CustomersObject.txt",false));
for (int i=0;i<customers.size();i++) {
out.writeObject(customers.get(i));
}
out.close();
System.out.println("\nCustomer save successful\n");
} catch (FileNotFoundException e) {
System.out.println("File not found");
e.printStackTrace();
} catch (IOException e) {
System.out.println("IO exception ");
e.printStackTrace();
}
}
I do similar load and store in Shelf class.
The problem is that you are storing your two databases using two separate files using two separate object serialization streams.
The reason that your current approach doesn't work for you is that the Object Serialization protocol stores object references as simple numbers. It numbers the objects in a stream as 1, 2, 3, and so on, and uses those numbers to refer to previously serialized objects in the stream.
This works within the context of one stream. But when you have two or more streams, a given object is likely to have different numbers in each stream. That means that there is no way for the readObject method to know that two objects in two different streams should actually be linked together.
The simple solution is to do is something like this:
ObjectInputStream in = new ObjectInputStream(new FileInputStream("DB.txt"));
for (int i = 0; i < customers.size(); i++) {
out.writeObject(customers.get(i));
}
for (int i = 0; i < shelves.size(); i++) {
out.writeObject(shelves.get(i));
}
Or better still, serialize the customers and shelves lists directly, because that will simplify the deserialization.
(Yes ... this messes up you current code structure. You would have to load and save the Customer and Shelf databases in the same place, or pass an open stream around the place.)
It is also possible to store the objects in two separate files, but you need to use custom serialization methods to do it.
First you need to decide which file you want each shared child object to go into; i.e. do the PhysicalMedia objects get stored in the Customer database or the Shelf database?
For each shared object class, define an identifier field of an appropriate type; e.g. String or `long. Modify your code-base so that the field is populated with values that are unique in the context of the application.
For the database where you want to objects to live, serialize as normal.
For the other databases, use Object Serialization advanced functionality to:
on write, replace the objects with (just) their identifiers, and
on read, lookup the identifiers in the first database and then insert the object looked up into the deserialized object.
The last will require deep knowledge of the Object Serialization APIs, custom serialization, and so on. In your example, you have the added complexity that the objects you want to substitute are keys in Map objects. Since you can't add custom readObject and writeObject for a standard map class, you will need to do deep tricks in custom subclasses of the Stream classes themselves. (And ... no, I can't give you an example!)
Aside: this kind of thing is one of the reasons why it is inadvisable to use object serialization to implement a database. It is better to use a real database, and ORM technology such as JPA (e.g. Hibernate). There are other reasons too.
I would like to retrieve multiple hashmap values with only specified fields. So I opted-in to Redis pipeline.
While testing the below code, i see redisResponse1 is always null, where as redisResponse2 has value.
getRedisTemplate().executePipelined(new RedisCallback<Object>() {
#Override
public Object doInRedis(RedisConnection connection) throws DataAccessException {
List<byte[]> redisResponse1 = connection.hMGet(key.getBytes(), params);
List<byte[]> redisResponse2 = getRedisTemplate().getConnectionFactory().getConnection().hMGet(key.getBytes(), specificParams);
return null;
}
});
When I look into the code and found that below, where
a) redisResponse2 is not executed with pipeline option
b) redisResponse1 is executed with pipeline (isPipelined() == true) but returning always null.
public List<byte[]> hMGet(byte[] key, byte[]... fields) {
try {
if (isPipelined()) {
pipeline(new JedisResult(pipeline.hmget(key, fields)));
return null;
}
if (isQueueing()) {
transaction(new JedisResult(transaction.hmget(key, fields)));
return null;
}
return jedis.hmget(key, fields);
} catch (Exception ex) {
throw convertJedisAccessException(ex);
}
}
So questions are
1) How do I achieve my use case with pipeline option?
2) What is the impact accessing getRedisTemplate().getConnectionFactory().getConnection() within this RedisCallback?
3) How this whole pipeline concept is working? Is it like dynamic Lua? where this Java code is converted as Lua script and send to Redis as script, executed in Redis and come back? Surprised on within this callback; the code is accessing/updating the outer class variables as well, so what will happen to all that variables? All those outer class variables also send to redis in lua?
4) I see many examples about doInRedis API is returning null; Why so? How to return/get valid Object from that?
The majority of your questions are available within the Spring Data Redis reference documentation.
Before digging into Pipelining, a single multi-get from one Hash does not require Pipelining because it's only a single command. Pipelining won't improve performance/stability/… of your Redis interaction.
Pipelining is arranged as callback and intended to issue multiple commands without awaiting the result immediately – think of it as a batch where you get all results later. Because pipelining synchronizes responses at the very end, you don't receive result values within the callback but at the very end, when the pipelining session is synchronized and executePipelined(…) terminates.
Your code should rather look like:
List<Object> results = getRedisTemplate().executePipelined(new RedisCallback<Object>() {
#Override
public Object doInRedis(RedisConnection connection) {
connection.hMGet(key.getBytes(), params);
return null;
}
});
List<Object> hmget = (List<Object>) results.get(0);
You have to use only the connection that you receive as callback argument because the connection has entered pipelining mode. Obtaining a connection from outside the callback (like template.getConnectionFactory().getConnection()) will open a new connection and execute Redis commands with awaiting responses – no pipelining is applied to any external obtained connection.
You can also use methods of RedisTemplate instead of working with the plain connection. executePipelined(…) binds the connection used in the callback to the current Thread and reuses that bound connection if you call template API methods.
Regarding your Lua question: The code/method calls are not transposed to Lua.
Regarding question:
I see many examples about doInRedis API is returning null; Why so? How to return/get valid Object from that?
Here is the answer:
https://docs.spring.io/spring-data/redis/docs/current/reference/html/#pipeline
//pop a specified number of items from a queue
List<Object> results = stringRedisTemplate.executePipelined(
new RedisCallback<Object>() {
public Object doInRedis(RedisConnection connection) throws DataAccessException {
StringRedisConnection stringRedisConn = (StringRedisConnection)connection;
for(int i=0; i< batchSize; i++) {
stringRedisConn.rPop("myqueue");
}
return null;
}
});
Note that the value returned from the RedisCallback is required to be null, as this value is discarded in favor of returning the results of the pipelined commands.
I have some input that I add to a serialized object.
Now when I read the serialized object, I want to check if it exists... If not loop till it has a value in it.
How do i modify the deserialization function to handle that.
There is basically a delay in populating my serializable object. So in the meantime if i were to read that object, it is going to be empty. I want to put a check to read only when it has data in it. if not it should wait till it has some data
public String _displayResults(){
String SomeData = "";
try {
FileInputStream fis = new FileInputStream("SomeDataobj");
ObjectInputStream ois = new ObjectInputStream(fis);
SomeData = (String)ois.readObject();
ois.close();
}
catch(Exception e) {
System.out.println("Exception during deserialization: ");
}
return SomeData;
}
What I tried:
added a wait condition for 2 secs for 10 times... Is there a cleaner way.
while ( ois.readObject().toString().equalsIgnoreCase("") && i <10){
Thread.sleep(2000);
i++;
}
Java provides an API called Externalizable, which allows you to customize the (de) serialization. Serialiazable is marker interface and that indicates the object can be wrote to output stream. Externalizable provides two methods readExternal() and writeExternal() where you can override the behavior.
Your question is not so clear about what you want to achieve, so I am not sure if the above information is helpful for you
this is my code snippet.
data is a 2D array of type Object.I have previously saved data in JTable.Now i have written
code to delete entry.But if go by this code only first entry gets deleted from JTable.
I am unable to understand the reason behind this.
please help me out in this.
-snippet:
public void deleteActionPerformed(ActionEvent ae) {
String delname=tf4.getText();
int c=0;
try {
ObjectInputStream ois=new ObjectInputStream(new FileInputStream("data.txt"));
data=(Object[][])ois.readObject();
for(;;c++) {
String x=(String)data[c][0];
if(x.equals(delname)) {
System.out.println("if working");
data[c][0]=null;
data[c][1]=null;
data[c][2]=null;
try {
ObjectOutputStream oos=new ObjectOutputStream(new FileOutputStream("data.txt"));
oos.writeObject(data);
} catch(Exception exc) {
System.out.println("error deleting data from"+" "+c+" row");
}
c++;
JOptionPane.showMessageDialog(new JFrame(),"contact deleted");
try {
ObjectInputStream oist=new ObjectInputStream(new FileInputStream("data.txt"));
data=(Object[][])oist.readObject();
JTable tb=new JTable(data,headers);
ObjectOutputStream oost=new ObjectOutputStream(new FileOutputStream("contacts.txt"));
oost.writeObject(tb);
} catch(Exception exc) {
System.out.println("error updating after deleting");
}
}
else
System.out.println("else working");
}
} catch(Exception exc) {
System.out.println("error reading data.txt for deleting");
}
}
After properly indenting your code, it becomes more obvious ...
You are re-creating (and re-reading) the file in your loop, instead of after it - why?
You are writing out a new JTable object after every deletion.
Why writing it out at all?
How are you reading them?
Additionally, you have two c++ inside your loop, so the second element of the array is skipped.
Third, in case of exceptions you simply go on (and even don't print out the whole exception).
You can only run this algorithm to delete once before getting a NullPointerException
If in a previous run you deleted any item, you'll get a NPE each time you compare it.
String x=(String)data[c][0];
if(x.equals(delname)) {
It would be safer if you do delname.equals(x) (assuming you have checked that delname is not null before) or check for x == null in the if before equals.
A better solution would be, after the loop, copying the array to a new one without the deleted elements. Even better, after deserializing the object, pass it to a List class and work with it, and only pass it back to array in order to serialize it when you are finished with the program (or directly serialize the List, if you can).
That said, I cannot find anything that would work with the first element but not the others. Maybe it is that when you run your test you first try with the first element and then, after corrupting the array so you'll have a NPE, you test the other elements.
EDIT: Didn't see the extra c++ (probably it was a while before?). Only will work with odd element indexes.