Context
I'm doing my student project and building a testing tool for regression testing.
Main idea: capture all constructors/methods/functions invocations using AOP during runtime and record all data into a database. Later retrieve the data, run constructors/methods/functions in the same order, and compare return values.
Problem
I'm trying to serialize objects (and arrays of objects) into a byte array, record it into PostgreSQL as a blob, and later (in another runtime) retrieve that blob and deserialize it back to object. But when I deserialize data in another runtime it changes and, for example, instead of boolean, I retrieve int. If I do exactly the same operations in the same runtime (serialize - insert into the database - SELECT from the database - deserialize) everything seems to work correctly.
Here is how I record data:
private void writeInvocationRecords(InvocationData invocationData, boolean isConstructor) {
final List<InvocationData> invocationRecords = isConstructor ? constructorInvocationRecords : methodInvocationRecords;
final String recordsFileName = isConstructor ? "constructor_invocation_records.json" : "method_invocation_records.json";
byte[] inputArgsBytes = null;
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutputStream out = null;
try {
out = new ObjectOutputStream(bos);
out.writeObject(invocationData.inputArgs);
out.flush();
inputArgsBytes = bos.toByteArray();
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
bos.close();
} catch (IOException ex) {
// ignore close exception
}
}
byte[] returnValueBytes = null;
ByteArrayOutputStream rvBos = new ByteArrayOutputStream();
ObjectOutputStream rvOut = null;
try {
rvOut = new ObjectOutputStream(rvBos);
rvOut.writeObject(invocationData.returnValue);
rvOut.flush();
returnValueBytes = rvBos.toByteArray();
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
rvBos.close();
} catch (IOException ex) {
// ignore close exception
}
}
invocationRecords.add(invocationData);
if (invocationRecords.size() >= (isConstructor ? CONSTRUCTORS_CACHE_SIZE : METHODS_CACHE_SIZE)) {
List<InvocationData> tempRecords = new ArrayList<InvocationData>(invocationRecords);
invocationRecords.clear();
try {
for (InvocationData record : tempRecords) {
SerialBlob blob = new javax.sql.rowset.serial.SerialBlob(inputArgsBytes);
SerialBlob rvBlob = new javax.sql.rowset.serial.SerialBlob(returnValueBytes);
psInsert.setString(1, record.className);
psInsert.setString(2, record.methodName);
psInsert.setArray(3, conn.createArrayOf("text", record.inputArgsTypes));
psInsert.setBinaryStream(4, blob.getBinaryStream());
psInsert.setString(5, record.returnValueType);
psInsert.setBinaryStream(6, rvBlob.getBinaryStream());
psInsert.setLong(7, record.invocationTimeStamp);
psInsert.setLong(8, record.invocationTime);
psInsert.setLong(9, record.orderId);
psInsert.setLong(10, record.threadId);
psInsert.setString(11, record.threadName);
psInsert.setInt(12, record.objectHashCode);
psInsert.setBoolean(13, isConstructor);
psInsert.executeUpdate();
}
conn.commit();
} catch (Exception e) {
e.printStackTrace();
}
}
}
Here is how I retrieve data:
List<InvocationData> constructorsData = new LinkedList<InvocationData>();
List<InvocationData> methodsData = new LinkedList<InvocationData>();
Statement st = conn.createStatement();
ResultSet rs = st.executeQuery(SQL_SELECT);
while (rs.next()) {
Object returnValue = new Object();
byte[] returnValueByteArray = new byte[rs.getBinaryStream(7).available()];
returnValueByteArray = rs.getBytes(7);
final String returnType = rs.getString(6);
ByteArrayInputStream rvBis = new ByteArrayInputStream(returnValueByteArray);
ObjectInputStream rvIn = null;
try {
rvIn = new ObjectInputStream(rvBis);
switch (returnType) {
case "boolean":
returnValue = rvIn.readBoolean();
break;
case "double":
returnValue = rvIn.readDouble();
break;
case "int":
returnValue = rvIn.readInt();
break;
case "long":
returnValue = rvIn.readLong();
break;
case "char":
returnValue = rvIn.readChar();
break;
case "float":
returnValue = rvIn.readFloat();
break;
case "short":
returnValue = rvIn.readShort();
break;
default:
returnValue = rvIn.readObject();
break;
}
rvIn.close();
rvBis.close();
} catch (IOException e) {
e.printStackTrace();
} catch (ClassNotFoundException e) {
e.printStackTrace();
} finally {
try {
if (rvIn != null) {
rvIn.close();
}
} catch (IOException ex) {
// ignore close exception
}
}
Object[] inputArguments = new Object[0];
byte[] inputArgsByteArray = new byte[rs.getBinaryStream(5).available()];
rs.getBinaryStream(5).read(inputArgsByteArray);
ByteArrayInputStream bis = new ByteArrayInputStream(inputArgsByteArray);
ObjectInput in = null;
try {
in = new ObjectInputStream(bis);
inputArguments = (Object[])in.readObject();
} catch (IOException e) {
e.printStackTrace();
} catch (ClassNotFoundException e) {
e.printStackTrace();
} finally {
try {
if (in != null) {
in.close();
}
} catch (IOException ex) {
// ignore close exception
}
}
InvocationData invocationData = new InvocationData(
rs.getString(2),
rs.getString(3),
(String[])rs.getArray(4).getArray(),
inputArguments,
rs.getString(6),
returnValue,
rs.getLong(8),
rs.getLong(9),
rs.getLong(10),
rs.getLong(11),
rs.getString(12),
rs.getInt(13)
);
if (rs.getBoolean(14)) {
constructorsData.add(invocationData);
} else {
methodsData.add(invocationData);
}
}
st.close();
rs.close();
conn.close();
An explosion of errors and misguided ideas inherent in this question:
Your read and write code is broken.
available() doesn't work. Well, it does what the javadoc says it does, and if you read the javadoc, and read it very carefully, you should come to the correct conclusion that what that is, is utterly useless. If you ever call available(), you've messed up. You're doing so here. More generally your read and write code doesn't work. For example, .read(byteArr) also doesn't do what you think it does. See below.
The entire principle behind what you're attempting to do, doesn't work
You can't 'save the state' of arbitrary objects, and if you want to push the idea, then if you can, then certainly not in the way you're doing it, and in general this is advanced java that involves hacking the JDK itself to get at it: Think of an InputStream that represents data flowing over a network connection. What do you imagine the 'serialization' of this InputStream object should look like? If you consider serialization as 'just represent the underlying data in memory', then what you'd get is a number that represents the OS 'pipe handle', and possibly some IP, port, and sequence numbers. This is a tiny amount of data, and all this data is completely useless - it doesn't say anything meaningful about that connection and this data cannot be used to reconstitute it, at all. Even within the 'scope' of a single session (i.e. where you serialize, and then deserialize almost immediately afterwards), as networks are a stream and once you grab a byte (or send a byte), it's gone. The only useful, especially for the notion of 'lets replay everything that happened as a test', serialization strategy involves actually 'recording' all the bytes that were picked up, as it happens, on the fly. This is not a thing that you can do as a 'moment in time' concept, it's continuous. You need a system that is recording all the things (it needs to be recording every inputstream, every outputstream, every time System.currentTimeMillis() in invoked, every time a random number is generated, etc), and then needs to use the results of recording it all when your API is asked to 'save' an arbitrary state.
Serialization instead is a thing that objects need to opt into, and where they may have to write custom code to properly deal with it. Not all objects can even be serialized (an InputStream representing a network pipe, as above, is one example of an object that cannot be serialized), and for some, serializing them requires some fancy footwork, and the only hope you have is that the authors of the code that powers this object put in that effort. If they didn't, there is nothing you can do.
The serialization framework of java awkwardly captures both of these notions. It does mean that your code, even if you fix the bugs in it, will fail on most objects that can exist in a JVM. Your testing tool can only be used to test the most simplistic code.
If you're okay with that, read on. But if not, you need to completely rethink what you're going to do with this.
ObjectOutputStream sucks
This is not just my opinion, the openjdk team itself is broadly in agreement (they probably wouldn't quite put it like that, of course). The data emitted by OOS is a weird, inefficient, and underspecced binary blob. You can't analyse this data in any feasible way other than spending a few years reverse engineering the protocol, or just deserializing it (which requires having all the classes, and a JVM - this can be an acceptable burden, depends on your use case).
Contrast to e.g. Jackson which serializes data into JSON, which you can parse with your eyeballs, or in any language, and even without the relevant class files. You can construct 'serialized JSON' yourself without the benefit of first having an object (for testing purposes this sounds like a good idea, no? You need to test this testing framework too!).
How do I fix this code?
If you understand all the caveats above and somehow still conclude that this project, as written and continuing to use the ObjectOutputStream API is still what you want to do (I really, really doubt that's the right call):
Use the newer APIs. available() does not return the size of that blob. read(someByteArray) is not guaranteed to fill the entire byte array. Just read the javadoc, it spells it out.
There is no way to determine the size of an inputstream by asking that inputstream. You may be able to ask the DB itself (usually, LENGTH(theBlobColumn) works great in a SELECT query.
If you somehow (e.g. using LENGTH(tbc)) know the full size, you can use InputStream's readFully method, which will actually read all bytes, vs. read, which reads at least 1, but is not guaranteed to read all of it. The idea is: It'll read the smallest chunk that is available. Imagine a network pipe where bytes are dribbling into the network card's buffer, one byte a second. If so far 250 bytes have dribbled in and you call .read(some500SizeByteArr), then you get 250 bytes (250 of the 500 bytes are filled in, and 250 is returned). If you call .readFully(some500SizeByteArr), then the code will wait about 250 seconds, and then returns 500, and fills in all 500 bytes. That's the difference, and that explains why read works the way it does. Said differently: If you do not check what read() is returning, your code is definitely broken.
If you do not know how much data there is, your only option involves a while loop, or to call a helper method that does that. You need to make a temporary byte array, then in a loop keep calling read until it returns -1. For every loop, take the bytes in that array from 0 to (whatever the read call returned), and send these bytes someplace else. For example, a ByteArrayOutputStream.
Class matching
when I deserialize data in another runtime it changes and, for example, instead of boolean, I retrieve int
The java serialization system isn't magically changing your stuff on you. Well, put a pin that. Most likely the class file available in the first run (where you saved the blob in the db) was different vs what it looked like in your second run. Voila, problem.
More generally this is a problem in serialization. If you serialize, say, class Person {Date dob; String name;}, and then in a later version of the software you realize that using a j.u.Date to store a date of birth is a very silly idea, as Date is an unfortunately named class (it represents an instant in time and not a date at all), so you replace it with a LocalDate instead, thus ending up with class Person{LocalDate dob; String name;}, then how do you deal with the problem that you now want to deserialize a BLOB that was made back when the Person.class file still had the broken Date dob; field?
The answer is: You can't. Java's baked in serialization mechanism will flat out throw an exception here, it will not try to do this. This is the serialVersionUID system: Classes have an ID and changing anything about them (such as that field) changes this ID; the ID is stored in the serialized data. If the IDs don't match, deserialization cannot be done. You can force the ID (make a field called serialVersionUID - you can search the web for how to do that), but then you'd still get an error, java's deserializer will attempt to deserialize a Date object into a LocalDate dob; field and will of course fail.
Classes can write their own code to solve this problem. This is non-trivial and is irrelevant to you, as you're building a framework and presumably can't pop in and write code for your testing framework's userbase's custom class files.
I told you to put a pin in 'the serialization mechanism isnt going to magically change types on you'. Put in sufficient effort with overriding serialVersionUID and such and you can end up there. But that'd be because you wrote code that confuses types, e.g. in your readObject implementation (again, search the web for java's serialization mechanism, readObject/writeObject - or just start reading the javadoc of java.io.Serializable, that's a good starting-off point).
Style issues
You create objects for no purpose, you seem to have some trouble with the distinction between a variable/reference and an object. You aren't using try-with-resources. The way your SELECT calls are made suggests you have an SQL injection security issue. e.printStackTrace() as line line in a catch block is always incorrect.
I've got a text file that has 173,139 words delimited by newlines. Basically, I need to load this 1.7mb file into an array of strings, for easy access. I'm trying to do this in onCreate() for the main activity, which might cause problems of its own because it might make starting the application really slow, but right now I'm just trying to load in the dictionary, I suppose.
I looked up the problem and found that I should use asset manager for this problem, so here's what I have in my Dictionary class:
public class Dictionary {
private String[] dictionary = new String[173139];
private String[] acceptedWordsList = new String[173139];
private String acceptedWords = "";
public Dictionary(Context context){
AssetManager am = context.getAssets();
try {
String rawText;
int element = 0;
InputStream is = am.open("words.txt");
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
while((rawText=reader.readLine()) != null){
dictionary[element++] = rawText;
}
} catch (IOException e) {
e.printStackTrace();
}
}
and I'm instantiating a Dictionary in my Main Activity in pretty much the first line in the class:
public class MainActivity extends Activity {
Dictionary dictionary = new Dictionary(this);
//rest of application code
}
What am I doing wrong here? I've also tried instantiating a Dictionary like this:
Dictionary dictionary = new Dictionary(this.getApplicationContext());
as well as:
Dictionary dictionary = new Dictionary(getApplicationContext());
but all of these seem to cause fatal errors in my application.
So two final questions: is it okay to instantiate in the beginning of the application code like that? And if so, why doesn't that code work when I'm trying to do so?
edit: I was asked for logcat output. My apologies, this is my first "real" android program. I don't know if this is exactly what you're looking for, but I don't want to omit anything that might be helpful either:
http://pastebin.com/raw.php?i=c99LFZzx
Just a thought: your context.getAssets call throws the NPE (possibly because context is null).
I think that's because you did not put Dictionary dictionary = new Dictionary(this); in the onCreate method actually.
I would expect it to work (provided the file is there) if you put that code on onCreate, after super.onCreate(bundle).
If you are worried about app initialization loading time, I suggest you use an AsyncTask to load your dictionary in a splash activity, so you can provide user feedback.
I am trying to read info from a file and create objects out of that information. Every 6 or so lines of the file is a different unit, meaning that the first set of lines are relevant to object A, the next set to object B, and so on.
I can read from the file and create my object just fine--for the first set. My problem is that I don't know how to get the reader to pick up from the spot it left off at when creating the next object...
(Note: the read() method which creates the file is part of the new object being created, not in a main() or anything like that). Here are the relevant bits of code:
The driver:
public class CSD{
public static void main (String[] argv){
Vector V=new Vector(10);
CoS jon=new CoS();
jon.display();
}//end main
}
which calls CoS, whose constructor is:
public CoS(){
try{
String fileName=getFileName();
FileReader freader=new FileReader(fileName);
BufferedReader inputFile=new BufferedReader(freader);
this.read(inputFile);
setDegree(major);
setStatus(credits);
} catch(FileNotFoundException ex){
}//end catch
}
Which calls both read() and getFileName():
public void read(BufferedReader inputFile){
try{
int n;
super.read(inputFile);
String str=inputFile.readLine();
if (str!=null){
n=Integer.parseInt(str);
setCredits(n);
str=inputFile.readLine();
setMajor(str);
}//end if
}catch(IOException ex){}
}//end method
public String getFileName() {
Scanner scan = new Scanner(System.in);
String filename;
System.out.print("Enter the file name and path ==> ");
filename = scan.nextLine();
System.out.println("");
return filename;
}
Thanks in advance, guys!
Why not use ObjectInputStream and ObjectOutputStream? Or any kind of real serialization?
javadoc: http://docs.oracle.com/javase/6/docs/api/java/io/ObjectOutputStream.html
example code: http://www.javadb.com/writing-objects-to-file-with-objectoutputstream
Basically, since you write your objects to a file and want to take care of the lines where they are located, I'll suggest a few other serialization alternatives.
One is the Object * Stream - you create a ObjectStream on a File and just write objects thru it. Later when you read, you read the objects in the reverse order you wrote them and they will come back just as you wrote them.
Another is to implement Serializable. Remember that transient keyword? Use it on fields you do not want to save to the file.
And then there's the raw "by hand" approach where you save only the things you want to save and reconstruct the objects later by passing these initialization values to their constructor. Kinda like people suggested that you make the file line a argument to the ctor :)
EDIT:
guess writing with Object*Streams requires you to implement Serializable or Externalizable.
but if the example code isn't clear enough, ask :)
I can't think of a way of describing it, basically I'm wondering if it is possible to write a method makes the current object become the decoded serialized XML.
I know that the saveTillXML works, as it generates an XML file which matches the state of the current Till object [but really I don't know that it does work, as the decoding fails]. However, I get these exceptions when I call the loadTillXML() method:
java.lang.reflect.InvocationTargetException
java.lang.IllegalStateException
java.lang.ArrayIndexOutOfBoundsException
I call the saveTillXML() method when the window is closed, & the loadTillXML method in the Till's constructor.
I have all the gets & sets & argumentless constructors in the Till, & in all the clases which the Till uses.
public void saveTillXML() throws IOException{
XMLEncoder encoder=new XMLEncoder(new BufferedOutputStream(new FileOutputStream("till.xml")));
encoder.writeObject(this);
encoder.close();
}
public static void loadTillXML() throws IOException{
XMLDecoder decoder=new XMLDecoder(new BufferedInputStream(new FileInputStream("till.xml")));
decoder.readObject();
decoder.close();
}
The only thing I can think of is the fact that decoder.readObject() doesn't go into anything, which is probably should if I want to make the current instance be the XML it's decoding.
Thanks in advance.
Okay, I've managed to solve my problem. The loadTillXML needed a return method - Till - which then prompted me to think that it needs to be called outside Till. So I used it instead of private Till till=new Till(); which was in the window.
The working method:
public static Till loadTillXML() {
try{
XMLDecoder decoder=new XMLDecoder(new BufferedInputStream(new FileInputStream("till.xml")));
return (Till) decoder.readObject();
}
catch(FileNotFoundException fNFE){
System.out.println("Cannot find file");
Till till=new Till();
return till;
}
}
Say, for example, I have a complex dynamically allocated structure (such as a binary tree) that needs to be written to a file made up of different sections. I would like to first write the size of the structure as a dword followed by the structure itself, however the size of the structure is only known after I have written the structure to the file. It is difficult, in this case, to pre-determine the size of the structure in memory.
Is it best to write the size as 0, then write the structure, then seek back and overwrite the size with the correct value? I don't like that idea, though. Is there a better/proper way to do it?
Just an idea: write the data to a ByteArrayOutputStream, after that, you should be able to call size() to get the actual length in bytes and call toByteArray() to get the byte buffer, that can be written to a file.
Code example
public static void main (String[] args) throws java.lang.Exception {
ArrayList objects = new ArrayList();
objects.add("Hello World");
objects.add(new Double(42.0));
System.out.println(sizeof(objects));
}
public static int sizeof(Serializable object) {
ObjectOutputStream out = null;
ByteArrayOutputStream baos = null;
try {
baos = new ByteArrayOutputStream();
out = new ObjectOutputStream(baos);
out.writeObject(object);
} catch (Exception e) {
e.printStackTrace();
} finally {
if (out != null) {
try {
out.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
return baos != null? baos.size() : -1;
}
This just demonstrate a sizeof emulator (which is different from the c implementation, because it calculates the size of a serialized object - the implementation for raw bytes would be slightly different).
Did you looked at Random Access Files yet?
Why do you need to write the size at all? Won't the file be the size of the structure after you have written it?
If you have variable components like arrays or lists, you can write the sizes of those as you write the data. However the total length is redundant and not very useful.
If you really have to, you can write the data to a ByteArrayOutputStream first to get the length. (But I seriously doubt it)
Please refer the below url http://www.javapractices.com/topic/TopicAction.do?Id=83 for calculating size of object .This utility seems worthful for your need.
To measure the size of a particular object containing data, measure JVM memory use before and after building the object.