We Keep Coding

Split big object before persisting it to file

I have an object that occupies approximately 15GB in my java app's heap, and I need to keep it persistent between JVM restarts. I use ObjectOutputStream's writeObject method to write it to file on the disk every time interval. Since the writing process is very long (a few minutes) and causes some GC issues, I would like to split the object somehow to persist each part separately to a different file and not in a single action to a single file.
Is there a way to do this (and of course to retrieve it back from the files when I need it)?
FileOutputStream fos = null;
GZIPOutputStream gos = null;
ObjectOutputStream oos = null;
try {
fos = new FileOutputStream("some_path");
gos = new GZIPOutputStream(fos);
oos = new ObjectOutputStream(gos);
oos.writeObject(myLargeObject);
oos.flush();
gos.close();
fos.close();
} catch (Exception e) {
e.printStackTrace();
}

You may want to take a look at this answer. The reverse process of recomposing the split files is nothing more than reading from each individual file and appending to a "master" one.

ObjectInputStream - reading large binary file - problems with memory

Before I proceed to my question : please note that I am not working on any client-server application that would require serialization, but the program I am trying to customize stores one big instance of one big class in a .dat file. I have read about this issue (memory leak in ObjectOutputStream and ObjectInputStream)and the fact that I could probably need to :
use the ObjectOutputStream.reset() method after writing the class instance in the .dat file, so that it doesn't hold the reference anymore;
re-write the code without using serialization;
split the file and read it in chunks;
change the JVM memory parameter by using -Xmx;
So, I was provided with one class that generates a language model and saves it with a .dat extension; the code was probably optimized for small model files (there are 2 model files provided as examples, both around 10MB ), but I generated a much larger model class, and it is around 40MB. Then, there is another class in another folder, totally independent on the first one, that uses this model, and the model has to be loaded using ObjectInputStream. Here comes the problem : a classic "OutOfMemoryError : Java heap space".
Writing the object:
try {
// Create an output stream to the file.
FileOutputStream file_output = new FileOutputStream (file);
ObjectOutputStream o = new ObjectOutputStream( file_output );
o.writeObject(this);
file_output.close ();
}
catch (IOException e) {
System.err.println ("IO exception = " + e );
}
Reading the object:
InputStream model = null;
ModelGeneration oRead = null;
ObjectInputStream p = null;
try {
model = new FileInputStream(filename);
BufferedInputStream buf = new BufferedInputStream(model);
p = new ObjectInputStream(buf);
oRead = (ModelGeneration) p.readObject();
p.reset();
} catch (IOException e) {
e.printStackTrace();
} catch (ClassNotFoundException e) {
e.printStackTrace();
} finally {
try {
model.close();
} catch (Exception e) {
e.printStackTrace();
}
}
I tried to use the reset() method, but it is useless because we load only one instance of one class at a time, nothing else needed. This is why I can't split the file, too: only one class instance is stored in the .dat file.
Changing the heap space seems like a worse solution than optimizing the code.
I would really appreciate your advice on what I can do.
Btw the code is here : http://svn.apache.org/repos/asf/uima/addons/trunk/Tagger/, I only implemented the required classes for a different language.
P.S. Works fine if I create a smaller model, but I would prefer the bigger one.

Writing a serialized object to file garbaged?

try
{
File dataFile = new File("C:/Users/keatit/Desktop/players.txt");
if(!dataFile.exists())
{
dataFile.createNewFile();
}
FileOutputStream fos = new FileOutputStream("C:/Users/keatit/Desktop/players.txt");
ObjectOutputStream oos = new ObjectOutputStream(fos);
oos.writeObject(players);
oos.close();
}
catch(FileNotFoundException fnfex)
{
System.out.println(fnfex.getMessage());
}
catch(IOException ioex)
{
System.out.println(ioex.getMessage());
}
I have a class player which implement Serializable but when I write objects to files the text is messed up and looks like the following. Any help would be much appreciated. Thank you.
"¬í sr java.util.ArrayListxÒ™Ça I sizexp w sr players.playerÌ`~%×êòœ I ageL firstNamet Ljava/lang/String;xp t Trevorsq ~ t Michaelax"

This is binary serialization. It's not meant to be writing to a human-readable text file. For that, you should look into something like JSON or YAML. I'd strongly recommend against writing to a .txt file using ObjectOutputStream - it gives the wrong impression.
The point of binary serialization is to be able to deserialize it later with the same serialization protocol - so in this case you'd use ObjectInputStream. You should find that that is able to correctly deserialize the object stored in your file.
(Side-note: FileOutputStream will create a new file automatically if it doesn't exist - you don't need to do so yourself. Additionally, you should use a try-with-resources statement to clean up automatically, rather than just calling close() outside a finally block.)

How to keep data stored in array after application has ended?

I'm working on an Android app and I got to a point where I need to add some Store objects to an ArrayList favoriteStores. The problem is that I want this list to persist after closing the application, because my list of favorite stores must stay there until I chose to delete particular items inside it. Anyone got any idea what type of implementation I might use? Thanks in advance,

If you don't want to save arraylist to database, you can save it to file. It is a great way if you just want to save arraylist and don't want to touch sqlite.
You can save arraylist to file with this method
public static <E> void SaveArrayListToSD(Context mContext, String filename, ArrayList<E> list){
try {
FileOutputStream fos = mContext.openFileOutput(filename + ".dat", mContext.MODE_PRIVATE);
ObjectOutputStream oos = new ObjectOutputStream(fos);
oos.writeObject(list);
fos.close();
} catch (Exception e) {
e.printStackTrace();
}
}
And you can read that saved file to arraylist with this method
public static Object ReadArrayListFromSD(Context mContext,String filename){
try {
FileInputStream fis = mContext.openFileInput(filename + ".dat");
ObjectInputStream ois = new ObjectInputStream(fis);
Object obj= (Object) ois.readObject();
fis.close();
return obj;
} catch (Exception e) {
e.printStackTrace();
return new ArrayList<Object>();
}
}
Hope this help.

You can either use a database like SQLite (howto here) or use some serialization technique. There is a related question to serialization here.
General information about storing data in Android can be found here.

Here's a nice post about data storage options in Android, read it carefully and select the option that you find the most appropriate.

create object output stream from an object

I want to create on ObjectOutputStream, but I don't want to persist the object in a file, so how to do that? All the tutorials(that I found) say only about the file way:
FileOutputStream fos = new FileOutputStream("t.tmp");
ObjectOutputStream oos = new ObjectOutputStream(fos);
oos.writeObject(new Date());
oos.close();
I want to store the object into a database, so I need to specify a stream in method setBinaryStream() from class PreparedStatement.
Thanks for answering...

Store it in a byte array instead. You can use ByteArrayOutputStream for this. This way you can use PreparedStatement#setBytes().
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(new Date());
oos.close();
// ...
preparedStatement.setBytes(i, baos.toByteArray());
That said, this is pretty a good smell. Are you sure that you need to serialize Java objects into a database? This way they are unindexable and unsearchable. If you for example store each Person serialized in the DB, you cannot do SELECT * FROM person WHERE name = 'John' anymore. The normal practice is to do a 1:1 mapping of the entity and the DB table. The Date for example can perfectly be stored in a DATETIME/TIMESTAMP column.

ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutputStream os = new ObjectOutputStream(bos);
os.writeObject(new Date());
os.close();
byte[] data = bos.toByteArray();
So now you have a byte array and do what you want with it.

you specifically need to use an outputstream to write to a database? I would seriously consider looking at the persistence api before attempting to write an outputstream implementation.. since connection details etc, might get tricky to manage.
have a look at link text and remember it can be used in J2SE as well.