We Keep Coding

Deserializing returns null object Java

Currently writing an application like Booking and I am in the stage where i want to store my infomation in files. I have created a Serializable Database class which has a field with a pathname and 2 methods for read/write . I have about 8 other classes which extend the Database and each holds a Hashmap and some querying methods. Naturally I read the Databases from my files before i start the application and i write before exiting, but I ran into a problem where the objects I am reading are all null. I've been at it for 2 hours now and I need a second opinion. This is the database's read/write methods:
public void write() {
try {
File temp = new File(this.filename);
temp.createNewFile(); // create file if not present
FileOutputStream fileOut = new FileOutputStream(this.filename);
ObjectOutputStream objectOut = new ObjectOutputStream(fileOut);
objectOut.writeObject(this);
objectOut.close();
System.out.println("The Object was successfully written to a file");
} catch (Exception ex) {
ex.printStackTrace();
}
}
public Object read() {
Object obj = null;
try {
File temp = new File(this.filename);
temp.createNewFile(); // create file if not present
FileInputStream fileIn = new FileInputStream(this.filename);
ObjectInputStream objectIn = new ObjectInputStream(fileIn);
obj = objectIn.readObject();
System.out.println("The Object was successfully read from the file");
objectIn.close();
} catch (EOFException ex) {
return obj;
} catch (IOException | ClassNotFoundException e) {
e.printStackTrace();
}
return null;
}
And here is the way I load them (Which is probably the problem) in my Application class
private void loadData() {
accommodationReviewsDatabase = (AccommodationReviews) accommodationReviewsDatabase.read();
brokerAccommodationsDatabase = (BrokerAccommodations) brokerAccommodationsDatabase.read();
credentialsUserDatabase = (CredentialsUser) credentialsUserDatabase.read();
customerReviewsDatabase = (CustomerReviews) customerReviewsDatabase.read();
userConfirmationsDatabase = (UserConfirmations) userConfirmationsDatabase.read();
userMessagesDatabase = (UserMessages) userMessagesDatabase.read();
}
private void writeData() {
accommodationReviewsDatabase.write();
brokerAccommodationsDatabase.write();
credentialsUserDatabase.write();
customerReviewsDatabase.write();
userConfirmationsDatabase.write();
userMessagesDatabase.write();
}
Some extra information that may be asked :
All my classes that I am storing are serializable
The files I am storing the databases are all *.ser (Thats the extension I found)
The files are stored inside the project

If your read() method completes without an EOFException, it ends with return null;. You should return obj;, the object you read.
You should not expect that EOFException will be thrown if your read succeeds. The EOFException would indicate that it ran out of data while it was trying to read your object, and could not complete successfully.
If you do get an EOFException, it is probably a good idea to give some indication instead of silently returning. Silent catch blocks deny you information that could be useful for debugging.

How to avoid OutOfMemory exception while reading large files in Java

I am working on the application which reads large amounts of data from a file. Basically, I have a huge file (around 1.5 - 2 gigs) containing different objects (~5 to 10 millions of them per file). I need to read all of them and put them to different maps in the app. The problem is that the app runs out of memory while reading the objects at some point. Only when I set it to use -Xmx4096m - it can handle the file. But if the file will be larger, it won't be able to do that anymore.
Here's the code snippet:
String sampleFileName = "sample.file";
FileInputStream fileInputStream = null;
ObjectInputStream objectInputStream = null;
try{
fileInputStream = new FileInputStream(new File(sampleFileName));
int bufferSize = 16 * 1024;
objectInputStream = new ObjectInputStream(new BufferedInputStream(fileInputStream, bufferSize));
while (true){
try{
Object objectToRead = objectInputStream.readUnshared();
if (objectToRead == null){
break;
}
// doing something with the object
}catch (EOFException eofe){
eofe.printStackTrace();
break;
} catch (Exception e) {
e.printStackTrace();
continue;
}
}
} catch (Exception e){
e.printStackTrace();
}finally{
if (objectInputStream != null){
try{
objectInputStream.close();
}catch (Exception e2){
e2.printStackTrace();
}
}
if (fileInputStream != null){
try{
fileInputStream.close();
}catch (Exception e2){
e2.printStackTrace();
}
}
}
First of all, I was using objectInputStream.readObject() instead of objectInputStream.readUnshared(), so it solved the issue partially. When I increased the memory from 2048 to 4096, it started parsing the file. BufferedInputStream is already in use. From the web I've found only examples how to read lines or bytes, but nothing regarding objects, performance wise.
How can I read the file without increasing the memory for JVM and avoiding the OutOfMemory exception? Is there any way to read objects from the file, not keeping anything else in the memory?

When reading big files, parsing objects and keeping them in memory there are several solutions with several tradeoffs:
You can fit all parsed objects into memory for that app deployed on one server. It either requires to store all objects in very zipped way, for example using byte or integer to store 2 numbers or some kind of shifting in other data structures. In other words fitting all objects in possible minimum space. Or increase memory for that server(scale vertically)
a) However reading the files can take too much memory, so you have to read them in chunks. For example this is what I was doing with json files:
JsonReader reader = new JsonReader(new InputStreamReader(in, "UTF-8"));
if (reader.hasNext()) {
reader.beginObject();
String name = reader.nextName();
if ("content".equals(name)) {
reader.beginArray();
parseContentJsonArray(reader, name2ContentMap);
reader.endArray();
}
name = reader.nextName();
if ("ad".equals(name)) {
reader.beginArray();
parsePrerollJsonArray(reader, prerollMap);
reader.endArray();
}
}
The idea is to have a way to identify when certain object starts and ends and read only that part.
b) You can also split files to smaller ones at the source if you can, then it will be easier to read them.
You can't fit all parsed objects for that app on one server. In this case you have to shard based on some object property. For example split data based on US state into multiple servers.
Hopefully it helps in your solution.

How to reset ObjectInputStream objects graph cache to avoid memory bloat

When loading huge files with ObjectInputStream, all read objects are buffered by stream for object graph resolving.
This cause huge memory overhead which isn't needed in my case (all objects read are interdependent).
Is there an equivalent to the reset() method of ObjectOutputStream which reset this buffer?
Code example:
try (FileInputStream fileInputStream = new FileInputStream(filename);
BufferedInputStream bufferedInputStream = new BufferedInputStream(fileInputStream);
ObjectInputStream objectInputStream = new ObjectInputStream(bufferedInputStream)) {
while (object = objectInputStream.readObject()) {
System.Out.println(object.toString());
}
}
There is actually a reset method on the class but it does a complete different thing.
See Java APIs which cause memory bloat

It's up to the sender to decide when to break the integrity of sent object graphs, by calling ObjectOutputStream.reset(). Not the receiver.
NB your code doesn't compile, and wouldn't be valid if it did:
while (object = objectInputStream.readObject()) {
}
This should be
try {
while (true) {
object = objectInputStream.readObject();
// ...
}
}
catch (EOFException exc) {
// end of stream
}
There is a misconception abroad that readObject() returns null at end of stream. It doesn't. It throws EOFException. It can return null any time you wrote a null.

Hmm it seems you need to use some sort of lazy loading techniques where you only load necessairy components of the object graph, not everything.

What is referring to my file, preventing me from calling .delete() successfully?

I am making a game and have started to work on a class called SaveManager, so far I have had no problems except when it comes to deleting the game.
My problem is this, .delete() returns false the majority of times, without calling an exception.
If I make a loop (below) it does eventually delete, despite me not actually changing any variables or references and the game loop itself definitely doesn't deal with the files at all, so it isn't "releasing" the file itself at any point.
public void delete(File save)
{
while (save.exists() && !save.delete())
{
i++;
}
new infoBox("Times delete called: " + i, "delete method in saveManager");
So my question is: If the reference is not from one of my classes how do I find it and remove it, allowing me to remove the above loop, which basically hijacks the game until the file is gone.
Here is the code which does deal with the file, in case I am in fact doing something wrong (God Forbid!) I'm pretty sure this does "release" the save file in question...
private void save()
{
ObjectOutputStream OOS = null;
BufferedOutputStream BOS = null;
FileOutputStream FOS = null;
try
{
File saveFile = new File(saveDirectory + hub.world.saveName + ".dat");
OOS = new ObjectOutputStream(new BufferedOutputStream(new FileOutputStream(saveFile, false)));
OOS.write... (List of Objects & Strings here)
OOS.flush();
OOS.close();
game.saved = true; //Used for warning message on exit
this.setSavesList();
}
catch (Exception e)
{
new errorWindow(e, "SaveFile, I/O error");
}
finally
{
try
{
FOS.flush();
FOS.close();
BOS.flush();
BOS.close();
OOS.flush();
OOS.close();
}
catch (IOException e)
{
hub.frame.errorWindow(e, "Problem closing streams");
}
}
}

One general thing to check while dealing with files is if you're closing any opened streams before attempting to delete it.
You can't delete a file unless the stream opened is closed.
Take a look at this.

You are creating three streams, and only closing the top-most one (OOS).
You should close each stream manually. I think that will fix your problem.
Also, it's often considered good practice to put the close() in a finally block, so it definitely happens.
FileOutputStream fos = new FileOutputStream(fileName);
try
{
BufferedOutputStream bos = new BufferedOutputStream(fos);
OOS = new ObjectOutputStream(bos);
OOS.write... (List of Objects & Strings here)
OOS.flush();
OOS.close();
bos.flush();
bos.close();
game.saved = true; //Used for warning message on exit
this.setSavesList();
} catch (IOException ioe)
{
}
finally
{
fos.close();
}
Otherwise, you might flush the object output stream, and the data just gets stuck in the buffered output stream, and never gets written to the file output stream.

Here's a food for thought...from the code above, you have used a try-catch block...my guess is that one of the above operations is throwing an exception because of which the out.close() operation is getting skipped...why not add a "finally" block and close the stream there...it would definitely help you save the trouble.

Are you sure your file is actually being created? In the code snippet above you have while(!save.delete()) but later on you seem to refer to it as saveFile, these may be two different files! I would first check to see if the file even exists on the disk by doing an
if(!saveFile.exists()) {
do something here
}
if it doesnt exist, a delete will just return false, you cannot delete a file that does not exist.

If you are checking the result of delete you also need to check whether the files exists. Obviously if there is not file or it has already been deleted, delete() will return false. ;)

How do I update a File created by openFileOutput

I'm currently building an application where the user will generate data over time and, should he/she has an internet connection, transmit it to the web. However, if he doesn't have web access, I need to store this data in the phone until the user recovers his access, when I'll need to recover this data to be transmitted. However, I'm facing lots of troubles to do this, as per below.
Note: before anything, I'm using a local java-created file because I know no other way to save/restore this data on the device. If you happen to know any other way to store/access this data from within the device please feel free to comment here.
Just for reference,
phantoms is an ArrayList containing objects with the data I need to
store,
Arquivador is the class that I'm using to make my data persistent and to recover it,
Funcionario is the class with the data generated by the program (just a few strings and numbers)
I am able to write a file to the file system through the code below, on my Activity:
try {
arq = new Arquivador();
arq.addFirstObjectInFile(
openFileOutput("dados.jlog", MODE_WORLD_WRITEABLE),
phantoms.get(0));
phantoms.remove(phantoms.get(0));
for (Funcionario func : phantoms) {
arq.addObjectInFile(openFileOutput("dados.jlog", MODE_APPEND),
func);
}
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
}
Here is the code inside Arquivador that adds the data to a file:
public void addObjectInFile(FileOutputStream arquivo,
Object objetoAAdicionar) {
try {
ObjectOutputStream aoos = new ObjectOutputStream(arquivo);
aoos.writeObject(objetoAAdicionar);
aoos.close();
} catch (IOException ioe) {
Log.d(TAG_NAME, "Erro no Appendable OOS.");
}
}
public void addFirstObjectInFile(FileOutputStream arquivo,
Object objetoAAdicionar) {
try {
AppendableObjectOutputStream aoos = new AppendableObjectOutputStream(
arquivo);
aoos.writeObject(objetoAAdicionar);
aoos.close();
} catch (IOException ioe) {
Log.d(TAG_NAME, "Erro no Appendable OOS.");
}
}
You will notice that I'm adding data to persistence in 2 steps, the first Object and the rest of them. This was an idea I saw on this post, here in StackOverflow, to allow appending data to a Java generated file. I have no problem with this code, it works perfectly.
Later on, back on my Activity, the internet connection is detected and I try to recover the file saved on the disk:
phantoms = new ArrayList<Funcionario>();
Object obj = arq.readObjectFromFile(openFileInput("dados.jlog"));
Funcionario func = null;
if (obj instanceof Funcionario) {
func = (Funcionario) obj;
}
while (func != null) {
phantoms.add(func);
arq.removeObjectFromFile(openFileInput("dados.jlog"), func,
getApplicationContext());
func = (Funcionario) arq
.readObjectFromFile(openFileInput("dados.jlog"));
}
The original idea was to read 1 object at a time, then attempt to transmit it and, if successful, erase the object from the file (so it didn't get retransmitted). However, I was having too many error messages with this. Instead, I decided to load all the objects at once, one by one, to see where my problem was more clearly.
Back to the Arquivador class:
public Object readObjectFromFile(FileInputStream arquivo) {
Object retorno = null;
if (arquivo.equals(null)) {
Log.e(TAG_NAME, "FIS is null!");
}
ObjectInputStream ois = null;
try {
ois = new ObjectInputStream(arquivo);
retorno = ois.readObject();
} catch (IOException ioex) {
} catch (ClassNotFoundException e) {
} finally {
try {
if (ois != null) ois.close();
} catch (IOException e) {
}
}
return retorno;
}
public void removeObjectFromFile(FileInputStream arqPrincipal,
Object objetoARemover, Context contexto) {
try {
// Construct the new file that will later be renamed to the original
// filename.
ObjectOutputStream oos = new ObjectOutputStream(
contexto.openFileOutput("dados.jlog.temp",
contexto.MODE_APPEND));
ObjectInputStream ois = new ObjectInputStream(arqPrincipal);
Object obj = null;
// Read from the original file and write to the new
// unless content matches data to be removed.
try {
while ((obj = ois.readObject()) != null) {
if (!(objetoARemover.equals(obj))) {
oos.writeObject(obj);
oos.flush();
}
}
} catch (EOFException eof) {
} finally {
oos.close();
ois.close();
// Delete the original file
File aDeletar = contexto.getFileStreamPath("dados.jlog");
File aRenomear = contexto.getFileStreamPath("dados.jlog.tmp");
if (!aDeletar.delete()) {
return;
} else {
// Rename the new file to the filename the original file
// had.
if (!aRenomear.renameTo(aDeletar)) Log.d(TAG_NAME,
"Error renaming file");
else Log.d(TAG_NAME, "Renaming successful");
}
}
} catch (FileNotFoundException ex) {
ex.printStackTrace();
Log.d(TAG_NAME, "Arquivo não encontrado");
} catch (IOException ex) {
ex.printStackTrace();
Log.d(TAG_NAME, "Erro de entrada/saída");
} catch (ClassNotFoundException e) {
Log.d(TAG_NAME, "Classe Não Encontrada.");
}
}
The method readObjectFromFile() seems to work just fine. I can even convert the read Object to Funcionario class and read its data.
My problems appear when I use removeObjectFromFile(). The idea is to create a temporary file to store objects from "dados.jlog" file other than the one that has been already loaded in the main program, then once this temp file is created the file "dados.jlog" should be deleted and the temporary file should be renamed to replace it.
The first thing I found out to be strange here is that the ois.readobject() keeps throwing an EOFException. While this makes sense, the tutorial I read on the internet doesn't mention this error. In fact, their code indicates that when the readObject() method reaches the EOF, it would return a reference to null, but instead this class throws this EOFException. I handled this exception in the code - though I'm not sure if this would be the right way to do it.
Another thing I find strange is the fact that this code fails to recognize the object that it should NOT copy. When I compare the object read from the file to the one received as argument, no matter what I try ( == , equals(), etc) they seem different objects to the compiler. Funcionario class is serializable has a serialversionUID, so the object read from the file should be identical to the one I stored. Worse than this, these 2 Objects being compared are read from the same file. They should be identical, right?
After creating the temporary file, I try to delete the original file and rename the temporary file. Though this seems to be working, once the removeObjectFromFile() ends the first time, the program is unable to read the data from the file "dados.jlog" again. I can't read the remaining data from the file and the program enters on an endless loop - since the 1st object is never removed from the list in the file.
Please enlighten me with this matter.

Personally I'd use an SQLLite database. Store each object in a row in the database. Once you've successfully transmitted you can remove the row from the database.
You can even reuse most of your code that you've already done. The easiest way to get there from where you are is to use a separate file for each object and store only the filename of the object in the database. You can then iterate over the rows in the database. Each time you transmit an object to your server simply delete that row from the database (and remove the file from the filesystem!). No rows in the database means no objects remain to be transmitted.