We Keep Coding

Flatbuffers: how do you build nested tables?

I have a 3-level nested Java POJO that looks like this in the schema file:
struct FPathSegment {
originIata:ushort;
destinationIata:ushort;
}
table FPathConnection {
segments:[FPathSegment];
}
table FPath {
connections:[FPathConnection];
}
When I try to serialize a Java POJO to the Flatbuffer equivalent I pretty much get "nested serialzation is not allowed" error every time I try to use a common FlatBufferBuilder to build this entire object graph.
There is no clue in the docs to state if I have a single builder for the entire graph? A separate one for every table/struct? If separate, how do you import the child objects into the parent?
There are all these methods like create/start/add various vectors, but no explanation what builders go in there. Painfully complicated.
Here is my Java code where I attempt to serialize my Java POJO into Flatbuffers equivalent:
private FPath convert(Path path) {
FlatBufferBuilder bld = new FlatBufferBuilder(1024);
// build the Flatbuffer object
FPath.startFPath(bld);
FPath.startConnectionsVector(bld, path.getConnections().size());
for(Path.PathConnection connection : path.getConnections()) {
FPathConnection.startFPathConnection(bld);
for(Path.PathSegment segment : connection.getSegments()) {
FPathSegment.createFPathSegment(bld,
stringCache.getPointer(segment.getOriginIata()),
stringCache.getPointer(segment.getDestinationIata()));
}
FPathConnection.endFPathConnection(bld);
}
FPath.endFPath(bld);
return FPath.getRootAsFPath(bld.dataBuffer());
}
Every start() method throws a "FlatBuffers: object serialization must not be nested" exception, can't figure out what is the way to do this.

You use a single FlatBufferBuilder, but you must finish serializing children before starting the parents.
In your case, that requires you to move FPath.startFPath to the end, and FPath.startConnectionsVector to just before that. This means you need to store the offsets for each FPathConnection in a temp array.
This will make the nesting error go away.
The reason for this inconvenience is to allow the serialization process to proceed without any temporary data structures.

Java Serialization for Dummies

I'm still working on the project I already needed a bit of help with:
JavaFX - TableView doesn't update items
Now I want to understand how this whole Serialization process in Java works, because unfortunately, I don't really get it now.
Before I go on, first of all, I'm a student, I'm not a professional. Second, I'm neither familiar with using DBs, nor XML or JSON, so I'd just like to find solution to my approach, no matter how inelegant it might be in the end, it just needs to work. So please don't feel offended if I just reject any advice in using other techniques.
So here's what I want:
Saving three different class objects to separate files BUT maintaining backward compatibility to each of it. The objects are Settings, Statistics and a "database" object, containing all words in a list added to it. In the future I may add more statistics or settings, means adding new variables, mostly type of IntegerProperty or DoubleProperty.
Now the question is: is it possible to load old version saved files and then during the process just initiate new variables not found in the old version with just null but keep the rest as it has been saved?
All I know is that the first thing to do so is not to alter the serialVersionUID.
Another thing would be saving the whole Model object (which contains the three objects mentioned before), so I just have to implement stuff for one class instead of three. But how would that work then concerning backward compatibility? I mean the class itself would not change but it's attributes in their own class structure.
Finally, what approach should I go for? And most of all, how do I do this and maintaning backward compatibilty at the same time? I do best with some concrete examples rather than plain theory.
Here are two example methods, if it's of any help. I already have methods for each class to write and read an object.
public static void saveModel(Model model, String destination) throws IOException
{
try
{
fileOutput = new FileOutputStream(destination);
objectOutput = new ObjectOutputStream(fileOutput);
objectOutput.writeObject(model);
}
catch (IOException e)
{
e.printStackTrace();
}
finally
{
if (objectOutput != null)
try
{
objectOutput.close();
}
catch (IOException e) {}
if (fileOutput != null)
try
{
fileOutput.close();
}
catch (IOException e) {}
}
}
public static Settings readSettings(String destination) throws IOException, FileNotFoundException
{
Settings s = null;
try
{
fileInput = new FileInputStream(destination);
objectInput = new ObjectInputStream(fileInput);
Object obj = objectInput.readObject();
if (obj instanceof Settings)
{
s = (Settings)obj;
}
}
catch (IOException e)
{
e.printStackTrace();
}
catch (ClassNotFoundException e)
{
e.printStackTrace();
}
finally
{
if (objectInput != null) try { objectInput.close(); } catch (IOException e) {}
if (fileInput != null) try { fileInput.close(); } catch (IOException e) {}
}
return s;
}
Tell me if you need more of my current code.
Thank you in advance!

... you must be this tall
Best advice for Serialisation is to avoid it for application persistence, especially if backwards compatibility is desired property in your application.
Answers
Is it possible to load old version saved files and then during the process just initiate new variables not found in the old version with just null but keep the rest as it has been saved?
Yes. Deserialising objects saved using previous versions of the class into a new version of this class will work only if:
fully qualified name of the class has not changed (same name and package)
previous and current class have exactly the same serialVersionUID; if one of the versions is missing it, it will be calculated as a 'hash' of all fields and methods and upon a mismatch deserialisation will fail.
inheritance hierarchy has not changed for that class (the same ancestors)
no fields have been removed in the new version of the class
no fields have become static
no fields have become transient
I just have to implement stuff for one class instead of three. But how would that work then concerning backward compatibility?
Yes. Providing that all classes of all fields of Model and Model class itself adhere to the rules above.
Finally, what approach should I go for? And most of all, how do I do this and maintaning backward compatibilty at the same time?
Yes, as long as you can guarantee all of the above rules forever, you will be backwards compatible.
I am sure you can appreciate that forever, or even for next year can be very hard to guarantee, especially in software.
This is why people do application persistence using more robust data exchange formats, than binary representation of serialised Java objects.
Raw data for the table, could be saved using anything from CSV file to JSON docs stored as files or as documents in NoSQL database.
For settings / config have a look at Java's Properties class which could store and load properties to and from *.properties or *.xml files or separately have a look at YAML.
Finally for backwards compatibility, have a look at FlatBuffers
The field of application persistence is very rich and ripe, so happy exploring.

Serialization does not work with a custom linked list

So here is my problem:
for a project I had to create a custom linked list whereby I had to add nodes to it and save/load it to and from the disk using serialisation
here are some things about my system before I define the problem
I have a generic 'customer file' which acts as the node data
this is stored in a node object which acts as an element of the list
there is a customer file class where the information such as name and email address are stored as well as the various get and set methods for each - these work fine
there is a node class with get and set data and next methods for each whereby the next item is a node object and acts as the next item in the list
there is a singly linked class with add, remove, modify, sort, search etc... methods - IT IS A CUSTOM MADE CLASS AND SO DOES NOT IMPLEMENT ANY JAVA PREMADE LISTS.
a lot of testing has been on all classes separately and used together - these methods are foolproof - they work
there is a main class which is used for the main interface between the user and the system - it is a CLI system (command line)
it has a save list to file function and load list from file function (in the main class) whereby each function uses serialization or deserialization to save/load the list from the disk
all classes implement the serializable interface
there is a 'MAIN' method in the main class whereby a while loop operates which allows the user to modify the list in some way (eg add a record, remove a record etc...)
the list is loaded outside the loop so it is not cleared each time the loop iterates (a common problem amongst colleagues)
i have a password for the system whereby identical methods are used to save a string to another file location and that has worked for weeks - the password is saved at that location and can be accessed, changed and removed at will
these load/save methods have the appropriate try/catch methods to catch any exceptions
The problem is that each time i load up my programming environment and want to look at the list, I find that the list on file is 'empty' and contained no records from when i last added/removed stuff.
I add records and modify the list - easy peasy as the other classes work - once these are added, i call the print function which simply displays all items in the list and it is fine.
However, the minute i close the environment, they are lost and when i reopen the environment to look at the list again, it is empty!
Upon looking in the folder where these classes are saved, i have noticed each time i run the program that 'shells' are created and remain there until the program is closed/finished however the 'listData.ser' which should have the linked list saved does not have any data.
Likewise the password file contains the password which was saved fine - so i am a little confused as to why my code does not work.
Here is my save list method:
private static void saveListToFile(SinglyLinkedList lst) {
try {
ObjectOutputStream os = new ObjectOutputStream(new FileOutputStream("ListData.ser"));
os.writeObject(lst);
os.flush();
os.close();
}
catch (FileNotFoundException e) {
e.printStackTrace();
}
catch (IOException e) {
e.printStackTrace();
}
}
Likewise the load list method is similar but uses object input stream and file input stream.
Any suggestions?
P.S. My main while loop is over 400 lines of code long and therefore not feasible to post.
Update 1.
Deserialization code in load list method:
private static SinglyLinkedList loadListFromFile() {
SinglyLinkedList lst = null;
try {
ObjectInputStream is = new ObjectInputStream(new FileInputStream("ListData.ser"));
lst = (SinglyLinkedList) is.readObject();
is.close();
}
catch(FileNotFoundException e) {
e.printStackTrace();
}
catch(IOException e) {
e.printStackTrace();
}
catch(ClassNotFoundException e) {
e.printStackTrace();
}
return lst;
}
I dont think the singly linked list class itself is the problem (response to comment) and it is not feasible to copy as it is also over 300 lines of code (lots of methods).

Have you tried calling close() on the FileOutputStream when you are done writing the file/object?

I have solved it, instead of posting my code I tried to do it myself. I turns out there were a few static methods of the list class - these were changed to non-static and now the list saves as expected each time.
Thanks for the help

Locking file across services

What is the best way to share a file between two "writer" services in the same application?
Edit:
Sorry I should have given more details I guess.
I have a Service that saves entries into a buffer. When the buffer gets full it writes all the entries to the file (and so on). Another Service running will come at some point and read the file (essentially copy/compress it) and then empty it.

Here is a general idea of what you can do:
public class FileManager
{
private final FileWriter writer = new FileWriter("SomeFile.txt");
private final object sync = new object();
public void writeBuffer(string buffer)
{
synchronized(sync)
{
writer.write(buffer.getBytes());
}
}
public void copyAndCompress()
{
synchronized(sync)
{
// copy and/or compress
}
}
}
You will have to do some extra work to get it all to work safe, but this is just a basic example to give you an idea of how it looks.

A common method for locking is to create a second file in the same location as the main file. The second file may contain locking data or be blank. The benefit to having locking data (such as a process ID) is that you can easily detect a stale lockfile, which is an inevitability you must plan for. Although PID might not be the best locking data in your case.
example:
Service1:
creates myfile.lock
creates/opens myfile
Service2:
Notices that myfile.lock is present and pauses/blocks/waits
When myfile.lock goes away, it creates it and then opens myfile.
It would also be advantageous for you to double-check that the file contains your locking information (identification specific to your service) right after creating it - just in case two or more services are waiting and create a lock at the exact same time. The last one succeeds and so all other services should notice that their locking data is no longer in the file. Also - pause a few milliseconds before checking its contents.

Persistence on Java CI-Bayes object

Has anyone ever persisted a training set for CI-Bayes? I have sample code from this site: http://www.theserverside.com/news/thread.tss?thread_id=49773
here is the code:
FisherClassifier fc=new FisherClassifierImpl();
fc.train("The quick brown fox jumps over the lazy dog's tail","good");
fc.train("Make money fast!", "bad");
String classification=fc.getClassification("money", "unknown"); // should be "bad"
so I need to be able to store the training set in a local file.
Has anyone ever done this before?

To persist a java Object in a local file, the Object must first implement the Serializable interface.
import java.io.Serializable;
public class MyClass implements Serializable {...
Then, the class from which you would like to persist this training set, should include a method like:
public void persistTrainingSet(FisherClassifier fc) {
String outputFile = <path/to/output/file>;
try {
FileOutputStream fos = new FileOutputStream(outputFile);
ObjectOutputStream oos = new ObjectOutputStream(fos);
oos.writeObject(fc);
oos.close();
}
catch (IOException e) {
//handle exception
}
finally {
//do any cleaning up
}
}

I have. After doing a couple projects with CI-Bayes, I would recommend you look elsewhere (of course this was a long time ago). It is a very bad idea to use an inference engine that needs to be trained before each use and if you really consider the issue of state management, it's complicated (e.g. do you want to just store the training data? or perhaps the trained distributions? chains?).
CI-Bayes is also kind of a convoluted codebase. It was modeled off some Python code that appeared in a book about intelligence. The Java version is not very well designed. It also does not use TDD, does not really have JavaDoc to speak of.
That said, you can get a simple classifier going pretty quickly. The longer term goal is the one you asked about though.