We Keep Coding

Firestore admin "listens" to all documents again on reboot

TL;DR
Every time my Fiestore admin server reboots my document listener is triggered for all documents even if I have already listened to the document and processed it. How do I get around this?
End TL;DR
I'm working on building a backend for my Firestore chat application. The basic idea is that whenever a users enters a chat message through a client app the backend server listens for new messages and processes them.
The problem I'm running into is that whenever I reboot my app server the listener is triggered for all of the existing already processed chats. So, it will respond to each chat even though it has already responded previously. I would like the app server to only respond to new chats that it hasn't already responded to.
One idea I have for a work around is to put a boolean flag on each chat document. When the backend processes the chat document it will set the flag. The listener will then only reply to chats that don't have the flag set.
Is this a sound approach or is there a better method? One concern I have is that every time I reboot my app server I will be charged heavily to re-query all of the previous chats. Another concern I have is that listening seems memory bound? If my app scales massively will I have to store all chat documents in memory? That doesn't seem like it will scale well...
//Example listener that processes chats based on whether or not the "hasBeenRepliedTo" flag is set
public void startFirestoreListener() {
CollectionReference docRef = db.collection("chats");
docRef.addSnapshotListener(new EventListener<QuerySnapshot>() {
#Override
public void onEvent(#javax.annotation.Nullable QuerySnapshot queryDocumentSnapshots, #javax.annotation.Nullable FirestoreException e) {
if(e != null) {
logger.error("There was an error listening to changes in the firestore chats collection. E: "+e.getLocalizedMessage());
e.printStackTrace();
}
else if(queryDocumentSnapshots != null && !queryDocumentSnapshots.isEmpty()) {
for(ChatDocument chatDoc : queryDocumentSnapshots.toObjects(ChatDocument.class)) {
if(!chatDoc.getHasBeenRepliedTo() {
//Do some processing
chatDoc.setHasBeenRepliedTo(true); //Set replied to flag
}
else {
//No-op, we've already replied to this chat
}
}
}
}
});
}

Yes, to avoid getting each document all the time, you will have to construct a query that yields only the documents that you know have been processed.
No, you are not charged to query documents. You are charged only to read them, which will happen if your query yields documents.
Yes, you will have to be able to hold all the results of a query in memory.
Your problem will be much easier to solve if you use Cloud Functions to receive events for each new document in a collection. You won't have to worry about any of the above things, and instead just worry about writing a Firestore trigger that does what you want with each new document, and paying for those invocations.

Hibernate open-close SessionFactory on every query vs on app instance

I am currently working on a Java Swing application in NetBeans with Hibernate guided with this wonderful repo from GitHub.
From the example code found here, it basically urges new programmers to open and close SessionFactory connection every time certain queries have been executed:
try {
HibernateSessionFactory.Builder.configureFromDefaultHibernateCfgXml()
.createSessionFactory();
new MySqlExample().doSomeDatabaseStuff();
} catch (Throwable th) {
th.printStackTrace();
} finally {
HibernateSessionFactory.closeSessionFactory();
}
private void doSomeDatabaseStuff() {
deleteAllUsers();
insertUsers();
countUsers();
User user = findUser(USER_LOGIN_A);
LOG.info("User A: " + user);
}
Is this a good programming exercise? Isn't it more efficient to open the SessionFactory on app startup and close it on WindowClosing event? What are the drawbacks of each method?
Thanks.

Using a persistent connection means you are going to have as many opened connections on your database as opened clients, plus you'll have to make sure it stays open (very often it will be closed if it stays idle for a long time).
On the other hand, executing a query will be significantly faster if the connection is already opened.
So it really depends on often your clients will use the database. If they use it very rarely, a persistent connection is useless.

Handle multiple clients with one server

I try to get a connection to multiple clients using the Sockets in Java. Everything seems to work, but the problem is, that the server just listens to the first client. If there are multiple clients, the server can send them all messages, but he can just listen to the messages that came from the first client. I tried this all out (I'm at this problem since yesterday). So I'm pretty sure, that the fault has to be in the class "ClientListener".
Explanation:
There is a List with clients (connection to communicate with Strings). In the GUI there is a list, where I can choose, with which client I'd like to communicate. If I change the client, the variable currentClient (int) switches to another number
networkClients is an ArrayList, where all the different connections are "stored".
The first connected client is exactly the same as the other clients, there is nothing special about him. He is called, when the variable currentClient is set to 0 (per default). The variable-switching is working. Like I said, all the clients give me a response if I send them an order, but just networkClients.get(0) is heard by the server (ClientListener).
class ClientListener implements Runnable {
String request;
#Override
public void run() {
try {
while (networkClients.size() < 1) {
Thread.sleep(1000);
}
//***I'm pretty sure, that the problem is in this line
while ((request = networkClients.get(currentClient).getCommunicationReader().readLine()) != null) {
//***
myFileList.add(new MyFile(request));
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
I hope someone can help me. I tried many things, but nothing worked.
EDIT: Like I wrote in the code example, is it possible that the while-loop isn't able to switch the number of "currentClient" (which is handled by another Thread)? I tested/simulated something similar in a testclass and the result was, that a while-loop of course can can update the state in it (meaning, that if a variable changes in the () of a while loop, it will of course be checked after every repeat).

You should take a look at multithreading.
Your server program should be made out of:
- The main thread
- A thread that handles new connections.
(Upon creating a new connection, start a new thread and pass the connection on to that thread)
- A thread for each connected client, listening to the each client separately
Take a look at some examples like: (1) (2)

I found the solution:
The Thread sits in the declared method I mentioned in the starting post (in the code snippet) and waits unlimited time for a new response of the client.
So changing the index of the list "networkClients" won't do anything, because nothing will happen there, until there is a new order sent by the client (which lets the thread go further).
So you need to implement an extra listener for each client.

To get database updates using servlets or jsp

What I want is to get database updates.
i.e If any changes occur to the database or a new record is inserted it should notify to the user.
Up to know what I implemented is using jQuery as shown below
$(document).ready(function() {
var updateInterval = setInterval(function() {
$('#chat').load('Db.jsp?elect=<%=emesg%>');
},1000);
});
It worked fine for me, but my teacher told to me that it's not a good way to do recommended using comet or long polling technology.
Can anyone give me examples for getting database updates using comet or long polling
in servlets/jsp? I'm using Tomcat as server.

Just taking a shot in the dark since I don't know your exact environment... You could have the database trigger fire a call to a servlet each time a row is committed which would then run some code that looked like the following:
Get the script sessions that are active for the page that we want to update. This eliminates the need to check every reverse ajax script session that is running on the site. Once we have the script sessions we can use the second code block to take some data and update a table on the client side. All that the second code section does is send javascript to the client to be executed via the reverse ajax connection that is open.
String page = ServerContextFactory.get().getContextPath() + "/reverseajax/clock.html";
Browser.withPage(page, new Runnable() {
public void run() {
Util.setValue("clockDisplay", output);
}
});
// Creates a new Person bean.
Person person = new Person(true);
// Creates a multi-dimensional array, containing a row and the rows column data.
String[][] data = {
{person.getId(), person.getName(), person.getAddress(), person.getAge()+"", person.isSuperhero()+""}
};
// Call DWR's util which adds rows into a table. peopleTable is the id of the tbody and
// data conta
ins the row/column data.
Util.addRows("peopleTable", data);
Note that both of the above sections of code are pulled straight from the documentation examples # http://directwebremoting.org/dwr-demo/. These are only simple examples of how reverse ajax can sent data to the client, but your exact situation seems to be more dependent on how you receive the notification than how you update the client screen.
Without some type of database notification to the java code I think you will have to poll the system at set intervals. You could make the system a little more efficient even when polling by verifying that there are reverse ajax script sessions active for the page before polling the database for info.

How to synchronize concurring Web Service calls in Java

I'm currently developing some web services in Java (& JPA with MySQL connection) that are being triggered by an SAP System.
To simplify my problem I'm referring the two crucial entities as BlogEntry and Comment. A BlogEntry can have multiple Comments. A Comment always belongs to exactly one BlogEntry.
So I have three Services (which I can't and don't want to redefine, since they're defined by the WSDL I exported from SAP and used parallel to communicate with other Systems): CreateBlogEntry, CreateComment, CreateCommentForUpcomingBlogEntry
They are being properly triggered and there's absolutely no problem with CreateBlogEntry or CreateComment when they're called seperately.
But: The service CreateCommentForUpcomingBlogEntry sends the Comment and a "foreign key" to identify the "upcoming" BlogEntry. Internally it also calls CreateBlogEntry to create the actual BlogEntry. These two services are - due to their asynchronous nature - concurring.
So I have two options:
create a dummy BlogEntry and connect the Comment to it & update the BlogEntry, once CreateBlogEntry "arrives"
wait for CreateBlogEntry and connect the Comment afterwards to the new BlogEntry
Currently I'm trying the former but once both services are fully executed, I end up with two BlogEntries. One of them only has the ID delivered by CreateCommentForUpcomingBlogEntry but it is properly connected to the Comment (more the other way round). The other BlogEntry has all the other information (such as postDate or body), but the Comment isn't connected to it.
Here's the code snippet of the service implementation CreateCommentForUpcomingBlogEntry:
#EJB
private BlogEntryFacade blogEntryFacade;
#EJB
private CommentFacade commentFacade;
...
List<BlogEntry> blogEntries = blogEntryFacade.findById(request.getComment().getBlogEntryId().getValue());
BlogEntry persistBlogEntry;
if (blogEntries.isEmpty()) {
persistBlogEntry = new BlogEntry();
persistBlogEntry.setId(request.getComment().getBlogEntryId().getValue());
blogEntryFacade.create(persistBlogEntry);
} else {
persistBlogEntry = blogEntries.get(0);
}
Comment persistComment = new Comment();
persistComment.setId(request.getComment().getID().getValue());
persistComment.setBody(request.getComment().getBody().getValue());
/*
set other properties
*/
persistComment.setBlogEntry(persistBlogEntry);
commentFacade.create(persistComment);
...
And here's the code snippet of the implementation CreateBlogEntry:
#EJB
private BlogEntryFacade blogEntryFacade;
...
List<BlogEntry> blogEntries = blogEntryFacade.findById(request.getBlogEntry().getId().getValue());
BlogEntry persistBlogEntry;
Boolean update = false;
if (blogEntries.isEmpty()) {
persistBlogEntry = new BlogEntry();
} else {
persistBlogEntry = blogEntries.get(0);
update = true;
}
persistBlogEntry.setId(request.getBlogEntry().getId().getValue());
persistBlogEntry.setBody(request.getBlogEntry().getBody().getValue());
/*
set other properties
*/
if (update) {
blogEntryFacade.edit(persistBlogEntry);
} else {
blogEntryFacade.create(persistBlogEntry);
}
...
This is some fiddling that fails to make things happen as supposed.
Sadly I haven't found a method to synchronize these simultaneous service calls. I could let the CreateCommentForUpcomingBlogEntry sleep for a few seconds but I don't think that's the proper way to do it.
Can I force each instance of my facades and their respective EntityManagers to reload their datasets? Can I put my requests in some sort of queue that is being emptied based on certain conditions?
So: What's the best pracice to make it wait for the BlogEntry to exist?
Thanks in advance,
David
Info:
GlassFish Server 3.1.2
EclipseLink, version: Eclipse Persistence Services - 2.3.2.v20111125-r10461

If you are sure you are getting a CreateBlogEntry call, queue the CreateCommentForUpcomingBlogEntry calls and dequeue and process them once you receive the CreateBlogEntry call.
Since you are on an application server, for queues, you can probably use JMS queues that autoflush to storage or use the DB cache engine (Ehcache ?), in case you receive a lot of calls or want to provide a recovery mechanism across restarts.