We Keep Coding

NFS: Synchronized processing of files in cluster env

There is a process which dumps 10k files in a shared NFS drive. I need to read and process the data from files. I have written java code which works great in a single node env. But when the code is deployed in WAS cluster with 4 nodes, the nodes are picking and processing the same files.
How can I avoid this? Is there some sort of file lock feature that I can use to fix this issue? Any help is highly appreciated.
More info:
I am using org.apache.commons.io.monitor library to poll the NFS directory every 10secs. Then, we read and process the files and then move the file to a post process folder. As mentioned, this works great in a single node env. When deployed in cluster, the nodes are polling the same file and processing them which is causing multiple calls with same data to a backend service.
I am looking for optimal solution.
PS:The application which processes the files doesn't have access to any kind of database.
Thanks in advance

"Is there some sort of file lock feature that I can use to fix this issue?" Not without doing some work on your end. You could create another file with the same name ending in .lock and have the application check to see if a lock file exists by creating the lock file and if it succeeds then it will process the file. If it fails it then knows one of the other cluster members already grabbed the lock file.

How to monitor a network port that is being used for a ftp service?

Every time a ftp connection is established I want a java program to get triggered.
I am not sure how to check the FTP connections. Is there a way to monitor the port so that I can acknowledge when FTP gets the new connection?
I've researched a bit and found the FTPClient class which basically suggests I need to do my own ftpclient, but I want to know if there is another way.
Even an external program could be an option.

it would be running on the server side, basically the program will be reading the files that the client put on it.
Consider using a file watcher. It doesn't seem that you really care that the file gets there using FTP (in fact, in the future it might get there by some other means), but just that the file is present and needs to be processed.
See Watching a Directory for Changes for details on how to write one.

I have a program that does a similar task server-side with java. It checks if files exist in a folder and launches the rest of the code if so. Like this:
public static boolean needToConvert(){
//get list of companies
List<String> companies = getCompanyNames();
//check whether a file exists in folder of company
for(String company: companies){
if (new File(INPUT_LOCATION + company).list().length > 0){
return true;
}
}
return false;
}
This is a rather in-expensive task, so you could just set the server to "check"; say, every minute or so.

There are a variety of approaches:
Have the FTP service itself launch the java app (sounds expensive) or notify a running Java app / service. This will be the simplest approach ...
Use some kind of external monitoring; e.g.
A generic packet packet sniffing tool (like wireshark or tcpdump) might be able to trigger an external application. If not, you may be able to get it to log info to a file that you scan for connection events.
You could possible implement something using jpcap to do your own packet sniffing in Java
There is a Linux utility called tcpspy that logs TCP/IP connection details to syslog. You could configure it log to a specific file and monitor that file, alternatively, you could modify tcpspy to launch or notify a Java app directly.

Java creating server that is ran as one file with database?

I have a program that I've created that is meant to poll an html internal page with different IPs that update and then will run a telnet session to those IPs to see if the device still has a connection... I'm attempting to challenge myself in creating something further with a dynamic webpage instead of my program spitting out console output...
My Issue:
I dont know what technologies / libraries Java has to execute such things
I want:
A Local Server, to upload a page LOCALLY only (no security is needed as this will be strictly intranet)
My program to implement: A database of sorts to save "logs" essentially that a certain IP / device has had successful connections
in the past....maybe stored to an external file is fine i presume (my
program currently has to re-poll everytime i run it.. i want some kind
of "remembering"..
Is it possible this can all be done in one file? so if i want my computer to run this as soon as it starts up... it will run... grab
its current state of the database of IPs... poll them (periodically)
and then persist and save and update the HTML page dynamically....
I hope i'm being as descriptive as possible... Its a bit of an abstract.. I really just want some introduction to different libraries ... a friend recommended stuff like MongoDB or something but I want to stay strictly to Java programming

java logging/Log Server

Say I have DemoServer project that only logs anything and I created another project say LogServer. So here's what I want to do; I will run both project simultaneously, as the DemoServer is running it will just keep on logging anything and in the LogServer project it should be able to access the logs that the DemoServer generates and save it to a new text file every 1 minute. Since this is the first time I'm doing this I find it hard to figure out how will the two project communicate? Can someone explain in detail how can i achieve this? Please also post links that might help me solve my problem.

I'd go with a local socket which is quite straightforward to implement in Java and can be later used also when DemoServer and LogServer are on different machines.
Developing it should be quite easy:
create your own LogMessage class
create a simple client/server infrastructure by using TCP sockets, take a look here
wrap the socket streams by using ObjectInputStream and ObjectOutputStream to be able to use serialization
just send logs by encapsulating them in messages from one process to the other and you are done

I would suggest you dig into oVirt source code (you can git clone the source) and see what we do with the log collector application.
There are many options to solve your problem -
A. Have your application log into a shared storage (i.e - nfs share) - one that both it and the log server can access. A cron job will run a periodic script that will copy the files that were last accessed, let's say - one hour ago, into a folder, that the the log server can access.
B. Use log4j and write your own Appender that will send the stuff you want log server to collect also to log server (via a file as suggested in section A, or via any other mean) - this way you will be able to contol on a category level basis what stuff the Log server can actually read

Using own Java software´s custom update implementation

I have a server, and Client that are working fine, they transfer file to each other and handle commands with success. The thing is that once connected to the server, the server can ask the Client to update itself.
Since I did everything using sockets with a TCP connection, and I am NOT able to use java web start, I am looking for a way I can update the software.
One thing I thought was:
Making the java call another java software and close itself. Then the other java software, would connect to the server again and waits for the update request, so it could replace all files from the current version (lib directory and the jar main file - Im using Netbeans).
Maybe I could just send the connection Object to the Software that would update, but the main only accpets String.
So I ask two questions here:
Is this a good solution? Or is there a better one?
If I do that, how can I send the Client Object of my proto from one application to another?

I would suggest keep a property file on server having latest version of app.
Each time you start the app compare version if its updatable then download all new updates in a temp dir
After SUCCESSFUL download invoke another small application to copy those files to your lib file in order to update your app actually
At the end prompt user that app has been updated and launch the newly updated app. i would say.