We Keep Coding

Java thread pooled complex constraints data processing

I have Java EE (desktop) application that had to process data files generated by multiple sources (up to a 200 different sources). Each source periodically generates data file with unique name which also contains that source's unique ID.
I need to create a thread pool with 15 threads which will process and remove files with these constraints:
Multiple threads can't process files from the same source simultaneously.
Multiple files from the same source should be processed in order of it's creation timestamp.
No synchronization with the file generator sources is possible so it means that the next file(s) may be generated by source while it's previous file is processed or scheduled for processing.
Processing should be multi threaded because of performance reasons (single threaded processing is not enough so I'm planning to use 10-15 threads).
A file processing operation may be time consuming 3-15 seconds.
Any suggestions on architecture of such complex synchronization of the threads in pool are welcome.
P.S. Due to the limitation on simultaneous processing the design I've used in more simple situations earlier i.e. using ArrayBlockingQueue does not fit this case.

General idea:
You have a task-queue per source.
And you have a central queue which is effectively a queue of task-queues which is shared between all worker threads.
For each source you create a task-queue. And you stick these task-queue's in a hashtable based on the unique id. This way you get the guarantee that tasks from the same source are processed in order (requirement 2).
If a tasks is received, you look up (or create) the task-queue in the hashtable and you add the task to the taskqueue. If it was the first task added to the queue, you also add it to the central queue.
Then there are a bunch of worker-threads that take task-queues from this central queue and then take a single task from this task-queue they just took and process that task. Once they are done with the task, they need to decide if the task-queue needs to be reinserted back into the central-queue or not.
There are a few parts were things could easily go wrong:
You don't want to end up with a task-queue being inserted into the central-queue multiple times. That would violate your first requirement.
You don't want the task-queue not being reinserted into the central-queue even though a task is available.
So you need to take care of the appropriate synchronization and it might be a bit more complex than you would initially think. But seen the fact that the tasks are long running, I would start out with a regular mutex (could be per task-queue) e.g. synchronized or a lock and don't worry about making it non blocking.

So this is the skeleton/tester of the class I've created to solve my problem. See #pveentjer answer for some details on whats going on here.
package org.zur.test;
import java.io.File;
import java.util.Comparator;
import java.util.HashMap;
import java.util.PriorityQueue;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.Semaphore;
import java.util.concurrent.ThreadLocalRandom;
import java.util.concurrent.TimeUnit;
public class PmFileProcessor {
private static final int THREADS_COUNT = 15;
ArrayBlockingQueue<File> scheduledFiles = new ArrayBlockingQueue<>(10000, false);
HashMap<String, PmFileProcessingJob> allJobs = new HashMap<>();
ScheduledExecutorService jobsExecutorService = Executors.newScheduledThreadPool(THREADS_COUNT);
public PmFileProcessor() {
super();
SourceJobsManager fc = new SourceJobsManager();
fc.setDaemon(true);
fc.start();
}
public void scheduleFile(File f) {
try {
scheduledFiles.add(f);
} catch (Exception e) {
// TODO: handle exception
}
}
/**
* Assigns files to file source processing job.
*
* #author
* <ul>
* <li>Zur13</li>
* </ul>
*
*/
public class SourceJobsManager extends Thread {
#Override
public void run() {
// assigns scheduled files to per-source jobs and schedules job for additional execution
while ( true ) {
try {
File f = scheduledFiles.take();
PmFileProcessingJob job = getSourceJob(f);
job.scheduleSourceFile(f);
jobsExecutorService.execute(job); // schedules job execution
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
// TODO: check disk space periodically
}
}
/**
* Finds existing job for file source or creates a new one.
*
* #param f
* #return
*/
private PmFileProcessingJob getSourceJob(File f) {
// TODO: test code
String fname = f.getName();
String[] parts = fname.split("_");
String uid = parts[0];
PmFileProcessingJob res = allJobs.get(uid);
if ( res == null ) {
res = new PmFileProcessingJob(uid);
allJobs.put(uid, res);
}
return res;
}
}
/**
* Process first file from scheduledSourceFiles queue (i.e. each job execution processes a single file or
* reschedules itself for later execution if another thread already processes the file from the same source).
*
* #author
* <ul>
* <li>Zur13</li>
* </ul>
*
*/
public class PmFileProcessingJob implements Runnable {
public final String fileSourceUidString;
PriorityQueue<File> scheduledSourceFiles = new PriorityQueue<>(1000, new Comparator<File>() {
#Override
public int compare(File o1, File o2) {
// TODO Auto-generated method stub
return 0;
}
});
Semaphore onePassSemaphore = new Semaphore(1);
public PmFileProcessingJob(String fileSourceUid) {
super();
this.fileSourceUidString = fileSourceUid;
}
/**
* Schedules file from for processing by this job.
*/
public void scheduleSourceFile(File f) {
scheduledSourceFiles.add(f);
}
#Override
public void run() {
File f = null;
if ( scheduledSourceFiles.size() > 0 ) { // fail fast optimization 1
if ( onePassSemaphore.tryAcquire() ) { // fail fast optimization 2
try {
f = scheduledSourceFiles.poll();
if ( f != null ) {
// TODO: process the file
try {
System.err.println(f.getName() + "\t" + Thread.currentThread().getId());
Thread.sleep(1000);
return;
} catch (Exception e) {
// TODO: handle exception
return; // prevents reschedule loop for failing files
}
} else {
// scheduledSourceFiles queue is empty
return;
}
} finally {
onePassSemaphore.release();
}
}
if ( f == null && scheduledSourceFiles.size() > 0 ) {
// this thread did not process the scheduled file because another thread holds the critical section
// pass
// this thread should reschedule this Job to release this thread and try to process this job later
// with another thread
// reschedule the job with 4 seconds delay to prevent excess CPU usage
// System.err.println("RESCHEDULE");
jobsExecutorService.schedule(this, 3, TimeUnit.SECONDS);
}
}
}
#Override
public int hashCode() {
final int prime = 31;
int result = 1;
result = prime * result + ((this.fileSourceUidString == null) ? 0 : this.fileSourceUidString.hashCode());
return result;
}
#Override
public boolean equals(Object obj) {
if ( this == obj )
return true;
if ( !(obj instanceof PmFileProcessingJob) )
return false;
PmFileProcessingJob other = (PmFileProcessingJob) obj;
if ( this.fileSourceUidString == null ) {
if ( other.fileSourceUidString != null )
return false;
} else if ( !this.fileSourceUidString.equals(other.fileSourceUidString) )
return false;
return true;
}
}
public static void main(String[] args) {
PmFileProcessor fp = new PmFileProcessor();
fp.unitTest();
}
private void unitTest() {
// TODO Auto-generated method stub
int filesCount = 1000;
for (int i = 0; i < filesCount; i++) {
int sourceUid = ThreadLocalRandom.current().nextInt(1, 30);
File f = new File(sourceUid + "_" + i);
scheduleFile(f);
}
Thread.yield();
try {
Thread.sleep(999000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}

How to create a Socket connection between Unity/C# Server and Android/Java Client?

I'm working on an academic project that consists in controlling a simulated environment created in Unity3D, from a conventional Android application. I already created the scenario in Unity and the mobile application is completely finished, my problem is in the connection. I chose to use sockets because of its simplicity. I managed to connect the application to a server written in C # through conventional Sockets, I know that information can be sent, but when I implemented it in Unity, everything failed, so I decided to use TCP Listener instead of Sockets in Unity / C # (the client Android still uses a conventional TCP connection through Sockets), and in effect, the application connects but there is no information transfer, because the Unity console throws an error that says: ObjectDisposedException: Can not access to disposed object.
Object name: 'System.Net.Sockets.TcpClient'.(The error is presented in line number 56: stream = client.tcp.GetStream ();) The main objective is to obtain the flow of information that would be represented in something like: "1: 0: 1: 0" and being able to make a Split to that string and according to that value change the state of a light bulb or other element, nevertheless I need the essential thing: to establish the connection flow. I'm not an expert in C # and much less using Unity, I really do not know much about design, but I wanted to do it to deliver an innovative work, I hope someone can guide me.
PD: C# Socket (No Unity yet) server working with Android Java client:
![1] https://imgur.com/a/HuNcDC3
//This is my Unity3D/C# Server:
public class SocketManager : MonoBehaviour
{
private List<ServerClient> connectedClients;
private List<ServerClient> disconectedClients;
private TcpListener server;
private string data;
private NetworkStream stream;
private StreamReader stremaReader;
private bool serverStarted;
public int socketPort = 7691;
private void Start()
{
connectedClients = new List<ServerClient>();
disconectedClients = new List<ServerClient>();
try
{
server = new TcpListener(IPAddress.Parse("192.168.1.64"), socketPort);
server.Start();
serverListening();
serverStarted = true;
Debug.Log("Server started. Port: " + socketPort.ToString());
}
catch (Exception ex)
{
Debug.Log("Server socket error: " + ex);
}
}
private void Update()
{
if (!serverStarted)
return;
foreach (ServerClient client in connectedClients)
{
if (IsConnected(client.tcp))
{
client.tcp.Close();
disconectedClients.Add(client);
continue;
}
else
{
stream = client.tcp.GetStream();
if (stream.DataAvailable)
{
this.stremaReader = new StreamReader(stream, true);
data = stremaReader.ReadLine();
if (data != null)
OnIcomingData(client, data);
}
}
}
}
private void OnIcomingData(ServerClient client, string data)
{
Debug.Log(client.clientName + ": " + data);
}
private bool IsConnected(TcpClient tcp)
{
try
{
if (tcp != null && tcp.Client != null && tcp.Client.Connected)
{
if (tcp.Client.Poll(0, SelectMode.SelectRead))
{
return !(tcp.Client.Receive(new byte[1], SocketFlags.Peek) == 0);
}
return true;
}
else
{
return false;
}
}
catch
{
return false;
}
}
private void serverListening()
{
server.BeginAcceptTcpClient(AcceptTcpClient, server);
}
private void AcceptTcpClient(IAsyncResult asyncResult)
{
TcpListener tcplistener = (TcpListener)asyncResult.AsyncState;
connectedClients.Add(new ServerClient(tcplistener.EndAcceptTcpClient(asyncResult)));
serverListening();
}
}
public class ServerClient
{
public TcpClient tcp;
public string clientName;
public ServerClient(TcpClient tcp)
{
this.tcp = tcp;
this.clientName = "Android";
}
}
// This is my Android/Java Client:
public class SocketThread extends Thread {
private Socket adviser;
private DataOutputStream dataOut;
#Override
public void run() {
super.run();
Log.e("Status:", "Thread started");
try {
adviser = new Socket("192.168.1.64", 7691);
dataOut = new DataOutputStream(adviser.getOutputStream());
} catch (IOException ex) {
Logger.getLogger(SocketThread.class.getName()).log(Level.SEVERE, null, ex);
}
}
public void sendCommand(String text) {
try {
dataOut.writeUTF(text);
Log.e("Sended Text: ", text);
} catch (IOException ex) {
Logger.getLogger(SocketThread.class.getName()).log(Level.SEVERE, null, ex);
}
}

Here's a more detailed answer than my comment, for future reference.
The problem lies in the following if check if (IsConnected(client.tcp)).
Right now what is happening is that when the client is connected to the server (so the if returns true) you are Closing the connection using client.tcp.Close(); and adding the connection to your disconectedClients list (But you are not removing the client from your connectedClients list! Meaning the client is now in both lists). Then when the next iteration of the foreach loop foreach (ServerClient client in connectedClients) comes around the if (IsConnected(client.tcp)) will return false (the client is no longer connected to the server, since you called close() in the previous iteration.
You then try to call stream = client.tcp.GetStream(); on a stream that is already closed, resulting in an error telling you you cannot get the stream.
This should fix it:
if (IsConnected(client.tcp))
{
client.tcp.Close();
client.tcp.Dispose(); //Dispose of a stream to release it for GC
disconectedClients.Add(client);
connectedClients.Remove(client); //Note to also remove the disconnected client from the connectedClients list.
continue;
}
else
{ ...rest of logic }
Note that i added connectedClients.Remove(client); and client.tcp.Dispose(); to your logic closing the stream. If you don't do this it'll keep getting included in theforeachloop unnecessarily, and the programm will try to callclose()` on a connection that is already closed, resulting in more errors. Disposing of the stream will release it so the garbage collector can collect it, freeing up memory.

Force stop Java Files.copy() running on external thread

The answer here seemed to be a valid solution before Java 8:
How to cancel Files.copy() in Java?
But now it doesn't work, because ExtendedCopyOption.INTERRUPTIBLE is private.
Basically, I need to download a file from some given URL and save it to my local file-system using Files.copy().
Currently, I am using a JavaFX Service because I need to show the progress in a ProgressBar.
However, I don't know how to block the thread running Files.copy() if the operation takes too long.
Using Thread.stop() is at least not wanted. Even Thread.interrupt() fails.
I also want the operation to terminate gracefully if the internet connection becomes unavailable.
To test the case when no internet connection is available, I'm removing my ethernet cable and putting it back after 3 seconds.
Unfortunately, Files.copy() returns only when I put back the ethernet cable, while I would like it to fail immediately.
As I can see, internally Files.copy() is running a loop, which prevents the thread from exiting.
Tester(Downloading OBS Studio exe):
/**
* #author GOXR3PLUS
*
*/
public class TestDownloader extends Application {
/**
* #param args
*/
public static void main(String[] args) {
launch(args);
}
#Override
public void start(Stage primaryStage) throws Exception {
// Block From exiting
Platform.setImplicitExit(false);
// Try to download the File from URL
new DownloadService().startDownload(
"https://github.com/jp9000/obs-studio/releases/download/17.0.2/OBS-Studio-17.0.2-Small-Installer.exe",
System.getProperty("user.home") + File.separator + "Desktop" + File.separator + "OBS-Studio-17.0.2-Small-Installer.exe");
}
}
DownloadService:
Using #sillyfly comment with FileChannel and removing File.copy seems to work only with calling Thread.interrupt() but it is not exiting when the internet is not available..
import java.io.File;
import java.net.URL;
import java.net.URLConnection;
import java.nio.channels.Channels;
import java.nio.channels.FileChannel;
import java.nio.file.StandardOpenOption;
import java.util.logging.Level;
import java.util.logging.Logger;
import javafx.concurrent.Service;
import javafx.concurrent.Task;
/**
* JavaFX Service which is Capable of Downloading Files from the Internet to the
* LocalHost
*
* #author GOXR3PLUS
*
*/
public class DownloadService extends Service<Boolean> {
// -----
private long totalBytes;
private boolean succeeded = false;
private volatile boolean stopThread;
// CopyThread
private Thread copyThread = null;
// ----
private String urlString;
private String destination;
/**
* The logger of the class
*/
private static final Logger LOGGER = Logger.getLogger(DownloadService.class.getName());
/**
* Constructor
*/
public DownloadService() {
setOnFailed(f -> System.out.println("Failed with value: " + super.getValue()+" , Copy Thread is Alive? "+copyThread.isAlive()));
setOnSucceeded(s -> System.out.println("Succeeded with value: " + super.getValue()+" , Copy Thread is Alive? "+copyThread.isAlive()));
setOnCancelled(c -> System.out.println("Succeeded with value: " + super.getValue()+" , Copy Thread is Alive? "+copyThread.isAlive()));
}
/**
* Start the Download Service
*
* #param urlString
* The source File URL
* #param destination
* The destination File
*/
public void startDownload(String urlString, String destination) {
if (!super.isRunning()) {
this.urlString = urlString;
this.destination = destination;
totalBytes = 0;
restart();
}
}
#Override
protected Task<Boolean> createTask() {
return new Task<Boolean>() {
#Override
protected Boolean call() throws Exception {
// Succeeded boolean
succeeded = true;
// URL and LocalFile
URL urlFile = new URL(java.net.URLDecoder.decode(urlString, "UTF-8"));
File destinationFile = new File(destination);
try {
// Open the connection and get totalBytes
URLConnection connection = urlFile.openConnection();
totalBytes = Long.parseLong(connection.getHeaderField("Content-Length"));
// --------------------- Copy the File to External Thread-----------
copyThread = new Thread(() -> {
// Start File Copy
try (FileChannel zip = FileChannel.open(destinationFile.toPath(), StandardOpenOption.CREATE,
StandardOpenOption.TRUNCATE_EXISTING, StandardOpenOption.WRITE)) {
zip.transferFrom(Channels.newChannel(connection.getInputStream()), 0, Long.MAX_VALUE);
// Files.copy(dl.openStream(), fl.toPath(),StandardCopyOption.REPLACE_EXISTING)
} catch (Exception ex) {
stopThread = true;
LOGGER.log(Level.WARNING, "DownloadService failed", ex);
}
System.out.println("Copy Thread exited...");
});
// Set to Daemon
copyThread.setDaemon(true);
// Start the Thread
copyThread.start();
// -------------------- End of Copy the File to External Thread-------
// ---------------------------Check the %100 Progress--------------------
long outPutFileLength;
long previousLength = 0;
int failCounter = 0;
// While Loop
while ((outPutFileLength = destinationFile.length()) < totalBytes && !stopThread) {
// Check the previous length
if (previousLength != outPutFileLength) {
previousLength = outPutFileLength;
failCounter = 0;
} else
++failCounter;
// 2 Seconds passed without response
if (failCounter == 40 || stopThread)
break;
// Update Progress
super.updateProgress((outPutFileLength * 100) / totalBytes, 100);
System.out.println("Current Bytes:" + outPutFileLength + " ,|, TotalBytes:" + totalBytes
+ " ,|, Current Progress: " + (outPutFileLength * 100) / totalBytes + " %");
// Sleep
try {
Thread.sleep(50);
} catch (InterruptedException ex) {
LOGGER.log(Level.WARNING, "", ex);
}
}
// 2 Seconds passed without response
if (failCounter == 40)
succeeded = false;
// --------------------------End of Check the %100 Progress--------------------
} catch (Exception ex) {
succeeded = false;
// Stop the External Thread which is updating the %100
// progress
stopThread = true;
LOGGER.log(Level.WARNING, "DownloadService failed", ex);
}
//----------------------Finally------------------------------
System.out.println("Trying to interrupt[shoot with an assault rifle] the copy Thread");
// ---FORCE STOP COPY FILES
if (copyThread != null && copyThread.isAlive()) {
copyThread.interrupt();
System.out.println("Done an interrupt to the copy Thread");
// Run a Looping checking if the copyThread has stopped...
while (copyThread.isAlive()) {
System.out.println("Copy Thread is still Alive,refusing to die.");
Thread.sleep(50);
}
}
System.out.println("Download Service exited:[Value=" + succeeded + "] Copy Thread is Alive? "
+ (copyThread == null ? "" : copyThread.isAlive()));
//---------------------- End of Finally------------------------------
return succeeded;
}
};
}
}
Interesting questions:
1-> What does java.lang.Thread.interrupt() do?

I strongly encourage you to use a FileChannel.
It has the transferFrom() method which returns immediately when the thread running it is interrupted.
(The Javadoc here says that it should raise a ClosedByInterruptException, but it doesn't.)
try (FileChannel channel = FileChannel.open(Paths.get(...), StandardOpenOption.CREATE,
StandardOpenOption.WRITE)) {
channel.transferFrom(Channels.newChannel(new URL(...).openStream()), 0, Long.MAX_VALUE);
}
It also has the potential to perform much better than its java.io alternative.
(However, it turns out that the implementation of Files.copy() may elect to delegate to this method instead of actually performing the copy by itself.)
Here's an example of a reusable JavaFX Service that lets you fetch a resource from the internet and save it to your local file-system, with automatic graceful termination if the operation takes too long.
The service task (spawned by createTask()) is the user of the file-channel API.
A separate ScheduledExecutorService is used to handle the time constraint.
Always stick to the good practices for extending Service.
If you choose to use such an high-level method, you won't be able to track down the progress of the task.
If the connection becomes unavailable, transferFrom() should eventually return without throwing an exception.
To start the service (may be done from any thread):
DownloadService downloadService = new DownloadService();
downloadService.setRemoteResourceLocation(new URL("http://speedtest.ftp.otenet.gr/files/test1Gb.db"));
downloadService.setPathToLocalResource(Paths.get("C:", "test1Gb.db"));
downloadService.start();
and then to cancel it (otherwise it will be automatically cancelled after the time expires):
downloadService.cancel();
Note that the same service can be reused, just be sure to reset it before starting again:
downloadService.reset();
Here is the DownloadService class:
public class DownloadService extends Service<Void> {
private static final long TIME_BUDGET = 2; // In seconds
private final ScheduledExecutorService watchdogService =
Executors.newSingleThreadScheduledExecutor(new ThreadFactory() {
private final ThreadFactory delegate = Executors.defaultThreadFactory();
#Override
public Thread newThread(Runnable r) {
Thread thread = delegate.newThread(r);
thread.setDaemon(true);
return thread;
}
});
private Future<?> watchdogThread;
private final ObjectProperty<URL> remoteResourceLocation = new SimpleObjectProperty<>();
private final ObjectProperty<Path> pathToLocalResource = new SimpleObjectProperty<>();
public final URL getRemoteResourceLocation() {
return remoteResourceLocation.get();
}
public final void setRemoteResourceLocation(URL remoteResourceLocation) {
this.remoteResourceLocation.set(remoteResourceLocation);
}
public ObjectProperty<URL> remoteResourceLocationProperty() {
return remoteResourceLocation;
}
public final Path getPathToLocalResource() {
return pathToLocalResource.get();
}
public final void setPathToLocalResource(Path pathToLocalResource) {
this.pathToLocalResource.set(pathToLocalResource);
}
public ObjectProperty<Path> pathToLocalResourceProperty() {
return pathToLocalResource;
}
#Override
protected Task<Void> createTask() {
final Path pathToLocalResource = getPathToLocalResource();
final URL remoteResourceLocation = getRemoteResourceLocation();
if (pathToLocalResource == null) {
throw new IllegalStateException("pathToLocalResource property value is null");
}
if (remoteResourceLocation == null) {
throw new IllegalStateException("remoteResourceLocation property value is null");
}
return new Task<Void>() {
#Override
protected Void call() throws IOException {
try (FileChannel channel = FileChannel.open(pathToLocalResource, StandardOpenOption.CREATE,
StandardOpenOption.WRITE)) {
channel.transferFrom(Channels.newChannel(remoteResourceLocation.openStream()), 0, Long.MAX_VALUE);
}
return null;
}
};
}
#Override
protected void running() {
watchdogThread = watchdogService.schedule(() -> {
Platform.runLater(() -> cancel());
}, TIME_BUDGET, TimeUnit.SECONDS);
}
#Override
protected void succeeded() {
watchdogThread.cancel(false);
}
#Override
protected void cancelled() {
watchdogThread.cancel(false);
}
#Override
protected void failed() {
watchdogThread.cancel(false);
}
}

There is one important aspect not covered by the other answers/comments; and that is a wrong assumption of yours:
What I want is it to fail immediately when no internet connection is there.
It is not that easy. The TCP stack/state machine is actually a pretty complicated thing; and depending on your context (OS type; TCP stack implementation, kernel parameters, ...), there can be situations where a network partition takes place and a sender doesn't notice for 15 or more minutes. Listen here for more details on that.
In other words: "just pulling the plug" is no way equal to "immediately breaking" your existing TCP connection. And just for the record: you don't need to plug cables manually to simulate network outages. In a reasonable test setup, tools like iptables aka firewalls can do that for you.

You seem to need an Asynchronous/Cancellable HTTP GET which can be tough.
The problem is that if read stalls waiting for more data (cable is pulled) it won't quit until either the socket dies or new data comes in.
There are a few path you could follow, tinkering with socket factories to set a good timeout, using http client with timeouts and others.
I would have a look at Apache Http Components which has non blocking HTTP based on java NIO Sockets.

Sending and receiving objects using sockets and threads not working properly

I am currently creating a service allowing to send objects from a client to a server and vice-versa, but experiencing an issue that I unfortunately cannot explain and fix.
First of all, here are the useful classes (I haven't put all methods such as getters and setters in this post).
/**
* This launcher creates a NetworkInterface, waits for a connection, sends a message to the connected client and waits for an incoming message
*
*/
public class ServerLauncher {
public static void main(String[] args) {
try {
NetworkSystem n = new NetworkSystem(4096);
n.startServerManager();
while (n.getCommunications().isEmpty()) {
// this line is unexpectedly magic
System.out.println("Waiting for a new connection...");
}
do {
n.getCommunications().get(0).send(new String("Hello, are you available?"));
} while (n.getCommunications().get(0).getReceiveManager().getReadObjects().isEmpty());
System.out.println(n.getCommunications().get(0).getReceiveManager().getReadObjects().get(0));
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
/**
* This launcher creates a NetworkSystem, connects to the server, waits for an incoming message and anwers back
*
*/
public class ClientLauncher {
public static void main(String[] args) {
try {
NetworkSystem n = new NetworkSystem(8192);
n.instanciateCommunication(new Socket(InetAddress.getLocalHost(), 4096));
while (n.getCommunications().get(0).getReceiveManager().getReadObjects().isEmpty()) {
// this line is unexpectedly magic
System.out.println("Waiting for an object...");
}
System.out.println(n.getCommunications().get(0).getReceiveManager().getReadObjects().get(0));
n.getCommunications().get(0).getSendManager().send(new String("No, I am not! We will talk later..."));
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
/**
* This class handles every incoming messages.
*/
public class ReceiveManager implements Runnable {
private ObjectInputStream inputStream;
private CommunicationManager communicationManager;
private List readObjects;
private boolean receive;
public ReceiveManager(CommunicationManager communicationManager) throws IOException {
this.communicationManager = communicationManager;
this.inputStream = new ObjectInputStream(this.communicationManager.getSocket().getInputStream());
this.readObjects = new ArrayList();
this.receive = true;
}
#Override
public void run() {
Object object = null;
try {
while ((object = this.inputStream.readObject()) != null && this.hasToReceive()) {
this.readObjects.add(object);
}
} catch (ClassNotFoundException | IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
this.setContinueToReceive(false);
}
}
private boolean hasToReceive() {
return this.receive;
}
public void setContinueToReceive(boolean value) {
this.receive = value;
}
}
/**
* This class allows the user to send messages
*/
public class SendManager {
private ObjectOutputStream outputStream;
private CommunicationManager communicationManager;
public SendManager(CommunicationManager communicationManager) throws IOException {
this.communicationManager = communicationManager;
this.outputStream = new ObjectOutputStream(this.communicationManager.getSocket().getOutputStream());
}
public void send(Object object) throws IOException {
this.outputStream.writeObject(object);
this.outputStream.flush();
}
}
So basically, as you may have noticed in the ServerLauncher and the ClientLauncher, there are two "magic" instructions. When those two lines are commented and I run the server then the client, nothing happens. The server and the client are simply running and never stop. However, when I uncomment these two magic lines, every works like a charm: messages are properly sent and received.
Would you guys know the reason of this unexpected behaviour ?
Oh yeah, I forgot, if you guys want me to upload everything to test the project or whatever, just tell me :-)

You're starving the CPU with those spin loops. You should sleep or wait while the queues are empty, or better still just take()from blocking queues.
NB Your loop condition isn't correct:
readObject() doesn't return null at end of stream. It throws EOFException.
You should also test hasToReceive() before calling readObject() rather than afterwards. Otherwise you always do an extra read.

Java sockets game server tweaks

I'm making a multyplayer game and I'm using java sockets for the server, the server is working very well but I think it needs some tweaks regarding the way I'm parsing/handling the requests.
I would like to know if there is a better way of parse the requests instead of splitting input lines by (,) commas like I'm doing.
Should the data sent between the client and server be encrypted in some way? Just a small encryption to obfuscate the requests on sniffers eyes. What is the best way of doing that?
And regarding the thread safe is it OK with Collections.synchronizedList and synchronized blocks on iterations? Or is there a better/cleaner way?
Finally is there any java sockes library that will do all those things above? If so should I use it or is that an overkill for a small java sokets game server.
Is any thing else that I should improve?
Thanks :)
Bellow is the basic structure of my server.
public class Servidor extends Thread {
private ServerSocket serverSocket;
public static boolean LISTENING = true;
private final List<Client> clients = Collections.synchronizedList(new ArrayList<Client>());
private final List<Game> games = Collections.synchronizedList(new ArrayList<Game>());
public Servidor() {
try {
serverSocket = new ServerSocket(SERVER_PORT);
} catch (IOException e) {
Log.add("error starting server: " + e);
}
}
#Override
public void run() {
// Wait for players to connect
while (LISTENING) {
try {
Client c = new Client(serverSocket.accept());
clients.add(c);
c.start();
} catch (IOException e) {}
}
}
class Client extends Thread {
private Socket socket;
private PrintWriter out;
private BufferedReader in;
private boolean loggedin;
private Player player;
private Game game;
public Cliente(Socket sock) {
socket = sock;
try {
out = new PrintWriter(new BufferedWriter(new OutputStreamWriter(socket.getOutputStream(), "UTF-8")), true);
in = new BufferedReader(new InputStreamReader(socket.getInputStream(), "UTF-8"));
} catch (IOException e) {
Log.add("error connecting to player!");
}
}
/* Send to this player */
public void send(String s) {
out.println(s);
}
/* Send to all players on the server */
public void boardcast(String s) {
synchronized (clients) {
for (Client c : clients) {
c.send(s);
}
}
}
/* Get player by ID */
private Player getPlayerByID(int playerID) {
synchronized (clients) {
for (Client c : clients) {
if (c.player != null) {
if (c.player.getID() == playerID) {
return c.player;
}
}
}
}
return null;
}
/* Split inputLine */
private String[] splitInput(String input, int argsNumber) {
String[] args = null;
try {
args = input.split(",", -1);
if (args.length != argsNumber) {
args = null;
}
} catch (Exception ex) {
Log.add("error splitting input");
} finally {
return args;
}
}
#Override
public void run() {
char cmd;
String inputLine, outputLine;
String[] args;
try {
loop:
while ((inputLine = in.readLine()) != null) {
// check if inputLine have 2 chars (CMD_TYPE:)
if (inputLine.length() < 2) {
kickPlayer();
break loop;
}
// get CMD
cmd = inputLine.charAt(0);
// remove (CMD_TYPE:) from inputLine
inputLine = inputLine.substring(2);
// check e player is logged (L:username,password)
if (!loggedin) {
// check if the input string have 2 arguments
if ((args = splitInput(inputLine, 2)) == null) {
kickPlayer();
break loop;
}else{
// ... TESTE LOGIN ON DATABASE ...
// set player data
player = new Player(query.getInt("player_id"), query.getString("username"), query.getInt("level"))
}
}else{
// Commands
switch (cmd) {
// P:CARD_ID,TARGET_ID eg:(P:5:3)
case CMD_PLAY:
// check if the input string have 2 arguments
if ((args = splitInput(inputLine, 2)) == null) {
kickPlayer();
break loop;
} else {
// ... VALIDATE OTHER PARAMETERS ...
// update game
game.addCard(args[0], args[0]);
// update players
boardcast(CMD_PLAY + ":" + player.getID+ "," + game.LastCard());
}
break;
// ... TEST OTHER COMMANDS ...
default:
Log.add("invalid command";
break loop;
}
}
}
} catch (IOException e) {
Log.add("connection lost";
} finally {
removeClient();
}
}
}
}

I am a programmer for 3 years now but now I am really trying to learn network server programming.
Some tips that might help you:
You should take a look about some design patterns to make you code more organized and extensible. You can specify a class for every message type and wrap it. It can use text delimiters or message size for framing the messages.
Keep the message handling code separated from the Client class you created, this makes easier to add support to new messages.
This would be something like:
class ClientHandler
{
public void run()
{
while(isConnected())
messageRouter.handleMessage(getNextMessage());
}
}
interface MessageHandler
{
public boolean canHandle(Message m);
public void handleMessage(Message m);
}
class MessageRouter
{
private List<MessageHandler> handlers;
public void handleMessage(Message msg)
{
for(MessageHandler m : handlers)
{
if(m.canHandle(msg))
{
m.handle(msg);
return;
}
}
throw UnsupportedMessage();
}
}
Something like that...
BTW: I am Brazilian too
good luck with your project

Adding basic security to communication is fairly simple and can be achieved without a lot of hassle. It took me a day of "googling" to figure out how to use it ... I have forgotten most of the details but if you want I can upload the source of my project for you to have a look at ... I personally would recommend some encryption before actually releasing the game...
A link for java ssl sockets can be found here
A link to create your own keystore can be found here
That should help you get started and message me if you want me to upload my source code.
As far as the lists are concerned most of the time you will only be reading the lists so I would suggest that you make a custom lock so that writers must acquire write lock whereas multiple readers can acquire the lock concurrently.. Will require you to design a new class or perhaps you can find a solution on-line.
As far as parsing the string is concerned, to the best of my knowledge objects can be passed remotely so you can transmit array of string instead of having to parse them. I personally broke my communication into several calls to read and write to input streams (probably the slowest approach :) )
and any ways good luck with your project. Message me if you need the source :)
.