The answer here seemed to be a valid solution before Java 8:
How to cancel Files.copy() in Java?
But now it doesn't work, because ExtendedCopyOption.INTERRUPTIBLE is private.
Basically, I need to download a file from some given URL and save it to my local file-system using Files.copy().
Currently, I am using a JavaFX Service because I need to show the progress in a ProgressBar.
However, I don't know how to block the thread running Files.copy() if the operation takes too long.
Using Thread.stop() is at least not wanted. Even Thread.interrupt() fails.
I also want the operation to terminate gracefully if the internet connection becomes unavailable.
To test the case when no internet connection is available, I'm removing my ethernet cable and putting it back after 3 seconds.
Unfortunately, Files.copy() returns only when I put back the ethernet cable, while I would like it to fail immediately.
As I can see, internally Files.copy() is running a loop, which prevents the thread from exiting.
Tester(Downloading OBS Studio exe):
/**
* #author GOXR3PLUS
*
*/
public class TestDownloader extends Application {
/**
* #param args
*/
public static void main(String[] args) {
launch(args);
}
#Override
public void start(Stage primaryStage) throws Exception {
// Block From exiting
Platform.setImplicitExit(false);
// Try to download the File from URL
new DownloadService().startDownload(
"https://github.com/jp9000/obs-studio/releases/download/17.0.2/OBS-Studio-17.0.2-Small-Installer.exe",
System.getProperty("user.home") + File.separator + "Desktop" + File.separator + "OBS-Studio-17.0.2-Small-Installer.exe");
}
}
DownloadService:
Using #sillyfly comment with FileChannel and removing File.copy seems to work only with calling Thread.interrupt() but it is not exiting when the internet is not available..
import java.io.File;
import java.net.URL;
import java.net.URLConnection;
import java.nio.channels.Channels;
import java.nio.channels.FileChannel;
import java.nio.file.StandardOpenOption;
import java.util.logging.Level;
import java.util.logging.Logger;
import javafx.concurrent.Service;
import javafx.concurrent.Task;
/**
* JavaFX Service which is Capable of Downloading Files from the Internet to the
* LocalHost
*
* #author GOXR3PLUS
*
*/
public class DownloadService extends Service<Boolean> {
// -----
private long totalBytes;
private boolean succeeded = false;
private volatile boolean stopThread;
// CopyThread
private Thread copyThread = null;
// ----
private String urlString;
private String destination;
/**
* The logger of the class
*/
private static final Logger LOGGER = Logger.getLogger(DownloadService.class.getName());
/**
* Constructor
*/
public DownloadService() {
setOnFailed(f -> System.out.println("Failed with value: " + super.getValue()+" , Copy Thread is Alive? "+copyThread.isAlive()));
setOnSucceeded(s -> System.out.println("Succeeded with value: " + super.getValue()+" , Copy Thread is Alive? "+copyThread.isAlive()));
setOnCancelled(c -> System.out.println("Succeeded with value: " + super.getValue()+" , Copy Thread is Alive? "+copyThread.isAlive()));
}
/**
* Start the Download Service
*
* #param urlString
* The source File URL
* #param destination
* The destination File
*/
public void startDownload(String urlString, String destination) {
if (!super.isRunning()) {
this.urlString = urlString;
this.destination = destination;
totalBytes = 0;
restart();
}
}
#Override
protected Task<Boolean> createTask() {
return new Task<Boolean>() {
#Override
protected Boolean call() throws Exception {
// Succeeded boolean
succeeded = true;
// URL and LocalFile
URL urlFile = new URL(java.net.URLDecoder.decode(urlString, "UTF-8"));
File destinationFile = new File(destination);
try {
// Open the connection and get totalBytes
URLConnection connection = urlFile.openConnection();
totalBytes = Long.parseLong(connection.getHeaderField("Content-Length"));
// --------------------- Copy the File to External Thread-----------
copyThread = new Thread(() -> {
// Start File Copy
try (FileChannel zip = FileChannel.open(destinationFile.toPath(), StandardOpenOption.CREATE,
StandardOpenOption.TRUNCATE_EXISTING, StandardOpenOption.WRITE)) {
zip.transferFrom(Channels.newChannel(connection.getInputStream()), 0, Long.MAX_VALUE);
// Files.copy(dl.openStream(), fl.toPath(),StandardCopyOption.REPLACE_EXISTING)
} catch (Exception ex) {
stopThread = true;
LOGGER.log(Level.WARNING, "DownloadService failed", ex);
}
System.out.println("Copy Thread exited...");
});
// Set to Daemon
copyThread.setDaemon(true);
// Start the Thread
copyThread.start();
// -------------------- End of Copy the File to External Thread-------
// ---------------------------Check the %100 Progress--------------------
long outPutFileLength;
long previousLength = 0;
int failCounter = 0;
// While Loop
while ((outPutFileLength = destinationFile.length()) < totalBytes && !stopThread) {
// Check the previous length
if (previousLength != outPutFileLength) {
previousLength = outPutFileLength;
failCounter = 0;
} else
++failCounter;
// 2 Seconds passed without response
if (failCounter == 40 || stopThread)
break;
// Update Progress
super.updateProgress((outPutFileLength * 100) / totalBytes, 100);
System.out.println("Current Bytes:" + outPutFileLength + " ,|, TotalBytes:" + totalBytes
+ " ,|, Current Progress: " + (outPutFileLength * 100) / totalBytes + " %");
// Sleep
try {
Thread.sleep(50);
} catch (InterruptedException ex) {
LOGGER.log(Level.WARNING, "", ex);
}
}
// 2 Seconds passed without response
if (failCounter == 40)
succeeded = false;
// --------------------------End of Check the %100 Progress--------------------
} catch (Exception ex) {
succeeded = false;
// Stop the External Thread which is updating the %100
// progress
stopThread = true;
LOGGER.log(Level.WARNING, "DownloadService failed", ex);
}
//----------------------Finally------------------------------
System.out.println("Trying to interrupt[shoot with an assault rifle] the copy Thread");
// ---FORCE STOP COPY FILES
if (copyThread != null && copyThread.isAlive()) {
copyThread.interrupt();
System.out.println("Done an interrupt to the copy Thread");
// Run a Looping checking if the copyThread has stopped...
while (copyThread.isAlive()) {
System.out.println("Copy Thread is still Alive,refusing to die.");
Thread.sleep(50);
}
}
System.out.println("Download Service exited:[Value=" + succeeded + "] Copy Thread is Alive? "
+ (copyThread == null ? "" : copyThread.isAlive()));
//---------------------- End of Finally------------------------------
return succeeded;
}
};
}
}
Interesting questions:
1-> What does java.lang.Thread.interrupt() do?
I strongly encourage you to use a FileChannel.
It has the transferFrom() method which returns immediately when the thread running it is interrupted.
(The Javadoc here says that it should raise a ClosedByInterruptException, but it doesn't.)
try (FileChannel channel = FileChannel.open(Paths.get(...), StandardOpenOption.CREATE,
StandardOpenOption.WRITE)) {
channel.transferFrom(Channels.newChannel(new URL(...).openStream()), 0, Long.MAX_VALUE);
}
It also has the potential to perform much better than its java.io alternative.
(However, it turns out that the implementation of Files.copy() may elect to delegate to this method instead of actually performing the copy by itself.)
Here's an example of a reusable JavaFX Service that lets you fetch a resource from the internet and save it to your local file-system, with automatic graceful termination if the operation takes too long.
The service task (spawned by createTask()) is the user of the file-channel API.
A separate ScheduledExecutorService is used to handle the time constraint.
Always stick to the good practices for extending Service.
If you choose to use such an high-level method, you won't be able to track down the progress of the task.
If the connection becomes unavailable, transferFrom() should eventually return without throwing an exception.
To start the service (may be done from any thread):
DownloadService downloadService = new DownloadService();
downloadService.setRemoteResourceLocation(new URL("http://speedtest.ftp.otenet.gr/files/test1Gb.db"));
downloadService.setPathToLocalResource(Paths.get("C:", "test1Gb.db"));
downloadService.start();
and then to cancel it (otherwise it will be automatically cancelled after the time expires):
downloadService.cancel();
Note that the same service can be reused, just be sure to reset it before starting again:
downloadService.reset();
Here is the DownloadService class:
public class DownloadService extends Service<Void> {
private static final long TIME_BUDGET = 2; // In seconds
private final ScheduledExecutorService watchdogService =
Executors.newSingleThreadScheduledExecutor(new ThreadFactory() {
private final ThreadFactory delegate = Executors.defaultThreadFactory();
#Override
public Thread newThread(Runnable r) {
Thread thread = delegate.newThread(r);
thread.setDaemon(true);
return thread;
}
});
private Future<?> watchdogThread;
private final ObjectProperty<URL> remoteResourceLocation = new SimpleObjectProperty<>();
private final ObjectProperty<Path> pathToLocalResource = new SimpleObjectProperty<>();
public final URL getRemoteResourceLocation() {
return remoteResourceLocation.get();
}
public final void setRemoteResourceLocation(URL remoteResourceLocation) {
this.remoteResourceLocation.set(remoteResourceLocation);
}
public ObjectProperty<URL> remoteResourceLocationProperty() {
return remoteResourceLocation;
}
public final Path getPathToLocalResource() {
return pathToLocalResource.get();
}
public final void setPathToLocalResource(Path pathToLocalResource) {
this.pathToLocalResource.set(pathToLocalResource);
}
public ObjectProperty<Path> pathToLocalResourceProperty() {
return pathToLocalResource;
}
#Override
protected Task<Void> createTask() {
final Path pathToLocalResource = getPathToLocalResource();
final URL remoteResourceLocation = getRemoteResourceLocation();
if (pathToLocalResource == null) {
throw new IllegalStateException("pathToLocalResource property value is null");
}
if (remoteResourceLocation == null) {
throw new IllegalStateException("remoteResourceLocation property value is null");
}
return new Task<Void>() {
#Override
protected Void call() throws IOException {
try (FileChannel channel = FileChannel.open(pathToLocalResource, StandardOpenOption.CREATE,
StandardOpenOption.WRITE)) {
channel.transferFrom(Channels.newChannel(remoteResourceLocation.openStream()), 0, Long.MAX_VALUE);
}
return null;
}
};
}
#Override
protected void running() {
watchdogThread = watchdogService.schedule(() -> {
Platform.runLater(() -> cancel());
}, TIME_BUDGET, TimeUnit.SECONDS);
}
#Override
protected void succeeded() {
watchdogThread.cancel(false);
}
#Override
protected void cancelled() {
watchdogThread.cancel(false);
}
#Override
protected void failed() {
watchdogThread.cancel(false);
}
}
There is one important aspect not covered by the other answers/comments; and that is a wrong assumption of yours:
What I want is it to fail immediately when no internet connection is there.
It is not that easy. The TCP stack/state machine is actually a pretty complicated thing; and depending on your context (OS type; TCP stack implementation, kernel parameters, ...), there can be situations where a network partition takes place and a sender doesn't notice for 15 or more minutes. Listen here for more details on that.
In other words: "just pulling the plug" is no way equal to "immediately breaking" your existing TCP connection. And just for the record: you don't need to plug cables manually to simulate network outages. In a reasonable test setup, tools like iptables aka firewalls can do that for you.
You seem to need an Asynchronous/Cancellable HTTP GET which can be tough.
The problem is that if read stalls waiting for more data (cable is pulled) it won't quit until either the socket dies or new data comes in.
There are a few path you could follow, tinkering with socket factories to set a good timeout, using http client with timeouts and others.
I would have a look at Apache Http Components which has non blocking HTTP based on java NIO Sockets.
Related
I have Java EE (desktop) application that had to process data files generated by multiple sources (up to a 200 different sources). Each source periodically generates data file with unique name which also contains that source's unique ID.
I need to create a thread pool with 15 threads which will process and remove files with these constraints:
Multiple threads can't process files from the same source simultaneously.
Multiple files from the same source should be processed in order of it's creation timestamp.
No synchronization with the file generator sources is possible so it means that the next file(s) may be generated by source while it's previous file is processed or scheduled for processing.
Processing should be multi threaded because of performance reasons (single threaded processing is not enough so I'm planning to use 10-15 threads).
A file processing operation may be time consuming 3-15 seconds.
Any suggestions on architecture of such complex synchronization of the threads in pool are welcome.
P.S. Due to the limitation on simultaneous processing the design I've used in more simple situations earlier i.e. using ArrayBlockingQueue does not fit this case.
General idea:
You have a task-queue per source.
And you have a central queue which is effectively a queue of task-queues which is shared between all worker threads.
For each source you create a task-queue. And you stick these task-queue's in a hashtable based on the unique id. This way you get the guarantee that tasks from the same source are processed in order (requirement 2).
If a tasks is received, you look up (or create) the task-queue in the hashtable and you add the task to the taskqueue. If it was the first task added to the queue, you also add it to the central queue.
Then there are a bunch of worker-threads that take task-queues from this central queue and then take a single task from this task-queue they just took and process that task. Once they are done with the task, they need to decide if the task-queue needs to be reinserted back into the central-queue or not.
There are a few parts were things could easily go wrong:
You don't want to end up with a task-queue being inserted into the central-queue multiple times. That would violate your first requirement.
You don't want the task-queue not being reinserted into the central-queue even though a task is available.
So you need to take care of the appropriate synchronization and it might be a bit more complex than you would initially think. But seen the fact that the tasks are long running, I would start out with a regular mutex (could be per task-queue) e.g. synchronized or a lock and don't worry about making it non blocking.
So this is the skeleton/tester of the class I've created to solve my problem. See #pveentjer answer for some details on whats going on here.
package org.zur.test;
import java.io.File;
import java.util.Comparator;
import java.util.HashMap;
import java.util.PriorityQueue;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.Semaphore;
import java.util.concurrent.ThreadLocalRandom;
import java.util.concurrent.TimeUnit;
public class PmFileProcessor {
private static final int THREADS_COUNT = 15;
ArrayBlockingQueue<File> scheduledFiles = new ArrayBlockingQueue<>(10000, false);
HashMap<String, PmFileProcessingJob> allJobs = new HashMap<>();
ScheduledExecutorService jobsExecutorService = Executors.newScheduledThreadPool(THREADS_COUNT);
public PmFileProcessor() {
super();
SourceJobsManager fc = new SourceJobsManager();
fc.setDaemon(true);
fc.start();
}
public void scheduleFile(File f) {
try {
scheduledFiles.add(f);
} catch (Exception e) {
// TODO: handle exception
}
}
/**
* Assigns files to file source processing job.
*
* #author
* <ul>
* <li>Zur13</li>
* </ul>
*
*/
public class SourceJobsManager extends Thread {
#Override
public void run() {
// assigns scheduled files to per-source jobs and schedules job for additional execution
while ( true ) {
try {
File f = scheduledFiles.take();
PmFileProcessingJob job = getSourceJob(f);
job.scheduleSourceFile(f);
jobsExecutorService.execute(job); // schedules job execution
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
// TODO: check disk space periodically
}
}
/**
* Finds existing job for file source or creates a new one.
*
* #param f
* #return
*/
private PmFileProcessingJob getSourceJob(File f) {
// TODO: test code
String fname = f.getName();
String[] parts = fname.split("_");
String uid = parts[0];
PmFileProcessingJob res = allJobs.get(uid);
if ( res == null ) {
res = new PmFileProcessingJob(uid);
allJobs.put(uid, res);
}
return res;
}
}
/**
* Process first file from scheduledSourceFiles queue (i.e. each job execution processes a single file or
* reschedules itself for later execution if another thread already processes the file from the same source).
*
* #author
* <ul>
* <li>Zur13</li>
* </ul>
*
*/
public class PmFileProcessingJob implements Runnable {
public final String fileSourceUidString;
PriorityQueue<File> scheduledSourceFiles = new PriorityQueue<>(1000, new Comparator<File>() {
#Override
public int compare(File o1, File o2) {
// TODO Auto-generated method stub
return 0;
}
});
Semaphore onePassSemaphore = new Semaphore(1);
public PmFileProcessingJob(String fileSourceUid) {
super();
this.fileSourceUidString = fileSourceUid;
}
/**
* Schedules file from for processing by this job.
*/
public void scheduleSourceFile(File f) {
scheduledSourceFiles.add(f);
}
#Override
public void run() {
File f = null;
if ( scheduledSourceFiles.size() > 0 ) { // fail fast optimization 1
if ( onePassSemaphore.tryAcquire() ) { // fail fast optimization 2
try {
f = scheduledSourceFiles.poll();
if ( f != null ) {
// TODO: process the file
try {
System.err.println(f.getName() + "\t" + Thread.currentThread().getId());
Thread.sleep(1000);
return;
} catch (Exception e) {
// TODO: handle exception
return; // prevents reschedule loop for failing files
}
} else {
// scheduledSourceFiles queue is empty
return;
}
} finally {
onePassSemaphore.release();
}
}
if ( f == null && scheduledSourceFiles.size() > 0 ) {
// this thread did not process the scheduled file because another thread holds the critical section
// pass
// this thread should reschedule this Job to release this thread and try to process this job later
// with another thread
// reschedule the job with 4 seconds delay to prevent excess CPU usage
// System.err.println("RESCHEDULE");
jobsExecutorService.schedule(this, 3, TimeUnit.SECONDS);
}
}
}
#Override
public int hashCode() {
final int prime = 31;
int result = 1;
result = prime * result + ((this.fileSourceUidString == null) ? 0 : this.fileSourceUidString.hashCode());
return result;
}
#Override
public boolean equals(Object obj) {
if ( this == obj )
return true;
if ( !(obj instanceof PmFileProcessingJob) )
return false;
PmFileProcessingJob other = (PmFileProcessingJob) obj;
if ( this.fileSourceUidString == null ) {
if ( other.fileSourceUidString != null )
return false;
} else if ( !this.fileSourceUidString.equals(other.fileSourceUidString) )
return false;
return true;
}
}
public static void main(String[] args) {
PmFileProcessor fp = new PmFileProcessor();
fp.unitTest();
}
private void unitTest() {
// TODO Auto-generated method stub
int filesCount = 1000;
for (int i = 0; i < filesCount; i++) {
int sourceUid = ThreadLocalRandom.current().nextInt(1, 30);
File f = new File(sourceUid + "_" + i);
scheduleFile(f);
}
Thread.yield();
try {
Thread.sleep(999000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
I'm working on an academic project that consists in controlling a simulated environment created in Unity3D, from a conventional Android application. I already created the scenario in Unity and the mobile application is completely finished, my problem is in the connection. I chose to use sockets because of its simplicity. I managed to connect the application to a server written in C # through conventional Sockets, I know that information can be sent, but when I implemented it in Unity, everything failed, so I decided to use TCP Listener instead of Sockets in Unity / C # (the client Android still uses a conventional TCP connection through Sockets), and in effect, the application connects but there is no information transfer, because the Unity console throws an error that says: ObjectDisposedException: Can not access to disposed object.
Object name: 'System.Net.Sockets.TcpClient'.(The error is presented in line number 56: stream = client.tcp.GetStream ();) The main objective is to obtain the flow of information that would be represented in something like: "1: 0: 1: 0" and being able to make a Split to that string and according to that value change the state of a light bulb or other element, nevertheless I need the essential thing: to establish the connection flow. I'm not an expert in C # and much less using Unity, I really do not know much about design, but I wanted to do it to deliver an innovative work, I hope someone can guide me.
PD: C# Socket (No Unity yet) server working with Android Java client:
![1] https://imgur.com/a/HuNcDC3
//This is my Unity3D/C# Server:
public class SocketManager : MonoBehaviour
{
private List<ServerClient> connectedClients;
private List<ServerClient> disconectedClients;
private TcpListener server;
private string data;
private NetworkStream stream;
private StreamReader stremaReader;
private bool serverStarted;
public int socketPort = 7691;
private void Start()
{
connectedClients = new List<ServerClient>();
disconectedClients = new List<ServerClient>();
try
{
server = new TcpListener(IPAddress.Parse("192.168.1.64"), socketPort);
server.Start();
serverListening();
serverStarted = true;
Debug.Log("Server started. Port: " + socketPort.ToString());
}
catch (Exception ex)
{
Debug.Log("Server socket error: " + ex);
}
}
private void Update()
{
if (!serverStarted)
return;
foreach (ServerClient client in connectedClients)
{
if (IsConnected(client.tcp))
{
client.tcp.Close();
disconectedClients.Add(client);
continue;
}
else
{
stream = client.tcp.GetStream();
if (stream.DataAvailable)
{
this.stremaReader = new StreamReader(stream, true);
data = stremaReader.ReadLine();
if (data != null)
OnIcomingData(client, data);
}
}
}
}
private void OnIcomingData(ServerClient client, string data)
{
Debug.Log(client.clientName + ": " + data);
}
private bool IsConnected(TcpClient tcp)
{
try
{
if (tcp != null && tcp.Client != null && tcp.Client.Connected)
{
if (tcp.Client.Poll(0, SelectMode.SelectRead))
{
return !(tcp.Client.Receive(new byte[1], SocketFlags.Peek) == 0);
}
return true;
}
else
{
return false;
}
}
catch
{
return false;
}
}
private void serverListening()
{
server.BeginAcceptTcpClient(AcceptTcpClient, server);
}
private void AcceptTcpClient(IAsyncResult asyncResult)
{
TcpListener tcplistener = (TcpListener)asyncResult.AsyncState;
connectedClients.Add(new ServerClient(tcplistener.EndAcceptTcpClient(asyncResult)));
serverListening();
}
}
public class ServerClient
{
public TcpClient tcp;
public string clientName;
public ServerClient(TcpClient tcp)
{
this.tcp = tcp;
this.clientName = "Android";
}
}
// This is my Android/Java Client:
public class SocketThread extends Thread {
private Socket adviser;
private DataOutputStream dataOut;
#Override
public void run() {
super.run();
Log.e("Status:", "Thread started");
try {
adviser = new Socket("192.168.1.64", 7691);
dataOut = new DataOutputStream(adviser.getOutputStream());
} catch (IOException ex) {
Logger.getLogger(SocketThread.class.getName()).log(Level.SEVERE, null, ex);
}
}
public void sendCommand(String text) {
try {
dataOut.writeUTF(text);
Log.e("Sended Text: ", text);
} catch (IOException ex) {
Logger.getLogger(SocketThread.class.getName()).log(Level.SEVERE, null, ex);
}
}
Here's a more detailed answer than my comment, for future reference.
The problem lies in the following if check if (IsConnected(client.tcp)).
Right now what is happening is that when the client is connected to the server (so the if returns true) you are Closing the connection using client.tcp.Close(); and adding the connection to your disconectedClients list (But you are not removing the client from your connectedClients list! Meaning the client is now in both lists). Then when the next iteration of the foreach loop foreach (ServerClient client in connectedClients) comes around the if (IsConnected(client.tcp)) will return false (the client is no longer connected to the server, since you called close() in the previous iteration.
You then try to call stream = client.tcp.GetStream(); on a stream that is already closed, resulting in an error telling you you cannot get the stream.
This should fix it:
if (IsConnected(client.tcp))
{
client.tcp.Close();
client.tcp.Dispose(); //Dispose of a stream to release it for GC
disconectedClients.Add(client);
connectedClients.Remove(client); //Note to also remove the disconnected client from the connectedClients list.
continue;
}
else
{ ...rest of logic }
Note that i added connectedClients.Remove(client); and client.tcp.Dispose(); to your logic closing the stream. If you don't do this it'll keep getting included in theforeachloop unnecessarily, and the programm will try to callclose()` on a connection that is already closed, resulting in more errors. Disposing of the stream will release it so the garbage collector can collect it, freeing up memory.
I have a java application running on my raspberry pi but it crashes most of the time. Whenever it crashes it usually has a very high CPU usage (> 100%) from java. How my application works: I have a RFID reader that reads tags and whenever a tag is read, a messageReceived method is called. That method stores the read tags in a specific set. Then I create a new thread which listens to a socket and while the socket is open and when the set has changed, the thread calls some javafx methods to open new screens. However, when I deploy the application to my raspberry pi, it crashes randomly and has a high CPU usage with java. Feel free to ask any questions if I forgot to explain anything .
Edit 1: my thread class.
Edit 2: My question now is: why do I have such a high CPU usage and how can I fix it.
public class RFIDThread implements Runnable {
/**
* The socket for the connection to the LLRP Reader
*/
private Socket socket;
private JSONArray valid_tags;
private JSONArray found_tags;
private TagsListController controller;
private RFIDSet rfidset;
/**
* Thread for constant reading of the stream
*
* #param socket
* #param controller
* #param tags
*/
public RFIDThread(Socket socket, TagsListController controller, JSONArray tags, RFIDSet rfidset) {
this.socket = socket;
this.controller = controller;
this.rfidset = rfidset;
this.found_tags = new JSONArray();
this.valid_tags = tags;
}
/**
* Runnable for this thread.
* First get all the found tags from the xml controller
* Then loop over the rfid set to find any new tags.
* If there are any, display them.
*/
#Override
public void run() {
CopyOnWriteArrayList<Tag> originalSet = new CopyOnWriteArrayList<>();
originalSet.addAll(rfidset.getSet());
boolean started = true;
if (socket.isConnected()) {
while (!socket.isClosed()) {
CopyOnWriteArrayList<Tag> set = new CopyOnWriteArrayList<>();
set.addAll(rfidset.getSet());
if(started || !originalSet.equals(set)) {
started = false;
CopyOnWriteArrayList<String> found_set = new CopyOnWriteArrayList<>();
found_set.addAll(controller.found_tags_list.getItems());
this.found_tags.clear();
this.found_tags.addAll(found_set);
for (Tag found_tag : set) {
if (found_tags.indexOf(found_tag.getId()) < 0) {
Integer index = valid_tags.indexOf(found_tag.getId());
if (index >= 0) {
Platform.runLater(() -> controller.showValid(found_tag.getId()));
} else {
Platform.runLater(() -> controller.showError(found_tag.getId()));
}
found_tags.add(found_tag.getId());
pause(5000);
}
}
originalSet = set;
pause(5000);
}
}
}
}
/**
* Close the socket
*/
public void shutdown() {
try {
this.socket.close();
} catch (IOException e) {
e.printStackTrace();
}
}
private void pause(long ms) {
try {
Thread.sleep(ms);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
Try moving the pause(5000); outside the if (started || !originalSet.equals(set)) { statement.
High cpu usage is usually a tight loop with no pause or I/O or waiting for stuff. In your case whenever the originalSet.equals(set) you will not pause.
You may prefer to just use:
if (started || !originalSet.equals(set)) {
// ...
} else {
pause(0);
}
or similar.
I launched my instance overnight to see how it handled things and when I came by this morning, I was facing a
Exception in thread "pool-535-thread-7" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:691)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:992)[info] application - Connecting to server A
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
The aim of my code is quite simple : Every 5 minutes, I connect to a list of remote server, send a request (via socket) and that's it.
Here's my code :
My "cron" task :
/** will create a new instance of ExecutorService every 5 minutes, loading all the websites in the database to check their status **/
/** Maybe that's where the problem is ? I need to empty (GC ?) this ExecutorService ? **/
Akka.system().scheduler().schedule(
Duration.create(0, TimeUnit.MILLISECONDS), // Initial delay 0 milliseconds
Duration.create(5, TimeUnit.MINUTES), // Frequency 5 minutes
new Runnable() {
public void run() {
// We get the list of websites to check
Query<Website> query = Ebean.createQuery(Website.class, "WHERE disabled = false AND removed IS NULL");
query.order("created ASC");
List<Website> websites = query.findList(); // Can be 1, 10, 100, 1000. In my test case, I had only 9 websites.
ExecutorService executor = Executors.newFixedThreadPool(NTHREDS);
for (Website website : websites) {
CheckWebsite task = new CheckWebsite(website);
executor.execute(task);
}
// This will make the executor accept no new threads
// and finish all existing threads in the queue
executor.shutdown();
}
},
Akka.system().dispatcher()
);
My CheckWebsite class :
public class CheckWebsite implements Runnable {
private Website website;
public CheckWebsite(Website website) {
this.website = website;
}
#Override
public void run() {
WebsiteLog log = website.checkState(); // This is where the request is made, I copy paste the code just after
if (log == null) {
Logger.error("OHOH, WebsiteLog should not be null for website.checkState() in CheckWebsite class :s");
return;
}
try {
log.save();
catch (Exception e) {
Logger.info ("An error occured :/");
Logger.info(e.getMessage());
e.printStackTrace();
}
}
}
My checkState() method in Website.class :
public WebsiteLog checkState() {
// Since I use Socket and the connection can hang indefinitely, I use an other ExecutorService in order to limit the time spent
// The duration is defined via Connector.timeout, Which will be the next code.
ExecutorService executor = Executors.newFixedThreadPool(1);
Connector connector = new Connector(this);
try {
final long startTime = System.nanoTime();
Future<String> future = executor.submit(connector);
String response = future.get(Connector.timeout, TimeUnit.MILLISECONDS);
long duration = System.nanoTime() - startTime;
return PlatformLog.getLastOccurence(this, response, ((int) duration/ 1000000));
}
catch (Exception e) {
return PlatformLog.getLastOccurence(this, null, null);
}
}
Here's the Connector.class. I removed useless part here (like Catches) :
public class Connector implements Callable<String> {
public final static int timeout = 2500; // WE use a timeout of 2.5s, which should be enough
private Website website;
public Connector(Website website) {
this.website = website;
}
#Override
public String call() throws Exception {
Logger.info ("Connecting to " + website.getAddress() + ":" + website.getPort());
Socket socket = new Socket();
try {
socket.connect(new InetSocketAddress(website.getIp(), website.getPort()), (timeout - 50));
BufferedReader input = new BufferedReader(new InputStreamReader(socket.getInputStream()));
String response = input.readLine();
socket.close();
return response;
}
catch (Exception e) {
e.printStackTrace();
throw e;
}
finally {
// I take the precaution to close the socket here in order to avoid a memory leak
// But if the previous ExecutorService force the close of this thread before
// I can't guarantee it will be closed :/
if (socket != null && !socket.isClosed()) {
socket.close();
}
}
}
}
I'm new to Java multithreading so I probably made big mistake. I suspect some area that could be potentially the reason, but my lack of knowledge requires me to ask for your help :)
As a summary, here's the potentials areas :
Creating a new ExecutorService every 5 minutes. Maybe I can reuse the old one ? Or do I need to close the current one when finished (if so, how ?).
The fact that I create an ExecutorService that will create an ExecutorService (in the checkstate() method)
The fact that the Connector class can be (violently) stopped by the ExecutorService running it, if it takes too long, resulting in a socket not closed (and then a memory leak) ?
Also, as you can see, the exception occured for the thread "pool-535-thread-7" which mean it didn't happen soon.
I store the last_occured check in the database, and the creation of the log entry (in WebsiteLog), the delta is around 5 hours (so, for every 5 minutes, the thread crashed after around 60 calls).
Update : Here's the revisited checkState method to include the shutdown call :
public PlatformLog checkState() {
ExecutorService executor = Executors.newFixedThreadPool(1);
Connector connector = new Connector(this);
String response = null;
Long duration = null;
try {
final long startTime = System.nanoTime();
Future<String> future = executor.submit(connector);
response = future.get(Connector.timeout, TimeUnit.MILLISECONDS);
duration = System.nanoTime() - startTime;
}
catch (Exception e) {}
executor.shutdown();
if (duration != null) {
return WebsiteLog.getLastOccurence(this, response, (duration.intValue()/ 1000000));
}
else {
return WebsiteLog.getLastOccurence(this, response, null);
}
}
I'm not sure this is the only problem, but you are creating an ExecutorService in your checkState() method but you don't shut it down.
According to the JavaDocs for Executors.newFixedThreadPool():
The threads in the pool will exist until it is explicitly shutdown.
The threads staying alive will cause the ExecutorService not to be garbage collected (which would call shutdown() on your behalf. Hence you are leaking a thread each time this is called.
I'm having an issue with a multi-threaded server I'm building as an academic exercise, more specifically with getting a connection to close down gracefully.
Each connection is managed by a Session class. This class maintains 2 threads for the connection, a DownstreamThread and an UpstreamThread.
The UpstreamThread blocks on the client socket and encodes all incoming strings into messages to be passed up to another layer to deal with. The DownstreamThread blocks on a BlockingQueue into which messages for the client are inserted. When there's a message on the queue, the Downstream thread takes the message off the queue, turns it into a string and sends it to the client. In the final system, an application layer will act on incoming messages and push outgoing messages down to the server to send to the appropriate client, but for now I just have a simple application that sleeps for a second on an incoming message before echoing it back as an outgoing message with a timestamp appended.
The problem I'm having is getting the whole thing to shut down gracefully when the client disconnects. The first issue I'm contending with is a normal disconnect, where the client lets the server know that it's ending the connection with a QUIT command. The basic pseudocode is:
while (!quitting) {
inputString = socket.readLine () // blocks
if (inputString != "QUIT") {
// forward the message upstream
server.acceptMessage (inputString);
} else {
// Do cleanup
quitting = true;
socket.close ();
}
}
The upstream thread's main loop looks at the input string. If it's QUIT the thread sets a flag to say that the client has ended communications and exits the loop. This leads to the upstream thread shutting down nicely.
The downstream thread's main loop waits for messages in the BlockingQueue for as long as the connection closing flag isn't set. When it is, the downstream thread is also supposed to terminate. However, it doesn't, it just sits there waiting. Its psuedocode looks like this:
while (!quitting) {
outputMessage = messageQueue.take (); // blocks
sendMessageToClient (outputMessage);
}
When I tested this, I noticed that when the client quit, the upstream thread shut down, but the downstream thread didn't.
After a bit of head scratching, I realised that the downstream thread is still blocking on the BlockingQueue waiting for an incoming message that will never come. The upstream thread doesn't forward the QUIT message any further up the chain.
How can I make the downstream thread shut down gracefully? The first idea that sprang to mind was setting a timeout on the take() call. I'm not too keen on this idea though, because whatever value you select, it's bound to be not entirely satisfactory. Either it's too long and a zombie thread sits there for a long time before shutting down, or it's too short and connections that have idled for a few minutes but are still valid will be killed. I did think of sending the QUIT message up the chain, but that requires it to make a full round trip to the server, then the application, then back down to the server again and finally to the session. This doesn't seem like an elegant solution either.
I did look at the documentation for Thread.stop() but that's apparently deprecated because it never worked properly anyway, so that looks like it's not really an option either. Another idea I had was to force an exception to be triggered in the downstream thread somehow and let it clean up in its finally block, but this strikes me as a horrible and kludgey idea.
I feel that both threads should be able to gracefully shutdown on their own, but I also suspect that if one thread ends it must also signal the other thread to end in a more proactive way than simply setting a flag for the other thread to check. As I'm still not very experienced with Java, I'm rather out of ideas at this point. If anyone has any advice, it would be greatly appreciated.
For the sake of completeness, I've included the real code for the Session class below, though I believe the pseudocode snippets above cover the relevant parts of the problem. The full class is about 250 lines.
import java.io.*;
import java.net.*;
import java.util.concurrent.*;
import java.util.logging.*;
/**
* Session class
*
* A session manages the individual connection between a client and the server.
* It accepts input from the client and sends output to the client over the
* provided socket.
*
*/
public class Session {
private Socket clientSocket = null;
private Server server = null;
private Integer sessionId = 0;
private DownstreamThread downstream = null;
private UpstreamThread upstream = null;
private boolean sessionEnding = false;
/**
* This thread handles waiting for messages from the server and sending
* them to the client
*/
private class DownstreamThread implements Runnable {
private BlockingQueue<DownstreamMessage> incomingMessages = null;
private OutputStreamWriter streamWriter = null;
private Session outer = null;
#Override
public void run () {
DownstreamMessage message;
Thread.currentThread ().setName ("DownstreamThread_" + outer.getId ());
try {
// Send connect message
this.sendMessageToClient ("Hello, you are client " + outer.getId ());
while (!outer.sessionEnding) {
message = this.incomingMessages.take ();
this.sendMessageToClient (message.getPayload ());
}
// Send disconnect message
this.sendMessageToClient ("Goodbye, client " + getId ());
} catch (InterruptedException | IOException ex) {
Logger.getLogger (DownstreamThread.class.getName ()).log (Level.SEVERE, ex.getMessage (), ex);
} finally {
this.terminate ();
}
}
/**
* Add a message to the downstream queue
*
* #param message
* #return
* #throws InterruptedException
*/
public DownstreamThread acceptMessage (DownstreamMessage message) throws InterruptedException {
if (!outer.sessionEnding) {
this.incomingMessages.put (message);
}
return this;
}
/**
* Send the given message to the client
*
* #param message
* #throws IOException
*/
private DownstreamThread sendMessageToClient (CharSequence message) throws IOException {
OutputStreamWriter osw;
// Output to client
if (null != (osw = this.getStreamWriter ())) {
osw.write ((String) message);
osw.write ("\r\n");
osw.flush ();
}
return this;
}
/**
* Perform session cleanup
*
* #return
*/
private DownstreamThread terminate () {
try {
this.streamWriter.close ();
} catch (IOException ex) {
Logger.getLogger (DownstreamThread.class.getName ()).log (Level.SEVERE, ex.getMessage (), ex);
}
this.streamWriter = null;
return this;
}
/**
* Get an output stream writer, initialize it if it's not active
*
* #return A configured OutputStreamWriter object
* #throws IOException
*/
private OutputStreamWriter getStreamWriter () throws IOException {
if ((null == this.streamWriter)
&& (!outer.sessionEnding)) {
BufferedOutputStream os = new BufferedOutputStream (outer.clientSocket.getOutputStream ());
this.streamWriter = new OutputStreamWriter (os, "UTF8");
}
return this.streamWriter;
}
/**
*
* #param outer
*/
public DownstreamThread (Session outer) {
this.outer = outer;
this.incomingMessages = new LinkedBlockingQueue ();
System.out.println ("Class " + this.getClass () + " created");
}
}
/**
* This thread handles waiting for client input and sending it upstream
*/
private class UpstreamThread implements Runnable {
private Session outer = null;
#Override
public void run () {
StringBuffer inputBuffer = new StringBuffer ();
BufferedReader inReader;
Thread.currentThread ().setName ("UpstreamThread_" + outer.getId ());
try {
inReader = new BufferedReader (new InputStreamReader (outer.clientSocket.getInputStream (), "UTF8"));
while (!outer.sessionEnding) {
// Read whatever was in the input buffer
inputBuffer.delete (0, inputBuffer.length ());
inputBuffer.append (inReader.readLine ());
System.out.println ("Input message was: " + inputBuffer);
if (!inputBuffer.toString ().equals ("QUIT")) {
// Forward the message up the chain to the Server
outer.server.acceptMessage (new UpstreamMessage (sessionId, inputBuffer.toString ()));
} else {
// End the session
outer.sessionEnding = true;
}
}
} catch (IOException | InterruptedException e) {
Logger.getLogger (Session.class.getName ()).log (Level.SEVERE, e.getMessage (), e);
} finally {
outer.terminate ();
outer.server.deleteSession (outer.getId ());
}
}
/**
* Class constructor
*
* The Core Java volume 1 book said that a constructor such as this
* should be implicitly created, but that doesn't seem to be the case!
*
* #param outer
*/
public UpstreamThread (Session outer) {
this.outer = outer;
System.out.println ("Class " + this.getClass () + " created");
}
}
/**
* Start the session threads
*/
public void run () //throws InterruptedException
{
Thread upThread = new Thread (this.upstream);
Thread downThread = new Thread (this.downstream);
upThread.start ();
downThread.start ();
}
/**
* Accept a message to send to the client
*
* #param message
* #return
* #throws InterruptedException
*/
public Session acceptMessage (DownstreamMessage message) throws InterruptedException {
this.downstream.acceptMessage (message);
return this;
}
/**
* Accept a message to send to the client
*
* #param message
* #return
* #throws InterruptedException
*/
public Session acceptMessage (String message) throws InterruptedException {
return this.acceptMessage (new DownstreamMessage (this.getId (), message));
}
/**
* Terminate the client connection
*/
private void terminate () {
try {
this.clientSocket.close ();
} catch (IOException e) {
Logger.getLogger (Session.class.getName ()).log (Level.SEVERE, e.getMessage (), e);
}
}
/**
* Get this Session's ID
*
* #return The ID of this session
*/
public Integer getId () {
return this.sessionId;
}
/**
* Session constructor
*
* #param owner The Server object that owns this session
* #param sessionId The unique ID this session will be given
* #throws IOException
*/
public Session (Server owner, Socket clientSocket, Integer sessionId) throws IOException {
this.server = owner;
this.clientSocket = clientSocket;
this.sessionId = sessionId;
this.upstream = new UpstreamThread (this);
this.downstream = new DownstreamThread (this);
System.out.println ("Class " + this.getClass () + " created");
System.out.println ("Session ID is " + this.sessionId);
}
}
Instead of calling Thread.stop use Thread.interrupt. That will cause the take method to throw an InterruptedException which you can use to know that you should shut down.
Can you just create "fake" quit message instead of setting outer.sessionEnding to true when "QUIT" appears. Putting this fake message in queue will wake the DownstreamThread and you can end it. In that case you can even eliminate this sessionEnding variable.
In pseudo code this could look like this:
while (true) {
outputMessage = messageQueue.take (); // blocks
if (QUIT == outputMessage)
break
sendMessageToClient (outputMessage);
}