We Keep Coding

One Producer ten consumers file-processing with Executors.newSingleThreadExecutor()

I have a LinkedBlockingQueue with an arbitrarily picked capacity of 10, and an input file with 1000 lines. I have one ExecutorService-type variable in the main method of the service class that, to my knowledge, first handles--using Executors.newSingleThreadExecutor()--a single thread to call buffer.readline() until file line == null, and then handles--within a loop using Executors.newSingleThreadExecutor()--ten threads to process lines and write them to output files, until !queue.take().equals("Stop"). However, after writing some lines to files, when I am in the debug mode, I see that the capacity of the queue eventually reaches max (10), and the processing threads do not execute queue.take(). All threads are in the running state, but the process halts after queue.put(). What would cause this problem, and is it solvable using some combination of thread-pooling or multiple ExecutorServicehandler variables, instead of a single variable?
Outline for current state of main method in service:
//app settings to get values for keys within a properties file
AppSettings appSettings = new AppSettings();
BlockingQueue<String> queue = new LinkedBlockingQueue<String>(10);
maxProdThreads = 1;
maxConsThreads = 10;
ExecutorService execSvc = null;
for (int i = 0; i < maxProdThreads; i++) {
execSvc = Executors.newSingleThreadExecutor();
execSvc.submit(new ReadJSONMessage(appSettings,queue));
}
for (int i = 0; i < maxConsThreads; i++) {
execSvc = Executors.newSingleThreadExecutor();
execSvc.submit(new ProcessJSONMessage(appSettings,queue));
}
Reading method code:
buffer = new BufferedReader(new FileReader(inputFilePath));
while((line = buffer.readLine()) != null){
line = line.trim();
queue.put(line);
}
Processing and Writing code:
while(!(line=queue.take()).equals("Stop")){
if(line.length() > 10)
{
try {
if(processMessage(line, outputFilePath) == true)
{
++count;
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
public boolean processMessage(String line, String outputFilePath){
CustomObject cO = new CustomObject();
cO.setText(line);
writeToFile1(cO,...);
writeToFile2(cO,...);
}
public void writeOutputAToFile(CustomObject cO,...){
synchronized(cO){
...
org.apache.commons.io.FileUtils.writeStringToFile(...)
}
}
public void writeOutputBToFile(CustomObject cO,...){
synchronized(cO){
...
org.apache.commons.io.FileUtils.writeStringToFile(...)
}
}

In the Processing and writing code..ensure that all resources are closed properly..Probably the resources might not be closed properly due to which the thread keeps running and the ExecutorService can not find an idle thread...

ExecutorService concurrency acts erratic and unstable

I have a project about concurrency and I have some trouble with the behaviour of my code. I am walking a file tree to find all files, and if I find a file which ends on .txt I submit a task to the executor. A thread will open the file and check what number in the file is the biggest. I then create an object, which holds the path for the file and the biggest number for that file. I append the object to a synchronized arraylist. But when I run the code, my arraylist sometimes have 1 object in it or 5 or 112 or 64. There should be 140 objects every time I run it. I hope you guys knows what the problem is.
public static List< Result > AllFiles( Path dir ) throws InterruptedException{
final List<Result> resultlist = new ArrayList<Result>();
final List<Result> synclist;
synclist = Collections.synchronizedList(resultlist);
ExecutorService exec
= Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors() + 1);
try {
Files.walk(dir).forEach(i -> {
String pathfile = i.getFileName().toString();
if (pathfile.contains(".txt")) {
exec.submit(() -> {
int high = findHighest(i);
ResultObj obj = new ResultObj(i, high);
synclist.add(obj);
});
}
});
exec.shutdown();
try {
exec.awaitTermination(1, TimeUnit.NANOSECONDS);
} catch (InterruptedException ex) {}
} catch (IOException ex) {}
System.out.println(synclist);
System.out.println(synclist.size());
return synclist;
}

You are only waiting 1 nanosecond in your awaitTermination call for your ExecutorService to shut down. As a result, you may be printing synclist before some of your files have been processed.

Modify running Java program

I have launched my program that have two loop like this :
for(int i=0;i<100;i++)
String[] articles = getArticles(i);
for(int j=0;j<articles.length;j++)
process(articles[j]);
Can I modify the running program so it stop at i = 1? If how to do that?
The program will take days to finish and I want to stop but I must know where it was stoped so the next time I can resume it.

Unfortunately you won't be able to stop a running program at a predictable spot unless debugger is on.
Stop your program, change the code, and start it up again.

It is possible (lots of things are). But it can be very, very hard - unless you are skilled at looking at Java bytecode and know your OS internals, I would not try.
Assuming you are running under linux, you can suspend and restart processes via kill -STOP <pid> and kill -CONT <pid>. You can also take a stack-trace of a running java process via jstack, and look at the running threads and decompiled code.

You could do this
static boolean stopLoop = false;
static int[] runLoop(int i, int j) {
for(;i<100;i++) {
if(stopLoop) return new int[i, j];
String[] articles = getArticles(i);
for(;j<articles.length;j++) {
process(articles[i]);
}
j = 0;
}
}

(You probably wanted to write process(articles[j]); in your inner loop.)
You can save last value of i and j into a file after processing of each article, then you can terminate your program anytime you want, and next time just read the values of i and j from your file and you don't have to process them again.
Or set 100 article processes per program run, or process for 20 minutes, there are so many possibilities.

There is no straightforward way to do this, but with a little extra legwork it can be done. One way is to output the latest successful iteration to persistent storage (like a file), then read the file if it exists to find out where to start. The example below shows one way to do it:
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.io.PrintWriter;
public class Test
{
public static void main(String[] args)
{
int start;
// Try to get the last successful run
File file = new File(System.getenv("HOME"), ".myprog");
if(file.exists()) {
try(BufferedReader reader = new BufferedReader(new FileReader(file))) {
String line = reader.readLine();
if(line == null) {
throw new NumberFormatException("Empty file, no number");
}
start = Integer.parseInt(line);
} catch(IOException | NumberFormatException ex) {
System.err.println("Unable to read progress: " + ex.getMessage());
System.err.println("Starting from the beginning");
start = 0;
}
} else {
start = 0;
}
// ... Your declarations go here
for(int i = start; i < 100; i++) {
String[] articles = getArticles(i);
for(int j=0;j<articles.length;j++) {
process(articles[i]);
}
// Record last successful run (this reopens the file every time)
try(PrintWriter writer = new PrintWriter(file)) {
writer.println(i);
} catch(IOException ex) {
System.err.println("Unable to record progress: " + ex.getMessage());
}
}
}
}
Note that I run on a Linux machine, so I called my file ~/.myprog. You may want to change that.

How to append Published DDS content in existing file on Subscriber side?

I have created the normal publishers and subscribers implemented using java , which works as reading the contents by size as 1MB of total size 5MB and published on every 1MB to the subscriber.Data is getting published successfully .Now 'm facing the issue on appending the content to the existing file .Finally i could find only the last 1MB of data in the file.So please let me to know how to solve this issue ? and also i have attached the source code for publisher and subscriber.
Publisher:
public class MessageDataPublisher {
static StringBuffer fileContent;
static RandomAccessFile randomAccessFile ;
public static void main(String[] args) throws IOException {
MessageDataPublisher msgObj=new MessageDataPublisher();
String fileToWrite="test.txt";
msgObj.towriteDDS(fileToWrite);
}
public void towriteDDS(String fileName) throws IOException{
DDSEntityManager mgr=new DDSEntityManager();
String partitionName="PARTICIPANT";
// create Domain Participant
mgr.createParticipant(partitionName);
// create Type
BinaryFileTypeSupport binary=new BinaryFileTypeSupport();
mgr.registerType(binary);
// create Topic
mgr.createTopic("Serials");
// create Publisher
mgr.createPublisher();
// create DataWriter
mgr.createWriter();
// Publish Events
DataWriter dwriter = mgr.getWriter();
BinaryFileDataWriter binaryWriter=BinaryFileDataWriterHelper.narrow(dwriter);
int bufferSize=1024*1024;
File readfile=new File(fileName);
FileInputStream is = new FileInputStream(readfile);
byte[] totalbytes = new byte[is.available()];
is.read(totalbytes);
byte[] readbyte = new byte[bufferSize];
BinaryFile binaryInstance;
int k=0;
for(int i=0;i<totalbytes.length;i++){
readbyte[k]=totalbytes[i];
k++;
if(k>(bufferSize-1)){
binaryInstance=new BinaryFile();
binaryInstance.name="sendpublisher.txt";
binaryInstance.contents=readbyte;
int status = binaryWriter.write(binaryInstance, HANDLE_NIL.value);
ErrorHandler.checkStatus(status, "MsgDataWriter.write");
ErrorHandler.checkStatus(status, "MsgDataWriter.write");
k=0;
}
}
if(k < (bufferSize-1)){
byte[] remaingbyte = new byte[k];
for(int j=0;j<(k-1);j++){
remaingbyte[j]=readbyte[j];
}
binaryInstance=new BinaryFile();
binaryInstance.name="sendpublisher.txt";
binaryInstance.contents=remaingbyte;
int status = binaryWriter.write(binaryInstance, HANDLE_NIL.value);
ErrorHandler.checkStatus(status, "MsgDataWriter.write");
}
is.close();
try {
Thread.sleep(4000);
} catch (InterruptedException e) {
e.printStackTrace();
}
// clean up
mgr.getPublisher().delete_datawriter(binaryWriter);
mgr.deletePublisher();
mgr.deleteTopic();
mgr.deleteParticipant();
}
}
Subscriber:
public class MessageDataSubscriber {
static RandomAccessFile randomAccessFile ;
public static void main(String[] args) throws IOException {
DDSEntityManager mgr = new DDSEntityManager();
String partitionName = "PARTICIPANT";
// create Domain Participant
mgr.createParticipant(partitionName);
// create Type
BinaryFileTypeSupport msgTS = new BinaryFileTypeSupport();
mgr.registerType(msgTS);
// create Topic
mgr.createTopic("Serials");
// create Subscriber
mgr.createSubscriber();
// create DataReader
mgr.createReader();
// Read Events
DataReader dreader = mgr.getReader();
BinaryFileDataReader binaryReader=BinaryFileDataReaderHelper.narrow(dreader);
BinaryFileSeqHolder binaryseq=new BinaryFileSeqHolder();
SampleInfoSeqHolder infoSeq = new SampleInfoSeqHolder();
boolean terminate = false;
int count = 0;
while (!terminate && count < 1500) {
// To run undefinitely
binaryReader.take(binaryseq, infoSeq, 10,
ANY_SAMPLE_STATE.value, ANY_VIEW_STATE.value,ANY_INSTANCE_STATE.value);
for (int i = 0; i < binaryseq.value.length; i++) {
toWrtieXML(binaryseq.value[i].contents);
terminate = true;
}
try
{
Thread.sleep(200);
}
catch(InterruptedException ie)
{
}
++count;
}
binaryReader.return_loan(binaryseq,infoSeq);
// clean up
mgr.getSubscriber().delete_datareader(binaryReader);
mgr.deleteSubscriber();
mgr.deleteTopic();
mgr.deleteParticipant();
}
private static void toWrtieXML(byte[] bytes) throws IOException {
// TODO Auto-generated method stub
File Writefile=new File("samplesubscriber.txt");
if(!Writefile.exists()){
randomAccessFile = new RandomAccessFile(Writefile, "rw");
randomAccessFile.write(bytes, 0, bytes.length);
randomAccessFile.close();
}
else{
randomAccessFile = new RandomAccessFile(Writefile, "rw");
long i=Writefile.length();
randomAccessFile.seek(i);
randomAccessFile.write(bytes, 0, bytes.length);
randomAccessFile.close();
}
}
}
Thanks in advance

It is hard to give a conclusive answer to your question, because your issue could be the result of several different causes. Also, once the cause of the problem has been identified, you will probably have multiple options to mitigate it.
The first place to look is at the reader side. The code does a take() in a loop with a 200 millisecond pause between each take. Depending on your QoS settings on the DataReader, you might be facing a situation where your samples get overwritten in the DataReader while your application is sleeping for 200 milliseconds. If you are doing this over a gigabit ethernet, then a typical DDS product would be able to do those 5 chunks of 1 megabyte within that sleep period, meaning that your default, one-place buffer will get overwritten 4 times during your sleep.
This scenario would be likely if you used the default history QoS settings for your BinaryFileDataReader, which means history.kind = KEEP_LAST and history.depth = 1. Increasing the latter to a larger value, for example to 20, would result in a queue capable of holding 20 chunks of your file while you are sleeping. That should be sufficient for now.
If this does not resolve your issue, other possible causes can be explored.

Issue with wait - notify implementation

I am working on Java multithreading , where I am starting 4 threads after assigning 4 different files to them , to be uploaded to the server.
My objective is , when one thread completes file upload , I need to start another thread assigning a new file to it.
After each file upload , I receive a notification from the server.
// The code for adding the first set of files
for (int count = 0; count < 4; count++) {
if (it.hasNext()) {
File current = new File((String) it.next());
try {
Thread t = new Thread(this, current );
t.start();
t.sleep(100);
} catch (Exception e) {
}
}
}
Now , I am assigning another thread with a file & keeping the thread in a wait state .
When a previous thread notifies , the current thread should start upload.
if (tempThreadCounter == 4 ) {
if (it.hasNext()) {
File current = new File((String) it.next());
try {
Thread t = new Thread(this, current);
t.start();
synchronized (this) {
t.wait();
}
tempThreadCounter++;
} catch (Exception e) {
}
}
}
On the final statement on the run method , I am adding the following statement.
public void run (){
// Performing different operations
//Final statement of the run method below
synchronized (this) {
this.notifyAll();
}
}
Currently , all the 5 threads are starting uploading at the same time.
It should be that the first 4 threads should start uploading & the fifth thread should start only when it it notified by any thread that it had completed its operation.
Any suggestions on the incorrect Thread implementation.

You can use ExecutorService with newFixedThreadPool and specify a concurrency of 1. But really, then why do you need multiple threads? One thread doing all the uploads so the user interface remains responsive should be enough.
ExecutorService exec = Executors.newFixedThreadPool(1); //1 thread at a time
for (int count = 0; count < 4; count++) {
if (it.hasNext()) {
File current = new File((String) it.next());
exec.execute(new Runnable() {
#Override
public void run() {
upload(current);
}
});
}
}
exec.shutdown();
exec.awaitTermination(900, TimeUnit.SECONDS);

Throw it all away and use java.util.concurrent.Executor.

you can join on the thread instead of waiting on it
try {
t.join();
}catch(InterruptedException e){
throw new RuntimeException(e);
}
tempThreadCounter++;