I'm writing a little Java program which uses PsExec.exe from cmd launched using ProcessBuilder to copy and install an application on networked PC (the number of PC that will need to be installed can vary from 5 to 50).
The program works fine if I launched ProcessBuilder for each PC sequentially.
However to speed things up I would like to implement some form of MultiThreading which could allow me to install 5 PC's at the time concurrently (one "batch" of 5 X Processbuilder processes untill all PC's have been installed).
I was thinking of using a Fixed Thread Pool in combination with a Callable interface (each execution of PsExec returns a value which indicates if the execution was succesfull and which I have to evaluate).
The code used for the ProcessBuilder is:
// Start iterating over all PC in the list:
for(String pc : pcList)
{
counter++;
logger.info("Starting the installation of remote pc: " + pc);
updateMessage("Starting the installation of remote pc: " + pc);
int exitVal = 99;
logger.debug("Exit Value set to 99");
try
{
ProcessBuilder pB = new ProcessBuilder();
pB.command("cmd", "/c",
"\""+psExecPath+"\"" + " \\\\" + pc + userName + userPassword + " -c" + " -f" + " -h" + " -n 60 " +
"\""+forumViewerPath+"\"" + " -q "+ forumAddress + remotePath + "-overwrite");
logger.debug(pB.command().toString());
pB.redirectError();
Process p = pB.start();
InputStream stErr = p.getErrorStream();
InputStreamReader esr = new InputStreamReader(stErr);
BufferedReader bre = new BufferedReader(esr);
String line = null;
line = bre.readLine();
while (line != null)
{
if(!line.equals(""))
logger.info(line);
line = bre.readLine();
}
exitVal = p.waitFor();
} catch (IOException ex)
{
logger.info("Exception occurred during installation of PC: \n"+pc+"\n "+ ex);
notInstalledPc.add(pc);
}
if(exitVal != 0)
{
notInstalledPc.add(pc);
ret = exitVal;
updateMessage("");
updateMessage("The remote pc: " + pc + " was not installed");
logger.info("The remote pc: " + pc + " was not installed. The error message returned was: \n"+getError(exitVal) + "\nProcess exit code was: " + exitVal);
}
else
{
updateMessage("");
updateMessage("The remote pc: " + pc + " was succesfully installed");
logger.info("The remote pc: " + pc + " was succesfully installed");
}
Now I've read some info on how to implement Callable and I would like to enclose my ProcessBuilder in a Callable interface and then submit all the Tasks for running in the for loop.
Am I on the right track?
You can surely do that. I suppose you want to use Callable instead of runnable to get the result of your exitVal ?
It doesn't seem like you have any shared data between your threads, so I think you should be fine. Since you even know how many Callables you are going to make you could create a collection of your Callables and then do
List<Future<SomeType>> results = pool.invokeAll(collection)
This would allow for easier handling of your result. Probably the most important thing you need to figure out when deciding on whether or not to use a threadpool is what to do if your program terminates while threads are still running; Do you HAVE to finish what you're doing in the threads, do you need to have seamless handling of errors etc.
Check out java threadpools doc: https://docs.oracle.com/javase/tutorial/essential/concurrency/pools.html
or search the web, there are tons of posts/blogs about when or not to use threadpools.
But seems like you're on the right track!
Thank you for your reply! It definitely put me on the right track. I ended up implementing it like this:
ThreadPoolExecutor executor = (ThreadPoolExecutor) Executors.newFixedThreadPool(5); //NEW
List<Future<List<String>>> resultList = new ArrayList<>();
updateMessage("Starting the installation of all remote pc entered...");
// Start iterating over all PC in the list:
for(String pc : pcList)
{
counter++;
logger.debug("Starting the installation of remote pc: " + pc);
psExe p = new psExe(pc);
Future<List<String>> result = executor.submit(p);//NEW
resultList.add(result);
}
for(Future<List<String>> future : resultList)
{.......
in the last for loop I read the result of my operations and write them on screen or act according to the result returned.
I still have a couple of questions as it is not really clear to me:
1 - If I have 20 PC and submit all the callable threads to the pool in my first For loop, do I get it correctly that only 5 threads will be started (threadpool size = 5) but all will already be created and put in Wait status, and only as soon as the first running Thread is complete and returns a result value the next one will automatically start until all PC are finished?
2 - What is the difference (advantage) of using invokeall() as you suggested compared to the method I used (submit() inside the for loop)?
Thank you once more for your help...I really Love this Java stuff!! ;-)
Related
I have the following code onCreate()
Log.d(TAG, "Setting priority background");
Process.setThreadPriority(Process.THREAD_PRIORITY_BACKGROUND);
Log.d(TAG, Thread.currentThread().getId() + ": " + Thread.currentThread().getPriority());
Log.d(TAG, Process.getThreadPriority(Process.myTid()) + " - myTid() " + Process.myTid() + " Thread.getId() = " + Thread.currentThread().getId());
// start a thread here
Thread thread = new Thread(() -> {
Log.d(TAG, " In new thread");
Log.d(TAG, Thread.currentThread().getId() + ": " + Thread.currentThread().getPriority());
Log.d(TAG, Process.getThreadPriority(Process.myTid()) + " - myTid() " + Process.myTid() + "Thread.getId() = " + Thread.currentThread().getId());
}
The output is:
Setting priority background
1: 5
10 - myTid() 8798 Thread.getId() = 1
In new thread
7534: 10
-8 - myTid() 8819 Thread.getId() = 7534
Can someone please explain:
1) Even though I set setThreadPriority(Process.THREAD_PRIORITY_BACKGROUND); the log in the next line shows priority 5. Why?
2) Where is the -8 coming from in the output for the priority of the second thread?
3) Where is the 10 for the new thread coming from too?
Update:
If I remove the line Process.setThreadPriority(Process.THREAD_PRIORITY_BACKGROUND);
the output remains the same
Reading here, THREAD_PRIORITY_BACKGROUND delegates a lower than normal priority to the task on that thread.
Taking information from here, using Thread.setPriority() contains a value from MIN_PRIORITY(1) to MAX_PRIORITY(10) whereas Process.setThreadPriority() supports value from -20 to 19.
The image below shows the Android Scheduler and the Linux Scheduler and their priority levels. Calling Process and Thread to get thread priority will return two diffrent values as they are two different schedulers, like in your logs.
Update:
This post provided some insight into this:
Process.myTid() is the linux thread ID
Thread.getId() is a simple
sequential long number.
"Thread.getId() is simply a "java layer" static long being auto-increment for each thread."
ThreadID is not equal to ProcessID.
However, casting the long to int and providing the ThreadID to the Process returns an expcted value Process.getThreadPriority((int) Thread.currentThread().getId()).
I am launching WebTorrent-CLI from within my Java application as a separate process. I am using zt-exec for managing the process. When WebTorrent is launched with the following command, it is supposed to exit after the file at given index (value of --select) has been downloaded.
"D:\downloadmanager\node\webtorrent.cmd" download "magnet:?xt=urn:btih:08ada5a7a6183aae1e09d831df6748d566095a10&dn=Sintel" --select 0 --out "D://nf/"
As expected, webtorrent-cli does exit after downloading 0th file when the command above is used to launch it from command line. But when I try the same from within my Java app, it completely ignores the --select option and continues downloading other files in the torrent.
Basically, when launched as a process from Java, webtorrent ignores all the options set (--select, --out or whatever). I should mention that there is nothing wrong with the library because recently I've tried replacing it with commons-exec and that solved nothing. Also, to make sure that the right command is passed while starting the process, I'm printing the command right before calling executor.start(). The command above is copied from the output retrieved from printing the command before the process starts.
This is how the process is started:
#Override
public synchronized void start() throws IOException {
if (mWasDownloadStarted || mWasDownloadFinished) return;
mExec.getCommand().listIterator().forEachRemaining(s -> {
System.out.print(s + " ");
});
mExec.start();
setProcessId();
mWasDownloadStarted = true;
mWasDownloadStopped = false;
}
This is how the command is prepared:
private String buildCommand() {
List <String> command = new ArrayList<>();
command.add("\"" + mManager.mWTLocation + "\"");
command.add("download");
command.add("\"" + mManager.mMagnetUrl + "\"");
if (mManager.mFileIndex >= 0) {
command.add("--select " + mManager.mFileIndex);
}
if (mManager.mSaveTo != null) {
command.add("--out \"" + mManager.mSaveTo + "\"");
}
mManager.mExec.command(command);
String cmdStr = "";
for (String s : command) {
cmdStr = cmdStr.concat(s + " ");
}
return cmdStr.trim();
}
What might be wrong?
Okay, so I was able to fix this issue.
The / character following the path specified as value of --out was causing the problem. In order to fix this, I added a line in node_modules/webtorrent-cli/bin/cmd.js to print the arguments passed to webtorrent:
console.log(process.argv)
With the /, output of this line was something like the following:
[ 'D:\\downloadmanager\\node\\node.exe',
'D:\\downloadmanager\\node\\node_modules\\webtorrent-cli\\bin\\cmd.js',
'download',
'magnet:?xt=urn:btih:08ada5a7a6183aae1e09d831df6748d566095a10&dn=Sintel',
'--select',
'0',
'--out',
'D:\\nf"' ]
Note the " that is included in the path after D:\\nf. When / is removed from the path, the quote disappears and webtorrent behaves as expected.
I doubt that this is a bug in webtorrent. I think zt-exec (or maybe I) was doing something stupid.
Somewhat unrelated, but I think I should also mention that I had to enclose every value for each option with quotes, even the index, to get rid of other nasty errors (e.g.: Error 87, the parameter is incorrect)
I am using JAVA ExecutorService threads to send amazon emails, this helps me to make concurrent connection with AmazonSES via API and sends mails at lightning speed. So amazon accepts some number of connection in a sec, so for me its 50 requests in a second. So I execute 50 threads in a second and send around 1million emails daily.
This is working pretty good, but now the number of mails is going to be increased. And I don't want to invest more into RAM and processors.
One of my friend suggested me to use RabbitMQ Workers instead of threads, so instead of 50 threads, I ll be having 50 workers which will do that job.
So before changing some code to test the resource management, I just want to know will there be any huge difference in consumption? So currently when I execute my threads, JAVA consumes 20-30% of memory. So if I used workers will it be low or high?
Or is their any alternative option to this?
Here is my thread email sending function:
#Override
public void run() {
Destination destination = new Destination().withToAddresses(new String[] { this.TO });
Content subject = new Content().withData(SUBJECT);
Content textBody = new Content().withData(BODY);
Body body = new Body().withHtml(textBody);
Message message = new Message().withSubject(subject).withBody(body);
SendEmailRequest request = new SendEmailRequest().withSource(FROM).withDestination(destination).withMessage(message);
Connection connection = new Connection();
java.util.Date dt = new java.util.Date();
java.text.SimpleDateFormat sdf = new java.text.SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
String insert = "";
try {
System.out.println("Attempting to send an email to " + this.TO);
ctr++;
client.sendEmail(request);
insert = "INSERT INTO email_histories (campaign_id,contact_id,group_id,is_opened,mail_sent_at,mail_opened_at,ip_address,created_at,updated_at,is_sent) VALUES (" + this.campaign_id + ", " + this.contact_id + ", " + this.group_id + ", false, '" + sdf.format(dt) + "', null, null, '" + sdf.format(dt) + "', '" + sdf.format(dt) + "', true);";
connection.insert(insert);
System.out.println("Email sent! to " + this.TO);
} catch (Exception ex) {
System.out.println("The email was not sent.");
System.out.println("Error message: " + ex.getMessage());
}
}
I have no experience with RabbitMQ, so I'll have to leave that for others to answer.
Or is their any alternative option to this?
Instead of using one thread per mail, move that code inside a runable. Add a shared Semaphore with the number of permits = the number of mails you want to send per second. Take one permit per mail, refill permits every second from another thread (i.e. a separate SchedledExecutorService or a Timer). Then adjust the Executor thread pool size to whatever your server can handle.
From a RabbitMQ perspective there is a small amount of memory and network resource consumed, although pretty constant for each connection. If you use a pool of worker threads to read off of the RabbitMQ queue or queues it is possible that it will save you some resources because you are not garbage collecting the individual threads. As far as alternatives are concerned I would use a Thread Pool in any case. Although perhaps too heavyweight for your use, Spring Framework has a very good thread pool that I have used in the past.
I'm experiencing java.net.ConnectException in random ways.
My servlet runs in Tomcat 6.0 (JDK 1.6).
The servlet periodically fetches data from 4-5 third-party web servers.
The servlet uses a ScheduledExecutorService to fetch the data.
Run locally, all is fine and dandy. Run on my prod server, I see semi-random failures to fetch data from 1 of the third parties (Canadian weather data).
These are the URLs that are failing (plain RSS feeds):
http://weather.gc.ca/rss/city/pe-1_e.xml
http://weather.gc.ca/rss/city/pe-2_e.xml
http://weather.gc.ca/rss/city/pe-3_e.xml
http://weather.gc.ca/rss/city/pe-4_e.xml
http://weather.gc.ca/rss/city/pe-5_e.xml
http://weather.gc.ca/rss/city/pe-6_e.xml
http://meteo.gc.ca/rss/city/pe-1_f.xml
http://meteo.gc.ca/rss/city/pe-2_f.xml
http://meteo.gc.ca/rss/city/pe-3_f.xml
http://meteo.gc.ca/rss/city/pe-4_f.xml
http://meteo.gc.ca/rss/city/pe-5_f.xml
http://meteo.gc.ca/rss/city/pe-6_f.xml
Strange: each cycle, when I periodically fetch this data, the success/fail is all over the map: some succeed, some fail, but it never seems to be the same twice. So, I'm not completely blocked, just randomly blocked.
I slowed down my fetches, by introducing a 61s pause between each one. That had no effect.
The guts of the code that does the actual fetch:
private static final int TIMEOUT = 60*1000; //msecs
public String fetch(String aURL, String aEncoding /*UTF-8*/) {
String result = "";
long start = System.currentTimeMillis();
Scanner scanner = null;
URLConnection connection = null;
try {
URL url = new URL(aURL);
connection = url.openConnection(); //this doesn't talk to the network yet
connection.setConnectTimeout(TIMEOUT);
connection.setReadTimeout(TIMEOUT);
connection.connect(); //actually connects; this shouldn't be needed here
scanner = new Scanner(connection.getInputStream(), aEncoding);
scanner.useDelimiter(END_OF_INPUT);
result = scanner.next();
}
catch (IOException ex) {
long end = System.currentTimeMillis();
long time = end - start;
fLogger.severe(
"Problem connecting to " + aURL + " Encoding:" + aEncoding +
". Exception: " + ex.getMessage() + " " + ex.toString() + " Cause:" + ex.getCause() +
" Connection Timeout: " + connection.getConnectTimeout() + "msecs. Read timeout:" +
connection.getReadTimeout() + "msecs."
+ " Time taken to fail: " + time + " msecs."
);
}
finally {
if (scanner != null) scanner.close();
}
return result;
}
Example log entry showing a failure:
SEVERE: Problem connecting to http://weather.gc.ca/rss/city/pe-5_e.xml Encoding:UTF-8.
Exception: Connection timed out java.net.ConnectException: Connection timed out
Cause:null
Connection Timeout: 60000msecs.
Read timeout:60000msecs.
Time taken to fail: 15028 msecs.
Note that the time to fail is always 15s + a tiny amount.
Also note that it fails to reach the configured 60s timeout for the connection.
The host-server admins (Environment Canada) state that they don't have any kind of a blacklist for the IP address of misbehaving clients.
Also important: the code had been running for several months without this happening.
Someone suggested that instead I should use curl, a bash script, and cron. I implemented that, and it works fine.
I'm not able to solve this problem using Java.
This question already has an answer here:
Closed 12 years ago.
Possible Duplicate:
Why do the outputs differ when I run this code using NetBeans 6.8 and Eclipse?
When I am running the following code using Eclipse and NetBeans 6.8. I want to see the available COM ports on my computer. When running in Eclipse it is returning me all available COM ports, but when running it in NetBeans, it does not seem to find any ports ..
public static void test() {
Enumeration lists=CommPortIdentifier.getPortIdentifiers();
System.out.println(lists.hasMoreElements());
while (lists.hasMoreElements()) {
CommPortIdentifier cn =
(CommPortIdentifier)lists.nextElement();
if ((CommPortIdentifier.PORT_SERIAL==cn.getPortType())) {
System.out.println(
"Name is serail portzzzz " +
cn.getName()+
" Owned status " +
cn.isCurrentlyOwned());
try {
SerialPort port1=(SerialPort)cn.open("ComControl",800000);
port1.setSerialPortParams(
9600,
SerialPort.DATABITS_8,
SerialPort.STOPBITS_1,
SerialPort.PARITY_NONE);
System.out.println("Before get stream");
OutputStream out=port1.getOutputStream();
InputStream input=port1.getInputStream();
System.out.println("Before write");
out.write("AT".getBytes());
System.out.println("After write");
int sample=0;
//while((( sample=input.read())!=-1)){
System.out.println("Before read");
//System.out.println(input.read() + "TEsting ");
//}
System.out.println("After read");
System.out.println(
"Receive timeout is " +
port1.getReceiveTimeout());
}
catch(Exception e) {
System.err.println(e.getMessage());
}
}
else {
System.out.println(
"Name is parallel portzzzz " +
cn.getName() +
" Owned status " +
cn.isCurrentlyOwned() +
cn.getPortType() +
" ");
}
}
}
Output with Netbeans,
false
Output using Eclipse,
true
Name is serail portzzzz COM1 Owned status false
Before get stream
Before write
After write
Before read
After read
Receive timeout is -1
Name is serail portzzzz COM2 Owned status false
Before get stream
Before write
After write
Before read
After read
Receive timeout is -1
Name is parallel portzzzz LPT1 Owned status false2
Name is parallel portzzzz LPT2 Owned status false2
An initial guess would be that the library you use use native code enclosed in a DLL and that code cannot be found giving an error earlier you have missed, and the code falls back to a dummy behaviour.
I would have a closer look at the initialization code to see what happens there.