fairly complex code but it's a very simple issue.
I have a thread, this is its run method:
public void run() //gets pages and writes to them
{ // i printed the pageId of every process to check they are running at the same time and competing for resources
for(ProcessCycle currentCycle : processCycles.getProcessCycles())
{
Long[] longArray = new Long[currentCycle.getPages().size()];
try {
Page<byte[]>[] newPages = mmu.getPages(currentCycle.getPages().toArray(longArray));
for(int i = 0; i < newPages.length; i++)
{
MMULogger.getInstance().write("GP:P" + id + " " + currentCycle.getPages().get(i) + " " + Arrays.toString(currentCycle.getData().get(i)), Level.INFO);
}
List<byte[]> currentPageData = currentCycle.getData();
System.out.println("process id " + id);
for(int i = 0; i < newPages.length;i++)
{
byte[] currentData = currentPageData.get(i);
newPages[i].setContent(currentData);
}
Thread.sleep(currentCycle.getSleepMs());
} catch (ClassNotFoundException | IOException | InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
don't bother reading all of it. just notice that after the mmu.getpages there is a for loop.
While a process is inside the for loop, i want to lock access to mmu.getpages for all other threads. synchronized is no good since my original process is no longer in mmu, but in the for loop, and reentrantlock might be a good idea but I'm unfamiliar with the syntax and ran into some issues.
long story short - how do i make all other threads wait while some thread is inside the for loop after mmu.getpages?
Usually I chose an approach like this:
private Object lock = new Object();
public void run() //gets pages and writes to them
{ // i printed the pageId of every process to check they are running at the same time and competing for resources
for(ProcessCycle currentCycle : processCycles.getProcessCycles())
{
Long[] longArray = new Long[currentCycle.getPages().size()];
try {
synchrnonized(lock) {
Page<byte[]>[] newPages = mmu.getPages(currentCycle.getPages().toArray(longArray));
for(int i = 0; i < newPages.length; i++)
{
MMULogger.getInstance().write("GP:P" + id + " " + currentCycle.getPages().get(i) + " " + Arrays.toString(currentCycle.getData().get(i)), Level.INFO);
}
}
List<byte[]> currentPageData = currentCycle.getData();
System.out.println("process id " + id);
for(int i = 0; i < newPages.length;i++)
{
byte[] currentData = currentPageData.get(i);
newPages[i].setContent(currentData);
}
Thread.sleep(currentCycle.getSleepMs());
} catch (ClassNotFoundException | IOException | InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
Not sure if there is a better way. This will only work as expected when all threads share the same instance of this object, otherwise you have to make lock a static member variable.
In my opinion a ReadWriteLock might be a best approach.
Something like this:
public class MmuClass {
private ReadWriteLock blockGetPages = new ReentrantReadWriteLock();
public byte [] getPages(...) {
try{
blockGetPages.readLock().lock();
// ...
// ...
// ...
return result;
finally{
blockGetPages.readLock().unlock();
}
}
public void lockAccessToGetPages(){
blockGetPages.writeLock().lock();
}
public void unlockAccessToGetPages(){
blockGetPages.writeLock().unlock();
}
}
and
Page<byte[]>[] newPages = mmu.getPages(currentCycle.getPages().toArray(longArray));
try{
mmu.lockAccessToGetPages();
for(int i = 0; i < newPages.length; i++) {
MMULogger.getInstance().write("GP:P" + id + " " + currentCycle.getPages().get(i) + " " + Arrays.toString(currentCycle.getData().get(i)), Level.INFO);
}
} finally{
mmu.unlockAccessToGetPages();
}
In this solutions all "readers" can simultaneously call getPages(), the access is blocked after calling lockAccessToGetPages() and unblocked after calling unlockAccessToGetPages(). If one thread locks the object in write mode, only this thread has access to the method. If some thread tries to lock it in write mode, must wait until all readers, which are currently "inside" the metod, finish their fork and leave the method.
Related
Could some one take a look this below program ?.
It is working fine for small process, but not exiting the program after completing large process.
Note: If it is small size query, about 50 records (retrieving and updating), the program is Exiting normally....
The purpose of this program is to get the data from the database, go to cloud to read JSON, validate the data and update the record in database with result.
public class ThreadLauncher
{
public static void main(String args[])
{
final ExecutorService service = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors()); // or hardcode a number
List<Future<Runnable>> futures = new ArrayList<Future<Runnable>>();
for (int n = 0; n < 10; n++)
{
Future f = service.submit(new Task(n));
futures.add(f);
}
// wait for all tasks to complete before continuing
for (Future<Runnable> f : futures)
{
try {
f.get();
//shut down the executor service so that this thread can exit
} catch (InterruptedException e) {
System.out.println("Exiting with InterruptedException : " + e.getMessage());
e.printStackTrace();
} catch (ExecutionException e) {
System.out.println("Exiting with ExecutionException : " + e.getMessage());
e.printStackTrace();
}
}
service.shutdownNow();
System.out.println("Exiting normally...");
}
}
final class Task
implements Runnable
{
private int loopCounter;
private int totalLoops = 5;
public Task(int counter)
{
this.loopCounter = counter;
}
#Override
public void run()
{
try {
GCPJSON.getInstance().getGCPDataFromJSON(PRODDataAccess.getInstance().getDataToProcess(loopCounter,totalLoops));
System.out.println("Task ID : " + this.loopCounter + " performed by " +
Thread.currentThread().getName());
} catch (Exception e) {
e.printStackTrace();
}
}
}
Here is my updated code.I have changed it from Future to FutureTask and added few lines items. I am hoping all these 10 tasks run in parallel.
List<FutureTask<Runnable>> futures = new ArrayList<FutureTask<Runnable>>();
for (int n = 0; n < 10; n++)
{
FutureTask f = (FutureTask) service.submit(new Task(n));
futures.add(f);
}
// wait for all tasks to complete before continuing
// for (FutureTask<Runnable> f : futures)
for (int i=0; i< futures.size(); i++)
{
FutureTask f = (FutureTask)futures.get(i) ;
//System.out.println("Number of futureTasks: " + i);
try {
if(!f.isDone()){
//wait indefinitely for future task to complete
f.get();
//System.out.println("FutureTask output="+f.get());
}else{
System.out.println("Task number :" + i + "Done.");
}
} catch (InterruptedException | ExecutionException e) {
System.out.println("Exiting with InterruptedException : " + e.getMessage());
e.printStackTrace();
}
}
//If we come out from the loop, we must have completed all the tasks. e.e. In above case , 10 tasks ( 10 loop submites)
try {
if (!service.awaitTermination(10000000, TimeUnit.MICROSECONDS)) {
System.out.println("Exiting normally...");
service.shutdownNow();
System.exit(0);
}
} catch (InterruptedException e) {
e.printStackTrace();
}
if(!service.isShutdown()){
System.exit(0);
}
It's because when you call shutdown or shutdownNow on executorService it only try to stop active threads and it will return list of active tasks based on Java documentation:
Attempts to stop all actively executing tasks, halts the processing of
waiting tasks, and returns a list of the tasks that were awaiting
execution.
This method does not wait for actively executing tasks to terminate.
Use {awaitTermination} to do that.
As documentation says you need to call awaitTermination to make sure every thread has finished or this method will kill them at the end of timeout.
UPDATE:
If you have no idea about timing estimation you can add following lines to make sure that all threads have been finished successfully.
int filesCount = getFileCount();//you know the files count, right?
AtomicInteger finishedFiles = new AtomicInteger(0);
ExecutorService executorService = Executors.newFixedThreadPool(threadCount);
for (int i = 0; i < threadCount; i++)
executorService.submit(() -> {
//do you work
//at the end of each file process
finishedFiles.incrementAndGet();
}
while (finishedFiles.get() < filesCount) { //let's wait until all files have been processed
Thread.sleep(100);
}
executorService.shutdown();
executorService.awaitTermination(1, TimeUnit.MINUTES);//anyway they already should have finished
I use jna to run WMI queries.
The following code queries WMI SELECT Caption,Capabilities from Win32_DiskDrive. The Type of Win32_DiskDrive.Capabilities is uint16[] and result.getValue returns a SAFEARRAY Instance.
System.out.println("Var Type(3 expected): " + value.getVarType().intValue());
returns randomly 0 or 3 if I start the process several times.
System.out.println("Size (>0 expected): " + (value.getUBound(0) - value.getLBound(0)));
is correct, but
Object el = value.getElement(0);
fails.
value.accessData();
returns null which is unexpected as well, so I cannot use OaIdlUtil#toPrimitiveArray (Nullpointer)
Unfortunately, the code does not work, and I have no idea what might be wrong. Any Ideas?
enum Win32_DiskDrive_Values {
Caption,
Capabilities
}
public static void main(String[] args) throws IOException, InterruptedException {
try {
WmiQuery<Win32_DiskDrive_Values> serialNumberQuery = new WmiQuery<Win32_DiskDrive_Values>("Win32_DiskDrive", Win32_DiskDrive_Values.class);
Ole32.INSTANCE.CoInitializeEx(null, Ole32.COINIT_MULTITHREADED);
WmiResult<Win32_DiskDrive_Values> result = serialNumberQuery.execute();
for (int i = 0; i < result.getResultCount(); i++) {
System.out.println(result.getValue(Win32_DiskDrive_Values.Caption, i));
SAFEARRAY value = (SAFEARRAY) result.getValue(Win32_DiskDrive_Values.Capabilities, i);
// According to https://learn.microsoft.com/en-us/windows/desktop/cimwin32prov/win32-diskdrive, the type of Capabilities
// should be uint16[] which should be Variant.VT_I2 (2-byte integer)
// however, it is not constant. sometimes it is 0, sometimes Variant.VT_I2 (3);
System.out.println("Var Type(3 expected): " + value.getVarType().intValue());
System.out.println("Size (>0 expected): " + (value.getUBound(0) - value.getLBound(0)));
Object el = value.getElement(0);
System.out.println("Element 0 (!=null expected): " + el);
Pointer pointer = value.accessData();
System.out.println("pointer (!=null expected): " + pointer);
}
} catch (Throwable e) {
e.printStackTrace();
} finally {
Ole32.INSTANCE.CoUninitialize();
}
}
The WMI code that I submitted to the JNA project is only set up to handle primitive values and Strings, not arrays. The problem you are encountering is that WMI is returning the pointer address to the array (either an empty array with VT_EMPTY = 0, or a 32-bit poniter with VT_I4 = 3). But the WMI result is released after the iteration, so you cannot use the WmiResult to fetch the object.
You need to write your own code (using the JNA implementation as a starting point) to grab the SAFEARRAY during iteration. You asked this question on the JNA website and #matthiasblaesing posted the following snippet which works for your use case:
public static void main(String[] args) throws IOException, InterruptedException {
Ole32.INSTANCE.CoInitializeEx(null, Ole32.COINIT_MULTITHREADED);
// Connect to the server
Wbemcli.IWbemServices svc = WbemcliUtil.connectServer("ROOT\\CIMV2");
// Send query
try {
Wbemcli.IEnumWbemClassObject enumerator = svc.ExecQuery("WQL", "SELECT Caption, Capabilities, CapabilityDescriptions FROM Win32_DiskDrive",
Wbemcli.WBEM_FLAG_FORWARD_ONLY | Wbemcli.WBEM_FLAG_RETURN_IMMEDIATELY, null);
try {
IWbemClassObject[] result;
VARIANT.ByReference pVal = new VARIANT.ByReference();
IntByReference pType = new IntByReference();
IntByReference plFlavor = new IntByReference();
while(true) {
result = enumerator.Next(0, 1);
if(result.length == 0) {
break;
}
COMUtils.checkRC(result[0].Get("Caption", 0, pVal, pType, plFlavor));
System.out.println("---------" + pVal.getValue() + "-------------");
OleAuto.INSTANCE.VariantClear(pVal);
COMUtils.checkRC(result[0].Get("CapabilityDescriptions", 0, pVal, pType, plFlavor));
SAFEARRAY safeArray = (SAFEARRAY) pVal.getValue();
for(int i = safeArray.getLBound(0); i<=safeArray.getUBound(0); i++) {
System.out.println("\t" + safeArray.getElement(i));
}
OleAuto.INSTANCE.VariantClear(pVal);
COMUtils.checkRC(result[0].Get("Capabilities", 0, pVal, pType, plFlavor));
safeArray = (SAFEARRAY) pVal.getValue();
for(int i = safeArray.getLBound(0); i<=safeArray.getUBound(0); i++) {
System.out.println("\t" + safeArray.getElement(i));
}
OleAuto.INSTANCE.VariantClear(pVal);
result[0].Release();
}
} finally {
// Cleanup
enumerator.Release();
}
} finally {
// Cleanup
svc.Release();
}
Ole32.INSTANCE.CoUninitialize();
}
This question already has answers here:
How to use an ExecutorCompletionService
(2 answers)
Closed 7 years ago.
public static void getTestData() {
try {
filename = "InventoryData_" + form_id;
PrintWriter writer = new PrintWriter("/Users/pnroy/Documents/" +filename + ".txt");
pids = new ArrayList<ProductId>();
GetData productList = new GetData();
System.out.println("Getting productId");
pids = productList.GetProductIds(form_id);
int perThreadSize = pids.size() / numberOfCrawlers;
ArrayList<ArrayList<ProductId>> perThreadData = new
ArrayList<ArrayList<ProductId>>(numberOfCrawlers);
for (int i = 1; i <= numberOfCrawlers; i++) {
perThreadData.add(new ArrayList<ProductId>(perThreadSize));
for (int j = 0; j < perThreadSize; j++) {
ProductId ids = new ProductId();
ids.setEbProductID((pids.get(((i - 1) * perThreadSize + j))).getEbProductID());
ids.setECProductID((pids.get(((i - 1) * perThreadSize + j))).getECProductID());
perThreadData.get(i - 1).add(ids);
}
}
BlockingQueue<String> q = new LinkedBlockingQueue<String>();
Consumer c1 = new Consumer(q);
Thread[] thread = new Thread[numberOfCrawlers];
for (int k = 0; k <= numberOfCrawlers; k++) {
// System.out.println(k);
GetCombinedData data = new GetCombinedData();
thread[k] = new Thread(data);
thread[k].setDaemon(true);
data.setVal(perThreadData.get(k), filename, q);
thread[k].start();
// writer.println(data.getResult());
}
new Thread(c1).start();
for (int l = 0; l <= numberOfCrawlers; l++) {
thread[l].join();
}
} catch (Exception e) {
}
}
Here number of crawlers is the number of threads.
The run method of GetCombined class has the following code:
The pids is passed as perThreadData.get(k-1) from the main method
The class CassController queries a API and i get a string result after some processing.
public void run(){
try{
for(int i=0;i<pids.size();i++){
//System.out.println("before cassini");
CassController cass = new CassController();
String result=cass.getPaginationDetails(pids.get(i));
queue.put(result);
// System.out.println(result);
Thread.sleep(1000);
}
writer.close();
}catch(Exception ex){
}
Consumer.java has the following code :
public class Consumer implements Runnable{
private final BlockingQueue queue;
Consumer(BlockingQueue q) { queue = q; }
public void run(){
try {
while (queue.size()>0)
{
consume(queue.take());
}
} catch (InterruptedException ex)
{
}
}
void consume(Object x) {
try{
PrintWriter writer = new PrintWriter(new FileWriter("/Users/pnroy/Documents/Inventory", true));
writer.println(x.toString());
writer.close();
}catch(IOException ex){
}
}
So if i set the number of crawlers to 10 and if there are 500 records each thread will process 50 records.I need to write the results into a file.I am confused how i can achieve this since its a array of thread and each thread is doing a bunch of operations.
I tried using blocking queue but that is printing repetitive results.I am new to multi threading and not sure how can i handle the case.
Can you please suggest.
With the introduction of many useful high-level concurrency classes, it now recommended not to directly use the Thread class anymore. Even the BlockingQueue class is rather low-level.
Instead, you have a nice application for the CompletionService, which builds upon the ExecutorService. The below example shows how to use it.
You want to replace the code in PartialResultTask (that's where the main processing happens) and System.out.println (that's where you probably want to write your result to a file).
public class ParallelProcessing {
public static void main(String[] args) {
ExecutorService executionService = Executors.newFixedThreadPool(10);
CompletionService<String> completionService = new ExecutorCompletionService<>(executionService);
// submit tasks
for (int i = 0; i < 500; i++) {
completionService.submit(new PartialResultTask(i));
}
// collect result
for (int i = 0; i < 500; i++) {
String result = getNextResult(completionService);
if (result != null)
System.out.println(result);
}
executionService.shutdown();
}
private static String getNextResult(CompletionService<String> completionService) {
Future<String> result = null;
while (result == null) {
try {
result = completionService.take();
} catch (InterruptedException e) {
// ignore and retry
}
}
try {
return result.get();
} catch (ExecutionException e) {
e.printStackTrace();
return null;
} catch (InterruptedException e) {
e.printStackTrace();
return null;
}
}
static class PartialResultTask implements Callable<String> {
private int n;
public PartialResultTask(int n) {
this.n = n;
}
#Override
public String call() {
return String.format("Partial result %d", n);
}
}
}
Firstly, yes I'm calling this from a web browser. It's quite a long piece of code but I've tried shortening it as much as possible.
Basically, I need to wait let's say 1 second for every iteration in the loop. Tried pretty much everything (.sleep() etc.) but it just doesn't seem to be pausing. The reason why I need to do this is because the SimpleSocketClient is calling a socket which has a low limit per second allowed.
#Override
public String execute(HttpServletRequest request, HttpServletResponse response) {
String forwardToJsp = null;
HttpSession session = request.getSession();
String allUrls = request.getParameter("domains");
ArrayList domainList = new ArrayList<String>();
Scanner sc = new Scanner(allUrls);
while (sc.hasNextLine()) {
String line = sc.nextLine();
domainList.add(line);
// process the line
}
sc.close();
String pageHtml = null;
String domain = "";
String status = "";
String registrant = "";
String dates = "";
String tag = "";
String email = "";
ArrayList domains = new ArrayList<Domain>();
Domain theDomain;
String ipAddress = request.getHeader("X-FORWARDED-FOR");
if (ipAddress == null) {
ipAddress = request.getRemoteAddr();
}
for (int i = 0; i < domainList.size(); i++) {
//NEED TO WAIT 1 SECOND HERE / ANYWHERE IN LOOP
String singleDomain = domainList.get(i).toString();
SimpleSocketClient tester = new SimpleSocketClient(singleDomain,ipAddress);
pageHtml = tester.getResult();
try {
String whoIs2 = ipAddress + " " + ipAddress + " " + singleDomain + "\r\n";
byte[] data = whoIs2.getBytes();
//details of each domain
//domain name
domain = singleDomain;
//status
status = "FTR";
//registrant
registrant = "N/A";
//dates
dates = "N/A";
//tag
tag = "N/A";
//email
email = "N/A";
}
} catch (Exception e) {
Logger.getLogger("ip is " + ipAddress + bulkWhoIsCommand.class.getName()).log(Level.SEVERE, null, e);
forwardToJsp = "index.jsp";
return forwardToJsp;
}
//single one
theDomain = new Domain(domain,status,registrant,dates,tag,email);
//now add to arrayList
domains.add(theDomain);
// try {
// Thread.sleep(230000);
// } catch (InterruptedException ex) {
// Logger.getLogger(bulkWhoIsCommand.class.getName()).log(Level.SEVERE, null, ex);
// }
// try {
// pause.poll(100 * 300, TimeUnit.MILLISECONDS);
// } catch (InterruptedException ex) {
// Logger.getLogger(bulkWhoIsCommand.class.getName()).log(Level.SEVERE, null, ex);
// }
}
EDIT - Friend recommended to use ajax to poll updates but surely there's a way of just using java.
Your can try to set a while-loop in the while-loop, to pause it. Should like this:
while(!done)
{
long start = System.currentTimeMillis();
while(System.currentTimeMillis() - start < 1000L){}
}
Didn't test it but the approach counts. I had the idea to do a combination of both. So every time Thread.Sleep() crashes, you have to take the loop. Something like this:
while(!done)
{
long start = System.currentTimeMillis();
try {
Thread.sleep(1000);
} catch (InterruptedException ex) {
System.err.println(e);
}
while(System.currentTimeMillis() - start < 1000L){}
}
When Thread.Sleep() worked it just get called once. Otherwise you need some CPU time. Could be the cpu economical version.
Preliminary Info: I am writing a program that has 2 groups of threads. Group 1 threads each have a string that they take in and compute a SHA-256 hash and stick that hash into a shared object. Group 2 threads each have a username and (hashed) password associated with them. Their job is to compare their hashed passwords with the ones in the shared object to "crack" the passwords.
Problem: I currently have that working, but am trying to find a way to make sure that the group 2 threads output the users in the same order of the original file. One issue is that it's a requirement for this program to print the username and password (if a match is found) as soon as the hash is computed. How can I manage to keep this requirement but still print them out in the order of the original file? The following is my code for the 2 groups of threads (group 2 is must be created before group 1)
GROUP 1 THREADS:
private static class Group1Th implements Runnable {
private String passToHash;
private SharedDict shared;
private Lock lock;
public Group1Th(String passToHash, Lock lock, SharedDict shared) {
this.passToHash = passToHash;
this.shared = shared;
this.lock = lock;
}
public void run() {
MessageDigest md = null;
try {md = MessageDigest.getInstance ("SHA-256");}
catch (NoSuchAlgorithmException e1) {e1.printStackTrace();}
String password = this.passToHash;
byte[] data = null;
try {data = password.getBytes ("UTF-8");}
catch (UnsupportedEncodingException e) {e.printStackTrace();}
for (int i = 1; i <= 100000; i++) {
md.update(data);
data = md.digest();
}
String hexData = Hex.toString(data);
//System.out.println("Hash for (" + password + "): " + hexData);
{synchronized(shared) {
shared.hashed.add(hexData);
shared.unhashed.add(password);
shared.notifyAll();
}}
}
}
GROUP 2 THREADS:
private static class Group2Th implements Runnable {
private String dbUser;
private String dbHashedPass;
private SharedDict shared = new SharedDict();
//lprivate int current = -1;
public Group2Th(String dbUser, String dbHashedPass, SharedDict shared) {
this.dbUser = dbUser;
this.dbHashedPass = dbHashedPass;
this.shared = shared;
//this.current = -1;
}
public void run() {
System.out.println("Hello from G2 Thread: " + this.dbUser + " ==> " + this.dbHashedPass);
{synchronized(shared) {
boolean broken = false;
while (broken == false) {
try {
shared.wait();
for (int i=0; i < shared.hashed.size(); i++) {
if (shared.hashed.get(i).equals(dbHashedPass)) {
System.out.println(dbUser + " " + shared.unhashed.get(i));
broken = true;
break;
}
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}}
}
}
You can ensure ordering by having Group2Th wait for Group1Th to finish its job and die.
Just look at the join() method.