Preliminary Info: I am writing a program that has 2 groups of threads. Group 1 threads each have a string that they take in and compute a SHA-256 hash and stick that hash into a shared object. Group 2 threads each have a username and (hashed) password associated with them. Their job is to compare their hashed passwords with the ones in the shared object to "crack" the passwords.
Problem: I currently have that working, but am trying to find a way to make sure that the group 2 threads output the users in the same order of the original file. One issue is that it's a requirement for this program to print the username and password (if a match is found) as soon as the hash is computed. How can I manage to keep this requirement but still print them out in the order of the original file? The following is my code for the 2 groups of threads (group 2 is must be created before group 1)
GROUP 1 THREADS:
private static class Group1Th implements Runnable {
private String passToHash;
private SharedDict shared;
private Lock lock;
public Group1Th(String passToHash, Lock lock, SharedDict shared) {
this.passToHash = passToHash;
this.shared = shared;
this.lock = lock;
}
public void run() {
MessageDigest md = null;
try {md = MessageDigest.getInstance ("SHA-256");}
catch (NoSuchAlgorithmException e1) {e1.printStackTrace();}
String password = this.passToHash;
byte[] data = null;
try {data = password.getBytes ("UTF-8");}
catch (UnsupportedEncodingException e) {e.printStackTrace();}
for (int i = 1; i <= 100000; i++) {
md.update(data);
data = md.digest();
}
String hexData = Hex.toString(data);
//System.out.println("Hash for (" + password + "): " + hexData);
{synchronized(shared) {
shared.hashed.add(hexData);
shared.unhashed.add(password);
shared.notifyAll();
}}
}
}
GROUP 2 THREADS:
private static class Group2Th implements Runnable {
private String dbUser;
private String dbHashedPass;
private SharedDict shared = new SharedDict();
//lprivate int current = -1;
public Group2Th(String dbUser, String dbHashedPass, SharedDict shared) {
this.dbUser = dbUser;
this.dbHashedPass = dbHashedPass;
this.shared = shared;
//this.current = -1;
}
public void run() {
System.out.println("Hello from G2 Thread: " + this.dbUser + " ==> " + this.dbHashedPass);
{synchronized(shared) {
boolean broken = false;
while (broken == false) {
try {
shared.wait();
for (int i=0; i < shared.hashed.size(); i++) {
if (shared.hashed.get(i).equals(dbHashedPass)) {
System.out.println(dbUser + " " + shared.unhashed.get(i));
broken = true;
break;
}
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}}
}
}
You can ensure ordering by having Group2Th wait for Group1Th to finish its job and die.
Just look at the join() method.
Related
I have created a password cracker in Java that cracks passwords from a text file list. It then generates a dictionary that contains the following pairs: the word hashed and the original word. I am looking for a way to speed up the program as having it read all of the words from the file and then use multithreading to generate the hashes. How can I break up the list of words so that it is in four separate partitions that I can then have multiple threads operate on in the createDictionary method? Here is what I have so far:
public class Main {
private static final String FNAME = "words.txt";
private final static String PASSWDFNAME = "passwd.txt";
private static Map<String, String> dictionary = new HashMap<>();
public static void main(String[] args) {
// Create dictionary of plain / hashed passwords from list of words
System.out.println("Generating dictionary ...");
long start = System.currentTimeMillis();
createDictionary(FNAME);
System.out.println("Generated " + dictionary.size() + " hashed passwords in dictionary");
long stop = System.currentTimeMillis();
System.out.println("Elapsed time: " + (stop - start) + " milliseconds");
// Read password file, hash plaintext passwords and lookup in dictionary
System.out.println("\nCracking password file ...");
start = System.currentTimeMillis();
crackPasswords(PASSWDFNAME);
stop = System.currentTimeMillis();
System.out.println("Elapsed time: " + (stop - start) + " milliseconds");
}
private static void createDictionary(String fname) {
// Read in list of words
List<String> words = new ArrayList<>();
try (Scanner input = new Scanner(new File(fname));) {
while (input.hasNext()) {
String s = input.nextLine();
if (s != null && s.length() >= 4) {
words.add(s);
}
}
} catch (FileNotFoundException e) {
System.out.println("File " + FNAME + " not found");
e.printStackTrace();
System.exit(-1);
}
// Generate dictionary from word list
for (String word : words) {
generateHashes(word);
}
}
private static void crackPasswords(String fname) {
File pfile = new File(fname);
try (Scanner input = new Scanner(pfile);) {
while (input.hasNext()) {
String s = input.nextLine();
String[] t = s.split(",");
String userid = t[0];
String hashedPassword = t[1];
String password = dictionary.get(hashedPassword);
if (password != null) {
System.out.println("CRACKED - user: "+userid+" has password: "+password);
}
}
} catch (FileNotFoundException ex) {
System.out.println(ex.getMessage());
ex.printStackTrace();
System.exit(-1);
}
}
private static void generateHashes(String word) {
// Convert word to lower case, generate hash, store dictionary entry
String s = word.toLowerCase();
String hashedStr = HashUtils.hashPassword(s);
dictionary.put(hashedStr, s);
// Capitalize word, generate hash, store dictionary entry
s = s.substring(0, 1).toUpperCase() + s.substring(1);
hashedStr = HashUtils.hashPassword(s);
dictionary.put(hashedStr, s);
}
}
It's very simple, check this out:
public static void main(String[] args) {
List<String> words = new ArrayList<>();
List<Thread> threads = new ArrayList<>();
int numThreads = 4;
int threadsSlice = words.size() / numThreads;
for(int i = 0; i < numThreads; i++) {
Thread t = new Thread(new WorkerThread(i * threadsSlice, (i + 1) * threadsSlice, words));
t.start();
threads.add(t);
}
threads.forEach(t -> {
try {
t.join();
} catch (InterruptedException e) {
e.printStackTrace();
}
});
}
static class WorkerThread implements Runnable {
private final int left;
private final int right;
private final List<String> words;
public WorkerThread(int left, int right, List<String> words) {
this.left = left;
this.right = right;
this.words = words;
}
#Override
public void run() {
for (int i = left; i < right; i++) {
generateHashes(words.get(i));
}
}
}
This code is creating 4 threads, each one scanning one partition of your list, and applying the generateHashes method on all the words in the partition.
You can put the words in the heap memory to avoid passing it to each thread via constructor param.
Also make sure to use a ConcurrentMap for your dictionary in the generateHashes method
I have below code where I am testing thread synchronization for elastic search, but somehow I am not able to success in it, can any one let me know where I am going wrong?
If I enable Thread sleep inside 'startThreadProcess' method then everything works fine because it sleeps for certain amount of time period. Which I don't want to, I want to get proper lock for thread without using thread sleep.
but what is happening in above code I have used Executor for pooling threads. There I am running for loop for count 4 so as to initiate 4 threads from pool. so when my thread submitted using executor submit which calls synchronized method and inside that synchronized method I am giving call to other synchronized method from where I am getting total number of count in particular node and proceeding ahead with that count by incrementing to insert new document where my first thread is yet not completed 2nd thread enters and try to get total number of count from the method which is being called from synchronized method so there I am getting wrong count for thread 2 as my first thread will insert 10000 json documents inside node so I am expecting Thread 2 should get count 10000 and then it should process for insert but here my Thread 2 enters in between and get count any random number and start inserting by incrementing from that number which is not expected scenario
package com.acn.adt.main;
public class ESTest {
private static final String dataPath = "C:\\Elastic Search\\Data.json";
static ESTest esTest = new ESTest();
private static TransportClient client = null;
private Properties elasticPro = null;
private InputStream input = null;
ElasticSearchCrud esCRUD = null;
private final Object lock = new Object();
public static void main(String[] args) {
String strArray[] = new String[] {"1"};
esTest.startProcess(strArray);
}
public void startProcess(String strArray[]) {
try {
input = new FileInputStream(ElasticSearchConstants.ELASTIC_PROPERTIES);
elasticPro = new Properties();
//elasticPro.load(ElasticSearchClient.class.getResourceAsStream(ElasticSearchConstants.ELASTIC_PROPERTIES));
elasticPro.load(input);
System.out.println(elasticPro.getProperty("homeDir"));
long startTime = System.currentTimeMillis();
Settings setting = Settings.builder()
//.put("client.transport.ping_timeout", "100s")
.put("cluster.name", elasticPro.getProperty("cluster"))
//.put("node.name", elasticPro.getProperty("node"))
//.put("client.transport.sniff", Boolean.valueOf(elasticPro.getProperty("transport.sniff")))
.put("client.transport.sniff", false)
.put("cluster.routing.allocation.enable", "all")
.put("cluster.routing.allocation.allow_rebalance", "always")
//.put("client.transport.ignore_cluster_name", true)
.build();
client = new PreBuiltTransportClient(setting)
.addTransportAddress(new TransportAddress(InetAddress.getByName("localhost"),
Integer.valueOf("9300")));
long endTime = System.currentTimeMillis();
System.out.println("Time taken for connecting " + TimeUnit.MILLISECONDS.toSeconds((endTime - startTime)));
ExecutorService executorService = Executors.newFixedThreadPool(10);
for(int i = 1; i <=4; i++) {
if(i==1) {
strArray = new String [] {"1"};
}else if(i == 2) {
strArray = new String [] {"1"};
}else if(i == 3) {
strArray = new String [] {"1"};
}else if(i == 4) {
strArray = new String [] {"1"};
}
executorService.execute(new ESThread(esTest,strArray,i));
}
}catch(Exception e) {
}
}
public class ESThread implements Runnable {
private final Object lock = new Object();
ESTest esTester = null;
String strArr [] = null;
int i =0;
public ESThread(ESTest esTester,String[] strArr,int i) {
this.esTester = esTester;
this.strArr = strArr;
this.i = i;
}
#Override
public void run() {
System.out.println("Name of Current thread is Thread_"+i);
synchronized(lock) {
esTester.startCRUDProcess(strArr);
}
System.out.println("Thread_"+i+" done.");
}
}
public void startCRUDProcess(String [] strArr) {
SearchAPI esSearch = new SearchAPIImpl();
boolean caseFlg = false;
String _indexName = "gcindex";
String _indexType = "gctype";
String _ids = "501,602,702###1,10000,10001";
String _id = "10000";
String[] _strIds = new String[] {"10000","9999"};
System.out.println("Insert Multiple Process is started...");
System.out.println("--------------------------------------");
try {
caseFlg = insertMultipleDocument(dataPath,client,_indexName,_indexType);
} catch (IOException | ParseException e) {
e.printStackTrace();
caseFlg = false;
}
}
public synchronized boolean insertMultipleDocument(String dataPath,TransportClient client,String _indexName,String _indexType) throws FileNotFoundException, ParseException {
try {
JSONParser parser = new JSONParser();
// we know we get an array from the example data
JSONArray jsonArray = (JSONArray) parser.parse( new FileReader( dataPath ) );
BulkRequestBuilder bulkDocument = client.prepareBulk();
#SuppressWarnings("unchecked")
Iterator<JSONObject> it = jsonArray.iterator();
int i = 0;
i = _getTotalHits(client,_indexName,_indexType);
System.out.println("Total number of hits inside index = "+_indexName+" of type = "+_indexType+" are : "+i);
System.out.println("-------------------------------------------------------------------------------------");
while( it.hasNext() ) {
i++;
JSONObject json = it.next();
System.out.println("Insert document for "+i+": " + json.toJSONString() );
// either use client#prepare, or use Requests# to directly build index/delete requests
bulkDocument.add(client.prepareIndex(_indexName, _indexType, i+"")
.setSource(json.toJSONString(), XContentType.JSON )
);
}
BulkResponse bulkResponse = bulkDocument.get();
if (bulkResponse.hasFailures()) {
System.out.println("process failures by iterating through each bulk response item : "+bulkResponse.buildFailureMessage());
return false;
} else {
System.out.println("All Documents inserted successfully...");
/*if(bulkResponse.getItems()!=null) {
for(BulkItemResponse response:bulkResponse.getItems()) {
System.out.println(response.toString());
System.out.println(response.getResponse());
}
}*/
return true;
}
} catch (IOException ex) {
System.out.println("Exception occurred while get Multiple Document : " + ex/*, ex*/);
return false;
}
}
public synchronized int _getTotalHits(TransportClient client,String _indexName,String _indexType){
SearchHits hits = null;
int recCount = 0;
long totalCount = 0;
try {
SearchResponse seacrhResponse = client.prepareSearch(_indexName)
.setTypes(_indexType)
.setSearchType(SearchType.QUERY_THEN_FETCH)
.get();
if (seacrhResponse != null) {
hits = seacrhResponse.getHits();
totalCount = hits.getTotalHits();
System.out.println("count = "+totalCount);
}
recCount = Integer.parseInt(totalCount+"");
}catch(Exception ex) {
System.out.println("Exception occurred while search Index : " + ex/*, ex*/);
}
return recCount;
}
}
fairly complex code but it's a very simple issue.
I have a thread, this is its run method:
public void run() //gets pages and writes to them
{ // i printed the pageId of every process to check they are running at the same time and competing for resources
for(ProcessCycle currentCycle : processCycles.getProcessCycles())
{
Long[] longArray = new Long[currentCycle.getPages().size()];
try {
Page<byte[]>[] newPages = mmu.getPages(currentCycle.getPages().toArray(longArray));
for(int i = 0; i < newPages.length; i++)
{
MMULogger.getInstance().write("GP:P" + id + " " + currentCycle.getPages().get(i) + " " + Arrays.toString(currentCycle.getData().get(i)), Level.INFO);
}
List<byte[]> currentPageData = currentCycle.getData();
System.out.println("process id " + id);
for(int i = 0; i < newPages.length;i++)
{
byte[] currentData = currentPageData.get(i);
newPages[i].setContent(currentData);
}
Thread.sleep(currentCycle.getSleepMs());
} catch (ClassNotFoundException | IOException | InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
don't bother reading all of it. just notice that after the mmu.getpages there is a for loop.
While a process is inside the for loop, i want to lock access to mmu.getpages for all other threads. synchronized is no good since my original process is no longer in mmu, but in the for loop, and reentrantlock might be a good idea but I'm unfamiliar with the syntax and ran into some issues.
long story short - how do i make all other threads wait while some thread is inside the for loop after mmu.getpages?
Usually I chose an approach like this:
private Object lock = new Object();
public void run() //gets pages and writes to them
{ // i printed the pageId of every process to check they are running at the same time and competing for resources
for(ProcessCycle currentCycle : processCycles.getProcessCycles())
{
Long[] longArray = new Long[currentCycle.getPages().size()];
try {
synchrnonized(lock) {
Page<byte[]>[] newPages = mmu.getPages(currentCycle.getPages().toArray(longArray));
for(int i = 0; i < newPages.length; i++)
{
MMULogger.getInstance().write("GP:P" + id + " " + currentCycle.getPages().get(i) + " " + Arrays.toString(currentCycle.getData().get(i)), Level.INFO);
}
}
List<byte[]> currentPageData = currentCycle.getData();
System.out.println("process id " + id);
for(int i = 0; i < newPages.length;i++)
{
byte[] currentData = currentPageData.get(i);
newPages[i].setContent(currentData);
}
Thread.sleep(currentCycle.getSleepMs());
} catch (ClassNotFoundException | IOException | InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
Not sure if there is a better way. This will only work as expected when all threads share the same instance of this object, otherwise you have to make lock a static member variable.
In my opinion a ReadWriteLock might be a best approach.
Something like this:
public class MmuClass {
private ReadWriteLock blockGetPages = new ReentrantReadWriteLock();
public byte [] getPages(...) {
try{
blockGetPages.readLock().lock();
// ...
// ...
// ...
return result;
finally{
blockGetPages.readLock().unlock();
}
}
public void lockAccessToGetPages(){
blockGetPages.writeLock().lock();
}
public void unlockAccessToGetPages(){
blockGetPages.writeLock().unlock();
}
}
and
Page<byte[]>[] newPages = mmu.getPages(currentCycle.getPages().toArray(longArray));
try{
mmu.lockAccessToGetPages();
for(int i = 0; i < newPages.length; i++) {
MMULogger.getInstance().write("GP:P" + id + " " + currentCycle.getPages().get(i) + " " + Arrays.toString(currentCycle.getData().get(i)), Level.INFO);
}
} finally{
mmu.unlockAccessToGetPages();
}
In this solutions all "readers" can simultaneously call getPages(), the access is blocked after calling lockAccessToGetPages() and unblocked after calling unlockAccessToGetPages(). If one thread locks the object in write mode, only this thread has access to the method. If some thread tries to lock it in write mode, must wait until all readers, which are currently "inside" the metod, finish their fork and leave the method.
This question already has answers here:
How to use an ExecutorCompletionService
(2 answers)
Closed 7 years ago.
public static void getTestData() {
try {
filename = "InventoryData_" + form_id;
PrintWriter writer = new PrintWriter("/Users/pnroy/Documents/" +filename + ".txt");
pids = new ArrayList<ProductId>();
GetData productList = new GetData();
System.out.println("Getting productId");
pids = productList.GetProductIds(form_id);
int perThreadSize = pids.size() / numberOfCrawlers;
ArrayList<ArrayList<ProductId>> perThreadData = new
ArrayList<ArrayList<ProductId>>(numberOfCrawlers);
for (int i = 1; i <= numberOfCrawlers; i++) {
perThreadData.add(new ArrayList<ProductId>(perThreadSize));
for (int j = 0; j < perThreadSize; j++) {
ProductId ids = new ProductId();
ids.setEbProductID((pids.get(((i - 1) * perThreadSize + j))).getEbProductID());
ids.setECProductID((pids.get(((i - 1) * perThreadSize + j))).getECProductID());
perThreadData.get(i - 1).add(ids);
}
}
BlockingQueue<String> q = new LinkedBlockingQueue<String>();
Consumer c1 = new Consumer(q);
Thread[] thread = new Thread[numberOfCrawlers];
for (int k = 0; k <= numberOfCrawlers; k++) {
// System.out.println(k);
GetCombinedData data = new GetCombinedData();
thread[k] = new Thread(data);
thread[k].setDaemon(true);
data.setVal(perThreadData.get(k), filename, q);
thread[k].start();
// writer.println(data.getResult());
}
new Thread(c1).start();
for (int l = 0; l <= numberOfCrawlers; l++) {
thread[l].join();
}
} catch (Exception e) {
}
}
Here number of crawlers is the number of threads.
The run method of GetCombined class has the following code:
The pids is passed as perThreadData.get(k-1) from the main method
The class CassController queries a API and i get a string result after some processing.
public void run(){
try{
for(int i=0;i<pids.size();i++){
//System.out.println("before cassini");
CassController cass = new CassController();
String result=cass.getPaginationDetails(pids.get(i));
queue.put(result);
// System.out.println(result);
Thread.sleep(1000);
}
writer.close();
}catch(Exception ex){
}
Consumer.java has the following code :
public class Consumer implements Runnable{
private final BlockingQueue queue;
Consumer(BlockingQueue q) { queue = q; }
public void run(){
try {
while (queue.size()>0)
{
consume(queue.take());
}
} catch (InterruptedException ex)
{
}
}
void consume(Object x) {
try{
PrintWriter writer = new PrintWriter(new FileWriter("/Users/pnroy/Documents/Inventory", true));
writer.println(x.toString());
writer.close();
}catch(IOException ex){
}
}
So if i set the number of crawlers to 10 and if there are 500 records each thread will process 50 records.I need to write the results into a file.I am confused how i can achieve this since its a array of thread and each thread is doing a bunch of operations.
I tried using blocking queue but that is printing repetitive results.I am new to multi threading and not sure how can i handle the case.
Can you please suggest.
With the introduction of many useful high-level concurrency classes, it now recommended not to directly use the Thread class anymore. Even the BlockingQueue class is rather low-level.
Instead, you have a nice application for the CompletionService, which builds upon the ExecutorService. The below example shows how to use it.
You want to replace the code in PartialResultTask (that's where the main processing happens) and System.out.println (that's where you probably want to write your result to a file).
public class ParallelProcessing {
public static void main(String[] args) {
ExecutorService executionService = Executors.newFixedThreadPool(10);
CompletionService<String> completionService = new ExecutorCompletionService<>(executionService);
// submit tasks
for (int i = 0; i < 500; i++) {
completionService.submit(new PartialResultTask(i));
}
// collect result
for (int i = 0; i < 500; i++) {
String result = getNextResult(completionService);
if (result != null)
System.out.println(result);
}
executionService.shutdown();
}
private static String getNextResult(CompletionService<String> completionService) {
Future<String> result = null;
while (result == null) {
try {
result = completionService.take();
} catch (InterruptedException e) {
// ignore and retry
}
}
try {
return result.get();
} catch (ExecutionException e) {
e.printStackTrace();
return null;
} catch (InterruptedException e) {
e.printStackTrace();
return null;
}
}
static class PartialResultTask implements Callable<String> {
private int n;
public PartialResultTask(int n) {
this.n = n;
}
#Override
public String call() {
return String.format("Partial result %d", n);
}
}
}
Firstly, yes I'm calling this from a web browser. It's quite a long piece of code but I've tried shortening it as much as possible.
Basically, I need to wait let's say 1 second for every iteration in the loop. Tried pretty much everything (.sleep() etc.) but it just doesn't seem to be pausing. The reason why I need to do this is because the SimpleSocketClient is calling a socket which has a low limit per second allowed.
#Override
public String execute(HttpServletRequest request, HttpServletResponse response) {
String forwardToJsp = null;
HttpSession session = request.getSession();
String allUrls = request.getParameter("domains");
ArrayList domainList = new ArrayList<String>();
Scanner sc = new Scanner(allUrls);
while (sc.hasNextLine()) {
String line = sc.nextLine();
domainList.add(line);
// process the line
}
sc.close();
String pageHtml = null;
String domain = "";
String status = "";
String registrant = "";
String dates = "";
String tag = "";
String email = "";
ArrayList domains = new ArrayList<Domain>();
Domain theDomain;
String ipAddress = request.getHeader("X-FORWARDED-FOR");
if (ipAddress == null) {
ipAddress = request.getRemoteAddr();
}
for (int i = 0; i < domainList.size(); i++) {
//NEED TO WAIT 1 SECOND HERE / ANYWHERE IN LOOP
String singleDomain = domainList.get(i).toString();
SimpleSocketClient tester = new SimpleSocketClient(singleDomain,ipAddress);
pageHtml = tester.getResult();
try {
String whoIs2 = ipAddress + " " + ipAddress + " " + singleDomain + "\r\n";
byte[] data = whoIs2.getBytes();
//details of each domain
//domain name
domain = singleDomain;
//status
status = "FTR";
//registrant
registrant = "N/A";
//dates
dates = "N/A";
//tag
tag = "N/A";
//email
email = "N/A";
}
} catch (Exception e) {
Logger.getLogger("ip is " + ipAddress + bulkWhoIsCommand.class.getName()).log(Level.SEVERE, null, e);
forwardToJsp = "index.jsp";
return forwardToJsp;
}
//single one
theDomain = new Domain(domain,status,registrant,dates,tag,email);
//now add to arrayList
domains.add(theDomain);
// try {
// Thread.sleep(230000);
// } catch (InterruptedException ex) {
// Logger.getLogger(bulkWhoIsCommand.class.getName()).log(Level.SEVERE, null, ex);
// }
// try {
// pause.poll(100 * 300, TimeUnit.MILLISECONDS);
// } catch (InterruptedException ex) {
// Logger.getLogger(bulkWhoIsCommand.class.getName()).log(Level.SEVERE, null, ex);
// }
}
EDIT - Friend recommended to use ajax to poll updates but surely there's a way of just using java.
Your can try to set a while-loop in the while-loop, to pause it. Should like this:
while(!done)
{
long start = System.currentTimeMillis();
while(System.currentTimeMillis() - start < 1000L){}
}
Didn't test it but the approach counts. I had the idea to do a combination of both. So every time Thread.Sleep() crashes, you have to take the loop. Something like this:
while(!done)
{
long start = System.currentTimeMillis();
try {
Thread.sleep(1000);
} catch (InterruptedException ex) {
System.err.println(e);
}
while(System.currentTimeMillis() - start < 1000L){}
}
When Thread.Sleep() worked it just get called once. Otherwise you need some CPU time. Could be the cpu economical version.