This question already has answers here:
How to use an ExecutorCompletionService
(2 answers)
Closed 7 years ago.
public static void getTestData() {
try {
filename = "InventoryData_" + form_id;
PrintWriter writer = new PrintWriter("/Users/pnroy/Documents/" +filename + ".txt");
pids = new ArrayList<ProductId>();
GetData productList = new GetData();
System.out.println("Getting productId");
pids = productList.GetProductIds(form_id);
int perThreadSize = pids.size() / numberOfCrawlers;
ArrayList<ArrayList<ProductId>> perThreadData = new
ArrayList<ArrayList<ProductId>>(numberOfCrawlers);
for (int i = 1; i <= numberOfCrawlers; i++) {
perThreadData.add(new ArrayList<ProductId>(perThreadSize));
for (int j = 0; j < perThreadSize; j++) {
ProductId ids = new ProductId();
ids.setEbProductID((pids.get(((i - 1) * perThreadSize + j))).getEbProductID());
ids.setECProductID((pids.get(((i - 1) * perThreadSize + j))).getECProductID());
perThreadData.get(i - 1).add(ids);
}
}
BlockingQueue<String> q = new LinkedBlockingQueue<String>();
Consumer c1 = new Consumer(q);
Thread[] thread = new Thread[numberOfCrawlers];
for (int k = 0; k <= numberOfCrawlers; k++) {
// System.out.println(k);
GetCombinedData data = new GetCombinedData();
thread[k] = new Thread(data);
thread[k].setDaemon(true);
data.setVal(perThreadData.get(k), filename, q);
thread[k].start();
// writer.println(data.getResult());
}
new Thread(c1).start();
for (int l = 0; l <= numberOfCrawlers; l++) {
thread[l].join();
}
} catch (Exception e) {
}
}
Here number of crawlers is the number of threads.
The run method of GetCombined class has the following code:
The pids is passed as perThreadData.get(k-1) from the main method
The class CassController queries a API and i get a string result after some processing.
public void run(){
try{
for(int i=0;i<pids.size();i++){
//System.out.println("before cassini");
CassController cass = new CassController();
String result=cass.getPaginationDetails(pids.get(i));
queue.put(result);
// System.out.println(result);
Thread.sleep(1000);
}
writer.close();
}catch(Exception ex){
}
Consumer.java has the following code :
public class Consumer implements Runnable{
private final BlockingQueue queue;
Consumer(BlockingQueue q) { queue = q; }
public void run(){
try {
while (queue.size()>0)
{
consume(queue.take());
}
} catch (InterruptedException ex)
{
}
}
void consume(Object x) {
try{
PrintWriter writer = new PrintWriter(new FileWriter("/Users/pnroy/Documents/Inventory", true));
writer.println(x.toString());
writer.close();
}catch(IOException ex){
}
}
So if i set the number of crawlers to 10 and if there are 500 records each thread will process 50 records.I need to write the results into a file.I am confused how i can achieve this since its a array of thread and each thread is doing a bunch of operations.
I tried using blocking queue but that is printing repetitive results.I am new to multi threading and not sure how can i handle the case.
Can you please suggest.
With the introduction of many useful high-level concurrency classes, it now recommended not to directly use the Thread class anymore. Even the BlockingQueue class is rather low-level.
Instead, you have a nice application for the CompletionService, which builds upon the ExecutorService. The below example shows how to use it.
You want to replace the code in PartialResultTask (that's where the main processing happens) and System.out.println (that's where you probably want to write your result to a file).
public class ParallelProcessing {
public static void main(String[] args) {
ExecutorService executionService = Executors.newFixedThreadPool(10);
CompletionService<String> completionService = new ExecutorCompletionService<>(executionService);
// submit tasks
for (int i = 0; i < 500; i++) {
completionService.submit(new PartialResultTask(i));
}
// collect result
for (int i = 0; i < 500; i++) {
String result = getNextResult(completionService);
if (result != null)
System.out.println(result);
}
executionService.shutdown();
}
private static String getNextResult(CompletionService<String> completionService) {
Future<String> result = null;
while (result == null) {
try {
result = completionService.take();
} catch (InterruptedException e) {
// ignore and retry
}
}
try {
return result.get();
} catch (ExecutionException e) {
e.printStackTrace();
return null;
} catch (InterruptedException e) {
e.printStackTrace();
return null;
}
}
static class PartialResultTask implements Callable<String> {
private int n;
public PartialResultTask(int n) {
this.n = n;
}
#Override
public String call() {
return String.format("Partial result %d", n);
}
}
}
Related
I would like to create 5 million csv files, I have waiting for almost 3 hours, but the program is still running. Can somebody give me some advice, how to speed up the file generation.
After these 5 million files generation complete, I have to upload them to s3 bucket.
It would be better if someone know how to generate these files through AWS, thus, we can move files to s3 bucket directly and ignore network speed issue.(Just start to learning AWS, there are lots of knowledge need to know)
The following is my code.
public class ParallelCsvGenerate implements Runnable {
private static AtomicLong baseID = new AtomicLong(8160123456L);
private static ThreadLocalRandom random = ThreadLocalRandom.current();
private static ThreadLocalRandom random2 = ThreadLocalRandom.current();
private static String filePath = "C:\\5millionfiles\\";
private static List<String> headList = null;
private static String csvHeader = null;
public ParallelCsvGenerate() {
headList = generateHeadList();
csvHeader = String.join(",", headList);
}
#Override
public void run() {
for(int i = 0; i < 1000000; i++) {
generateCSV();
}s
}
private void generateCSV() {
StringBuilder builder = new StringBuilder();
builder.append(csvHeader).append(System.lineSeparator());
for (int i = 0; i < headList.size(); i++) {
if(i < headList.size() - 1) {
builder.append(i % 2 == 0 ? generateRandomInteger() : generateRandomStr()).append(",");
} else {
builder.append(i % 2 == 0 ? generateRandomInteger() : generateRandomStr());
}
}
String fileName = String.valueOf(baseID.addAndGet(1));
File csvFile = new File(filePath + fileName + ".csv");
FileWriter fileWriter = null;
try {
fileWriter = new FileWriter(csvFile);
fileWriter.write(builder.toString());
fileWriter.flush();
} catch (Exception e) {
System.err.println(e);
} finally {
try {
if(fileWriter != null) {
fileWriter.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
private static List<String> generateHeadList() {
List<String> headList = new ArrayList<>(20);
String baseFiledName = "Field";
for(int i = 1; i <=20; i++) {
headList.add(baseFiledName + i);
}
return headList;
}
/**
* generate a number in range of 0-50000
* #return
*/
private Integer generateRandomInteger() {
return random.nextInt(0,50000);
}
/**
* generate a string length is 5 - 8
* #return
*/
private String generateRandomStr() {
int strLength = random2.nextInt(5, 8);
String str="abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
int length = str.length();
StringBuilder builder = new StringBuilder();
for (int i = 0; i < strLength; i++) {
builder.append(str.charAt(random.nextInt(length)));
}
return builder.toString();
}
Main
ParallelCsvGenerate generate = new ParallelCsvGenerate();
Thread a = new Thread(generate, "A");
Thread b = new Thread(generate, "B");
Thread c = new Thread(generate, "C");
Thread d = new Thread(generate, "D");
Thread e = new Thread(generate, "E");
a.run();
b.run();
c.run();
d.run();
e.run();
Thanks for your guys advice, just refactor the code, and generate 3.8million files using 2.8h, which is much better.
Refactor code:
public class ParallelCsvGenerate implements Callable<Integer> {
private static String filePath = "C:\\5millionfiles\\";
private static String[] header = new String[]{
"FIELD1","FIELD2","FIELD3","FIELD4","FIELD5",
"FIELD6","FIELD7","FIELD8","FIELD9","FIELD10",
"FIELD11","FIELD12","FIELD13","FIELD14","FIELD15",
"FIELD16","FIELD17","FIELD18","FIELD19","FIELD20",
};
private String fileName;
public ParallelCsvGenerate(String fileName) {
this.fileName = fileName;
}
#Override
public Integer call() throws Exception {
try {
generateCSV();
} catch (IOException e) {
e.printStackTrace();
}
return 0;
}
private void generateCSV() throws IOException {
CSVWriter writer = new CSVWriter(new FileWriter(filePath + fileName + ".csv"), CSVWriter.DEFAULT_SEPARATOR, CSVWriter.NO_QUOTE_CHARACTER);
String[] content = new String[]{
RandomGenerator.generateRandomInteger(),
RandomGenerator.generateRandomStr(),
RandomGenerator.generateRandomInteger(),
RandomGenerator.generateRandomStr(),
RandomGenerator.generateRandomInteger(),
RandomGenerator.generateRandomStr(),
RandomGenerator.generateRandomInteger(),
RandomGenerator.generateRandomStr(),
RandomGenerator.generateRandomInteger(),
RandomGenerator.generateRandomStr(),
RandomGenerator.generateRandomInteger(),
RandomGenerator.generateRandomStr(),
RandomGenerator.generateRandomInteger(),
RandomGenerator.generateRandomStr(),
RandomGenerator.generateRandomInteger(),
RandomGenerator.generateRandomStr(),
RandomGenerator.generateRandomInteger(),
RandomGenerator.generateRandomStr(),
RandomGenerator.generateRandomInteger(),
RandomGenerator.generateRandomStr()
};
writer.writeNext(header);
writer.writeNext(content);
writer.close();
}
}
Main
public static void main(String[] args) {
System.out.println("Start generate");
long start = System.currentTimeMillis();
ThreadPoolExecutor threadPoolExecutor = new ThreadPoolExecutor(8, 8,
0L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<Runnable>());
List<ParallelCsvGenerate> taskList = new ArrayList<>(3800000);
for(int i = 0; i < 3800000; i++) {
taskList.add(new ParallelCsvGenerate(i+""));
}
try {
List<Future<Integer>> futures = threadPoolExecutor.invokeAll(taskList);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("Success");
long end = System.currentTimeMillis();
System.out.println("Using time: " + (end-start));
}
You could write directly into the file (without allocating the whole file in one StringBuilder). (I think this is the biggest time+memory bottleneck here: builder.toString())
You could generate each file in parallel.
(little tweaks:) Omit the if's inside loop.
if(i < headList.size() - 1) is not needed, when you do a more clever loop + 1 extra iteration.
The i % 2 == 0 can be eliminated by a better iteration (i+=2) ..and more labor inside the loop (i -> int, i + 1 -> string)
If applicable prefer append(char) to append(String). (Better append(',') than append(",")!)
...
You can use Fork/Join framework (java 7 and above) to make your process in parallel and use multi core of your Cpu.
I'll take an example for you.
import java.util.concurrent.ForkJoinPool;
import java.util.concurrent.ForkJoinTask;
import java.util.concurrent.RecursiveTask;
import java.util.stream.LongStream;
public class ForkJoinAdd extends RecursiveTask<Long> {
private final long[] numbers;
private final int start;
private final int end;
public static final long threshold = 10_000;
public ForkJoinAdd(long[] numbers) {
this(numbers, 0, numbers.length);
}
private ForkJoinAdd(long[] numbers, int start, int end) {
this.numbers = numbers;
this.start = start;
this.end = end;
}
#Override
protected Long compute() {
int length = end - start;
if (length <= threshold) {
return add();
}
ForkJoinAdd firstTask = new ForkJoinAdd(numbers, start, start + length / 2);
firstTask.fork(); //start asynchronously
ForkJoinAdd secondTask = new ForkJoinAdd(numbers, start + length / 2, end);
Long secondTaskResult = secondTask.compute();
Long firstTaskResult = firstTask.join();
return firstTaskResult + secondTaskResult;
}
private long add() {
long result = 0;
for (int i = start; i < end; i++) {
result += numbers[i];
}
return result;
}
public static long startForkJoinSum(long n) {
long[] numbers = LongStream.rangeClosed(1, n).toArray();
ForkJoinTask<Long> task = new ForkJoinAdd(numbers);
return new ForkJoinPool().invoke(task);
}
}
use this example
And if you want to read more about it, Guide to the Fork/Join Framework in Java | Baeldung
and Fork/Join (The Java™ Tutorials
can help you to better understand and better design your app.
be lucky.
Remove the for(int i = 0; i < 1000000; i++) loop from run method (leave a single generateCSV() call.
Create 5 million ParallelCsvGenerate objects.
Submit them to a ThreadPoolExecutor
Converted main:
public static void main(String[] args) {
ThreadPoolExecutor ex = (ThreadPoolExecutor) Executors.newFixedThreadPool(8);
for(int i = 0; i < 5000000; i++) {
ParallelCsvGenerate generate = new ParallelCsvGenerate();
ex.submit(generate);
}
ex.shutdown();
}
It takes roughly 5 minutes to complete on my laptop (4 physical cores with hyperthreading, SSD drive).
EDIT:
I've replaced FileWriter with AsynchronousFileChannel using the following code:
Path file = Paths.get(filePath + fileName + ".csv");
try(AsynchronousFileChannel asyncFile = AsynchronousFileChannel.open(file,
StandardOpenOption.WRITE,
StandardOpenOption.CREATE)) {
asyncFile.write(ByteBuffer.wrap(builder.toString().getBytes()), 0);
} catch (IOException e) {
e.printStackTrace();
}
to achieve 30% speedup.
I believe that the main bottleneck is the hard drive and filesystem itself. Not much more can be achieved here.
I have below code where I am testing thread synchronization for elastic search, but somehow I am not able to success in it, can any one let me know where I am going wrong?
If I enable Thread sleep inside 'startThreadProcess' method then everything works fine because it sleeps for certain amount of time period. Which I don't want to, I want to get proper lock for thread without using thread sleep.
but what is happening in above code I have used Executor for pooling threads. There I am running for loop for count 4 so as to initiate 4 threads from pool. so when my thread submitted using executor submit which calls synchronized method and inside that synchronized method I am giving call to other synchronized method from where I am getting total number of count in particular node and proceeding ahead with that count by incrementing to insert new document where my first thread is yet not completed 2nd thread enters and try to get total number of count from the method which is being called from synchronized method so there I am getting wrong count for thread 2 as my first thread will insert 10000 json documents inside node so I am expecting Thread 2 should get count 10000 and then it should process for insert but here my Thread 2 enters in between and get count any random number and start inserting by incrementing from that number which is not expected scenario
package com.acn.adt.main;
public class ESTest {
private static final String dataPath = "C:\\Elastic Search\\Data.json";
static ESTest esTest = new ESTest();
private static TransportClient client = null;
private Properties elasticPro = null;
private InputStream input = null;
ElasticSearchCrud esCRUD = null;
private final Object lock = new Object();
public static void main(String[] args) {
String strArray[] = new String[] {"1"};
esTest.startProcess(strArray);
}
public void startProcess(String strArray[]) {
try {
input = new FileInputStream(ElasticSearchConstants.ELASTIC_PROPERTIES);
elasticPro = new Properties();
//elasticPro.load(ElasticSearchClient.class.getResourceAsStream(ElasticSearchConstants.ELASTIC_PROPERTIES));
elasticPro.load(input);
System.out.println(elasticPro.getProperty("homeDir"));
long startTime = System.currentTimeMillis();
Settings setting = Settings.builder()
//.put("client.transport.ping_timeout", "100s")
.put("cluster.name", elasticPro.getProperty("cluster"))
//.put("node.name", elasticPro.getProperty("node"))
//.put("client.transport.sniff", Boolean.valueOf(elasticPro.getProperty("transport.sniff")))
.put("client.transport.sniff", false)
.put("cluster.routing.allocation.enable", "all")
.put("cluster.routing.allocation.allow_rebalance", "always")
//.put("client.transport.ignore_cluster_name", true)
.build();
client = new PreBuiltTransportClient(setting)
.addTransportAddress(new TransportAddress(InetAddress.getByName("localhost"),
Integer.valueOf("9300")));
long endTime = System.currentTimeMillis();
System.out.println("Time taken for connecting " + TimeUnit.MILLISECONDS.toSeconds((endTime - startTime)));
ExecutorService executorService = Executors.newFixedThreadPool(10);
for(int i = 1; i <=4; i++) {
if(i==1) {
strArray = new String [] {"1"};
}else if(i == 2) {
strArray = new String [] {"1"};
}else if(i == 3) {
strArray = new String [] {"1"};
}else if(i == 4) {
strArray = new String [] {"1"};
}
executorService.execute(new ESThread(esTest,strArray,i));
}
}catch(Exception e) {
}
}
public class ESThread implements Runnable {
private final Object lock = new Object();
ESTest esTester = null;
String strArr [] = null;
int i =0;
public ESThread(ESTest esTester,String[] strArr,int i) {
this.esTester = esTester;
this.strArr = strArr;
this.i = i;
}
#Override
public void run() {
System.out.println("Name of Current thread is Thread_"+i);
synchronized(lock) {
esTester.startCRUDProcess(strArr);
}
System.out.println("Thread_"+i+" done.");
}
}
public void startCRUDProcess(String [] strArr) {
SearchAPI esSearch = new SearchAPIImpl();
boolean caseFlg = false;
String _indexName = "gcindex";
String _indexType = "gctype";
String _ids = "501,602,702###1,10000,10001";
String _id = "10000";
String[] _strIds = new String[] {"10000","9999"};
System.out.println("Insert Multiple Process is started...");
System.out.println("--------------------------------------");
try {
caseFlg = insertMultipleDocument(dataPath,client,_indexName,_indexType);
} catch (IOException | ParseException e) {
e.printStackTrace();
caseFlg = false;
}
}
public synchronized boolean insertMultipleDocument(String dataPath,TransportClient client,String _indexName,String _indexType) throws FileNotFoundException, ParseException {
try {
JSONParser parser = new JSONParser();
// we know we get an array from the example data
JSONArray jsonArray = (JSONArray) parser.parse( new FileReader( dataPath ) );
BulkRequestBuilder bulkDocument = client.prepareBulk();
#SuppressWarnings("unchecked")
Iterator<JSONObject> it = jsonArray.iterator();
int i = 0;
i = _getTotalHits(client,_indexName,_indexType);
System.out.println("Total number of hits inside index = "+_indexName+" of type = "+_indexType+" are : "+i);
System.out.println("-------------------------------------------------------------------------------------");
while( it.hasNext() ) {
i++;
JSONObject json = it.next();
System.out.println("Insert document for "+i+": " + json.toJSONString() );
// either use client#prepare, or use Requests# to directly build index/delete requests
bulkDocument.add(client.prepareIndex(_indexName, _indexType, i+"")
.setSource(json.toJSONString(), XContentType.JSON )
);
}
BulkResponse bulkResponse = bulkDocument.get();
if (bulkResponse.hasFailures()) {
System.out.println("process failures by iterating through each bulk response item : "+bulkResponse.buildFailureMessage());
return false;
} else {
System.out.println("All Documents inserted successfully...");
/*if(bulkResponse.getItems()!=null) {
for(BulkItemResponse response:bulkResponse.getItems()) {
System.out.println(response.toString());
System.out.println(response.getResponse());
}
}*/
return true;
}
} catch (IOException ex) {
System.out.println("Exception occurred while get Multiple Document : " + ex/*, ex*/);
return false;
}
}
public synchronized int _getTotalHits(TransportClient client,String _indexName,String _indexType){
SearchHits hits = null;
int recCount = 0;
long totalCount = 0;
try {
SearchResponse seacrhResponse = client.prepareSearch(_indexName)
.setTypes(_indexType)
.setSearchType(SearchType.QUERY_THEN_FETCH)
.get();
if (seacrhResponse != null) {
hits = seacrhResponse.getHits();
totalCount = hits.getTotalHits();
System.out.println("count = "+totalCount);
}
recCount = Integer.parseInt(totalCount+"");
}catch(Exception ex) {
System.out.println("Exception occurred while search Index : " + ex/*, ex*/);
}
return recCount;
}
}
I have the following code to read from Chronicle queue (it's written in Kotlin but that does not matter):
val queue = ChronicleQueueBuilder.single(path).build()
val tailer = queue.createTailer()
tailer.toEnd()
// // This code is wrong
// val lastIndex = tailer.index()
//
// val shift = lastIndex - 10
// if (shift > 0) {
// tailer.moveToIndex(lastIndex)
// }
while (true) {
val text = await(tailer)
if (prefix == null) {
println(text)
} else {
if (text.startsWith(prefix)) {
// Would be nice without additional allocation ...
println(text.substring(prefix.length + 1))
}
}
}
How can I modify the commented code to read previous 10 records from the queue and continue on?
Rationale: It is useful in situations where the queue is used for displaying logs. You want to see a few previous logging statements and see new logging statements as they come.
I have written a test for you. Please run the same it should work.
public class ChronicleTest {
private String chroniclePath = "/tmp/chronicle-test";
private int msgCount = 10;
private int i = 0;
#Test
public void writeToQ() {
ChronicleQueue queue = ChronicleQueueBuilder.single(chroniclePath).build();
ExcerptAppender appender = queue.acquireAppender();
for (i = 1; i <= msgCount; i++) {
appender.writeBytes(b -> {
b.writeInt(i);
});
}
ExcerptTailer tailer = queue.createTailer();
tailer.toEnd();
long lastIndex = tailer.index();
tailer.moveToIndex(lastIndex - 5);
while (tailer.readBytes(b -> {
int value = b.readInt();
System.out.println("Received:" + value);
}))
System.out.println("Completed");
}
}
In addition to directly using the index, you could use the direction property of the ExcerptTailer:
final SingleChronicleQueue queue = createQueue();
final int totalRecords = 20;
final int tailRecords = 10;
final ExcerptAppender appender = queue.acquireAppender();
for (int i = 0; i < totalRecords; i++) {
try(final DocumentContext ctx = appender.writingDocument()) {
ctx.wire().writeText(Integer.toString(i));
}
}
final ExcerptTailer tailer = queue.createTailer();
tailer.direction(TailerDirection.BACKWARD).toEnd();
int rewind = tailRecords;
final int endCycle = tailer.cycle();
while(--rewind != 0) {
try(final DocumentContext ctx = tailer.readingDocument()) {
if (!ctx.isPresent()) {
break;
}
if (endCycle != tailer.cycle()) {
System.out.println("Rewound past beginning of cycle");
}
}
}
tailer.direction(TailerDirection.FORWARD);
for (int i = 0; i < tailRecords; i++) {
try(final DocumentContext ctx = tailer.readingDocument()) {
if (!ctx.isPresent()) {
break;
}
System.out.println(ctx.wire().readText());
}
}
fairly complex code but it's a very simple issue.
I have a thread, this is its run method:
public void run() //gets pages and writes to them
{ // i printed the pageId of every process to check they are running at the same time and competing for resources
for(ProcessCycle currentCycle : processCycles.getProcessCycles())
{
Long[] longArray = new Long[currentCycle.getPages().size()];
try {
Page<byte[]>[] newPages = mmu.getPages(currentCycle.getPages().toArray(longArray));
for(int i = 0; i < newPages.length; i++)
{
MMULogger.getInstance().write("GP:P" + id + " " + currentCycle.getPages().get(i) + " " + Arrays.toString(currentCycle.getData().get(i)), Level.INFO);
}
List<byte[]> currentPageData = currentCycle.getData();
System.out.println("process id " + id);
for(int i = 0; i < newPages.length;i++)
{
byte[] currentData = currentPageData.get(i);
newPages[i].setContent(currentData);
}
Thread.sleep(currentCycle.getSleepMs());
} catch (ClassNotFoundException | IOException | InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
don't bother reading all of it. just notice that after the mmu.getpages there is a for loop.
While a process is inside the for loop, i want to lock access to mmu.getpages for all other threads. synchronized is no good since my original process is no longer in mmu, but in the for loop, and reentrantlock might be a good idea but I'm unfamiliar with the syntax and ran into some issues.
long story short - how do i make all other threads wait while some thread is inside the for loop after mmu.getpages?
Usually I chose an approach like this:
private Object lock = new Object();
public void run() //gets pages and writes to them
{ // i printed the pageId of every process to check they are running at the same time and competing for resources
for(ProcessCycle currentCycle : processCycles.getProcessCycles())
{
Long[] longArray = new Long[currentCycle.getPages().size()];
try {
synchrnonized(lock) {
Page<byte[]>[] newPages = mmu.getPages(currentCycle.getPages().toArray(longArray));
for(int i = 0; i < newPages.length; i++)
{
MMULogger.getInstance().write("GP:P" + id + " " + currentCycle.getPages().get(i) + " " + Arrays.toString(currentCycle.getData().get(i)), Level.INFO);
}
}
List<byte[]> currentPageData = currentCycle.getData();
System.out.println("process id " + id);
for(int i = 0; i < newPages.length;i++)
{
byte[] currentData = currentPageData.get(i);
newPages[i].setContent(currentData);
}
Thread.sleep(currentCycle.getSleepMs());
} catch (ClassNotFoundException | IOException | InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
Not sure if there is a better way. This will only work as expected when all threads share the same instance of this object, otherwise you have to make lock a static member variable.
In my opinion a ReadWriteLock might be a best approach.
Something like this:
public class MmuClass {
private ReadWriteLock blockGetPages = new ReentrantReadWriteLock();
public byte [] getPages(...) {
try{
blockGetPages.readLock().lock();
// ...
// ...
// ...
return result;
finally{
blockGetPages.readLock().unlock();
}
}
public void lockAccessToGetPages(){
blockGetPages.writeLock().lock();
}
public void unlockAccessToGetPages(){
blockGetPages.writeLock().unlock();
}
}
and
Page<byte[]>[] newPages = mmu.getPages(currentCycle.getPages().toArray(longArray));
try{
mmu.lockAccessToGetPages();
for(int i = 0; i < newPages.length; i++) {
MMULogger.getInstance().write("GP:P" + id + " " + currentCycle.getPages().get(i) + " " + Arrays.toString(currentCycle.getData().get(i)), Level.INFO);
}
} finally{
mmu.unlockAccessToGetPages();
}
In this solutions all "readers" can simultaneously call getPages(), the access is blocked after calling lockAccessToGetPages() and unblocked after calling unlockAccessToGetPages(). If one thread locks the object in write mode, only this thread has access to the method. If some thread tries to lock it in write mode, must wait until all readers, which are currently "inside" the metod, finish their fork and leave the method.
SOLVED IT
I've written a program that loads Strings after an equal sign, and has it count how many times its done this. After counting, I tell it to tell me how large the int is. The value I'm looking for is 3, and it tells me, 3. I then change it to an String, the value stays three. Then, I put it into an 4d array, and It tells me the value is 2. What happened?
The Code:
int times=0;
else if (list.equals("Weapon")) {//If the word weapon is before the =
weapon = value; //take the string after the = and put it into String weapon
troopStats[times][1][weaponTimes][0] = weapon;
weaponTimes++;
System.out.println(weaponTimes+"weapontimes"+times);
}
weaponTimesStr = Integer.toString(weaponTimes);
System.out.println(weaponTimesStr+"string");
troopStats[times][1][0][1] = weaponTimesStr;
System.out.println(troopStats[times][1][0][1]+"InArray");
times++
//loops
The Output:
3weapontimes //Counted the equals sign 3 times, Note that this is from the part of the
omitted code
3string // Changed the integer to a string and got 3
2InArray // Put it into an array, and got 2 back
What Is going on?
(I know that I could just add 1 to the value, but I want to use this code for a unknown number of things later on)
To help, I've posted the entire code:
public class TroopLoader {
static String[][][][] troopStats;
static int times = 0;
static int weaponTimes = 0;
static int armorTimes = 0;
static int animalTimes = 0;
static String weaponTimesStr;
static String armorTimesStr;
static String animalTimesStr;
static String troop;
static String weapon;
static String armor;
static String animal;
static String speed;
static int total = 0;
/*
* [][][]
*
* [total number of troops (total)]
*
* [stats] 0= name 1= weapon 2= armor 3= animal 4= speed
*
* [different things within stat]
*/
public void readTroop() {
File file = new File("resources/objects/troops.txt");
BufferedReader reader = null;
try {
reader = new BufferedReader(new FileReader(file));
String text = null;
// repeat until all lines is read
while ((text = reader.readLine()) != null) {
StringTokenizer troops = new StringTokenizer(text, "=");
if (troops.countTokens() == 2) {
String list = troops.nextToken();
if (list.equals("Troop")) {
total++;
}
else {
}
} else {
}
}
troopStats = new String[total][5][10][2];
}
catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
finally {
try {
if (reader != null) {
reader.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
File file2 = new File("resources/objects/troops.txt");
BufferedReader reader2 = null;
try {
reader2 = new BufferedReader(new FileReader(file2));
String text = null;
// repeat until all lines is read
while ((text = reader2.readLine()) != null) {
StringTokenizer troops = new StringTokenizer(text, "=");
if (troops.countTokens() == 2) {
String list = troops.nextToken();
String value = troops.nextToken();
if (list.equals("Troop")) {
troop = value;
troopStats[times][0][0][0] = troop;
}
else if (list.equals("Weapon")) {
weapon = value;
troopStats[times][1][weaponTimes][0] = weapon;
weaponTimes++;
System.out.println(weaponTimes+"weapontimes"+times);
}
else if (list.equals("Armor")) {
armor = value;
troopStats[times][2][armorTimes][0] = armor;
armorTimes++;
}
else if (list.equals("Animal")) {
animal = value;
troopStats[times][3][animalTimes][0] = animal;
animalTimes++;
}
else if (list.equals("Speed")) {
speed = value;
troopStats[times][4][0][0] = speed;
}
else if (list.equals("Done")) {
weaponTimesStr = Integer.toString(weaponTimes);
System.out.println(weaponTimesStr+"string");
armorTimesStr = Integer.toString(armorTimes);
animalTimesStr = Integer.toString(animalTimes);
troopStats[times][1][0][1] = weaponTimesStr;
troopStats[times][1][0][1] = armorTimesStr;
troopStats[times][1][0][1] = animalTimesStr;
System.out.println(troopStats[times][1][0][1]+"InArray"+times);
times++;
troop = "";
weapon = "";
armor = "";
animal = "";
speed = "";
weaponTimes = 0;
armorTimes = 0;
animalTimes = 0;
}
else {
}
} else {
}
}
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
finally {
try {
if (reader2 != null) {
reader2.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
In the earlier part of the code, I had the program store a value in the location on the array with the weaponTimes variable, not storing the weaponTimes variable. My mistake, sorry for wasting your time.
I wrote a SSCCE with what you posted and it prints what you would expect:
public static void main(String[] args) {
String[][][][] troopStats = new String[4][4][4][4];
int times = 2;
int weaponTimes = 3;
String weaponTimesStr = Integer.toString(weaponTimes);
System.out.println(weaponTimesStr + "string"); //prints 3string
troopStats[times][1][0][1] = weaponTimesStr;
System.out.println(troopStats[times][1][0][1] + "InArray"); //prints 3InArray
}
So the problem is most likely something/somewhere else.
The following:
public class Foo {
public static void main(String[] args) {
String[][][][] troopStats = new String[2][2][2][2];
String weaponTimesStr = Integer.toString(3);
System.out.println(weaponTimesStr+"string");
troopStats[0][1][0][1] = weaponTimesStr;
// You said in a comment that 'times' is equal to 0 in this case so have subbed that in
System.out.println(troopStats[0][1][0][1]+"InArray");
}
}
Gives me the expected output:
3string
3InArray
Sorry I've wasted your time, my mistake was because I stored values in the array using the values of weaponTimes, and not storing weaponTimes in the array.
troopStats[times][1][weaponTimes][0] = weapon;
That was the mistake.