We Keep Coding

Multiple Threads in a thread pool writing data in same List

I'm having multiple threads running in my threadPool Each thread reads a huge file and returns the data from this file in a List.
Code looks like :
class Writer{
ArrayList finalListWhereDataWillBeWritten = new Array<Integer>()
for(query q : allQueries){ //all the read queries to read file
threadPool.submit(new GetDataFromFile(fileName,filePath));
}//all the read queries have been submitted.
}
Now I know that following section of code will occur some where in my code but I don't know where to place it.
Because if I place it just after submit() in for loop it'll not add it because each file is very huge and may not have completed its processing.
synchronized(finalListWhereDataWillBeWritten){
//process the data obtained from single file and add it to target list
finalListWhereDataWillBeWritten.addAll(dataFromSingleThread);
}
So can anyone please tell me that where do I place this chunk of code and what other things I need to make sure of so that Critical Section Problem donot occur.
class GetDataFromFile implements Runnable<List<Integer>>{
private String fileName;
private String filePath;
public List<Integer> run(){
//code for streaming the file fileName
return dataObtainedFromThisFile;
}
}
And do i need to use wait() / notifyAll() methods in my code given that I'm only reading data from files parallely in threads and placing them in a shared List

Instead of reinventing the wheel you should simply implement Callable<List<Integer>> and submit it to the JDK's standard Executor Service. Then, as the futures complete, you collect the results into the list.
final ExecutorService threadPool =
Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors());
final List<Future<List<Integer>>> futures = new ArrayList<>();
for(query q : allQueries) {
futures.add(threadPool.submit(new GetDataFromFile(fileName, filePath)));
}
for (Future<List<Integer>> f : futures) {
finalListWhereDataWillBeWritten.addAll(f.get());
}
And this is all assuming you are below Java 8. With Java 8 you would of course use a parallel stream:
final List<Integer> finalListWhereDataWillBeWritten =
allQueries.parallelStream()
.flatMap(q -> getDataFromFile(q.fileName, q.filePath))
.collect(toList());

UPDATE Please consider the answer provided by Marko which is far better
If you want to ensure that your threads all complete before you work on your list, do the following:
import java.util.List;
import java.util.Vector;
public class ThreadWork {
public static void main(String[] args) {
int count = 5;
Thread[] threads = new ListThread[count];
List<String> masterList = new Vector<String>();
for(int index = 0; index < count; index++) {
threads[index] = new ListThread(masterList, "Thread " + (index + 1));
threads[index].start();
}
while(isOperationRunning(threads)) {
// do nothing
}
System.out.println("Done!! Print Your List ...");
for(String item : masterList){
System.out.println("[" + item + "]");
}
}
private static boolean isOperationRunning(Thread[] threads) {
boolean running = false;
for(Thread thread : threads) {
if(thread.isAlive()) {
running = true;
break;
}
}
return running;
}
}
class ListThread extends Thread {
private static String items[] = { "A", "B", "C", "D"};
private List<String> list;
private String name;
public ListThread(List<String> masterList, String threadName) {
list = masterList;
name = threadName;
}
public void run() {
for(int i = 0; i < items.length;++i) {
randomWait();
String data = "Thread [" + name + "][" + items[i] + "]";
System.out.println( data );
list.add( data );
}
}
private void randomWait() {
try {
Thread.currentThread();
Thread.sleep((long)(3000 * Math.random()));
}
catch (InterruptedException x) {}
}
}

Generics and type erasure in Java issue

I currently face the following issue:
I am trying to refactor a recursive algorithm to an iterative one. What this recursive algorithm does is this:
method1 is passed some initial parameters. Based on a processing that takes place at the beginning of method1, method2 is invoked with these parameters. Now method2 uses some conditions and based on the one that is satisfied method1 is invoked again with the appropriate parameters.
Now, based on the answer on the link I've provided above I did the same thing. But I have to pass parameters around so I did this:
Stack<ArrayList<Object>> stack (the stack for the ArrayList<Object> objects)
ArrayList<Object> parametersForSync = new ArrayList<Object>();
ArrayList<Object> paramForHandle = new ArrayList<Object>();
(Each array list of objects is a list of the parameters to be passed to both the methods. The first array list is for the first method and the second for the second method.)
Assuming I pop and push array lists down the stack correctly I face the following issue which is my main problem and the reason of this question:
Within method2 I have to check whether the object (that was on the array list and is passed to the method) is an instanceof another class of mine. Now I have some conditions there which do not get satisfied when in fact they should.
Is this because of java's type erasure?
Is there anyway to overcome this?
If am not clear at a certain point in my explanations please ask me to clarify.
Edit:
What follows is the code that replaces the recursion that goes like this:
syncWithServer(parameter set x){
handleResultArray(parameter set y);
};
handleResultArray(parameter set ){
syncWithServer(parameter set w)
}
===========================================================
Stack<ArrayList<Object>> stack = new Stack<ArrayList<Object>>();
ArrayList<Object> paramList = new ArrayList<Object>();
paramList.add(oeHelper);
paramList.add(false);
paramList.add(domain);
paramList.add(null);
paramList.add(true);
paramList.add(10000);
paramList.add(true);
stack.push(paramList);
int counter = 0;
ArrayList<Object> parametersForSync = new ArrayList<Object>();
ArrayList<Object> paramForHandle = new ArrayList<Object>();
while (!stack.isEmpty()) {
Log.d(TAG, "Loop: " + counter);
parametersForSync = stack.pop();
paramForHandle = ((OEHelper) parametersForSync.get(0))
.syncWithServer(
// why error here?
(boolean) parametersForSync.get(1),
(OEDomain) parametersForSync.get(2),
(List<Object>) parametersForSync.get(3),
(boolean) parametersForSync.get(4),
(int) parametersForSync.get(5),
(boolean) parametersForSync.get(6));
parametersForSync = ((OEHelper) paramForHandle.get(3))
.handleResultArray(
(OEFieldsHelper) paramForHandle.get(0),
(JSONArray) paramForHandle.get(1),
(boolean) paramForHandle.get(2));
if (parametersForSync.size() != 0) {
stack.push(parametersForSync);
}
counter++;
Now the first method:
public ArrayList<Object> syncWithServer(boolean twoWay, OEDomain domain,
List<Object> ids, boolean limitedData, int limits,
boolean removeLocalIfNotExists) {
Log.d(TAG, "syncWithServer");
List<OEColumn> dbCols = mDatabase.getDatabaseColumns();
List<OEColumn> dbFinalList = new ArrayList<OEColumn>();
ArrayList<Object> parametersList = new ArrayList<Object>();
Log.d(TAG, "Columns & finalList created");
for (OEColumn col : dbCols) {
if (!mOne2ManyCols.contains(col.getName())) {
dbFinalList.add(col);
}
}
OEFieldsHelper fields = new OEFieldsHelper(dbFinalList);
try {
if (domain == null) {
domain = new OEDomain();
}
if (ids != null) {
domain.add("id", "in", ids);
}
if (limitedData) {
mPref = new PreferenceManager(mContext);
int data_limit = mPref.getInt("sync_data_limit", 60);
domain.add("create_date", ">=",
OEDate.getDateBefore(data_limit));
}
if (limits == -1) {
limits = 50;
}
Log.d(TAG, "*****.search_read() started");
JSONObject result = *****.search_read(mDatabase.getModelName(),
fields.get(), domain.get(), 0, limits, null, null);
Log.d(TAG, "***.search_read() returned");
mAffectedRows = result.getJSONArray("records").length();
parametersList.add(fields);
parametersList.add(result.getJSONArray("records"));
parametersList.add(removeLocalIfNotExists);
parametersList.add(OEHelper.this);
//This parametersList contains the parameters that must be used to invoke the next method
Now the second method:
=================================================
public ArrayList<Object> handleResultArray(
OEFieldsHelper fields, JSONArray results,
boolean removeLocalIfNotExists) {
Log.d(TAG, "handleResultArray");
ArrayList<Object> parametersList = new ArrayList<Object>();
// ArrayList<Object> parameterStack = new ArrayList<Object>();
try {
fields.addAll(results);
List<OERelationData> rel_models = fields.getRelationData();
Log.d(TAG, "rel_models: "+rel_models.size());
for (OERelationData rel : rel_models) {
// Handling many2many records
if (rel.getDb().getClass()==OEManyToMany.class
/*instanceof OEManyToMany*/) {//TODO type erasure?
Log.v(TAG, "Syncing ManyToMany Records");
OEManyToMany m2mObj = (OEManyToMany) rel.getDb();
OEHelper oe = ((OEDatabase) m2mObj.getDBHelper())
.getOEInstance();
parametersList.add(oe);
parametersList.add(false);
parametersList.add(null);
parametersList.add(rel.getIds());
parametersList.add(false);
parametersList.add(0);
parametersList.add(false);
return parametersList;
} else if (rel.getDb().getClass()==OEManyToOne.class
/*instanceof OEManyToOne*/) {
// Handling many2One records
Log.v(TAG, "Syncing ManyToOne Records");
// M2OCounter++;
OEManyToOne m2oObj = (OEManyToOne) rel.getDb();
OEHelper oe = ((OEDatabase) m2oObj.getDBHelper())
.getOEInstance();
parametersList.add(oe);
parametersList.add(false);
parametersList.add(null);
parametersList.add(rel.getIds());
parametersList.add(false);
parametersList.add(0);
parametersList.add(false);
// parametersMap.put(Counter, parametersList);
// parameterStack.add(parametersList);
return parametersList;
} else if (rel.getDb().getClass()==OEOneToMany.class
/*instanceof OEOneToMany*/) {
Log.v(TAG, "Syncing OneToMany Records");
// O2MCounter++;
OEOneToMany o2mObj = (OEOneToMany) rel.getDb();
OEHelper oe = ((OEDatabase) o2mObj.getDBHelper())
.getOEInstance();
oe.setOne2ManyCol(o2mObj.getColumnName());
parametersList.add(oe);
parametersList.add(false);
parametersList.add(null);
parametersList.add(rel.getIds());
parametersList.add(false);
parametersList.add(0);
parametersList.add(false);
// parametersMap.put(Counter, parametersList);
// parameterStack.add(parametersList);
return parametersList;
} else {
Log.v(TAG, "Syncing records with no relations"
+ rel.getDb().getClass().getSimpleName());
OEHelper oe = ((OEDatabase) rel.getDb()).getOEInstance();
parametersList.add(oe);
parametersList.add(false);
parametersList.add(null);
parametersList.add(rel.getIds());
parametersList.add(false);
parametersList.add(0);
parametersList.add(false);
return parametersList;//TODO when nothing matches this returns
}
}
List<Long> result_ids = mDatabase.createORReplace(
fields.getValues(), removeLocalIfNotExists);
mResultIds.addAll(result_ids);
mRemovedRecordss.addAll(mDatabase.getRemovedRecords());
} catch (Exception e) {
e.printStackTrace();
}
return parametersList;
}
The second method is supposed to return from within one of the conditions but none of the conditions is met

No - it is not a type erasure problem. As Oracle says:
Type erasure ensures that no new classes are created for parameterized
types; consequently, generics incur no runtime overhead.
So at runtime your classes are just plain classes and your objects are what they are, but it is not going to remove the base type information.
Why not put a log statement in and print out the class of the object?

Java sanitizing Arraylist records suggestions

I am looking for an idea how to accomplish this task. So I'll start with how my program is working.
My program reads a CSV file. They are key value pairs separated by a comma.
L1234456,ygja-3bcb-iiiv-pppp-a8yr-c3d2-ct7v-giap-24yj-3gie
L6789101,zgna-3mcb-iiiv-pppp-a8yr-c3d2-ct7v-gggg-zz33-33ie
etc
Function takes a file and parses it into an arrayList of String[]. The function returns the ArrayList.
public ArrayList<String[]> parseFile(File csvFile) {
Scanner scan = null;
try {
scan = new Scanner(csvFile);
} catch (FileNotFoundException e) {
}
ArrayList<String[]> records = new ArrayList<String[]>();
String[] record = new String[2];
while (scan.hasNext()) {
record = scan.nextLine().trim().split(",");
records.add(record);
}
return records;
}
Here is the code, where I am calling parse file and passing in the CSVFile.
ArrayList<String[]> Records = parseFile(csvFile);
I then created another ArrayList for files that aren't parsed.
ArrayList<String> NotParsed = new ArrayList<String>();
So the program then continues to sanitize the key value pairs separated by a comma. So we first start with the first key in the record. E.g L1234456. If the record could not be sanitized it then it replaces the current key with "CouldNOtBeParsed" text.
for (int i = 0; i < Records.size(); i++) {
if(!validateRecord(Records.get(i)[0].toString())) {
Logging.info("Records could not be parsed " + Records.get(i)[0]);
NotParsed.add(srpRecords.get(i)[0].toString());
Records.get(i)[0] = "CouldNotBeParsed";
} else {
Logging.info(Records.get(i)[0] + " has been sanitized");
}
}
Next we do the 2nd key in the key value pair e.g ygja-3bcb-iiiv-pppp-a8yr-c3d2-ct7v-giap-24yj-3gie
for (int i = 0; i < Records.size(); i++) {
if(!validateRecordKey(Records.get(i)[1].toString())) {
Logging.info("Record Key could not be parsed " + Records.get(i)[0]);
NotParsed.add(Records.get(i)[1].toString());
Records.get(i)[1] = "CouldNotBeParsed";
} else {
Logging.info(Records.get(i)[1] + " has been sanitized");
}
}
The problem is that I need both keyvalue pairs to be sanitized, make a separate list of the keyValue pairs that could not be sanitized and a list of the ones there were sanitized so they can be inserted into a database. The ones that cannot will be printed out to the user.
I thought about looping thought the records and removing the records with the "CouldNotBeParsed" text so that would just leave the ones that could be parsed. I also tried removing the records from the during the for loop Records.remove((i)); However that messes up the For loop because if the first record could not be sanitized, then it's removed, the on the next iteration of the loop it's skipped because record 2 is now record 1. That's why i went with adding the text.
Atually I need two lists, one for the Records that were sanitized and another that wasn't.
So I was thinking there must be a better way to do this. Or a better method of sanitizing both keyValue pairs at the same time or something of that nature. Suggestions?

Start by changing the data structure: rather than using a list of two-element String[] arrays, define a class for your key-value pairs:
class KeyValuePair {
private final String key;
private final String value;
public KeyValuePair(String k, String v) { key = k; value = v; }
public String getKey() { return key; }
public String getValue() { return value; }
}
Note that the class is immutable.
Now make an object with three lists of KeyValuePair objects:
class ParseResult {
private final List<KeyValuePair> sanitized = new ArrayList<KeyValuePair>();
private final List<KeyValuePair> badKey = new ArrayList<KeyValuePair>();
private final List<KeyValuePair> badValue = new ArrayList<KeyValuePair>();
public ParseResult(List<KeyValuePair> s, List<KeyValuePair> bk, List<KeyValuePair> bv) {
sanitized = s;
badKey = bk;
badValue = bv;
}
public List<KeyValuePair> getSanitized() { return sanitized; }
public List<KeyValuePair> getBadKey() { return badKey; }
public List<KeyValuePair> getBadValue() { return badValue; }
}
Finally, populate these three lists in a single loop that reads from the file:
public static ParseResult parseFile(File csvFile) {
Scanner scan = null;
try {
scan = new Scanner(csvFile);
} catch (FileNotFoundException e) {
???
// Do something about this exception.
// Consider not catching it here, letting the caller deal with it.
}
final List<KeyValuePair> sanitized = new ArrayList<KeyValuePair>();
final List<KeyValuePair> badKey = new ArrayList<KeyValuePair>();
final List<KeyValuePair> badValue = new ArrayList<KeyValuePair>();
while (scan.hasNext()) {
String[] tokens = scan.nextLine().trim().split(",");
if (tokens.length != 2) {
???
// Do something about this - either throw an exception,
// or log a message and continue.
}
KeyValuePair kvp = new KeyValuePair(tokens[0], tokens[1]);
// Do the validation on the spot
if (!validateRecordKey(kvp.getKey())) {
badKey.add(kvp);
} else if (!validateRecord(kvp.getValue())) {
badValue.add(kvp);
} else {
sanitized.add(kvp);
}
}
return new ParseResult(sanitized, badKey, badValue);
}
Now you have a single function that produces a single result with all your records cleanly separated into three buckets - i.e. sanitized records, records with bad keys, and record with good keys but bad values.

BIRT: How to remove a dataset parameter programmatically

I want to modify an existing *.rptdesign file and save it under a new name.
The existing file contains a Data Set with a template SQL select statement and several DS parameters.
I'd like to use an actual SQL select statement which uses only part of the DS parameters.
However, the following code results in the exception:
Exception in thread "main" `java.lang.RuntimeException`: *The structure is floating, and its handle is invalid!*
at org.eclipse.birt.report.model.api.StructureHandle.getStringProperty(StructureHandle.java:207)
at org.eclipse.birt.report.model.api.DataSetParameterHandle.getName(DataSetParameterHandle.java:143)
at org.eclipse.birt.report.model.api.DataSetHandle$DataSetParametersPropertyHandle.removeParamBindingsFor(DataSetHandle.java:851)
at org.eclipse.birt.report.model.api.DataSetHandle$DataSetParametersPropertyHandle.removeItems(DataSetHandle.java:694)
--
OdaDataSetHandle dsMaster = (OdaDataSetHandle) report.findDataSet("Master");
HashSet<String> bindVarsUsed = new HashSet<String>();
...
// find out which DS parameters are actually used
HashSet<String> bindVarsUsed = new HashSet<String>();
...
ArrayList<OdaDataSetParameterHandle> toRemove = new ArrayList<OdaDataSetParameterHandle>();
for (Iterator iter = dsMaster.parametersIterator(); iter.hasNext(); ) {
OdaDataSetParameterHandle dsPara = (OdaDataSetParameterHandle)iter.next();
String name = dsPara.getName();
if (name.startsWith("param_")) {
String bindVarName = name.substring(6);
if (!bindVarsUsed.contains(bindVarName)) {
toRemove.add(dsPara);
}
}
}
PropertyHandle paramsHandle = dsMaster.getPropertyHandle( OdaDataSetHandle.PARAMETERS_PROP );
paramsHandle.removeItems(toRemove);
What is wrong here?
Has anyone used the DE API to remove parameters from an existing Data Set?

I had similar issue. Resolved it by calling 'removeItem' multiple times and also had to re-evaluate parametersIterator everytime.
protected void updateDataSetParameters(OdaDataSetHandle dataSetHandle) throws SemanticException {
int countMatches = StringUtils.countMatches(dataSetHandle.getQueryText(), "?");
int paramIndex = 0;
do {
paramIndex = 0;
PropertyHandle odaDataSetParameterProp = dataSetHandle.getPropertyHandle(OdaDataSetHandle.PARAMETERS_PROP);
Iterator parametersIterator = dataSetHandle.parametersIterator();
while(parametersIterator.hasNext()) {
Object next = parametersIterator.next();
paramIndex++;
if(paramIndex > countMatches) {
odaDataSetParameterProp.removeItem(next);
break;
}
}
if(paramIndex < countMatches) {
paramIndex++;
OdaDataSetParameter dataSetParameter = createDataSetParameter(paramIndex);
odaDataSetParameterProp.addItem(dataSetParameter);
}
} while(countMatches != paramIndex);
}
private OdaDataSetParameter createDataSetParameter(int paramIndex) {
OdaDataSetParameter dataSetParameter = StructureFactory.createOdaDataSetParameter();
dataSetParameter.setName("param_" + paramIndex);
dataSetParameter.setDataType(DesignChoiceConstants.PARAM_TYPE_INTEGER);
dataSetParameter.setNativeDataType(1);
dataSetParameter.setPosition(paramIndex);
dataSetParameter.setIsInput(true);
dataSetParameter.setIsOutput(false);
dataSetParameter.setExpressionProperty("defaultValue", new Expression("<evaluation script>", ExpressionType.JAVASCRIPT));
return dataSetParameter;
}

Variable g may not have been initialized

I have many questions about this project that I'm working on. It's a virtual database for films. I have a small MovieEntry class (to process individual entries) and a large MovieDatabase class that keeps track of all 10k+ entries. In my second searchYear method as well as subsequent methods I get the error "variable g (or d or whatever) might not have been initialized."
I also get a pop-up error that says Warnings from last compilation: unreachable catch clause. thrown type java.io.FileNotFoundException has already been caught. I'm positively stumped on both. Here's the code:
public class MovieDatabase
{
private ArrayList<MovieEntry> Database = new ArrayList<MovieEntry>();
public MovieDatabase(){
ArrayList<MovieDatabase> Database = new ArrayList<MovieDatabase>(0);
}
public int countTitles() throws IOException{
Scanner fileScan;
fileScan = new Scanner (new File("movies.txt"));
int count = 0;
String movieCount;
while(fileScan.hasNext()){
movieCount = fileScan.nextLine();
count++;
}
return count;
}
public void addMovie(MovieEntry m){
Database.add(m);
}
public ArrayList<MovieEntry> searchTitle(String substring){
for (MovieEntry title : Database)
System.out.println(title);
return null;
}
public ArrayList<MovieEntry> searchGenre(String substring){
for (MovieEntry genre : Database)
System.out.println(genre);
return null;
}
public ArrayList<MovieEntry> searchDirector (String str){
for (MovieEntry director : Database)
System.out.println(director);
return null;
}
public ArrayList<String> searchYear (int yr){
ArrayList <String> yearMatches = new ArrayList<String>();
for (MovieEntry m : Database)
m.getYear(yr);
if(yearMatches.contains(yr) == false){
String sYr = Integer.toString(yr);
yearMatches.add(sYr);
}
return yearMatches;
}
public ArrayList<MovieEntry> searchYear(int from, int to){
ArrayList <String> Matches = new ArrayList<String>();
for(MovieEntry m : Database);
m.getYear();
Matches.add();
return Matches;
}
public void readMovieData(String movies){
String info;
try{
Scanner fileReader = new Scanner(new File("movies"));
Scanner lineReader;
while(fileReader.hasNext()){
info = fileReader.nextLine();
lineReader = new Scanner(info);
lineReader.useDelimiter(":");
String title = lineReader.next();
String director = lineReader.next();
String genre = lineReader.next();
int year = lineReader.nextInt();
}
}catch(FileNotFoundException error){
System.out.println("File not found.");
}catch(IOException error){
System.out.println("Oops! Something went wrong.");
}
}
public int countGenres(){
ArrayList <String> gList = new ArrayList<String>();
for(MovieEntry m : Database){
String g = m.getGenre(g);
if(gList.contains(g) == false){
gList.add(g);
}
return gList.size();
}
}
public int countDirectors(){
ArrayList <String> dList = new ArrayList<String>();
for(MovieEntry m : Database){
String d = m.getDirector(d);
if(dList.contains(d) == false){
dList.add(d);
}
return dList.size();
}
}
public String listGenres(){
ArrayList <String> genreList = new ArrayList<String>();
}
}

catch(IOException error){
System.out.println("Oops! Something went wrong.");
}
Its telling you that the FileNotFoundException will deal with what the IOException is catching, so the IOException becomes unreachable as in it will never catch an IO exceltion, why just not catch an Exception instead
As for the initialization
public int countDirectors(){
ArrayList <String> dList = new ArrayList<String>();
for(MovieEntry m : Database){
String d = m.getDirector(d); //THIS LINE
if(dList.contains(d) == false){
dList.add(d);
}
return dList.size();
}
The line String d = m.getDirector(d); might be the problem, d wont be initialised unless there is something in the MovieEntry and as far as i can see there will never be anything because you are initialising it to an empty array list
ArrayList<MovieDatabase> Database = new ArrayList<MovieDatabase>(0);
Maybe you should be passing a array of movies to the constructor and then add these movies to the Database variable ?

Seems like there are a number of issues with this code.
What parameter does MovieEntry.getGenre() expect? You may not use g in that case because it has not been defined yet.
The exception issue you mentioned means that the exception was already caught, or possibly never thrown. I believe that in this case the IOException is never thrown out from the code within the try block.
There are a number of methods that are supposed to return a value but do not, example:
public String listGenres(){
ArrayList <String> genreList = new ArrayList<String>();
}
Also, it is a java naming convention to use lower case first characters (camel case) for values:
private ArrayList<MovieEntry> database = new ArrayList<MovieEntry>();
Oh, and do you need to re-initialize the database variable in the constructor?:
public MovieDatabase(){
ArrayList<MovieDatabase> Database = new ArrayList<MovieDatabase>(0);
}
Hope this is helpful.