We Keep Coding

Removing Stopwords stored in a Set from an ArrayList of Strings

I've been following various posts on here regarding removing stop words from an ArrayList (More so this one than others). But i've come across some issues when customising this code to suit my needs.
My code reads in two files, a textfile of stop words, and a textfile of data collected from Twitter. I store the stopwords in a HashSet and ultimately want to remove them from the textfile of Twitter data (that's stored in an ArrayList). But the problem i have with my code is that everything works (such as reading the files and appending the output to a file), except for the removal of stopwords.
The files i'm currently using for tests are here
public static void main(String[] args) {
ArrayList<String> listOfWords = new ArrayList<String>();
try {
// Read in sto pwords text file aswell as the textfile to edit
Scanner stopWordsFile = new Scanner(new File("stopwords_twitter.txt"));
Scanner textFile = new Scanner(new File("LiverpoolTest.txt"));
// Create a set for the stop words
Set<String> stopWords = new HashSet<String>();
// For each stopword split them and transform them to lowercase
while (stopWordsFile.hasNext()) {
stopWords.add(stopWordsFile.next().trim());
}
// Creates an empty list for the text files contents
ArrayList<String> words = new ArrayList<String>();
/* For each word in the file correct (removing words between the delimiters)
them and add them to the ArrayList */
while (textFile.hasNextLine()) {
for (String word : textFile.nextLine().trim().toLowerCase()
.replaceAll("/-/-/.*?/-/-/\\s*","").split("/")) {
words.add(word);
}
}
// Iterate over the ArrayList
for(String word : words) {
String wordCompare = word.toLowerCase();
// If the word isn't a stop word, add to listOfWords ArrayList
if (!stopWords.contains(wordCompare)) {
listOfWords.add(word);
}
}
stopWordsFile.close();
textFile.close();
} catch(FileNotFoundException e){
e.printStackTrace();
}
try {
File fileName;
FileWriter fw;
// Create a new textfile for listOfWords
fileName = new File("LiverpoolNoStopWords.txt");
fw = new FileWriter(fileName, true);
// Output listOfWords to a new textfile
for (String str : listOfWords) {
String word = str + "\n";
System.out.print(word);
fw.write(word);
}
fw.close();
} catch(IOException e){
System.err.println("Error. Cannot open file for writing.");
System.exit(1);
}
}

all it took is to debug the program. This is something OP could have done him/herself.
To test the loading of the stop words, I printed the contents of stopWords. It was correct.
To test the parsing of the twitter words, I printed wordCompare right after it is set:
String wordCompare = word.toLowerCase();
System.out.println("|"+wordCompare+"|");
and got this:
|the redmen tv : chris sat down with spanish journalist guillem balague to talk through liverpool’s season as a whole, how real madrid have been playing and how they are likely to play against liverpool tonight!|
||
|watch now:https:|
||
|t.co|
|oqmcx3zs9c|
|subscribe: https:|
||
|t.co|
|tbybrgabge https:|
||
|t.co|
|s6010yicen|
||
|the redmen tv : real madrid v liverpool | https:|
||
|t.co|
|jlwbp8q7bf|
||
|we have hours of build up content including interviews with;|
So obviously, the problem is that the split() doesn't split into words. and indeed, the split was expecting forward slash "/" as delimiter. changed to .split("\\s+") split by whitespace.
added a print in case stop words found
if (!stopWords.contains(wordCompare)) {
listOfWords.add(word);
} else {
System.out.println("####$$");
}
and got this:
|the|
####$$
|redmen|
|tv|
|:|
|chris|
|sat|
####$$
|down|
####$$
|with|
####$$
|spanish|

How to read a file based on string position and store it in variable

I need to read a file, completely and split the strings inside the file and store it in a variable using Java
See below example, my text file contains
devarajan 1000210 08754540275 600019
ramesh 1000210 08754540275 600019
udhay 1000210 08754540275 600019
I tired using string position but it is not working out.
Please find attached sample file as well. Regards
My Code:
public class Program {
public static void main(String[] args) {
String line = "devarajan 1000210 08754540275 600019 ";
String[] words = line.split("\\W+");
for (String word : words) {
System.out.println(word);
}
}
}
Output:
devarajan 1000210 08754540276
My file will contain the list of string 10-10 position will be name 20-30 position will be empid 30-40 will phone number. so while i used the previous snippet i am getting blank spaces "devarajan" " 1000210".. i should avoid that blank spaces.
In turn my code is splitting up as soon as it encounters blank space, instead of position
#Twelve, # Kick : I am getting the output as follows for your snippet
but imagine if i have a space in my name ex: "twelve dollar" instead of "
twelvedollar", then the name will get split and stored in different array position. and that is the reason, i have asked whether it is possible to split the string based on the position

just one way to do it ..
try {
Scanner inFile = new Scanner(new File("myInputFile.txt"));
String[] data;
ArrayList<String[]> arr = new ArrayList<String[]>();
while (inFile.hasNext()) {
data = inFile.nextLine().split("\\s+"); // or split("\t") if using tabs
System.out.println(Arrays.toString(data));
arr.add(data);
}
}
catch (FileNotFoundException fe) {
fe.printStackTrace();
}

Cannot read file using Scanner, never gets into loop

I am trying to read a text file in order to copy some parts of it into a new text file.
This is how I create my Scanner item :
// folder
File vMainFolder = new File(System.getProperty("user.home"),"LightDic");
if (!vMainFolder.exists() & !vMainFolder.mkdirs()) {
System.out.println("Missing LightDic folder.");
return;
}
// file
System.out.println("Enter the source file's name : ");
Scanner vSc = new Scanner(System.in);
String vNomSource = vSc.next();
Scanner vSource;
try {
vSource = new Scanner(new File(vMainFolder, vNomSource+".txt"));
} catch (final java.io.FileNotFoundException pExp) {
System.out.println("Dictionnary not found.");
return;
}
And this is how I wrote my while structure :
while (vSource.hasNextLine()) {
System.out.println("test : entering the loop");
String vMot = vSource.nextLine(); /* edit : I added this statement, which I've forgotten in my previous post */
}
When executing the program, it never prints "test : entering the loop".
Of course, this file I am testing is not empty, it is a list of words like so :
a
à
abaissa
abaissable
abaissables
abaissai
I don't understand what I did wrong, I've used this method a few times in the past.

Problem solved.
I don't really know why, but I solved the problem changing the encoding of my FRdic.txt file from ANSI to UTF-8, and then the file was read.
Thanks to everyone who tried to help me.
Is it normal that a text file encoded in ANSI is not read by the JVM ?

Getting values from a string

I'm making a simple paint program and am stuck with getting a certain part of a string.
Here's the trouble - When I save the 9-panel image, it stores the RBG values of each panel to a .txt file. Example:
java.awt.Color[r=0,g=0,b=0]
java.awt.Color[r=255,g=255,b=255]
java.awt.Color[r=255,g=0,b=0]
java.awt.Color[r=0,g=0,b=255]
java.awt.Color[r=0,g=0,b=0]
java.awt.Color[r=255,g=255,b=0]
java.awt.Color[r=255,g=255,b=0]
java.awt.Color[r=255,g=0,b=0]
java.awt.Color[r=0,g=0,b=255]
From here, I call a scanner to read the lines of our file. I just need to find the best way to extract the values inside the [ ] to a String. I've tried using a tokenizer to no avail, still being stuck with excess Strings. I've tried manipulating characters but again failed. What would be the best way to go about extracting the data from our brackets? AND would it be easier to store the individual r=xxx, b=xxx, g=xxx values to a String[]? Thanks, and here is the source i have so far:
import java.awt.Color;
import java.io.*;
import java.lang.*;
import java.util.*;
//when finished, organize imports (narrow down what imports were used)
public class SaveLoad {
private boolean tryPassed, tryPassed2;
private Formatter x;
//final String[] rawData; will be where the rgb raws are stored
private Scanner xReader;
public void save(Color[] c, String s) {
//s is the filename
int counter = c.length;
//Tries to create a file and, if it does, adds the data to it.
try{
x = new Formatter(s+".txt");
tryPassed = true;
while(counter>0) {
x.format("%s. %s\n", (c.length-(counter-1)), c[counter-1]);
counter--;
}
x.close();
}catch (Exception e){
e.printStackTrace();
tryPassed = false;
}
}
//load will take paramaters of a filename(string); NOTE:::: make the file loaded specify an appendix (ex] .pixmap)
//MAYBE add a load interface with a jDropdownmenu for the filetype? add parameter String filetype.
public void load(String s, String filetype) {
//loads the file and, if successful, attempts to read it.
try{
xReader = new Scanner(new File(s+filetype));
tryPassed2 = true;
}catch(Exception e){
e.printStackTrace();
tryPassed2 = false;
System.out.println(s+filetype+" is not a valid file");
}
while(xReader.hasNext()&&tryPassed2==true) {
String inBrackets = xReader.next().substring(17);
System.out.println(inBrackets);
}
}
}
Also, ignore my messy notations.

The best way is to change the storage format. At least two options:
comma-separate values. Store r,g,b on each line. For example 215,222,213. Then you can have line.split(",") to obtain a String[] of the values
serialize the whole Color array using ObjectOutputStream

I would advise to change format. But if you insists on your one use regex:
String st = "java.awt.Color[r=0,g=0,b=0]";
Pattern p = Pattern.compile("java.awt.Color\\[r=(.*),g=(.*),b=(.*)\\]");
Matcher m = p.matcher(st);
if (m.matches()) {
System.out.println("r=" + m.group(1));
System.out.println("g=" + m.group(2));
System.out.println("b=" + m.group(3));
}

Java: CSV file read & write

I'm reading 2 csv files: store_inventory & new_acquisitions.
I want to be able to compare the store_inventory csv file with new_acquisitions.
1) If the item names match just update the quantity in store_inventory.
2) If new_acquisitions has a new item that does not exist in store_inventory, then add it to the store_inventory.
Here is what i have done so far but its not very good. I added comments where i need to add taks 1 & 2.
Any advice or code to do the above tasks would be great! thanks.
File new_acq = new File("/src/test/new_acquisitions.csv");
Scanner acq_scan = null;
try {
acq_scan = new Scanner(new_acq);
} catch (FileNotFoundException ex) {
Logger.getLogger(mainpage.class.getName()).log(Level.SEVERE, null, ex);
}
String itemName;
int quantity;
Double cost;
Double price;
File store_inv = new File("/src/test/store_inventory.csv");
Scanner invscan = null;
try {
invscan = new Scanner(store_inv);
} catch (FileNotFoundException ex) {
Logger.getLogger(mainpage.class.getName()).log(Level.SEVERE, null, ex);
}
String itemNameInv;
int quantityInv;
Double costInv;
Double priceInv;
while (acq_scan.hasNext()) {
String line = acq_scan.nextLine();
if (line.charAt(0) == '#') {
continue;
}
String[] split = line.split(",");
itemName = split[0];
quantity = Integer.parseInt(split[1]);
cost = Double.parseDouble(split[2]);
price = Double.parseDouble(split[3]);
while(invscan.hasNext()) {
String line2 = invscan.nextLine();
if (line2.charAt(0) == '#') {
continue;
}
String[] split2 = line2.split(",");
itemNameInv = split2[0];
quantityInv = Integer.parseInt(split2[1]);
costInv = Double.parseDouble(split2[2]);
priceInv = Double.parseDouble(split2[3]);
if(itemName == itemNameInv) {
//update quantity
}
}
//add new entry into csv file
}
Thanks again for any help. =]

Suggest you use one of the existing CSV parser such as Commons CSV or Super CSV instead of reinventing the wheel. Should make your life a lot easier.

Your implementation makes the common mistake of breaking the line on commas by using line.split(","). This does not work because the values themselves might have commas in them. If that happens, the value must be quoted, and you need to ignore commas within the quotes. The split method can not do this -- I see this mistake a lot.
Here is the source of an implementation that does it correctly:
http://agiletribe.purplehillsbooks.com/2012/11/23/the-only-class-you-need-for-csv-files/

With help of the open source library uniVocity-parsers, you could develop with pretty clean code as following:
private void processInventory() throws IOException {
/**
* ---------------------------------------------
* Read CSV rows into list of beans you defined
* ---------------------------------------------
*/
// 1st, config the CSV reader with row processor attaching the bean definition
CsvParserSettings settings = new CsvParserSettings();
settings.getFormat().setLineSeparator("\n");
BeanListProcessor<Inventory> rowProcessor = new BeanListProcessor<Inventory>(Inventory.class);
settings.setRowProcessor(rowProcessor);
settings.setHeaderExtractionEnabled(true);
// 2nd, parse all rows from the CSV file into the list of beans you defined
CsvParser parser = new CsvParser(settings);
parser.parse(new FileReader("/src/test/store_inventory.csv"));
List<Inventory> storeInvList = rowProcessor.getBeans();
Iterator<Inventory> storeInvIterator = storeInvList.iterator();
parser.parse(new FileReader("/src/test/new_acquisitions.csv"));
List<Inventory> newAcqList = rowProcessor.getBeans();
Iterator<Inventory> newAcqIterator = newAcqList.iterator();
// 3rd, process the beans with business logic
while (newAcqIterator.hasNext()) {
Inventory newAcq = newAcqIterator.next();
boolean isItemIncluded = false;
while (storeInvIterator.hasNext()) {
Inventory storeInv = storeInvIterator.next();
// 1) If the item names match just update the quantity in store_inventory
if (storeInv.getItemName().equalsIgnoreCase(newAcq.getItemName())) {
storeInv.setQuantity(newAcq.getQuantity());
isItemIncluded = true;
}
}
// 2) If new_acquisitions has a new item that does not exist in store_inventory,
// then add it to the store_inventory.
if (!isItemIncluded) {
storeInvList.add(newAcq);
}
}
}
Just follow this code sample I worked out according to your requirements. Note that the library provided simplified API and significent performance for parsing CSV files.

The operation you are performing will require that for each item in your new acquisitions, you will need to search each item in inventory for a match. This is not only not efficient, but the scanner that you have set up for your inventory file would need to be reset after each item.
I would suggest that you add your new acquisitions and your inventory to collections and then iterate over your new acquisitions and look up the new item in your inventory collection. If the item exists, update the item. If it doesnt, add it to the inventory collection. For this activity, it might be good to write a simple class to contain an inventory item. It could be used for both the new acquisitions and for the inventory. For a fast lookup, I would suggest that you use HashSet or HashMap for your inventory collection.
At the end of the process, dont forget to persist the changes to your inventory file.

As Java doesn’t support parsing of CSV files natively, we have to rely on third party library. Opencsv is one of the best library available for this purpose. It’s open source and is shipped with Apache 2.0 licence which makes it possible for commercial use.
Here, this link should help you and others in the situations!

For writing to CSV
public void writeCSV() {
// Delimiter used in CSV file
private static final String NEW_LINE_SEPARATOR = "\n";
// CSV file header
private static final Object[] FILE_HEADER = { "Empoyee Name","Empoyee Code", "In Time", "Out Time", "Duration", "Is Working Day" };
String fileName = "fileName.csv");
List<Objects> objects = new ArrayList<Objects>();
FileWriter fileWriter = null;
CSVPrinter csvFilePrinter = null;
// Create the CSVFormat object with "\n" as a record delimiter
CSVFormat csvFileFormat = CSVFormat.DEFAULT.withRecordSeparator(NEW_LINE_SEPARATOR);
try {
fileWriter = new FileWriter(fileName);
csvFilePrinter = new CSVPrinter(fileWriter, csvFileFormat);
csvFilePrinter.printRecord(FILE_HEADER);
// Write a new student object list to the CSV file
for (Object object : objects) {
List<String> record = new ArrayList<String>();
record.add(object.getValue1().toString());
record.add(object.getValue2().toString());
record.add(object.getValue3().toString());
csvFilePrinter.printRecord(record);
}
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
fileWriter.flush();
fileWriter.close();
csvFilePrinter.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}

You can use Apache Commons CSV api.
FYI this anwser : https://stackoverflow.com/a/42198895/6549532
Read / Write Example