We Keep Coding

What is the best/fastest way to load these .csv files?

I've been attempting to load 6 .csv files. I've used the CSVReader library, and the BufferedReader one. I noticed that when using the BufferedReader library the files are loaded faster. Nonetheless, this caused me to have an OutOfMemory exception, so I had to set the max memory usage to 1024mb in Eclipse. On the other hand, when I use the CSVReader library I don't get this issue. I'm wondering if there's something wrong in my code, and what could be the best way to load the files considering it to be optimal in terms of speed and memory. Here is my code (I'm using BufferedReader here):
public void loadMovingViolations(int semestre)
{
try
{
String path = ".\\data\\Moving_Violations_Issued_in_";
String mth1Path = "";
String mth2Path = "";
String mth3Path = "";
String mth4Path = "";
String mth5Path = "";
String mth6Path = "";
if (semestre == 1)
{
mth1Path = path + "January_2018.csv";
mth2Path = path + "February_2018.csv";
mth3Path = path + "March_2018.csv";
mth4Path = path + "April_2018.csv";
mth5Path = path + "May_2018.csv";
mth6Path = path + "June_2018.csv";
}
else if (semestre == 2)
{
mth1Path = path + "July_2018.csv";
mth2Path = path + "August_2018.csv";
mth3Path = path + "September_2018.csv";
mth4Path = path + "October_2018.csv";
mth5Path = path + "November_2018.csv";
mth6Path = path + "December_2018.csv";
}
String[] mths = {mth1Path, mth2Path, mth3Path, mth4Path, mth5Path, mth6Path};
String cPath = "";
int numInfracs = 0;
int[] infracs = new int[6];
double xMin = Double.MAX_VALUE, yMin = Double.MAX_VALUE, xMax = 0, yMax = 0;
BufferedReader br = null;
int i = 0;
while (i < mths.length)
{
int tempInfrac = 0;
cPath = mths[i];
br = new BufferedReader(new FileReader(cPath));
String row = br.readLine();
while ( (row = br.readLine()) != null)
{
String[] columns = row.split(",");
String in1 = columns[0];
Integer objId = Integer.parseInt(in1);
String location = columns[2];
String in2 = columns[3];
int adressId = 0;
if ( !(in2.compareTo("") == 0) )
adressId = Integer.parseInt(in2);
String in3 = columns[4];
double streetId = 0;
if ( !(in3.compareTo("") == 0) )
streetId = Double.parseDouble(in3);
String in4 = columns[5];
Double xCord = Double.parseDouble(in4);
String in5 = columns[6];
Double yCord = Double.parseDouble(in5);
String ticketType = columns[7];
String in6 = columns[8];
Integer fineAmt = Integer.parseInt(in6);
String in7 = columns[9];
double totalPaid = Double.parseDouble(in7);
String in8 = columns[10];
Integer penalty1 = Integer.parseInt(in8);
String accident = columns[12];
String date = columns[13];
String vioCode = columns[14];
String vioDesc = columns[15];
VOMovingViolations vomv = new VOMovingViolations(objId, location, adressId, streetId, xCord, yCord, ticketType, fineAmt, totalPaid, penalty1, accident, date, vioCode, vioDesc);
movingViolationsQueue.enqueue(vomv);
tempInfrac++;
if (xCord > xMax)
xMax = xCord;
if (yCord > yMax)
yMax = yCord;
if (xCord < xMin)
xMin = xCord;
if (yCord < yMin)
yMin = yCord;
}
numInfracs += tempInfrac;
infracs[i] = tempInfrac;
i++;
br.close();
}
System.out.println();
int j = 0;
for (int current: infracs)
{
String[] sa = mths[j].substring(35).split("_");
String mth = sa[0];
System.out.println("En el mes " + mth + " se encontraron " +
current + " infracciones");
j++;
}
System.out.println();
System.out.println("Se encontraron " + numInfracs + " infracciones en el semestre.");
System.out.println();
System.out.println("Minimax: " + "("+xMin+", "+yMin+"), " + "("+xMax+", "+yMax+")");
System.out.println();
}
catch (Exception e)
{
e.printStackTrace();
System.out.println();
System.out.println("No se pudieron cargar los datos");
System.out.println();
}
}

Regarding the "better" way, as usual it depends.
You are reinventing the wheel. And it is surprisingly hard to write a fully functional csv parser that works with arbitrary input data. Your parser does a simple split on ",", this means it will fail as soon as one of your columns contains a string with a comma inside! You might also run into trouble when the separation character is changed.
Your code is faster because it omits a ton of things a csv parser can do. Therefore your code works with your table, but if somebody else gives you a valid csv file, your parser will throw exceptions at you. A real csv parser would accept any well formed input!
Thus: if the sole purpose of your code is read files with that given structure, sure, you can use your faster solution. But if you expect that your input data format will change over time, then every update will make you change your code. And worse, such updates might make your code more complicated over time. Therefore you have to carefully balance development efficiency against runtime performance.

Thank you for your answers. I've tried using another library and now it only takes about 1.2 seconds to load the files (I'm loading 600k objects approx.). But i'm still geting OutOfMemory: java heap exception when I don't put Xms512m and Xml1024m in the Eclipse commands. Is there any way to make my loading method use less memory?
public void loadMovingViolations(int semestre)
{
CsvParserSettings settings = new CsvParserSettings();
settings.getFormat().setLineSeparator("\n");
CsvParser parser = new CsvParser(settings);
String path = ".\\data\\Moving_Violations_Issued_in_";
String mth1Path = "";
String mth2Path = "";
String mth3Path = "";
String mth4Path = "";
String mth5Path = "";
String mth6Path = "";
if (semestre == 1)
{
mth1Path = path + "January_2018.csv";
mth2Path = path + "February_2018.csv";
mth3Path = path + "March_2018.csv";
mth4Path = path + "April_2018.csv";
mth5Path = path + "May_2018.csv";
mth6Path = path + "June_2018.csv";
}
else if (semestre == 2)
{
mth1Path = path + "July_2018.csv";
mth2Path = path + "August_2018.csv";
mth3Path = path + "September_2018.csv";
mth4Path = path + "October_2018.csv";
mth5Path = path + "November_2018.csv";
mth6Path = path + "December_2018.csv";
}
String[] mths = {mth1Path, mth2Path, mth3Path, mth4Path, mth5Path, mth6Path};
String cPath = "";
int numInfracs = 0;
int[] infracs = new int[6];
double xMin = Double.MAX_VALUE, yMin = Double.MAX_VALUE, xMax = 0, yMax = 0;
try
{
int i = 0;
while (i < mths.length)
{
int tempInfrac = 0;
cPath = mths[i];
parser.beginParsing(new FileReader(cPath));
parser.parseNext();
String[] row = null;
while((row = parser.parseNext()) != null)
{
String in1 = row[0].toString();
Integer objId = Integer.parseInt(in1);
String location = row[2].toString();
int addressId = 0;
if (row[3] != null)
{
String in2 = row[3].toString();
addressId = Integer.parseInt(in2);
}
double streetId = 0;
if (row[4] != null)
{
String in3 = row[4].toString();
streetId = Double.parseDouble(in3);
}
String in4 = row[5].toString();
Double xCord = Double.parseDouble(in4);
String in5 = row[6].toString();
Double yCord = Double.parseDouble(in5);
String ticketType = row[7].toString();
String in6 = row[8].toString();
Integer fineAmt = Integer.parseInt(in6);
String in7 = row[9].toString();
double totalPaid = Double.parseDouble(in7);
String in8 = row[10].toString();
Integer penalty1 = Integer.parseInt(in8);
String accident = row[12].toString();
String date = "";
if (row[13] != null)
date = row[13].toString();
String vioCode = row[14].toString();
String vioDesc = "";
if (row[15] != null)
vioDesc = row[15].toString();
VOMovingViolations vomv = new VOMovingViolations(objId, location, addressId, streetId, xCord, yCord, ticketType, fineAmt, totalPaid, penalty1, accident, date, vioCode, vioDesc);
queue.enqueue(vomv);
tempInfrac++;
if (xCord > xMax)
xMax = xCord;
if (yCord > yMax)
yMax = yCord;
if (xCord < xMin)
xMin = xCord;
if (yCord < yMin)
yMin = yCord;
}
numInfracs += tempInfrac;
infracs[i] = tempInfrac;
parser.stopParsing();
i++;
}
System.out.println();
int j = 0;
for (int current: infracs)
{
String[] sa = mths[j].substring(35).split("_");
String mth = sa[0];
System.out.println("En el mes " + mth + " se encontraron " +
current + " infracciones");
j++;
}
System.out.println();
System.out.println("Se encontraron " + numInfracs + " infracciones en el semestre.");
System.out.println();
System.out.println("Minimax: " + "("+xMin+", "+yMin+"), " + "("+xMax+", "+yMax+")");
System.out.println();
}
catch (FileNotFoundException e)
{
e.printStackTrace();
System.out.println();
System.out.println("No se encontró el archivo");
System.out.println();
}
}

Display the Shortest Path using BFS in java

This is my BFS algorithm code.
I can calculate the shortest path distance, but somehow I am not able to display the shortest path.
Like for example, I calculated the shortest path from source to destination is 2. But i would like to also display the pathway. (PlaceA -> PlaceB -> PlaceC) for example.
May i know how do i display out the shortest path out using Java?
Please do help me! Thank you!
public static void main(String[] args) {
// TODO Auto-generated method stub
chooseNumOfVertices();
chooseNumOfEdges();
generateGraph();
in.close();
}
private static void generateGraph() {
int source = 0;
int destination = 0;
int edge = chooseEdge;
// TODO Auto-generated method stub
ArrayList<LinkedList<Integer>> graph = new ArrayList<LinkedList<Integer>>();
for (int i = 0; i < limit; i++) {
LinkedList<Integer> vertex = new LinkedList<Integer>();
vertex.add(i);
graph.add(vertex);
}
while (chooseEdge > 0) {
// Randomize the value
int value = new Random().nextInt(cityMapping.size());
int value2 = new Random().nextInt(cityMapping.size());
//
if (value != value2 && !graph.get(value).contains(value2) && !graph.get(value2).contains(value)) {
graph.get(value).add(value2);
graph.get(value2).add(value);
chooseEdge--;
}
}
// Printing out the Nodes
for (int i = 0; i < graph.size(); i++) {
// Return each LinkedList nodes
// System.out.println(graph.get(i));
for (int j = 0; j < graph.get(i).size(); j++) {
// Return each individual nodes inside LinkedList
for (Entry<Integer, String> entry : cityMapping.entrySet()) {
if (entry.getKey() == graph.get(i).get(j)) {
//System.out.print(graph.get(i).get(j) + "-> ");
System.out.print(graph.get(i).get(j) + ". " + entry.getValue() + " -> ");
}
}
}
System.out.println();
}
do {
for (
int i = 0; i < limit; i++) {
int[] newArray = new int[limit];
distance.add(newArray);
predecessor.add(newArray);
}
long time = System.nanoTime();
System.out.println("Searching BFS");
System.out.println("--------------------------------------------");
for (int i = 0; i < limit; i++) {
BFS(graph, i);
}
long CPUTime = (System.nanoTime() - time);
System.out.println("CPU Time for BFS for " + limit + "vertices and " + edge + "edges (in ns): " + CPUTime);
System.out.print("Enter -1 to exit! Enter source vertex (between 0 to " + (limit - 1) + ") : ");
source = in.nextInt();
if (source == -1) {
System.out.print("System terminating...");
break;
}
System.out.print("Enter destination vertex (between 0 to " + (limit - 1) + ") : ");
destination = in.nextInt();
System.out.println("Distance from " + source + " to " + destination + " is: " +
getDistance(source, destination));
System.out.println("The Predecessor of the path from " + source + " to " + destination + " is: "
+ getPredecessor(source, destination));
} while (source != -1);
}
private static void BFS(ArrayList<LinkedList<Integer>> graph, int i) {
// TODO Auto-generated method stub
boolean[] mark = new boolean[graph.size()];
Queue<Integer> L = new ArrayBlockingQueue<Integer>(graph.size()); //Queue
L.add(i);
mark[i] = true;
Arrays.fill(predecessor.get(i), -1);
Arrays.fill(distance.get(i), -1);
distance.get(i)[i] = 0;
while (!L.isEmpty()) {
int vertex = L.remove();
for (int i1 = 0; i1 < graph.get(vertex).size(); i1++) {
int v = graph.get(vertex).get(i1);
if (!mark[v]) {
mark[v] = true;
predecessor.get(i)[v] = vertex;
L.add(v);
distance.get(i)[v] = distance.get(i)[predecessor.get(i)[v]] + 1;
}
}
}
}
public static int getDistance(int start, int end) {
return (distance.get(start)[end]);
}
public static int getPredecessor(int start, int end) {
return (predecessor.get(start)[end]);
}
private static void chooseNumOfEdges() {
System.out.println("Please input the number of Edges:");
chooseEdge = in.nextInt();
}
// Number of Vertices
private static void chooseNumOfVertices() {
in = new Scanner(System.in);
System.out.println("Please input the number of Vertices:");
limit = in.nextInt();
// Read CSV
List<String[]> content = readCsvFile();
// Map each number to a city name
cityMapping = new HashMap<>();
for (int i = 0; i < limit; i++) {
cityMapping.put(i, content.get(i)[0]);
}
// System.out.println(cityMapping);
}
// Read CSV file
public static List<String[]> readCsvFile() {
String csvFile = "./Lab 4/country.csv";
BufferedReader br = null;
ArrayList<String> names = new ArrayList<String>();
List<String[]> content = new ArrayList<>();
String cvsSplitBy = ",";
try {
String line = "";
br = new BufferedReader(new FileReader(csvFile));
while ((line = br.readLine()) != null) {
content.add(line.split(cvsSplitBy));
}
} catch (Exception e) {
e.printStackTrace();
}
Random r = new Random();
Collections.shuffle(content);
return content;
}
}

Storing numbers with nested Lists java

I have a .txt file with the next format:
I need to storage that values in this way, at index 0 i need to have
[[154.5,0.0],[13.3333333333333, 102.0],[6,272.0],[10.3333333333333,1860.0]...
(there will be more pairs cause the second column in txt is not ordered, and there will be zeros again later)]. I hope that u understand my question, i think i need arraylist of arraylist of arraylist, i just need an idea, thanks.
This is the code:
String working_directory = System.getProperty("user.dir");
File file = new File(working_directory + "\\src\\" + filename);
BufferedReader br = new BufferedReader(new FileReader(file));
ArrayList<ArrayList<Double>> array_A = new ArrayList<ArrayList<Double>>();
ArrayList<Double> aux = new ArrayList<Double>();
aux.add((double)0);
aux.add((double)0);
for ( int i = 0 ; i < matrixSize; i ++ ) {
array_A.add(aux);
}
String[] elements_in_line = new String[3];
String line;
while ((line = br.readLine()) != null) {
ArrayList<Double> value_col_index = new ArrayList<Double>();
ArrayList<Double> pair = new ArrayList<Double>();
elements_in_line = line.split(", ");
double value = Double.parseDouble(elements_in_line[0]);
int line_index = Integer.parseInt(elements_in_line[1]);
int column_index = Integer.parseInt(elements_in_line[2]);
boolean already_exists = false;
if (array_A.get(line_index).size() > 0) {
pair = array_A.get(line_index);
for (int i = 0; i < pair.size(); i++) {
if (column_index == pair.get(1)) {
already_exists = true;
break;
}
}
}
if (already_exists) {
pair.set(0, pair.get(0) + value);
} else {
if (limit_line != 0) {
if (array_A.get(line_index).size() == limit_line) {
System.out.println("Dimension of matrix exceeded!");
System.exit(1);
}
}
array_A.set(line_index,new ArrayList<Double>(Arrays.asList(value,(double)column_index)));
}
}
return array_A;
}

running time error when trying to implent a keyword frequency counter in my parser java

I want to implement my input reading method into my main class, I want use my code to parse. It's been fixed now. thanks.
String x;
int count = -1;=
while (str.hasMoreTokens()) {
count++;
x = str.nextToken();
word[count] = x;
System.out.println(count + ": " + word[count]);
}
System.out.println("---Frequency---");
// create unique words
for (int i = 0; i < 7; i++) {
if ((!Arrays.asList(unique).contains(word[i]))) {
unique[i] = word[i];
}
}
// measuring frequency
int[] measure = new int[10];
for (int a = 0; a < 7; a++) {
if (Arrays.asList(unique).contains(word[a])) {
measure[a] += 1;
System.out.println(unique[a] + " : " + measure[a]);
}
}
}
}
private List<String[]> termsDocsArray = new ArrayList<String[]>();
private List<String> allTerms = new ArrayList<String>(); //to hold all terms
private List<double[]> tfidfDocsVector = new ArrayList<double[]>();
/**

To start with your code
String text = "Professor, engineering, data, mining, research";
StringTokenizer str = new StringTokenizer(text);
String word[] = new String[10];
String unique[] = new String[10];
String x;
int count = -1;
while (str.hasMoreTokens()) {
count++;
x = str.nextToken();
word[count] = x;
System.out.println(count + ": " + word[count]);
}
System.out.println("---Frequency---");
// create unique words
for (int i = 0; i < 7; i++) {
if ((!Arrays.asList(unique).contains(word[i]))) {
unique[i] = word[i];
}
}
// measuring frequency
int[] measure = new int[10];
for (int a = 0; a < 7; a++) {
if (Arrays.asList(unique).contains(word[a])) {
measure[a] += 1;
System.out.println(unique[a] + " : " + measure[a]);
}
}
should be in it's own method like .
private void doSomething(){
//This variable will hold all terms of each document in an array.
String text = "Professor, engineering, data, mining, research";
StringTokenizer str = new StringTokenizer(text);
String word[] = new String[10];
String unique[] = new String[10];
String x;
int count = -1;
while (str.hasMoreTokens()) {
count++;
x = str.nextToken();
word[count] = x;
System.out.println(count + ": " + word[count]);
}
System.out.println("---Frequency---");
// create unique words
for (int i = 0; i < 7; i++) {
if ((!Arrays.asList(unique).contains(word[i]))) {
unique[i] = word[i];
}
}
// measuring frequency
int[] measure = new int[10];
for (int a = 0; a < 7; a++) {
if (Arrays.asList(unique).contains(word[a])) {
measure[a] += 1;
System.out.println(unique[a] + " : " + measure[a]);
}
}
}
Secondly in ur given code u have written like
int count = -1;=
which accounts to this error Syntax error on token "=", { expected.It should be
int count = -1;
And since all your code is simply written in class without any method so it is giving you the error saying { expected.
Please make sure you have copied the code correctly.

Searching in arraylist<String[]> from user input

I want to search in an arraylist from a user input but my if condition doesn't seem to work. Using boolean and .contains() doesn't work for my programme either. This is the coding:
String phone;
phone=this.text1.getText();
System.out.println("this is the phone: " + phone);
BufferedReader line = new BufferedReader(new FileReader(new File("C:\\Users\\Laura Sutardja\\Documents\\IB DP\\Computer Science HL\\cs\\data.txt")));
String indata;
ArrayList<String[]> dataArr = new ArrayList<String[]>();
while ((indata = line.readLine()) != null) {
String[] club = new String[2];
String[] value = indata.split(",", 2);
//for (int i = 0; i < 2; i++) {
int n = Math.min(value.length, club.length);
for (int i = 0; i < n; i++) {
club[i] = value[i];
}
boolean aa = dataArr.contains(this.text1.getText());
if(aa==true)
text2.setText("The data is found.");
else
text2.setText("The data is not found.");
dataArr.add(club);
}
for (int i = 0; i < dataArr.size(); i++) {
for (int x = 0; x < dataArr.get(i).length; x++) {
System.out.printf("dataArr[%d][%d]: ", i, x);
System.out.println(dataArr.get(i)[x]);
}
}
}
catch ( IOException iox )
{
System.out.println("Error");
}

Your dataArr is a list of String[], and you are searching for a String. The two are different kind of objects.
I don't really know how the content of the club array looks like, but you should either change dataArr in order to hold plain String, or to write a method which looks iteratively in dataArr for a String[] containing the output of this.text1.getText().

There is a lot wrong with the program. I assume you want to read a textfile and store each line in the arraylist. To do this you have to split each line of the textfile and store that array in the arrayList.
String[] value;
while ((indata = line.readLine()) != null) {
value = indata.split(",");
dataArr.add(value);
}
Now you have the contents of the file in the arrayList.
Next you want to compare the userinput with each element of the arraylist.
int j = 0;
for (int i = 0; i < dataArr.size(); i++) {
String[] phoneData = dataArr.get(i);
if (phoneData[1].equals(phone)) { // i am assuming here that the phone number is the 2nd element of the String[] array, since i dont know how the textfile looks.
System.out.println("Found number.");
club[j++] = phoneData[1];
} else if (i == dataArr.size()-1) {
System.out.println("Didn't find number.");
}
}
Edit:
As requested:
String phone;
phone = "38495";
System.out.println("this is the phone: " + phone);
BufferedReader line = new BufferedReader(new FileReader(new File("list.txt")));
String indata;
ArrayList<String[]> dataArr = new ArrayList<>();
String[] club = new String[2];
String[] value;// = indata.split(",", 2);
while ((indata = line.readLine()) != null) {
value = indata.split(",");
dataArr.add(value);
}
int j = 0;
for (int i = 0; i < dataArr.size(); i++) {
String[] phoneData = dataArr.get(i);
if (phoneData[1].equals(phone)) {
System.out.println("Found number.");
club[j++] = phoneData[1];
break;
} else if (i == dataArr.size()-1) {
System.out.println("Didn't find number.");
}
}
I hope this makes sense now.