We Keep Coding

How to speed up reading in input in Java

I am attempting to read in info from files to implement Dijkstra's algorithm. I believe that the double for loop is causing this to drastically slow down, is there anyway around this?
Edge[] edge = new Edge[127807];
int indexEdge = 0;
String line2 = "";
BufferedReader fileReader2 = new BufferedReader(new FileReader("Road.txt"));
String valueString = null;
String vertex1IDName = null;
String vertex2IDName = null;
String extra = null;
float value = 0;
int vertex1ID = 0;
int vertex2ID = 0;
//Read the file line by line
while ((line2 = fileReader2.readLine()) != null)
{
//Get all tokens available in line
String[] tokens2 = line2.split(DELIMITER);
for(String token1 : tokens2)
{
vertex1IDName = tokens2[0];
vertex2IDName = tokens2[1];
valueString = tokens2[2];
if(tokens2.length - 1 == 3) {
extra = tokens2[tokens2.length - 1];
}
else {
extra = "";
}
vertex1ID = Integer.parseInt(vertex1IDName);
vertex2ID = Integer.parseInt(vertex2IDName);
value = Float.parseFloat(valueString);
}
System.out.println("value: "+ value + " vertex1ID:"+ vertex1ID +" vertex2ID:"+ vertex2ID+ " extra:" + extra);
//if vertex 1 name or vertex 2 name in vertex.getID()
for(int i = 0; i< indexVertex; i++) {
for(int j = 0; j< indexVertex; j++) {
if(vertex1ID == vertex[i].getID() && vertex2ID == vertex[j].getID()) {
vertex[i].addNeighbour(edge[indexEdge] = new Edge(value,vertex[i],vertex[j],extra));
indexEdge++;
}
}
}
}

Speed up reading CSV in Java

I have a relatively inefficent CSVReader code, see below. It takes more than 30 seconds to read 30000+ lines. How to speed up this reading process as fast as possible?
public class DataReader {
private String csvFile;
private List<String> sub = new ArrayList<String>();
private List<List> master = new ArrayList<List>();
public void ReadFromCSV(String csvFile) {
String line = "";
String cvsSplitBy = ",";
try (BufferedReader br = new BufferedReader(new FileReader(csvFile))) {
System.out.println("Header " + br.readLine());
while ((line = br.readLine()) != null) {
// use comma as separator
String[] list = line.split(cvsSplitBy);
// System.out.println("the size is " + country[1]);
for (int i = 0; i < list.length; i++) {
sub.add(list[i]);
}
List<String> temp = (List<String>) ((ArrayList<String>) sub).clone();
// master.add(new ArrayList<String>(sub));
master.add(temp);
sub.removeAll(sub);
}
} catch (IOException e) {
e.printStackTrace();
}
System.out.println(master);
}
public List<List> getMaster() {
return master;
}
}
UPDATE: I have found that my code actually can finish the reading work in less than 1 second if run it separately. As this DataReader is a part used by my simulation model to initialize the relevant properties. And the following part is associated with the use of the data imported, WHICH TAKES 40 SECONDS TO FINISH! Anyone could help by looking at the generic part of the codes?
// add route network
Network<Object> net = (Network<Object>)context.getProjection("IntraCity Network");
IndexedIterable<Object> local_hubs = context.getObjects(LocalHub.class);
for (int i = 0; i <= CSV_reader_route.getMaster().size() - 1; i++) {
String source = (String) CSV_reader_route.getMaster().get(i).get(0);
String target = (String) CSV_reader_route.getMaster().get(i).get(3);
double dist = Double.parseDouble((String) CSV_reader_route.getMaster().get(i).get(6));
double time = Double.parseDouble((String) CSV_reader_route.getMaster().get(i).get(7));
Object source_hub = null;
Object target_hub = null;
Query<Object> source_query = new PropertyEquals<Object>(context, "hub_code", source);
for (Object o : source_query.query()) {
if (o instanceof LocalHub) {
source_hub = (LocalHub) o;
}
if (o instanceof GatewayHub) {
source_hub = (GatewayHub) o;
}
}
Query<Object> target_query = new PropertyEquals<Object>(context, "hub_code", target);
for (Object o : target_query.query()) {
if (o instanceof LocalHub) {
target_hub = (LocalHub) o;
}
if (o instanceof GatewayHub) {
target_hub = (GatewayHub) o;
}
}
// System.out.println(target_hub.getClass() + " " + time);
// Route this_route = (Route) net.addEdge(source_hub, target_hub);
// context.add(this_route);
// System.out.println(net.getEdge(source_hub, target_hub));
if (net.getEdge(source, target) == null) {
Route this_route = (Route) net.addEdge(source, target);
context.add(this_route);
// this_route.setDist(dist);
// this_route.setTime(time); }
}
}

In your code you are doing many write operation to just add the list of values from current row in your master list which is not required. You can replace the existing code with simple one as given below.
Existing code:
String[] list = line.split(cvsSplitBy);
// System.out.println("the size is " + country[1]);
for (int i = 0; i < list.length; i++) {
sub.add(list[i]);
}
List<String> temp = (List<String>) ((ArrayList<String>) sub).clone();
// master.add(new ArrayList<String>(sub));
master.add(temp);
sub.removeAll(sub);
Suggested code:
master.add(Arrays.asList(line.split(cvsSplitBy)));

I don't have a CSV that big, but you could try the following:
public static void main(String[] args) throws IOException {
Path csvPath = Paths.get("path/to/file.csv");
List<List<String>> master = Files.lines(csvPath)
.skip(1)
.map(line -> Arrays.asList(line.split(",")))
.collect(Collectors.toList());
}
EDIT: I tried it with a CSV sample with 50k entries and the code runs in less than one second.

With extends to the answer of #Alex R, you can process it in parallel as well like this:
public static void main(String[] args) throws IOException {
Path csvPath = Paths.get("path/to/file.csv");
List<List<String>> master = Files.lines(csvPath)
.skip(1).parallel()
.map(line -> Arrays.asList(line.split(",")))
.collect(Collectors.toList());
}

Storing numbers with nested Lists java

I have a .txt file with the next format:
I need to storage that values in this way, at index 0 i need to have
[[154.5,0.0],[13.3333333333333, 102.0],[6,272.0],[10.3333333333333,1860.0]...
(there will be more pairs cause the second column in txt is not ordered, and there will be zeros again later)]. I hope that u understand my question, i think i need arraylist of arraylist of arraylist, i just need an idea, thanks.
This is the code:
String working_directory = System.getProperty("user.dir");
File file = new File(working_directory + "\\src\\" + filename);
BufferedReader br = new BufferedReader(new FileReader(file));
ArrayList<ArrayList<Double>> array_A = new ArrayList<ArrayList<Double>>();
ArrayList<Double> aux = new ArrayList<Double>();
aux.add((double)0);
aux.add((double)0);
for ( int i = 0 ; i < matrixSize; i ++ ) {
array_A.add(aux);
}
String[] elements_in_line = new String[3];
String line;
while ((line = br.readLine()) != null) {
ArrayList<Double> value_col_index = new ArrayList<Double>();
ArrayList<Double> pair = new ArrayList<Double>();
elements_in_line = line.split(", ");
double value = Double.parseDouble(elements_in_line[0]);
int line_index = Integer.parseInt(elements_in_line[1]);
int column_index = Integer.parseInt(elements_in_line[2]);
boolean already_exists = false;
if (array_A.get(line_index).size() > 0) {
pair = array_A.get(line_index);
for (int i = 0; i < pair.size(); i++) {
if (column_index == pair.get(1)) {
already_exists = true;
break;
}
}
}
if (already_exists) {
pair.set(0, pair.get(0) + value);
} else {
if (limit_line != 0) {
if (array_A.get(line_index).size() == limit_line) {
System.out.println("Dimension of matrix exceeded!");
System.exit(1);
}
}
array_A.set(line_index,new ArrayList<Double>(Arrays.asList(value,(double)column_index)));
}
}
return array_A;
}

inefficient looping in java

This is my csv data:
Name,Code,Price,Colour,Type,Stock
A,1001,35000,Red,Car Paint,54
B,1002,56000,Blue,House Paint,90
As you can see, my coding is inefficient.
This is because all the textfields in netbeans do not allow same variable names, I have to give different variable names to each text field (Example: code1, code2, code3, name1, name2,name3)
Can someone help me on how to loop this data so they do it four times and i dont have to repeat the coding? and to skip the process if the fields are blank.
The following is my coding:
try
{
for(int z=0; z<4;z++)
{
String code1;
code1=this.text1.getText();
System.out.println("this is the code: " + code1);
String qty;
int qty1;
qty=this.quantity1.getText();
qty1=Integer.parseInt(qty);
System.out.println("quantity: "+qty1);
String code2;
code2=this.text2.getText();
System.out.println("this is the code: " + code2);
int qty2;
qty=this.quantity2.getText();
qty2=Integer.parseInt(qty);
System.out.println("quantity: "+qty2);
String code3;
code3=this.text3.getText();
System.out.println("this is the code: " + code3);
int qty3;
qty=this.quantity2.getText();
qty3=Integer.parseInt(qty);
System.out.println("quantity: "+qty3);
String code4;
code4=this.text4.getText();
System.out.println("this is the code: " + code4);
int qty4;
qty=this.quantity2.getText();
qty4=Integer.parseInt(qty);
System.out.println("quantity: "+qty4);
int sum=0;
BufferedReader line = new BufferedReader(new FileReader(new File("C:\\Users\\Laura Sutardja\\Documents\\IB DP\\Computer Science HL\\cs\\product.txt")));
String indata;
ArrayList<String[]> dataArr = new ArrayList<>();
String[] club = new String[6];
String[] value;
while ((indata = line.readLine()) != null) {
value = indata.split(",");
dataArr.add(value);
}
for (int i = 0; i < dataArr.size(); i++) {
String[] nameData = dataArr.get(i);
if (nameData[1].equals(code1)) {
System.out.println("Found name.");
name1.setText(""+ nameData[0]);
int price;
price=Integer.parseInt(nameData[2]);
int totalprice=qty1*price;
String total=Integer.toString(totalprice);
price1.setText(total);
sum=sum+totalprice;
break;
}
}
for (int i = 0; i < dataArr.size(); i++) {
String[] nameData = dataArr.get(i);
if (nameData[1].equals(code2)) {
System.out.println("Found name.");
name2.setText(""+ nameData[0]);
int price;
price=Integer.parseInt(nameData[2]);
int totalprice=qty2*price;
String total=Integer.toString(totalprice);
price2.setText(total);
sum=sum+totalprice;
break;
}
}
for (int i = 0; i < dataArr.size(); i++) {
String[] nameData = dataArr.get(i);
if (nameData[1].equals(code3)) {
System.out.println("Found name.");
name3.setText(""+ nameData[0]);
int price;
price=Integer.parseInt(nameData[2]);
int totalprice=qty3*price;
int totalprice3=totalprice;
String total=Integer.toString(totalprice);
price3.setText(total);
sum=sum+totalprice;
break;
}
}
for (int i = 0; i < dataArr.size(); i++) {
String[] nameData = dataArr.get(i);
if (nameData[1].equals(code4)) {
System.out.println("Found name.");
name4.setText(""+ nameData[0]);
int price;
price=Integer.parseInt(nameData[2]);
int totalprice=qty4*price;
int totalprice4=totalprice;
String total=Integer.toString(totalprice);
price4.setText(total);
sum=sum+totalprice;
break;
}
}
total1.setText("Rp. "+sum);
}
}
catch ( IOException iox )
{
System.out.println("Error");
}

Why don't you use a library like http://commons.apache.org/proper/commons-csv/

Solving this problem is actually rather straight forward if you break it down into separate parts.
First you need to solve the problem of loading the data into an internal data representation that is easy to use. Just loading the file into Java is rather simple and you have already done this:
BufferedReader csvFile = new BufferedReader(new FileReader(new File(path)));
String line = "start";
int count = 0;
while((line = csvFile.readLine()) != null){
System.out.println(line);
}
csvFile.close();
The next problem is splitting the line and store it in a meaningful way - for each line.
HashMap<Integer, String> record = new HashMap<Integer, String>();
String[] raw = line.split(",");
for(int i=0;i<raw.length; i++){
record.put(i, raw[i]);
}
Now you state you only want to store records that have non-empty fields so we need to check for that:
HashMap<Integer, String> record = new HashMap<Integer, String>();
String[] raw = line.split(",");
Boolean store = true;
for(int i=0;i<raw.length; i++){
if(raw[i].equals("") || raw[i].equals(null)){
store = false;
break;
}
record.put(i, raw[i]);
}
if(store)
csvData.add(record);
Now, you can load each record of the csv file as a dictionary that you can easily use. All that remains is to save a list of these dictionaries.
ArrayList<Map<Integer, String>> csvData = new ArrayList<Map<Integer, String>>();
BufferedReader csvFile = new BufferedReader(new FileReader(new File(path)));
String line = "start";
int count = 0;
while((line = csvFile.readLine()) != null){
if(count == 0){//skip first line
count++;
continue;
}
HashMap<Integer, String> record = new HashMap<Integer, String>();
String[] raw = line.split(",");
Boolean store = true;
for(int i=0;i<raw.length; i++){
if(raw[i].equals("") || raw[i].equals(null))
{
store = false;
break;
}
record.put(i, raw[i]);
}
if(store)
csvData.add(record);
}
csvFile.close();
Full code snippet that loads in data and easily access whatever information you want:
public class Main {
public static final int NAME = 0;
public static final int CODE = 1;
public static final int PRICE = 2;
public static final int COLOR = 3;
public static final int TYPE = 4;
public static final int STOCK = 5;
public static void main(String[] args) throws IOException{
ArrayList<Map<Integer, String>> csvData = loadCSVFile("C:\\path\\to\\file\\products.txt");
//Print some of the data
System.out.println("---------------------------");
for(Map<Integer, String> record : csvData){
printInfo(record);
}
}
public static ArrayList<Map<Integer, String>> loadCSVFile(String path) throws IOException{
ArrayList<Map<Integer, String>> csvData = new ArrayList<Map<Integer, String>>();
BufferedReader csvFile = new BufferedReader(new FileReader(new File(path)));
String line = "start";
int count = 0;
while((line = csvFile.readLine()) != null){
if(count == 0){
count++;
continue;
}
HashMap<Integer, String> record = new HashMap<Integer, String>();
String[] raw = line.split(",");
Boolean store = true;
for(int i=0;i<raw.length; i++){
if(raw[i].equals("") || raw[i].equals(null))
{
store = false;
break;
}
record.put(i, raw[i]);
}
if(store)
csvData.add(record);
}
csvFile.close();
return csvData;
}
public static void printInfo(Map<Integer, String> record){
System.out.println(record.get(CODE) + " : " + record.get(TYPE));
System.out.println(record.get(NAME) + " : " + record.get(STOCK) + " : " + record.get(PRICE));
System.out.println("---------------------------");
}
}

Getter within loop returning null after assigning values

I've got a slight problem with my program. Simply put, what I do is this:
Choose the option to "Add a new Student record".
When I try to display the newly added values (First Name, Last Name, Course), it all comes back as null.
I can't wrap my head around what's wrong, but I'll try to post the needed codes in order for all of you to get a better view.
Main class:
public static void main(String[] args) throws Exception{
String cont1=" ";
String X;
int x1,arr_count=1;
String[] option = {"First Name","Last Name","Course"};
BufferedReader reader1=new BufferedReader(new InputStreamReader(System.in));
do{
System.out.println("\nWelcome! Please select a task: \nA. View a record.\nB. Add a record.\nC. Edit a record.\nD. Exit the program.");
String option1 = reader1.readLine();
if("a".equals(option1)|| "A".equals(option1)){
System.out.println("Please enter the Student ID for the record to view.");
String a = reader1.readLine();
x1 = Integer.parseInt(a);
for(int y=0;y<3;y++){
System.out.print(Records.getRecord(x1,y) + " ");
}
System.out.print("\nSubjects and Grades: \n");
for(int y=3;y<6;y++){
System.out.print(Records.getRecord(x1,y) + " (" + Records.getRecord(x1,y+3) + ")\n");
}
System.out.println("\nReturn to Main Menu? (Y/N)");
cont1= reader1.readLine();
}
else if("b".equals(option1)||"B".equals(option1)){
arr_count++;
for (int y = 0; y<3;y++){
System.out.println("Enter Value for "+ option [y]+":" );
X = reader1.readLine();
X = Records.getRecord(arr_count,y);
}
System.out.println("\nNew Student added:\nID number: "+ arr_count);
for(int y=0;y<3;y++){
X = Records.getRecord(arr_count,y);
System.out.print(X + " ");
}
Records class:
public class Records {
public static String getRecord(int recordID,int index1)throws Exception{
//BufferedReader reader1=new BufferedReader(new InputStreamReader(System.in));
String records[][] = new String [10][9];
records[0][0] = "John"; records[0][3] = "English"; records[0][6] = "C";
records[0][1] = "Smith"; records[0][4] = "Math"; records[0][7] = "B";
records[0][2] = "BS IT"; records[0][5] = "Graphics"; records[0][8] = "A";
records[1][0] = "Juan"; records[1][3] = "Graphics"; records[1][6] = "B";
records[1][1] = "Ponce"; records[1][4] = "Math"; records[1][7] = "B";
records[1][2] = "BS ECE"; records[1][5] = "Robotics"; records[1][8] = "A";
/**records[2][0] = ""; records[2][3] = ""; records[2][6] = "";
records[2][1] = ""; records[2][4] = ""; records[2][7] = "";
records[2][2] = ""; records[2][5] = ""; records[2][8] = "";
records[3][0] = ""; records[3][3] = ""; records[3][6] = "";
records[3][1] = ""; records[3][4] = ""; records[3][7] = "";
records[3][2] = ""; records[3][5] = ""; records[3][8] = "";
records[4][0] = ""; records[4][3] = ""; records[4][6] = "";
records[4][1] = ""; records[4][4] = ""; records[4][7] = "";
records[4][2] = ""; records[4][5] = ""; records[4][8] = "";*/
return records[recordID][index1];
}
}
I've tried to set blank values to the records array, also putting the function which assigns the new values to the array into the Records class but I still can't find a way to make this work.

You never set the value of the record.
X = reader1.readLine();
X = Records.getRecord(arr_count,y);
Is discarding the value read. Create a setValue method in your Record class -
X = reader1.readLine();
Records.setValue(arr_count, y, X);
--- inRecord
setValue ( x, y, val ) {
records[x][y] = val;
}

It seems your Main class is not complete. What is the condition for the do while loop to stop? Is there anything else in the main method than this loop?