We Keep Coding

Read Tab-Separated-Columns into Lists - Java

Tab-Separated File:
2019-06-06 10:00:00 1.0
2019-06-06 11:00:00 2.0
I'd like to iterate over the file once and add the value of each column to a list.
My working approach would be:
import java.util.*;
import java.io.*;
public class Program {
public static void main(String[] args)
{
ArrayList<Double> List_1 = new ArrayList<Double>();
ArrayList<Double> List_2 = new ArrayList<Double>();
String[] values = null;
String fileName = "File.txt";
File file = new File(fileName);
try
{
Scanner inputStream = new Scanner(file);
while (inputStream.hasNextLine()){
try {
String data = inputStream.nextLine();
values = data.split("\\t");
if (values[1] != null && !values[1].isEmpty() == true) {
double val_1 = Double.parseDouble(values[1]);
List_1.add(val_1);
}
if (values[2] != null && !values[2].isEmpty() == true) {
double val_2 = Double.parseDouble(values[2]);
List_2.add(val_2);
}
}
catch (ArrayIndexOutOfBoundsException exception){
}
}
inputStream.close();
}
catch (FileNotFoundException e) {
e.printStackTrace();
}
System.out.println(List_1);
System.out.println(List_2);
}
}
I get:
[1.0]
[2.0]
It doesn't work without the checks for null, ìsEmpty and the ArrayIndexOutOfBoundsException.
I would appreciate any hints on how to save a few lines while keeping the scanner approach.

One option is to create a Map of Lists using column number as a key. This approach gives you "unlimited" number of columns and exactly the same output than one in the question.
public class Program {
public static void main(String[] args) throws Exception
{
Map<Integer, List<Double>> listMap = new TreeMap<Integer, List<Double>>();
String[] values = null;
String fileName = "File.csv";
File file = new File(fileName);
Scanner inputStream = new Scanner(file);
while (inputStream.hasNextLine()){
String data = inputStream.nextLine();
values = data.split("\\t");
for (int column = 1; column < values.length; column++) {
List<Double> list = listMap.get(column);
if (list == null) {
listMap.put(column, list = new ArrayList<Double>());
}
if (!values[column].isEmpty()) {
list.add(Double.parseDouble(values[column]));
}
}
}
inputStream.close();
for(List<Double> list : listMap.values()) {
System.out.println(list);
}
}
}

You can clean up your code some by using try-with resources to open and close the Scanner for you:
try (Scanner inputStream = new Scanner(file))
{
//your code...
}
This is useful because the inputStream will be closed automatically once the try block is left and you will not need to close it manually with inputStream.close();.
Additionally if you really want to "save lines" you can also combine these steps:
double val_2 = Double.parseDouble(values[2]);
List_2.add(val_2);
Into a single step each, since you do not actually use the val_2 anywhere else:
List_2.add(Double.parseDouble(values[2]));
Finally you are also using !values[1].isEmpty() == true which is comparing a boolean value to true. This is typically bad practice and you can reduce it to !values[1].isEmpty() instead which will have the same functionality. Try not to use == with booleans as there is no need.

you can do it like below:
BufferedReader bfr = Files.newBufferedReader(Paths.get("inputFileDir.tsv"));
String line = null;
List<List<String>> listOfLists = new ArrayList<>(100);
while((line = bfr.readLine()) != null) {
String[] cols = line.split("\\t");
List<String> outputList = new ArrayList<>(cols);
//at this line your expected list of cols of each line is ready to use.
listOfLists.add(outputList);
}
As a matter of fact, it is a simple code in java. But because it seems that you are a beginner in java and code like a python programmer, I decided to write a sample code to let you have a good start point. good luck

Writing array of objects to a file with Gson

public boolean wirteJson(Passenger passenger[]){
try {
file = new FileWriter(System.getProperty("user.dir")+"/"+fileName);
for (int i = 0; i < 10; i++) {
String json =gson.toJson(passenger[i]);
file.write(json);
}
file.flush();
file.close();
} catch (IOException ex) {
return false;
}
return false;
}
I'm trying to write the passenger array to a file with Gson. I'm open to alternatives.

You don't have to serialize the given input array by each element:
Gson can do it all itself;
your method assumes the input array is exactly 10 elements long (thus, throwing ArrayIndexOutOfBoundsException if the input array has less elements, or writing only first 10 elements);
your method does not write well-formed JSON: {},{},{} is invalid, whilst [{},{},{}] is;
your method, as suggested by RealSkeptic, does not need intermediate string representations.
All you need are just two methods Gson.toJson and Gson.fromJson.
So, here is a simple example:
final class Passenger {
final String name;
Passenger(final String name) {
this.name = name;
}
}
public static void main(final String... args)
throws IOException {
final File file = createTempFile("q43439637_", "_q43439637");
file.deleteOnExit();
try ( final FileWriter fileWriter = new FileWriter(file) ) {
final Passenger[] before = { new Passenger("Alice"), new Passenger("Bob") };
gson.toJson(before, fileWriter);
}
try ( final FileReader fileReader = new FileReader(file) ) {
final Passenger[] after = gson.fromJson(fileReader, Passenger[].class);
for ( final Passenger p : after ) {
System.out.println(p.name);
}
}
}
Output:
Alice
Bob
P.S. Your out-of-catch return seems to have to return true rather than false. Also, flush() is unnecessary before close().

Reading, comparing and merging multiple files in Java

Given there are some files Customer-1.txt, Customer-2.txt and Customer-3.txt and these files have the following content:
Customer-1.txt
1|1|MARY|SMITH
2|1|PATRICIA|JOHNSON
4|2|BARBARA|JONES
Customer-2.txt
1|1|MARY|SMITH
2|1|PATRICIA|JOHNSON
3|1|LINDA|WILLIAMS
4|2|BARBARA|JONES
Customer-3.txt
2|1|PATRICIA|JOHNSON
3|1|LINDA|WILLIAMS
5|2|ALEXANDER|ANDERSON
These files have a lot of duplicate data, but it is possible that each file contains some data that is unique.
And given that the actual files are sorted, big (a few GB each file) and there are many files...
Then what is the:
a) memory cheapest
b) cpu cheapest
c) fastest
way in Java to create one file out of these three files that will contain all the unique data of each file sorted and concatenated like such:
Customer-final.txt
1|1|MARY|SMITH
2|1|PATRICIA|JOHNSON
3|1|LINDA|WILLIAMS
4|2|BARBARA|JONES
5|2|ALEXANDER|ANDERSON
I looked into the following solution https://github.com/upcrob/spring-batch-sort-merge , but I would like to know if its possible to perhaps do it with the FileInputStream and/or a non spring batch solution.
A solution to use an in memory or real database to join them is not viable for my use case due to the size of the files and the absence of an actual database.

Since the input files are already sorted, a simple parallel iteration of the files, merging their content, is the memory cheapest, cpu cheapest, and fastest way to do it.
This is a multi-way merge join, i.e. a sort-merge join without the "sort", with elimination of duplicates, similar to a SQL DISTINCT.
Here is a version that can do unlimited number of input files (well, as many as you can have open files anyway). It uses a helper class to stage the next line from each input file, so the leading ID value only has to be parsed once per line.
private static void merge(StringWriter out, BufferedReader ... in) throws IOException {
CustomerReader[] customerReader = new CustomerReader[in.length];
for (int i = 0; i < in.length; i++)
customerReader[i] = new CustomerReader(in[i]);
merge(out, customerReader);
}
private static void merge(StringWriter out, CustomerReader ... in) throws IOException {
List<CustomerReader> min = new ArrayList<>(in.length);
for (;;) {
min.clear();
for (CustomerReader reader : in)
if (reader.hasData()) {
int cmp = (min.isEmpty() ? 0 : reader.compareTo(min.get(0)));
if (cmp < 0)
min.clear();
if (cmp <= 0)
min.add(reader);
}
if (min.isEmpty())
break; // all done
// optional: Verify that lines that compared equal by ID are entirely equal
out.write(min.get(0).getCustomerLine());
out.write(System.lineSeparator());
for (CustomerReader reader : min)
reader.readNext();
}
}
private static final class CustomerReader implements Comparable<CustomerReader> {
private BufferedReader in;
private String customerLine;
private int customerId;
CustomerReader(BufferedReader in) throws IOException {
this.in = in;
readNext();
}
void readNext() throws IOException {
if ((this.customerLine = this.in.readLine()) == null)
this.customerId = Integer.MAX_VALUE;
else
this.customerId = Integer.parseInt(this.customerLine.substring(0, this.customerLine.indexOf('|')));
}
boolean hasData() {
return (this.customerLine != null);
}
String getCustomerLine() {
return this.customerLine;
}
#Override
public int compareTo(CustomerReader that) {
// Order by customerId only. Inconsistent with equals()
return Integer.compare(this.customerId, that.customerId);
}
}
TEST
String file1data = "1|1|MARY|SMITH\n" +
"2|1|PATRICIA|JOHNSON\n" +
"4|2|BARBARA|JONES\n";
String file2data = "1|1|MARY|SMITH\n" +
"2|1|PATRICIA|JOHNSON\n" +
"3|1|LINDA|WILLIAMS\n" +
"4|2|BARBARA|JONES\n";
String file3data = "2|1|PATRICIA|JOHNSON\n" +
"3|1|LINDA|WILLIAMS\n" +
"5|2|ALEXANDER|ANDERSON\n";
try (
BufferedReader in1 = new BufferedReader(new StringReader(file1data));
BufferedReader in2 = new BufferedReader(new StringReader(file2data));
BufferedReader in3 = new BufferedReader(new StringReader(file3data));
StringWriter out = new StringWriter();
) {
merge(out, in1, in2, in3);
System.out.print(out);
}
OUTPUT
1|1|MARY|SMITH
2|1|PATRICIA|JOHNSON
3|1|LINDA|WILLIAMS
4|2|BARBARA|JONES
5|2|ALEXANDER|ANDERSON
The code merges purely by ID value, and doesn't verify that rest of line is actually equal. Insert code at the optional comment to check for that, if needed.

This might help:
public static void main(String[] args) {
String files[] = {"Customer-1.txt", "Customer-2.txt", "Customer-3.txt"};
HashMap<Integer, String> customers = new HashMap<Integer, String>();
try {
String line;
for(int i = 0; i < files.length; i++) {
BufferedReader reader = new BufferedReader(new FileReader("data/" + files[i]));
while((line = reader.readLine()) != null) {
Integer uuid = Integer.valueOf(line.split("|")[0]);
customers.put(uuid, line);
}
reader.close();
}
BufferedWriter writer = new BufferedWriter(new FileWriter("data/Customer-final.txt"));
Iterator<String> it = customers.values().iterator();
while(it.hasNext()) writer.write(it.next() + "\n");
writer.close();
} catch (Exception e) {
e.printStackTrace();
}
}
If you have any cquestions ask me.

save to and load from not working in java [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 6 years ago.
Improve this question
for the life of me I can't figure out what is wrong with these codes .. the save to keep overwrite itself and the load from doesn't load the already existing data .. I have searched for this code but it seems like people use different codes .. please help me end my headache
// Write to file
static void writeToFile(Customer c[], int number_of_customers) throws IOException {
// set up file for output
// pw used to write to file
File outputFile = new File("Customers.dat");
FileOutputStream fos = new FileOutputStream(outputFile);
PrintWriter pw = new PrintWriter(new OutputStreamWriter(fos));
int i = 0;
do {
pw.println(c[i].getName());
pw.println(c[i].getNumber());
i++;
} while (i < number_of_customers);
pw.println(0);
pw.println(0);
pw.close();
}
// Read from file
public static int readFromFile(Customer c[]) throws IOException {
// set up file for reading
// br used to read from file
File inputFile = new File("Customers.dat");
FileInputStream fis = new FileInputStream(inputFile);
BufferedReader br = new BufferedReader(new InputStreamReader(fis));
String cus;
int l = -1;
// Subtract AND assignment operator, It subtracts right operand from the
// left operand and assign the result to left operand
int all_customers = 0;
do {
l++;
c[l] = new Customer();
c[l].cus_name = br.readLine();
cus = br.readLine();
c[l].cus_no = Integer.parseInt(cus);
all_customers++;
} while (c[l].cus_no != 0); // end while
br.close(); // end ReadFile class
return all_customers - 1;
}

An alternative way to fix your write method would be to use a FileOutputStream constructor that lets you request that data be appended to the end of the file.
FileOutputStream fos = new FileOutputStream(outputFile, true);
This does assume that you always write a complete final record with an end of line after it, even under error conditions. You'll still have to deal with this type of situation with the other solution (read and merge), but with that one the subsequent run can detect and deal with it if necessary. So the append solution I describe is not as robust.

You have a number of issues with your code.
Looking first at your readFromFile method:
You're passing in an array that your method is filling up with all the records it finds. What happens if there are more customers in the file than there's room for in the array? (hint: ArrayIndexOutOfBoundsException is a thing)
You're parsing an integer read as a string from the file. What happens if the file is corrupt and the line read is not an integer?
The name of the file to read from is hard-coded. This should be a constant or configuration option. For the purpose of writing methods, it is best to make it a parameter.
You're opening the file and reading from it in the method. For purposes of unit testing, you should split this into separate methods.
In general, you should be using a Collections class instead of an array to hold a list of objects.
You're accessing the Customer attributes directly in the readFromFile method. You should be using an accessor method.
Collections-based approach
Here's my proposed rewrite based on using Collections APIs:
public static List<Customer> readFromFile(String filename) throws IOException {
// set up file for reading
// br used to read from file
File inputFile = new File(filename);
FileInputStream fis = new FileInputStream(inputFile);
BufferedReader br = new BufferedReader(new InputStreamReader(fis));
List<Customer> customers = readFromStream(br);
br.close(); // end ReadFile class
return customers;
}
This uses this method to actually read the contents:
public static List<Customer> readFromStream(BufferedReader br) throws IOException {
List<Customer> customerList = new LinkedList<>();
// Subtract AND assignment operator, It subtracts right operand from the
// left operand and assign the result to left operand
boolean moreCustomers = true;
while (moreCustomers) {
try {
Customer customer = new Customer();
customer.setName(br.readLine());
String sCustNo = br.readLine();
customer.setNumber(Integer.parseInt(sCustNo));
if (customer.getNumber() == 0) {
moreCustomers = false;
}
else {
customerList.add(customer);
}
}
catch (NumberFormatException x) {
// happens if the line is not a number.
// handle this somehow, e.g. by ignoring, logging, or stopping execution
// for now, we just stop reading
moreCustomers = false;
}
}
return customerList;
}
Using a similar approach for writeToFile, we get:
static void writeToFile(Collection<Customer> customers, String filename) throws IOException {
// set up file for output
// pw used to write to file
File outputFile = new File(filename);
FileOutputStream fos = new FileOutputStream(outputFile);
PrintWriter pw = new PrintWriter(new OutputStreamWriter(fos));
writeToStream(customers, pw);
pw.flush();
pw.close();
}
static void writeToStream(Collection<Customer> customers, PrintWriter pw) throws IOException {
for (Customer customer: customers) {
pw.println(customer.getName());
pw.println(customer.getNumber());
}
pw.println(0);
pw.println(0);
}
However, we still haven't addressed your main concern. It seems you want to merge the file content with the customers in memory when you call writeToFile. I suggest that you instead introduce a new method for this purpose. This keeps the existing methods simpler:
static void syncToFile(Collection<Customer> customers, String filename) throws IOException {
// get a list of existing customers
List<Customer> customersInFile = readFromFile(filename);
// use a set to merge
Set<Customer> customersToWrite = new HashSet<>();
// first add current in-memory cutomers
customersToWrite.addAll(customers);
// then add the ones from the file. Duplicates will be ignored
customersToWrite.addAll(customersInFile);
// then save the merged set
writeToFile(customersToWrite, filename);
}
Oh... I almost forgot: The magic of using a Set to merge the file and in-memory list relies on you to implement the equals() method in the Customer class. If you overwrite equals(), you should also overwrite hashCode(). For example:
public class Customer {
#Override
public boolean equals(Object obj) {
return (obj != null) && (obj instanceof Customer) && (getNumber() == ((Customer)obj).getNumber());
}
#Override
public int hashCode() {
return getNumber()+31;
}
};
CustomerList-based approach
If you cannot use Collections APIs, the second-best would be to write your own collection type that supports the same operations, but is backed by an array (or linked list, if you have learned that). In your case, it would be a list of customers. I'll call the type CustomerList:
Analyzing our existing code, we'll need a class that implements an add method and a way to traverse the list. Ignoring Iterators, we'll accomplish the latter with a getLength and a getCustomer (by index). For the synchronization, we also need a way to check if a customer is in the list, so we'll add a contains method:
public class CustomerList {
private static final int INITIAL_SIZE = 100;
private static final int SIZE_INCREMENT = 100;
// list of customers. We're keeping it packed, so there
// should be no holes!
private Customer[] customers = new Customer[INITIAL_SIZE];
private int numberOfCustomers = 0;
/**
* Adds a new customer at end. Allows duplicates.
*
* #param newCustomer the new customer to add
* #return the updated number of customers in the list
*/
public int add(Customer newCustomer) {
if (numberOfCustomers == customers.length) {
// the current array is full, make a new one with more headroom
Customer[] newCustomerList = new Customer[customers.length+SIZE_INCREMENT];
for (int i = 0; i < customers.length; i++) {
newCustomerList[i] = customers[i];
}
// we will add the new customer at end!
newCustomerList[numberOfCustomers] = newCustomer;
// replace the customer list with the new one
customers = newCustomerList;
}
else {
customers[numberOfCustomers] = newCustomer;
}
// we've added a new customer!
numberOfCustomers++;
return numberOfCustomers;
}
/**
* #return the number of customers in this list
*/
public int getLength() {
return numberOfCustomers;
}
/**
* #param i the index of the customer to retrieve
* #return Customer at index <code>i</code> of this list (zero-based).
*/
public Customer getCustomer(int i) {
//TODO: Add boundary check of i (0 <= i < numberOfCustomers)
return customers[i];
}
/**
* Check if a customer with the same number as the one given exists in this list
* #param customer the customer to check for (will use customer.getNumber() to check against list)
* #return <code>true</code> if the customer is found. <code>false</code> otherwise.
*/
public boolean contains(Customer customer) {
for (int i = 0; i < numberOfCustomers; i++) {
if (customers[i].getNumber() == customer.getNumber()) {
return true;
}
}
// if we got here, it means we didn't find the customer
return false;
}
}
With this implemented, the rewrite of the writeToFile method is exactly the same, except we use CustomerList instead of List<Customer>:
static void writeToFile(CustomerList customers, String filename) throws IOException {
// set up file for output
// pw used to write to file
File outputFile = new File(filename);
FileOutputStream fos = new FileOutputStream(outputFile);
PrintWriter pw = new PrintWriter(new OutputStreamWriter(fos));
writeToStream(customers, pw);
pw.flush();
pw.close();
}
The writeToStream is also very similar, except since we're not using an Iterator, we have to traverse the list manually:
static void writeToStream(CustomerList customers, PrintWriter pw) throws IOException {
for (int i = 0; i < customers.getLength(); i++) {
pw.println(customers.getCustomer(i).getName());
pw.println(customers.getCustomer(i).getNumber());
}
pw.println(0);
pw.println(0);
}
Similar for readFromFile -- pretty much the same except for the list type:
public static CustomerList readFromFile(String filename) throws IOException {
// set up file for reading
// br used to read from file
File inputFile = new File(filename);
FileInputStream fis = new FileInputStream(inputFile);
BufferedReader br = new BufferedReader(new InputStreamReader(fis));
CustomerList customers = readFromStream(br);
br.close(); // end ReadFile class
return customers;
}
The readFromStream is also pretty much the same, except for the type (the methods used on CustomerList has the same signature as the ones used on List<Customer>:
public static CustomerList readFromStream(BufferedReader br) throws IOException {
CustomerList customerList = new CustomerList();
// Subtract AND assignment operator, It subtracts right operand from the
// left operand and assign the result to left operand
boolean moreCustomers = true;
while (moreCustomers) {
try {
Customer customer = new Customer();
customer.setName(br.readLine());
String sCustNo = br.readLine();
customer.setNumber(Integer.parseInt(sCustNo));
if (customer.getNumber() == 0) {
moreCustomers = false;
}
else {
customerList.add(customer);
}
}
catch (NumberFormatException x) {
// happens if the line is not a number.
// handle this somehow, e.g. by ignoring, logging, or stopping execution
// for now, we just stop reading
moreCustomers = false;
}
}
return customerList;
}
The most different method is the syncToFile, as we don't have the Set type that guarantees no duplicates, we have to manually check each time we try to insert a customer from the file:
static void syncToFile(CustomerList customers, String filename) throws IOException {
// get a list of existing customers
CustomerList customersInFile = readFromFile(filename);
// use a set to merge
CustomerList customersToWrite = new CustomerList();
// first add current in-memory customers
for (int i = 0; i < customers.getLength(); i++) {
customersToWrite.add(customers.getCustomer(i));
}
// then add the ones from the file. But skip duplicates
for (int i = 0; i < customersInFile.getLength(); i++) {
if (!customersToWrite.contains(customersInFile.getCustomer(i))) {
customersToWrite.add(customersInFile.getCustomer(i));
}
}
// then save the merged set
writeToFile(customersToWrite, filename);
}
Something to note here is that we could have optimized the add operations by having an extra constructor for CustomerList that took the new capacity, but I'll leave at least something for you to figure out ;)

How do I read from a File to an array

I am trying to read from a file to an array. I tried two different styles and both aren't working. Below are the two styles.
Style 1
public class FileRead {
int i;
String a[] = new String[2];
public void read() throws FileNotFoundException {
//Z means: "The end of the input but for the final terminator, if any"
a[i] = new Scanner(new File("C:\\Users\\nnanna\\Documents\\login.txt")).useDelimiter("\\n").next();
for(i=0; i<=a.length; i++){
System.out.println("" + a[i]);
}
}
public static void main(String args[]) throws FileNotFoundException{
new FileRead().read();
}
}
Style 2
public class FileReadExample {
private int j = 0;
String path = null;
public void fileRead(File file){
StringBuilder attachPhoneNumber = new StringBuilder();
try{
FileReader read = new FileReader(file);
BufferedReader bufferedReader = new BufferedReader(read);
while((path = bufferedReader.readLine()) != null){
String a[] = new String[3];
a[j] = path;
j++;
System.out.println(path);
System.out.println(a[j]);
}
bufferedReader.close();
}catch(IOException exception){
exception.printStackTrace();
}
}
I need it to read each line of string and store each line in an array. But neither works. How do I go about it?

Do yourself a favor and use a library that provides this functionality for you, e.g.
Guava:
// one String per File
String data = Files.toString(file, Charsets.UTF_8);
// or one String per Line
List<String> data = Files.readLines(file, Charsets.UTF_8);
Commons / IO:
// one String per File
String data = FileUtils.readFileToString(file, "UTF-8");
// or one String per Line
List<String> data = FileUtils.readLines(file, "UTF-8");

It's not really clear exactly what you're trying to do (partly with quite a lot of code commented out, leaving other code which won't even compile), but I'd recommend you look at using Guava:
List<String> lines = Files.readLines(file, Charsets.UTF_8);
That way you don't need to mess around with the file handling yourself at all.