We Keep Coding

CSV Handling / FileReader and BufferedReader with UTF-8

I am having issues with reading a csv file and UTF-8. What I am doing is, reading the file to a String Array. Converting the String Array to a 2D String Array, depending on the line separator. Cleaning the line separator and displaying the 2D String Array in the console.
The issue is, that special characters like ö and ä are messed up on the console.
System.out.println("Default Charset=" + Charset.defaultCharset());
Tells me that my system is using UTF-8. What am I doing wrong?
I´m handling the CSV files with a 2D Array to get direct access to the rows and columns. Is there a better approach for that?
// Reading File
public static String[] readFile(String path) throws IOException {
FileReader input = new FileReader(path);
BufferedReader br = new BufferedReader(input);
String[] stringBuff = new String[countLines(path)];
int lines = countLines(path);
for (int j = 0; j < lines; j++) {
stringBuff[j] = br.readLine();
}
br.close();
return stringBuff;
}
// Converting to a 2D Array
public static String[][] convert(String[] array, String path) throws IOException {
int lines = countLines(path);
int[] locationarray;
int columns = (findcolumn(array)) + 1;
String[][] array2 = new String[lines][columns];
for (int i = 0; i < array.length; i++) {
locationarray = findlocation(array, i);
for (int j = 0; j < columns; j++) {
array2[i][j] = array[i].substring(locationarray[j], locationarray[j + 1]);
array2[i][j] = cleanup(array2[i][j]);
// System.out.print(array2[i][j]);
}
// System.out.println();
}
return array2;
}
// Show Array
public static void showarray(String[][] srcfinal) {
for (int i = 0; i < srcfinal.length; i++) {
for (int j = 0; j < srcfinal[i].length; j++) {
System.out.print(srcfinal[i][j]);
}
System.out.println();
}
}

Importing data from file into an int array that's called from a different class

I am working on a program that deals with inheritance. Below is one of the class that is extended from my super class bus.
public BusTrip(){
super();
int totalPassenger = 0;
fare = new int [3];
numOfPassenger = new int [3];
String destination = "";
}
I am having trouble in my driver class to import data into my ArrayList. I understand that it's reading everything in the data file position 4 all the way to the end of to the string. I have tried a while loop as well but that does not seem to go past the first set of data. Is tokenizer the only method? Is there a way for me to prevent the for loop from reading pass a certain point?
Data File
cr, greyhound, 2015, 30, 22, 44, 14, 10, 5, 15, New York
public static void readAllBus(ArrayList<Bus> busInformation){
File infile = null;
Scanner scan = null;
int[] fare = new int [3];
int[] numOfPassenger = new int [3];
try{
infile = new File("busData.csv");
scan = new Scanner(infile);
}//end try
catch(FileNotFoundException e){
System.out.println("Error! File not found!");
System.exit(0);
}//end catch
while(scan.hasNext()){
String [] str = scan.nextLine().split(",");
if(str[0].equals("cr")){
for(int i = 0; i < fare.length; i++)
for(int j = 4; j < str.length; j++)
fare[i] = Integer.parseInt(str[j]);
//error For input string: "Bahamas"
for(int i = 0; i < numOfPassenger.length; i++)
for(int j = 7; j < str.length; j++)
numOfPassenger[i] = Integer.parseInt(str[j]);
//error For input string: "Bahamas"
busInformation.add(new BusTrip(str[1],
Integer.parseInt(str[2]),
Integer.parseInt(str[3]),
fare,
numOfPassenger,
str[10]));
}//end if
}//end while
}//end readAllBus

I think you have a bug in the for loop. You are executing the inner loop multiple times when you just wanted to read the value only once. You could rewrite your logic as follows -
public static void readAllBus(ArrayList<Bus> busInformation){
File infile = null;
Scanner scan = null;
int[] fare = new int [3];
int[] numOfPassenger = new int [3];
try{
infile = new File("busData.csv");
scan = new Scanner(infile);
}//end try
catch(FileNotFoundException e){
System.out.println("Error! File not found!");
System.exit(0);
}//end catch
while(scan.hasNext()){
String [] str = scan.nextLine().split(",");
if(str[0].equals("cr")){
if(str.length>7){
for(int i = 0; i < 3; i++){
fare[i] = Integer.parseInt(str[i+4]);
}
}
if(str.length>10){
for(int i = 0; i < 3; i++){
numOfPassenger[i] = Integer.parseInt(str[i+7]);
}
}
busInformation.add(new BusTrip(str[1],
Integer.parseInt(str[2]),
Integer.parseInt(str[3]),
fare,
numOfPassenger,
str[10]));
}//end if
}//end while
}//end readAllBus

finding standard deviation in csv file

i am trying to find standard deviation(σ = √[(Σ(x - MEAN))2 ÷ n]) of single extracted column of csv file.csv file contain around 45000 instance and 17 attribute saperated with ';'.
for finding standard deviation it need MEAN value in every iteration of while loop for substact with Xi. so i think MEAN need before while loop iteration for find standard deviation.but i dont know how to do this or is there any way to do this. am getting stuck here. then i had puted code for replace old Xi with new Xi. and then write(generate) new csv file.
import java.io.File;
import java.io.FileNotFoundException;
import java.util.Scanner;
import java.util.logging.Level;
import java.util.logging.Logger;
import java.io.FileWriter;
import java.io.*;
import static java.lang.Math.sqrt;
public class Main {
public static void main(String[] args) throws IOException {
String filename = "ly.csv";
File file = new File(filename);
BufferedWriter writer = null;
try {
writer = new BufferedWriter(new FileWriter("bank-full_updated.csv"));
}
catch (IOException e) {
}
try {
double Tuple,avg;
double temp;
Tuple = 0;
double stddev=0;
Scanner inputStream = new Scanner(file);
inputStream.next();
while (inputStream.hasNext()) {
String data1 = inputStream.next();
String[] values = data1.split(";");
double Xi = Double.parseDouble(values[1]);
//now finding standard deviation
temp1 += (Xi-MEAN);
// temp2=(temp1*temp1);
// temp3=(temp2/count);
// standard deviation=Math.sqrt(temp3);
Xi=standard deviation * Xi
//now replace new Xi to original values1
values[1] = String.valueOf(Xi);
// iterate through the values and build a string out of them for write a new file
StringBuilder sb = new StringBuilder();
String newData = sb.toString();
for (int i = 0; i < values.length; i++) {
sb.append(values[i]);
if (i < values.length - 1) {
sb.append(";");
}
}
// get the new string
System.out.println(sb.toString());
writer.write(sb.toString()+"\n");
}
writer.close();
inputStream.close();
}
catch (FileNotFoundException ex) {
Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ex);
}
}
}

It is possible to calculate the standard deviation in a single pass. Professor Donald Knuth has an algorithm that does it using the Kahan summation. Here is the paper: http://researcher.ibm.com/files/us-ytian/stability.pdf
Here is another way but it suffers from rounding errors:
double std_dev2(double a[], int n) {
if(n == 0)
return 0.0;
double sum = 0;
double sq_sum = 0;
for(int i = 0; i < n; ++i) {
sum += a[i];
sq_sum += a[i] * a[i];
}
double mean = sum / n;
double variance = sq_sum / n - mean * mean;
return sqrt(variance);
}

How do I copy an array?

What I'm trying to do is I'm trying to take an image and make it a tiled image. The starting image should look like this.
http://i1146.photobucket.com/albums/o525/walroid/letter_Q_grayscale_zpsd3b567a7.jpg
And then what the image is turned into tiles then it should look like this:
http://i1146.photobucket.com/albums/o525/walroid/replicate_example_zps5e5248e8.jpg
In my code the pictures are saved into an array which is called into the method. What I want to do is copy that array and then put it into another array which will replicate the image. How do I do that?
Here's my entire code:
import java.awt.Color;
import java.awt.image.BufferedImage;
import java.io.*;
import javax.imageio.ImageIO;
public class ImageProcessor {
public static void main(String[] args) {
if (args.length < 3) {
System.out.println("Not enough arguments");
System.exit(-1);
}
String function = args[0];
if (function.equals("-reflectV")) {
String inputFileName = args[1];
String outputFileName = args[2];
int[][] imageArr = readGrayscaleImage(inputFileName);
int[][] reflectedArr = reflectV(imageArr);
writeGrayscaleImage(outputFileName, reflectedArr);
} else if (function.equals("-reflectH")) {
String inputFileName = args[1];
String outputFileName = args[2];
int[][] imageArr = readGrayscaleImage(inputFileName);
int[][] reflectedArr = reflectH(imageArr);
writeGrayscaleImage(outputFileName, reflectedArr);
} else if (function.equals("-ascii")) {
String inputFileName = args[1];
String outputFileName = args[2];
int[][] imageArr = readGrayscaleImage(inputFileName);
int[][] reflectedArr = reflectV(imageArr);
try {
PrintStream output = new PrintStream(new File("output.txt"));
} catch (java.io.FileNotFoundException ex) {
System.out.println("Error: File Not Found");
System.exit(-1);
}
} else if (function.equals("-adjustBrightness")) {
String amount = args[1];
int a = Integer.parseInt(amount);
System.out.print(a)
String inputFileName = args[1];
String outputFileName = args[2];
int[][] imageArr = readGrayscaleImage(inputFileName);
int[][] brightnessArr = adjustBrightness(imageArr);
writeGrayscaleImage(outputFileName, brightnessArr);
} else
System.out.println("That is not a valid choice");
system.exit(-1)
public static int[][] reflectV ( int[][] arr){
int[][] reflected = new int[arr.length][arr[0].length];
for (int i = 0; i < arr.length; i++) {
for (int j = 0; j < arr[i].length; j++) {
reflected[i][j] = arr[i][arr[i].length - 1 - j];
}
}
return reflected;
}
public static int[][] reflectH ( int[][] arr){
int[][] reflected = new int[arr.length][arr[0].length];
for (int i = 0; i < arr.length; i++) {
for (int j = 0; j < arr[i].length; j++) {
reflected[j][i] = arr[i][arr[j].length - 1 - j];
}
}
return reflected;
}
public static int[][] adjustBrightness ( int[][] arr){
int[][] brightness = new int[arr.length][arr[0].length];
for (int i = 0; i < arr.length; i++) {
for (int j = 0; j < arr[i].length; j++) {
RGB
}
}
return brightness;
}
public static int[][] readGrayscaleImage (String filename){
int[][] result = null; //create the array
try {
File imageFile = new File(filename); //create the file
BufferedImage image = ImageIO.read(imageFile);
int height = image.getHeight();
int width = image.getWidth();
result = new int[height][width]; //read each pixel value
for (int x = 0; x < width; x++) {
for (int y = 0; y < height; y++) {
int rgb = image.getRGB(x, y);
result[y][x] = rgb & 0xff;
}
}
} catch (IOException ioe) {
System.err.println("Problems reading file named " + filename);
System.exit(-1);
}
return result;
}
public static void writeGrayscaleImage(String filename, int[][] array) {
int width = array[0].length;
int height = array.length;
try {
BufferedImage image = new BufferedImage(width, height,
BufferedImage.TYPE_INT_RGB); //create the image
//set all its pixel values based on values in the input array
for (int x = 0; x < width; x++) {
for (int y = 0; y < height; y++) {
int rgb = array[y][x];
rgb |= rgb << 8;
rgb |= rgb << 16;
image.setRGB(x, y, rgb);
}
}
//write the image to a file
File imageFile = new File(filename);
ImageIO.write(image, "jpg", imageFile);
} catch (IOException ioe) {
System.err.println("Problems writing file named " + filename);
System.exit(-1);
}
}
}

You need to do a "deep copy" of the array. Simply copying your array to a new variable will only assign the reference (shallow copy), and if you manipulate data in one of the arrays, it will change both.
Shallow Copy:
String[] myArray2 = myArray1;
This will essentially have 2 references pointing at the same data. If you change anything in myArray2, it will also change in myArray1.
Deep Copy:
There are multiple ways to do a deep copy. The obvious way is to iterate over your array and copy each element one at a time into the new array.
String[] myArray2 = new String[myArray1.length];
for (int i = 0; i < myArray1.length; i++) {
myArray2[i] = myArray1[i];
}
A sometimes simpler/faster way is to serialize your array and then de-serialize it while it's still in memory. This causes the JVM to treat the de-serialized array as an entirely new array ("no strings attached" so-to-speak).
Here's an example from an old project of mine:
/**
* Clones an object creating a brand new
* object by value of input object. Accomplishes this
* by serializing the object, then deservializing it.
*
* #param obj Input Object to clone
* #return a new List<Product> type cloned from original.
* #throws IOException If IOException
* #throws ClassNotFoundException If ClassNotFoundException
*/
private static List<Product> cloneProdList(Object obj) throws IOException, ClassNotFoundException {
java.io.ByteArrayOutputStream bos = new java.io.ByteArrayOutputStream();
java.io.ObjectOutputStream obj_out = new java.io.ObjectOutputStream(bos);
obj_out.writeObject(obj);
java.io.ByteArrayInputStream bis = new java.io.ByteArrayInputStream(bos.toByteArray());
java.io.ObjectInputStream obj_in = new java.io.ObjectInputStream(bis);
#SuppressWarnings("unchecked")
List<Product> newObj = (List<Product>)obj_in.readObject();
bos.close();
bis.close();
obj_out.close();
obj_in.close();
return newObj;
}
This code takes a List type as the input (well, it actually casts to an Object type), serializes and then de-serializes while still in memory, then casts back into a List object and returns the new object back out of the method.
You can modify it to use an Array object instead pretty easily. (an array[] type will autobox into Array for you)

https://docs.oracle.com/javase/7/docs/api/java/lang/System.html#arraycopy%28java.lang.Object,%20int,%20java.lang.Object,%20int,%20int%29
You can use System.arraycopy in a for loop to copy from one array to other.

Using the Array class and calling the static method Array.copyOf(array, array.length) is quite handy so that if myArray1 is the previous array and myArray2 is the new array then myArray2 = Array.copyOf(myArray1, myArray1.length)

How to speed up external merge sort in Java

I am writing code for the external merge sort. The idea is that the input files contain too many numbers to be stored in an array so you read some of it and put it into files to be stored. Here's my code. While it runs fast, it is not fast enough. I was wondering if you can think of any improvements I can make on the code. Note that at first, I sort every 1m integers together so I skip iterations of the merging algorithm.
import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.security.DigestInputStream;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.Arrays;
public class ExternalSort {
public static void sort(String f1, String f2) throws Exception {
RandomAccessFile raf1 = new RandomAccessFile(f1, "rw");
RandomAccessFile raf2 = new RandomAccessFile(f2, "rw");
int fileByteSize = (int) (raf1.length() / 4);
int size = Math.min(1000000, fileByteSize);
externalSort(f1, f2, size);
boolean writeToOriginal = true;
DataOutputStream dos;
while (size <= fileByteSize) {
if (writeToOriginal) {
raf1.seek(0);
dos = new DataOutputStream(new BufferedOutputStream(
new MyFileOutputStream(raf1.getFD())));
} else {
raf2.seek(0);
dos = new DataOutputStream(new BufferedOutputStream(
new MyFileOutputStream(raf2.getFD())));
}
for (int i = 0; i < fileByteSize; i += 2 * size) {
if (writeToOriginal) {
dos = merge(f2, dos, i, size);
} else {
dos = merge(f1, dos, i, size);
}
}
dos.flush();
writeToOriginal = !writeToOriginal;
size *= 2;
}
if (writeToOriginal)
{
raf1.seek(0);
raf2.seek(0);
dos = new DataOutputStream(new BufferedOutputStream(
new MyFileOutputStream(raf1.getFD())));
int i = 0;
while (i < raf2.length() / 4){
dos.writeInt(raf2.readInt());
i++;
}
dos.flush();
}
}
public static void externalSort(String f1, String f2, int size) throws Exception{
RandomAccessFile raf1 = new RandomAccessFile(f1, "rw");
RandomAccessFile raf2 = new RandomAccessFile(f2, "rw");
int fileByteSize = (int) (raf1.length() / 4);
int[] array = new int[size];
DataInputStream dis = new DataInputStream(new BufferedInputStream(
new MyFileInputStream(raf1.getFD())));
DataOutputStream dos = new DataOutputStream(new BufferedOutputStream(
new MyFileOutputStream(raf2.getFD())));
int count = 0;
while (count < fileByteSize){
for (int k = 0; k < size; ++k){
array[k] = dis.readInt();
}
count += size;
Arrays.sort(array);
for (int k = 0; k < size; ++k){
dos.writeInt(array[k]);
}
}
dos.flush();
raf1.close();
raf2.close();
dis.close();
dos.close();
}
public static DataOutputStream merge(String file,
DataOutputStream dos, int start, int size) throws IOException {
RandomAccessFile raf = new RandomAccessFile(file, "rw");
RandomAccessFile raf2 = new RandomAccessFile(file, "rw");
int fileByteSize = (int) (raf.length() / 4);
raf.seek(4 * start);
raf2.seek(4 *start);
DataInputStream dis = new DataInputStream(new BufferedInputStream(
new MyFileInputStream(raf.getFD())));
DataInputStream dis3 = new DataInputStream(new BufferedInputStream(
new MyFileInputStream(raf2.getFD())));
int i = 0;
int j = 0;
int max = size * 2;
int a = dis.readInt();
int b;
if (start + size < fileByteSize) {
dis3.skip(4 * size);
b = dis3.readInt();
} else {
b = Integer.MAX_VALUE;
j = size;
}
while (i + j < max) {
if (j == size || (a <= b && i != size)) {
dos.writeInt(a);
i++;
if (start + i == fileByteSize) {
i = size;
} else if (i != size) {
a = dis.readInt();
}
} else {
dos.writeInt(b);
j++;
if (start + size + j == fileByteSize) {
j = size;
} else if (j != size) {
b = dis3.readInt();
}
}
}
raf.close();
raf2.close();
return dos;
}
public static void main(String[] args) throws Exception {
String f1 = args[0];
String f2 = args[1];
sort(f1, f2);
}
}

You might wish to merge k>2 segments at a time. This reduces the amount of I/O from n log k / log 2 to n log n / log k.
Edit: In pseudocode, this would look something like this:
void sort(List list) {
if (list fits in memory) {
list.sort();
} else {
sublists = partition list into k about equally big sublists
for (sublist : sublists) {
sort(sublist);
}
merge(sublists);
}
}
void merge(List[] sortedsublists) {
keep a pointer in each sublist, which initially points to its first element
do {
find the pointer pointing at the smallest element
add the element it points to to the result list
advance that pointer
} until all pointers have reached the end of their sublist
return the result list
}
To efficiently find the "smallest" pointer, you might employ a PriorityQueue.

I would use memory mapped files. It can be as much as 10x faster than using this type of IO. I suspect it will be much faster in this case as well. The mapped buffers use virtual memory rather heap space to store data and can be larger than your available physical memory.

We have implemented a public domain external sort in Java:
http://code.google.com/p/externalsortinginjava/
It might be faster than yours. We use strings and not integers, but you could easily modify our code by substituting integers for strings (the code was made hackable by design). At the very least, you can compare with our design.
Looking at your code, it seems like you are reading the data in units of integers. So IO will be a bottleneck I would guess. With external memory algorithms, you want to read and write blocks of data---especially in Java.

You are sorting integers so you should check out radix sort. The core idea of radix sort is that you can sort n byte integers with n passes through the data with radix 256.
You can combine this with merge sort theory.