We Keep Coding

Parsing data and removing invisible characters like carriage return

I am stuck and for the life of me I can't figure out why I am not getting this data to parse out into 5 tokens.
I load the data below in a file called test.dat:
Row1: 1236~~Tier 5~~54~~updated~~01/05/2019
Row2: 1255934~~Tier 1~~30~~Meeting series continued to review period.~~8/21/2018
Row3: 12556~~Tier 1~~30~~Team began to map out Customer Harm Scenarios in this ~~8/21/2018
Row4: 1255936~~Tier 1~~30~~week's calls.
As of now, calls set through August 8th (~~8/21/2018) end of test file.
See how Row 4 in this editor shows the return carriage? In my file it does not have a visible line break like it does here. It must be hidden characters.
This seems to be causing the problem. I tried to remove it, but nothing seems to work.
Here is my code:
import java.io.*;
import java.util.*;
class MergeData {
public static void main(String[] args) {
System.out.print("Enter master_file:");
String master = loadFile();
HashMap masterMap = readData(master);
}
public static String loadFile(){
BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
String fileName = new String();
try{
fileName = br.readLine();
System.out.print("opening file... "+ fileName+"\n");
}catch(IOException nfe){
System.err.println("Invalid Format!");
}
return fileName;
}
public static HashMap readData(String file){
HashMap dataMap = new HashMap();
try {
FileReader fileReader = new FileReader(file);
BufferedReader br = new BufferedReader(fileReader);
String line;
int rowCnt = 0;
while ((line = br.readLine()) != null) {
String clean = line.replaceAll("(?:\\n|\\r)", "");
String[] fields = clean.split("~~");
int size = fields.length;
System.out.println("delimiter size:" + size );
rowCnt = rowCnt +1;
String f1 = null;
String f2 = null;
String f3 = null;
String f4 = null;
String f5 = null;
if(size == 5){
f1 = fields[0];
f2 = fields[1];
f3 = fields[2];
f4 = fields[3];
f5 = fields[4];
} else {
System.out.println("!!! INVALID ROW: id---->" + fields[0]);
}
System.out.println(
"rowCnt:" +rowCnt+ " f1:" +f1+ "----f2:" +f2+
"----f3:" +f3+ "----f4:" + f4+ "----f5:" + f5 +"\n\n");
}
br.close();
fileReader.close();
} catch (IOException e) {
System.out.println("\n File Read Error \n" +e );
}
return dataMap;
}
}

combine two methods in Java together in this Java code

I want combine the two methods Just some error in my document parser, frequencyCounter and parseFiles thsi code.
I want all of frequencyCounter should be a function that should be executed from within parseFiles, and relevant information don't worry about the file's content should be passed to doSomething so that it knows what to print.
Right now I'm just keep messing up on how to put these two methods together, please give some advices
this is my main class:
public class Yolo {
public static void frodo() throws Exception {
int n; // number of keywords
Scanner sc = new Scanner(System.in);
System.out.println("number of keywords : ");
n = sc.nextInt();
for (int j = 0; j <= n; j++) {
Scanner scan = new Scanner(System.in);
System.out.println("give the testword : ");
String testWord = scan.next();
System.out.println(testWord);
File document = new File("path//to//doc1.txt");
boolean check = true;
try {
FileInputStream fstream = new FileInputStream(document);
BufferedReader br = new BufferedReader(new InputStreamReader(fstream));
String strLine;
strLine = br.readLine();
// Read File Line By Line
int count = 0;
while ((strLine = br.readLine()) != null) {
// check to see whether testWord occurs at least once in the
// line of text
check = strLine.toLowerCase().contains(testWord.toLowerCase());
if (check) {
// get the line
String[] lineWords = strLine.split("\\s+");
// System.out.println(strLine);
count++;
}
}
System.out.println(testWord + "frequency: " + count);
br.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
}

The code below gives you this output:
Professor frequency: 54
engineering frequency: 188
data frequency: 2
mining frequency: 2
research frequency: 9
Though this is only for doc1, you've to add a loop to iterate on all the 5 documents.
public class yolo {
public static void frodo() throws Exception {
String[] keywords = { "Professor" , "engineering" , "data" , "mining" , "research"};
for(int i=0; i< keywords.length; i++){
String testWord = keywords[i];
File document = new File("path//to//doc1.txt");
boolean check = true;
try {
FileInputStream fstream = new FileInputStream(document);
BufferedReader br = new BufferedReader(new InputStreamReader(fstream));
String strLine;
strLine = br.readLine();
// Read File Line By Line
int count = 0;
while ((strLine = br.readLine()) != null) {
// check to see whether testWord occurs at least once in the
// line of text
check = strLine.toLowerCase().contains(testWord.toLowerCase());
if (check) {
// get the line
String[] lineWords = strLine.split("\\s+");
count++;
}
}
System.out.println(testWord + "frequency: " + count);
br.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
hope this helps!

java read integers from text file

I am using eclipse ; I need to read integers from a text file that may have many lines of numbers separated by space : 71 57 99 ...
I need to get these numbers as 71 and 57 ...but my code produces numbers in the range 10 to 57
int size = 0;
int[] spect = null;
try {
InputStream is = this.getClass().getResourceAsStream("/dataset.txt");
size = is.available();
spect = new int[size];
for (int si = 0; si < size; si++) {
spect[si] = (int) is.read();// System.out.print((char)is.read() + " ");
}
is.close();
} catch (IOException e) {
System.out.print(e.getMessage());
}

read() reads single byte and then you are converting into int value, you need to read line by line using BufferedReader and then split() and Integer.parseInt()

Have you considered using a Scanner to do this? Scanner can take the name of the file as the parameter and can easily read out each individual number.
InputStream is = this.getClass().getResourceAsStream("/dataset.txt");
int[] spect = new int[is.available()];
Scanner fileScanner = new Scanner("/dataset.txt");
for(int i = 0; fileScanner.hasNextInt(); i++){
spect[i] = fileScanner.nextInt();
}

You may convert it to a BufferedReader and read and split the lines.
BufferedReader br = new BufferedReader(new InputStreamReader(is));
String line;
while((line = br.readLine()) != null) {
String[] strings = line.split(" ");
for (String str : strings) {
Integer foo = Integer.parseInt(str);
//do what you need with the Integer
}
}

Importing data from a large file into a 2d Array

I am trying to import a large data file and insert the information into a 2D array. The file has around 19,000 lines, and consists of 5 columns. My code is absolutely correct, there are no run time errors nor exceptions. Though, the problem is that when I try to print out data[15000][0], it says null. but my line does have 15,000 lines and it should print out the element inside the array. But when I print out data[5000][0], it works. What could possibly be wrong? I have 19,000 cities in 19,000 different lines, but it seems like when It goes around 10,000+ nothing gets stored in the 2d array. Help please
import java.io.BufferedReader;
import java.io.FileReader;
import java.io.FileNotFoundException;
import java.io.IOException;
public class Data1
{
public static void main(String[] args)
{
try{
FileReader file = new FileReader("/Users/admin/Desktop/population.csv");
BufferedReader in = new BufferedReader(file);
String title = in.readLine();
String[][] data = new String[20000][5];
int currentRow = 0;
String current;
int i = 0;
String temp;
while ((temp = in.readLine()) !=null)
{
String[] c = new String[5];
String line = in.readLine().replaceAll("\"", ""); //changing the format of the data input
c = line.split(",");
c[1] = c[1].replace(" ", "");
for (int j = 0; j <data[0].length; j++)
{
current = c[j];
data[i][j] = c[j];
}
i++;
}
System.out.println(data[15000][0]);
}
catch (FileNotFoundException ex)
{
ex.printStackTrace();
}
catch (IOException ex)
{
ex.printStackTrace();
}
catch (Exception ex)
{
ex.printStackTrace();
}
}
}

You're throwing away a line on each loop.
while (in.readLine() != null)
should be
String temp;
while ((temp = in.readLine()) != null)
And then no calls to .readLine() inside the loop but refer to "temp".

Read line only once...
String line=null;
while ((line=in.readLine()) !=null) // reading line once here
{
String[] c = new String[5];
line = line.replaceAll("\"", ""); //
c = line.split(",");
c[1] = c[1].replace(" ", "");

One of your errors are the loops
while (in.readLine() !=null)
{
String[] c = new String[5];
String line = in.readLine().replaceAll("\"", ""); //changing the format of the data input
c = line.split(",");
c[1] = c[1].replace(" ", "");
Each time you invoke in.readLine() it reads a line,so you are skipping one line each time since you are calling readline twice(thus reading two lines) but storing only the second line.
You should replace it with.
String line=in.readLine();
while (line !=null)
{
String[] c = new String[5];
line.replaceAll("\"", ""); //changing the format of the data input
c = line.split(",");
c[1] = c[1].replace(" ", "");
//whatever code you have
//last line of the loop
line=in.readLine();
Can you provide us with a couple of lines of your file? And are you sure that all the file is formatted correctly ?

How to assign a text file's lines of integers into 2D tables in Java?

I have the sample.txt which contains 100 Integers (range 0-9) in every line formatted like this:
9 2 0 3 4 1 0 7 5 3 7 8 6 2 0 1 4 4 5 9 0 3 2 1 7 (etc... 100 numbers)
I want to scan the file and put every line into a 10x10 table. So:
public void loadTableFromFile(String filepath){
try (Scanner s = new Scanner(new BufferedReader(new FileReader(filepath)))) {
String line;
while (s.hasNextLine()) {
// WHAT HERE? THIS BLOCK DOES NOT WORK
/* if (s.hasNextInt()) {
//take int and put it in the table in the right position procedure
} else {
s.next();
} */
// END OF NOT WORKING BLOCK
}
} catch (FileNotFoundException e){
}
}

How about something like this?
public void loadTableFromFile(String filepath) {
Scanner s = null; // Our scanner.
try {
s = new Scanner(new BufferedReader(
new FileReader(filepath))); // get it from the file.
String line;
while (s.hasNextLine()) { // while we have lines.
line = s.nextLine(); // get a line.
StringTokenizer st = new StringTokenizer(line, " ");
int i = 0;
while (st.hasMoreTokens()) {
if (i != 0) {
System.out.print(' '); // add a space between elements.
}
System.out.print(st.nextToken().trim()); // print the next element.
i++;
if (i % 10 == 0) { // Add a new line every ten elements.
System.out.println();
}
}
System.out.println(); // between lines.
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} finally {
if (s != null)
s.close();
}
}

Here is a solution that reads the line of the file into an array of strings using the split by whitespace method, and then reads them in using a for loop. I threw any exceptions that might have occurred in the method declaration, alternatively, use the try catch loop as above (might be better design, not sure about that.)
public void loadTableFromFile(String filePath) throws IOException {
BufferedReader br = new BufferedReader(new FileReader(filePath));
String[] line = br.readLine().split(" ");
br.close(); // file only has 1 line with 100 integers
int[][] mydata = new int[10][10];
for(int i = 0; i < line.length; i++) {
mydata[i % 10][(int) (i / 10)] = Integer.parseInt(line[i]);
}
}
Now, if the file has more than one line, you could instead read the entire file line by line, and then use the above idea like this:
BufferedReader br = new BufferedReader(new FileReader(filePath));
String line1;
while((line1 = br.readLine()) != null) {
String[] line = line1.split(" ");
... // do above stuff of reading in 1 line here
}
br.close();

Try,
try (Scanner s = new Scanner(new BufferedReader(new FileReader(filepath)))) {
String line;
while (s.hasNextLine()) {
String[] strArr=line.split(" ");
for(int i=0;i<strArr.length;i++){
System.out.print(" "+strArr[i]);
if((i+1)%10==0){
System.out.println();
}
}
}
} catch (FileNotFoundException e){
}