We Keep Coding

Why does Scanner read every other line of CSV file? Java

I am reading in a CSV file and putting each delimited element into a two-dimensional array. The code looks like this:
public DataProcess(String filename, String[][] contents, int n) {//n is 6 for contents, 5 for fiveMinContents
Scanner fileReader = null;
try {
fileReader = new Scanner(new File(filename));
} catch (FileNotFoundException ex) {
System.out.println(ex + " FILE NOT FOUND ");
}
fileReader.useDelimiter(",");
int rowIndex = 0;
while (fileReader.hasNext()) {
for (int j = 0; j < n; j++) {
contents[rowIndex][j] = fileReader.next();
System.out.println("At (" + rowIndex +", "+j+"): " +
contents[rowIndex][j]);
}
rowIndex++;
fileReader.nextLine();
}
}
I am not sure why it reads every other line of this particular CSV file because this is file 2/2 that is being read in this manner. The first one reads fine, but now this one skips every other line. Why would it work for one but not the other? I am running this on Eclipse's latest update.
I also checked out this answer and it did not help.

Because the last line of your loop reads a line and discards it. You need something like,
while (fileReader.hasNextLine()) {
String line = fileReader.nextLine();
contents[rowIndex] = fileReader.split(",\\s*");
System.out.println("At (" + rowIndex + "): "
+ Arrays.toString(contents[rowIndex]));
rowIndex++;
}
You could also print the multi-dimensional array with one call like
System.out.println(Arrays.deepToString(contents));

While the approach may work for you, it's not optimal. There are premade CSV readers for Java. One example is commons-csv:
Reader in = new FileReader("path/to/file.csv");
Iterable<CSVRecord> records = CSVFormat.EXCEL.parse(in);
for (CSVRecord record : records) {
String date = record.get(1);
String time = record.get(2);
// and so on, so forth
}
There are a small number of dependencies that have to be on your classpath. Hope that helps.

I found the issue to this problem.
First, I recommend using the external library that was suggested.
The issue was that since this second file was reading the entire row, whereas the first CSV file was reading what I wanted it to, but there was a column at the end of the file that I was ignoring. There must be a way that a CSV file is structured where the end of a row has a different delimiter or something along those lines--not sure. To fix this issue, I just added an extra column to the second file and I am not reading it in; it is just there.
In short, use an external CSV-reader library. If you don't want to do that, then just add a column directly after the last column in the file and do not read it.

How to read files with an offset from Hadoop using Java

Problem: I want to read a section of a file from HDFS and return it, such as lines 101-120 from a file of 1000 lines.
I don't want to use seek because I have read that it is expensive.
I have log files which I am using PIG to process down into meaningful sets of data. I've been writing an API to return the data for consumption and display by a front end. Those processed data sets can be large enough that I don't want to read the entire file out of Hadoop in one slurp to save wire time and bandwidth. (Let's say 5 - 10MB)
Currently I am using a BufferedReader to return small summary files which is working fine
ArrayList lines = new ArrayList();
...
for (FileStatus item: items) {
// ignoring files like _SUCCESS
if(item.getPath().getName().startsWith("_")) {
continue;
}
in = fs.open(item.getPath());
BufferedReader br = new BufferedReader(new InputStreamReader(in));
String line;
line = br.readLine();
while (line != null) {
line = line.replaceAll("(\\r|\\n)", "");
lines.add(line.split("\t"));
line = br.readLine();
}
}
I've poked around the interwebs quite a bit as well as Stack but haven't found exactly what I need.
Perhaps this is completely the wrong way to go about doing it and I need a completely separate set of code and different functions to manage this. Open to any suggestions.
Thanks!
As added noted based on research from the below discussions:
How does Hadoop process records records split across block boundaries?
Hadoop FileSplit Reading

I think SEEK is a best option for reading files with huge volumes. It did not cause any problems to me as the volume of data that i was reading was in the range of 2 - 3GB. I did not encounter any issues till today but we did use file splitting to handle the large data set. below is the code which you can use for reading purpose and test the same.
public class HDFSClientTesting {
/**
* #param args
*/
public static void main(String[] args) {
// TODO Auto-generated method stub
try{
//System.loadLibrary("libhadoop.so");
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
conf.addResource(new Path("core-site.xml"));
String Filename = "/dir/00000027";
long ByteOffset = 3185041;
SequenceFile.Reader rdr = new SequenceFile.Reader(fs, new Path(Filename), conf);
Text key = new Text();
Text value = new Text();
rdr.seek(ByteOffset);
rdr.next(key,value);
//Plain text
JSONObject jso = new JSONObject(value.toString());
String content = jso.getString("body");
System.out.println("\n\n\n" + content + "\n\n\n");
File file =new File("test.gz");
file.createNewFile();
}
catch (Exception e ){
throw new RuntimeException(e);
}
finally{
}
}
}

How to access values of a line, while reading in a text file in Java

I am trying to load in two files at the same time but also access the first gps1 file. I want to access the gps1 file line-by-line and depending on the sentence type which I will explain later I want to do different stuff with that line and then move to the next line.
Basically gps1 for example has multiple lines but each line falls under a couple of catagories all starting with $GPS(then other characters). Some of these types have a time stamp which I need to collect and some types do not have a time stamp.
File gps1File = new File(gpsFile1);
File gps2File = new File(gpsFile2);
FileReader filegps1 = new FileReader(gpsFile1);
FileReader filegps2 = new FileReader(gpsFile2);
BufferedReader buffer1 = new BufferedReader(filegps1);
BufferedReader buffer2 = new BufferedReader(filegps2);
String gps1;
String gps2;
while ((gps1 = buffer1.readLine()) != null) {
The gps1 data file is as follows
$GPGSA,A,3,28,09,26,15,08,05,21,24,07,,,,1.6,1.0,1.3*3A
$GPRMC,151018.000,A,5225.9627,N,00401.1624,W,0.11,104.71,210214,,*14
$GPGGA,151019.000,5225.9627,N,00401.1624,W,1,09,1.0,38.9,M,51.1,M,,0000*72
$GPGSA,A,3,28,09,26,15,08,05,21,24,07,,,,1.6,1.0,1.3*3A
Thanks

I don't really understand the problem you are facing but anyway, if you want to get your lines content you can use a StringTokenizer
StringTokenizer st = new StringTokenizer(gps1, ",");
And then access the data one by one
while(st.hasMoreToken)
String s = st.nextToken();
EDIT:
NB: the first token will be your "$GPXXX" attribute

Read the five score in a textfile then print them back with the new score added to the list - ANDROID

In my program when the player submits a score it gets added to a local text file called localHighScores. This is list of the top five score the player has achieved while on that specific device.
I wasn't sure how to write to a new line using FileOutputStream (if you know please share), so instead I've inputted a space in between each score. Therefore what I am trying to do is when the player clicks submit the program will open the file and read any current data is saved. It will save it to an String Array, each element being one of the five score in the text file and when it hits a 'space' in the fie it will add the score just read to the write array element
The code I currently have is as follows:
String space = " ";
String currentScoreSaved;
String[] score = new String[5];
int i = 0;
try
{
BufferedReader inputReader = new BufferedReader(new InputStreamReader(openFileInput("localHighScore.txt")));
String inputString;StringBuffer stringBuffer = new StringBuffer();
while ((inputString = inputReader.readLine()) != null && i < 6)
{
if((inputString = inputReader.readLine()) != space)
{
stringBuffer.append(inputString + "\n");
i++;
score[i] = stringBuffer.toString();
}
}
currentScoreSaved = stringBuffer.toString();
FileOutputStream fos = openFileOutput("localHighScore.txt", Context.MODE_PRIVATE);
while (i < 6)
{
i++;
fos.write(score[i].getBytes());
fos.write(space.getBytes());
}
fos.write(localHighScore.getBytes());
//fos.newLine(); //I thought this was how you did a new line but sadly I was mistaken
fos.close();
}
catch(Exception e)
{
e.printStackTrace();
}
Now you will notice this doesn't re arrange the score if a new highscore is achieved. That I am planning on doing next. For the moment I am just trying to get the program to do the main thing which is read in the current data, stick it in an Array then print it back to that file along with the new score
Any Ideas how this might work, as currently it's printing out nothing even when I had score in the textfile before hand

I'm only a first year student in Java programming and I am a new user here at stackoverflow.com, so pardon me if coding for android has some special rules I don't know about, which prevents this simple and humble example from working. But here is how I would read from a file in the simplest of ways.
File tempFile = new File("<SubdirectoryIfAny/name_of_file.txt");
Scanner readFile = new Scanner( tempFile );
// Assuming that you can structure the file as you please with fx each bit of info
// on a new line.
int counter = 0;
while ( readFile.hasNextLine() ) {
score[counter] = readFile.nextLine();
counter++;
}
As for the writing back to the file? Put it in an entirely different method and simply make a simplified toString-like method, that prints out all the values the exact way you want them in the file, then create a "loadToFile" like method and use the to string method to print back into the file with a printstream, something like below.
File tempFile = new File("<SubdirectoryIfAny/name_of_file.txt");
PrintStream write = new PrintStream(tempFile);
// specify code for your particular program so that the toString method gets the
// info from the string array or something like that.
write.print( <objectName/this>.toStringLikeMethod() );
// remember the /n /n in the toStringLikeMethod so it prints properly in the file.
Again if this is something you already know, which is just not possible in this context please ignore me, but if not I hope it was useful. As for the exceptions, you can figure that you yourself. ;)

Since you are a beginner, and I assume you are trying to get things off the ground as quickly as possible, I'd recommend using SharedPreferences. Basically it is just a huge persistent map for you to use! Having said that... you should really learn about all the ways of storage in Android, so check out this document:
http://developer.android.com/guide/topics/data/data-storage.html
The Android docs are awesome! FYI SharedPreferences may not be the best and awesomest way to do this... but I'm all for quick prototyping as a learner. If you want, write a wrapper class around SharedPreferences.

Java - Load file, replace string, save

I have a program that loads lines from a user file, then selects the last part of the String (which would be an int)
Here's the style it's saved in:
nameOfValue = 0
nameOfValue2 = 0
and so on. I have selected the value for sure - I debugged it by printing. I just can't seem to save it back in.
if(nameOfValue.equals(type)) {
System.out.println(nameOfValue+" equals "+type);
value.replace(value, Integer.toString(Integer.parseInt(value)+1));
}
How would I resave it? I've tried bufferedwriter but it just erases everything in the file.

My suggestion is, save all the contents of the original file (either in memory or in a temporary file; I'll do it in memory) and then write it again, including the modifications. I believe this would work:
public static void replaceSelected(File file, String type) throws IOException {
// we need to store all the lines
List<String> lines = new ArrayList<String>();
// first, read the file and store the changes
BufferedReader in = new BufferedReader(new FileReader(file));
String line = in.readLine();
while (line != null) {
if (line.startsWith(type)) {
String sValue = line.substring(line.indexOf('=')+1).trim();
int nValue = Integer.parseInt(sValue);
line = type + " = " + (nValue+1);
}
lines.add(line);
line = in.readLine();
}
in.close();
// now, write the file again with the changes
PrintWriter out = new PrintWriter(file);
for (String l : lines)
out.println(l);
out.close();
}
And you'd call the method like this, providing the File you want to modify and the name of the value you want to select:
replaceSelected(new File("test.txt"), "nameOfValue2");

I think most convenient way is:
Read text file line by line using BufferedReader
For each line find the int part using regular expression and replace
it with your new value.
Create a new file with the newly created text lines.
Delete source file and rename your new created file.
Please let me know if you need the Java program implemented above algorithm.

Hard to answer without the complete code...
Is value a string ? If so the replace will create a new string but you are not saving this string anywhere. Remember Strings in Java are immutable.
You say you use a BufferedWriter, did you flush and close it ? This is often a cause of values mysteriously disappearing when they should be there. This exactly why Java has a finally keyword.
Also difficult to answer without more details on your problem, what exactly are you trying to acheive ? There may be simpler ways to do this that are already there.