We Keep Coding

Java "Quicksave" Execution

The problem I seem to have hit is one relating to loading times; I'm not running on a particularly fast machine by any means, but I still want to dabble into neural networks. In short, I have to load 336,600,000 integers into one large array (I'm using the MNIST database; each image is 28x28, which amounts to 748 pixels per image, times 45,000 images). It works fine, and surprisingly I don't run out of RAM, but... it takes 4 and a half hours, just to get the data into an array.
I can supply the rest of the code if you want me to, but here's the function that runs through the file.
public static short[][] readfile(String fileName) throws FileNotFoundException, IOException {
short[][] array = new short[10000][784];
BufferedReader br = new BufferedReader(new FileReader(System.getProperty("user.dir") + "/MNIST/" + fileName + ".csv"));
br.readLine();
try {
for (short i = 1; i < 45000; i++) {
String line = br.readLine();
for (short j = 0; j < 784; j++) {
array[i][j] = Short.parseShort(line.split(",")[j]);
}
}
br.close();
} catch (IOException e) {
e.printStackTrace();
}
return array;
}
What I want to know is, is there some way to "quicksave" the execution of the program so that I don't have to rebuild the array for every small tweak?
Note: I haven't touched Java in a while, and my code is mostly chunked together from a lot of different sources. I wouldn't be surprised if there were some serious errors (or just Java "no-nos"), it would actually help me a lot if you could fix them if you answer.
Edit: Bad question, I'm just blind... sorry for wasting time
Edit 2: I've decided after a while that instead of loading all of the images, and then training with them one by one, I could simply train one by one and load the next. Thank you all for your ideas!

array[i][j] = Short.parseShort(line.split(",")[j]);
You are calling String#split() for every single integer.
Call it once outside the loop and copy the value into your 2d array.

Java NumberFormatException stops working if caught

UPDATE: sorry for misleading some of you, here is more detailed code:
I'm trying to parse String[] array to int[] array. I take String from JTextArea, make String[] and then int[].
array1String="asd, lala, 22";
array2String="1,2,3";
String [] arr1 = array1String.split(",");
String [] arr2 = array2String.split(",");
int[] array1 = new int[arr1.length];
for (int i=0; i<arr1.length; i++) {
try {
array1[i] = Integer.parseInt(arr1[i]);
} catch (Exception ex) {
resultLabel.setText(ex.getMessage());
}
}
int[] array2 = new int[arr2.length];
for (int i=0; i<arr2.length; i++) {
try {
array2[i] = Integer.parseInt(arr2[i]);
} catch (Exception ex) {
resultLabel.setText(ex.getMessage());
}
}
the parseInt() method interprets any gibberish as a "0". So instead of getting an exception, I get every int[] member corresponding to gibberish String[] member as a ZERO. But that's not what I want. I want to catch the exception to pass the message to the JLabel. I don't want it to happily eat any nonsense and make it "0". Am i missing something? I suspect that is not how this is supposed to work.
result arrays look like the
array1 [0,0,22]
array2 [1,2,3].
And nothing goes to Label, as if exception never happened. By i didn't change the code, only handling the exception changed.

The code setting the text of the JLabel is missing, but I guess it's after the try-catch-block and will simply use the value of x, which is 0 by default. The preferable solution to this problem is to update the text inside the try-catch-block like this:
try{
int x = ...
resultLabel.setText("" + x);
}catch(Exception ex){
resultLabel.setText(ex.getMessage());
}
In general, if generating data might throw an exception is recommended to process the data inside the try-catch-block the data is produced in.
EDIT:
There's one simple problem with this code: int doesn't have any properties like an undefined value. The simplest approach to this would be to use an Integer[] instead and null as undefined-value:
Integer x;
try{
x = ...;
}catch(Excpetion ex){
resultLabel.setText(ex.getMessage());
x = null;
}

The default value for int is 0. Since you get an exception and nothing is assigned to x, you get 0

Firstly, if you are using int x = Integer.parseInt(str); in TRY block then it means your x becomes local variable of your TRY block and you cannot use it outside of TRY, not even in CATCH block. So, the in case of exception, question of x=0 will not even come because x would have lost its scope.
And if you are declaring x outside of TRY block (so that you can use it inside CATCH or outside TRY block) then you have initialize it (ofcourse with value as 0 or -1) then x=0 or x=-1 should be expected value of x
Your concern:
I want to catch the exception to pass the message to the JLabel.
Consider below approach, where you are catching the exception, computing the error message and then passing/setting into your label field etc.
Advantage of this approach as opposed to other is that you need to have duplicate code of resultLabel.setText(), you are updating your labelMessage with either success message or failure message and using it.
Please feel free to let me know if this doesn't completely address your concern.
String labelMessage = null;
String str = "asd";
try {
int x = Integer.parseInt(str);
labelMessage = //Whatever you want...
} catch (NumberFormatException e) {
e.printStackTrace();
labelMessage = "Error message = " + e.getMessage(); // OR some other menaingful message you want
}
//Set to you label field...

Java use a piece of code from a text file

I am currently trying to figure something out. For my world editor I want my program to read a text file and use its content as code material. I've already made a decent file reader but now I've got a problem. In the console I am getting the right output, the file has only one line that says:
this.makeGrass(new Vector3f(0, 1, 2));
this is actually part of a code that tells my program to render a specific object to the scene, in this case it's a grass model. However instead of just printing this information to the console with
System.out.println(aryLines[i]);
I want to be able to use the information stored on the .txt file so I can actually add it to my rendering code. The entire method that prints the lines on the text file to the console is:
public void TextOutput()
{
String file_name = "C:/Text.txt";
try
{
StoreCoords file = new StoreCoords(file_name);
String[] aryLines = file.OpenFile();
int i;
for (i = 0; i < aryLines.length; i++)
{
System.out.println(aryLines[i]);
// !! How to use the information as part of my code ??
}
} catch(IOException e)
{
System.out.println(e.getMessage());
}
}
I hope you understand what I want: The content of my text file is a piece of code that I want to use further instead of having it just print to the console, I'm sure this is possible but I wouldn' know how.

As Java is a compiled language, you'd have to recompile at runtime and I am not sure that is even possible. If I were you, I'd hardcode in my own commands. You want to call a function called makeGrass, hardcode it in. Maybe in your text file you can have this:
makeGrass:0,1,2
Then have this right after the println:
if(aryLines[i].startsWith("makeGrass:")) {
String Arguments = aryLines[i].substring(aryLines[i].indexOf(":")+1, aryLines[i].length());
ArgArray = Arguments.split(",");
this.makeGrass(new Vector3f(Double.parseDouble(ArgArray[0]), Double.parseDouble(ArgArray[1]), Double.parseDouble(ArgArray[2])));
}
I'm going to leave my answer like this, assuming you are an experienced programmer. If I am wrong feel free to ask and I will explain it to you. I can also explain how to modify it to add different commands if you want.
Also, this is rather unsafe because if the input is in the wrong format it will crash the app. If you plan on letting users edit the file, then I can show you how to add on safeties.
Hope this helped,
Joseph Meadows

Okay, thanks to Joseph Meadows for the hint, I'm doing the following thing, right after the println statement I've added the code provided by him. To make ArgArray work I had to put String[] before it and also I had to create a new constructor in my Vector3f class to match the Double.parseDouble thingy..
public void TextOutput()
{
String file_name = "C:/Users/Server/Desktop/textText.txt";
try
{
StoreCoords file = new StoreCoords(file_name);
String[] aryLines = file.OpenFile();
int i;
for (i = 0; i < aryLines.length; i++)
{
System.out.println(aryLines[i]);
if(aryLines[i].startsWith("makeGrass:")) {
String Arguments = aryLines[i].substring(aryLines[i].indexOf(":")+1, aryLines[i].length());
String[] ArgArray = Arguments.split(",");
this.makeGrass(new Vector3f(Double.parseDouble(ArgArray[0]),
Double.parseDouble(ArgArray[1]),
Double.parseDouble(ArgArray[2])));
}
}
} catch(IOException e)
{
System.out.println(e.getMessage());
}
}
my original Vector3f constructor is:
public Vector3f(float x, float y, float z)
{
this.m_x = x;
this.m_y = y;
this.m_z = z;
}
and to make the code in the TextOutput method work I've added another constructor right below the original one..
public Vector3f(double parseDouble, double parseDouble2, double parseDouble3) {
this.m_x = (float) parseDouble;
this.m_y = (float) parseDouble2;
this.m_z = (float) parseDouble3;
}
Now everything works great, the console gives me the apropriate statement
makeGrass:0,1,2
and the rendering system creates the grass model at the apropriate coordinates, the only thing I want to change now is that I don't have to add an additional constructor to the Vector3f class, I'm sure I'll figure that out too.
In the picture provided in this link you can see exactly what's going on:
http://www.pic-upload.de/view-27720774/makeGrassEx.png.html
As you can see, the content of the text file is printed out in the console (the numbers below is the fps counter) and the coordinates provided by the text file are interpreted correctly, two grass models being displayed at the respective coordinates which is exactly what I wanted!
Thanks again for your help Joseph Meadows, this is exactly what I was looking for!

I am not sure if you solved this yet, but you did not need the second constructor. I was unsure of the data type you were using for the coordinates, and I assumed you use doubles because that is what I have grown accustomed to using.
In actuality, all types can be parsed from a string. Look here:
this.makeGrass(new Vector3f(Double.parseDouble(ArgArray[0]),
Double.parseDouble(ArgArray[1]),
Double.parseDouble(ArgArray[2])));
This right now is turning the string into a double. That is what
Double.parseDouble();
does.
It looks like you are using floats though, so you can always just use the float parsing method:
Float.parseFloat("String");
That would result with this:
this.makeGrass(new Vector3f(Float.parseFloat(ArgArray[0]),
Float.parseFloat(ArgArray[1]),
Float.parseFloat(ArgArray[2])));
Sorry for the late response, and you are surely welcome for the help. I just love being useful!

How to use CSVReader?

I'm trying to read my file in notepad using CSVReader but I cannot get it work. it says ArrayIndexOutOfBoundsException: 2 line "y[i][2]". eventhough I intiallize my array rows in 100. I'm currently new in using Java and CSVReader.
public static void main(String[] args) throws IOException {
double[][] x = new double[100][3];
double[][] y = new double[100][1];
String line[];
try{
CSVReader br=new CSVReader(new FileReader("ex2data2.txt"),',');
int i = 0;
while((line=br.readNext())!=null){
x[i][0] = Double.parseDouble(line[0]);
x[i][1] = Double.parseDouble(line[1]);
y[i][2] = Double.parseDouble(line[2]);
i++;
}
br.close();
}
catch (IOException e) {
e.printStackTrace();
}

It's completely unclear as to why you would be storing the data as you are after reading it from the file, but ...
double[][] y = new double[100][1];
This allocates an array of 100 double arrays, each with a length of 1
Here:
y[i][2] = Double.parseDouble(line[2]);
You attempt to store something at the third element of one of those 100 arrays. They aren't that large; you created them to have a length of one.
I suspect you meant to do:
y[i][0] = Double.parseDouble(line[2]);
since the only thing you're storing in the y arrays is that single value.
All that being said, this is a poor way to store these values. In general you are better served using a dynamic data structure so you don't have to worry about what the length (number of lines) of the file is. In addition, why would you need two different 2D arrays? Even a List<Double[]>, for example, would be better.

You have create
double[][] y = new double[100][1];
i.e. 100 rows and 1 column. but trying to put value at position y[i][2]. thats why you are getting ArrayIndexOutOfBoundsException. create like
double[][] y = new double[100][3];
or you can simply put value as (in this case you don't need to create 2D array given as above)
y[i][0] = Double.parseDouble(line[2]);

Reading Objects until End of File in java

I'm trying to write a program where the user can: 1) Add a person to the contact (name, phone, email), 2) Remove a person from the contacts, 3) Read all from contact.
The Way I'm doing this is I'm asking for the user for their choice and respectively does whatever. For writing, I simply write an object to the file. For removing, I think I'll be asking the user for "last name" which will be used as the KEY (since I'm using a TreeMap)and will remove the value (object) at the key.
So I'm having a problem with reading here. I'm trying to read the object like so:
public void readContact()
{
TreeMap<String, Contact> contactMap = new TreeMap<String, Contact>();
try
{
ObjectInputStream in = new ObjectInputStream(new BufferedInputStream(
new FileInputStream(file)));
while( in.available() > 0 ) //This line does NOT read
{
Contact c = (Contact)in.readObject();
contactMap.put(c.getLastName(), c);
}
for(Map.Entry contact : contactMap.entrySet() )
{
Contact con = contactMap.get( contact.getKey() );
System.out.println( con.getLastName() + ", " + con.getFirstName() + ": " + con.getPhoneNumber() + "\t" + con.getEmail());
}
}
catch(Exception e)
{
System.out.println("Exception caught");
}
}
Please do not suggest doing something like while(true) until I get the EOFException because:
that isn't what exception handling is for I believe
I still have more things to do after this so I can't have the program terminating'

Please do not suggest doing something like while(true) until I get the EOFException
That is exactly what I suggest. When you are searching for answers it is counter-productive to circumscribe the solution space according to arbitrary criteria like this.
because:
that isn't what exception handling is for I believe
When an API that you are calling throws an exception, as this one does, you don't have any choice but to catch it. Whatever you may think about 'what exception handling is for', you are subject to what the designers of the API thought when they designed the API.
I still have more things to do after this so I can't have the program terminating'
So don't terminate it. Catch EOFException, close the input, and break out of the loop.
I have seen more costly programming time wasted over 'what exception handling is for' than I can really credit.

I know that you are looking for an answer that is not using exception handling, but I believe in this case using EOFException to determine when all input has been read is the right way.
The JavaDoc of EOFException states that
This exception is mainly used by data input streams to signal end of stream. Note that many other input operations return a special value on end of stream rather than throwing an exception.
So, there are input streams that use other means to signal an end of file, but ObjectInputStream#readObject uses ObjectInputStream$BlockDataInputStream#peekByte to determine if there is more data to read, and peekByte throws an EOFException when the end of the stream has been reached.
So it is feasible to use this exception as an indicator that the end of the file has been reached.
To handle the exceptions without interrupting the program flow, some of the possible exceptions should be passed up in the hierarchy. They can be handled by a try - catch block in the code that calls readContact().
The EOFException can simply be used as an indicator that we are done reading the objects.
public TreeMap<String, Contact> readContact() throws FileNotFoundException,
IOException, ClassNotFoundException {
TreeMap<String, Contact> contactMap = new TreeMap<String, Contact>();
// The following call can throw a FileNotFoundException or an IOException.
// Since this is probably better dealt with in the calling function,
// readContact is made to throw these exceptions instead.
ObjectInputStream in = new ObjectInputStream(new BufferedInputStream(
new FileInputStream(file)));
while (true) {
try {
// Read the next object from the stream. If there is none, the
// EOFException will be thrown.
// This call might also throw a ClassNotFoundException, which can be passed
// up or handled here.
Contact c = (Contact) in.readObject();
contactMap.put(c.getLastName(), c);
for (Map.Entry<String, Contact> contact : contactMap.entrySet()) {
Contact con = contact.getValue();
System.out.println(con.getLastName() + ", "
+ con.getFirstName() + ": " + con.getPhoneNumber()
+ "\t" + con.getEmail());
}
} catch (EOFException e) {
// If there are no more objects to read, return what we have.
return contactMap;
} finally {
// Close the stream.
in.close();
}
}
}

- Exceptions are not only used in order to raise an alarm when something goes wrong while calling a method, but are also used in Threads and IO with various other uses.
- You can use Exception to indicate end of the file.
- Use the try-catch combo to work along with the above to keep the flow of the program smooth.

Why so much trouble while reading object from file, just save a hash map into a file and read the same once from file then perform any operation.
Also I would suggest to use any one of object oriented database like Db4o to do this quickly then you never worry about end of file exception

'ObjectInputStream.available returns 0' is a known problem, see http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4954570, and since we cannot use it I think EOFException would be a reasonable approach in your situation. Catching EOFExcepion will not terminate your program.
You could write the number of objects to your file with ObjectOutputStream.writeInt, and then you would read this number with ObjectInputStream.readInt and know how many objects to read
You could use null as EOF marker.
You could save your objects as an array or List or even Map and then read them with one readObject.

What you have discovered
You found out about FileInputStream.available() returning 0 even though there are unread bytes in the file! Why is this? This could happen (rendering FileInputStream.available() unreliable for a couple of reasons:
According to this documentation, FileInputStream.available() approximates the number of bytes that can be read without blocking
The hard disk drive itself might change its operation mid-read (go from spinning to not spinning)
The file you are trying to access is either a file on a remote system or a device file : May the FileInputStream.available foolish me?
The FileInputStream might be blocked
Some alternative way
As an alternative to relying on EOFException to close the file, you could use a (very-small!) binary file that keeps track of the number of Objects in your file. (Judging from your code, it looks like you are simply writing Objects to the file.) The way I have used this is just to
store the number of bytes the number itself will consume
using that number of bytes, store the number itself
For example, the first time the serialization file is created, I could make the binary file store 1 1 (which specifies that the number of Objects in the serialization file takes up 1 byte, and that number is 1). This way, after 255 Objects (remember, an unsigned byte can only store up to 28-1 == 255), if I write another Object (Object number 256 up to 2562-1 == 65535), the binary file will have, as contents, 2 1 0, which specifies that the number takes up 2 bytes and is 1*2561+0 == 256. Provided that the serialization is reliable (good luck on ensuring that: http://www.ibm.com/developerworks/library/j-serialtest/index.html), this method will let you store (and detect) up to 256255-1 bytes (which pretty much means that this method works indefinitely).
The code itself
How something like that would be implemented would be something like:
ObjectOutputStream yourOutputStream = new ObjectOutputStream(new FileOutputStream(workingDirectory + File.separatorChar + yourFileName); //The outputStream
File binaryFile = new File(workingDirectory + File.separatorChar + nameOfFile); //the binary file
int numOfObjects = 0, numOfBytes; //The number of Objects in the file
//reading the number of Objects from the file (if the file exists)
try
{
FileInputStream byteReader = new FileInputStream(binaryFile);
numOfBytes = byteReader.read();
//Read the rest of the bytes (the number itself)
for (int exponent = numOfBytes; exponent >= 0; exponent--)
{
numOfObjects += byteReader.read() * Math.pow(256,exponent);
}
}
catch (IOException exception)
{
//if an exception was thrown due to the file not existing
if (exception.getClass() == FileNotFoundException.class)
{
//we simply create the file (as mentioned earlier in this answer)
try
{
FileOutputStream fileCreator = new FileOutputStream(binaryFile);
//we write the integers '1','1' to the file
for (int x = 0; x < 2; x++) fileCreator.write(1);
//attempt to close the file
fileCreator.close();
}
catch (IOException innerException)
{
//here, we need to handle this; something went wrong
innerException.printStackTrace();
System.exit(-1);
}
}
else
{
exception.printStackTrace();
System.exit(-2);
}
}
Now, we have the number of Objects in the file (I leave it to you to figure out how to update the bytes to indicate that one more Object has been written when yourOutputStream calls writeObject(yourObject);; I have to go clock in.)
Edit: yourOutputStream is either going to write over all the data in the binaryFile or append data to it. I just found out that RandomAccessFile would be a way to insert data into anywhere in the file. Again, I leave details to you. However you want to do it.

You can write the last object as null.
And then iterate till you get a null at the reading side.
e.g.
while ((object = inputStream.readObject()) != null) {
// ...
}