Currently, I am having problems with the Backpropagation algorithm.
I am trying to implement it and use it to recognize the direction of faces (left, right, down, straight).
Basically, I have N images, read the pixels and change its values(0 to 255) to values from 0.0 to 1.0. All images are 32*30.
I have an input layer of 960 neurons, a hidden layer of 3 neurons and an output layer of 4 neurons. For example, the output <0.1,0.9,0.1,0.1> means that the person looks to the right.
I followed the pseudy-code. However, it doesn't work right - it does not compute the correct weights and consequently it can't handle the training and test examples.
Here are parts of the code:
// main function - it runs the algorithm
private void runBackpropagationAlgorithm() {
for (int i = 0; i < 900; ++i) {
for (ImageUnit iu : images) {
double [] error = calcOutputError(iu.getRatioMatrix(), iu.getClassification());
changeHiddenUnitsOutWeights(error);
error = calcHiddenError(error);
changeHiddenUnitsInWeights(error,iu.getRatioMatrix());
}
}
}
// it creates the neural network
private void createNeuroneNetwork() {
Random generator = new Random();
for (int i = 0; i < inHiddenUnitsWeights.length; ++i) {
for (int j = 0; j < hiddenUnits; ++j) {
inHiddenUnitsWeights[i][j] = generator.nextDouble();
}
}
for (int i = 0; i < hiddenUnits; ++i) {
for (int j = 0; j < 4; ++j) {
outHddenUnitsWeights[i][j] = generator.nextDouble();
}
}
}
// Calculates the error in the network. It runs through the whole network.
private double [] calcOutputError(double[][] input, double [] expectedOutput) {
int currentEdge = 0;
Arrays.fill(hiddenUnitNodeValue, 0.0);
for (int i = 0; i < input.length; ++i) {
for (int j = 0; j < input[0].length; ++j) {
for (int k = 0; k < hiddenUnits; ++k) {
hiddenUnitNodeValue[k] += input[i][j] * inHiddenUnitsWeights[currentEdge][k];
}
++currentEdge;
}
}
double[] out = new double[4];
for (int j = 0; j < 4; ++j) {
for (int i = 0; i < hiddenUnits; ++i) {
out[j] += outHddenUnitsWeights[i][j] * hiddenUnitNodeValue[i];
}
}
double [] error = new double [4];
Arrays.fill(error, 4);
for (int i = 0; i < 4; ++i) {
error[i] = ((expectedOutput[i] - out[i])*(1.0-out[i])*out[i]);
//System.out.println((expectedOutput[i] - out[i]) + " " + expectedOutput[i] + " " + out[i]);
}
return error;
}
// Changes the weights of the outgoing edges of the hidden neurons
private void changeHiddenUnitsOutWeights(double [] error) {
for (int i = 0; i < hiddenUnits; ++i) {
for (int j = 0; j < 4; ++j) {
outHddenUnitsWeights[i][j] += learningRate*error[j]*hiddenUnitNodeValue[i];
}
}
}
// goes back to the hidden units to calculate their error.
private double [] calcHiddenError(double [] outputError) {
double [] error = new double[hiddenUnits];
for (int i = 0; i < hiddenUnits; ++i) {
double currentHiddenUnitErrorSum = 0.0;
for (int j = 0; j < 4; ++j) {
currentHiddenUnitErrorSum += outputError[j]*outHddenUnitsWeights[i][j];
}
error[i] = hiddenUnitNodeValue[i] * (1.0 - hiddenUnitNodeValue[i]) * currentHiddenUnitErrorSum;
}
return error;
}
// changes the weights of the incomming edges to the hidden neurons. input is the matrix of ratios
private void changeHiddenUnitsInWeights(double [] error, double[][] input) {
int currentEdge = 0;
for (int i = 0; i < input.length; ++i) {
for (int j = 0; j < input[0].length; ++j) {
for (int k = 0; k < hiddenUnits; ++k) {
inHiddenUnitsWeights[currentEdge][k] += learningRate*error[k]*input[i][j];
}
++currentEdge;
}
}
}
As the algorithm works, it computes bigger and bigger weights, which finally approach infinity (NaN values). I checked the code. Alas, I didn't manage to solve my problem.
I will be firmly grateful to anyone who would try to help me.
I didn't check all of your code. I just want to give you some general advices. I don't know if your goal is (1) to learn the direction of faces or (2) to implement your own neural network.
In case (1) you should consider one of those libraries. They just work and give you much more flexible configuration options. For example, standard backpropagation is one of the worst optimization algorithms for neural networks. The convergence depends on the learning rate. I can't see which value you chose in your implementation, but it could be too high. There are other optimization algorithms that don't require a learning rate or adapt it during training. In addition, 3 neurons in the hidden layer is most likely not enough. Most of the neural networks that have been used for images have hundreds and sometimes even thousands of hidden units. I would suggest you first try to solve your problem with a fully developed library. If it does work, try implementing your own ANN or be happy. :)
In case (2) you should first try to solve a simpler problem. Take a very simple artificial data set, then take a standard benchmark and then try it with your data. A good way to verify that your backpropagation implementation works is a comparison with a numerical differentation method.
Your code is missing the transfer functions. It sounds like you want the logistic function with a softmax output. You need to include the following in calcOutputError
// Logistic transfer function for hidden layer.
for (int k = 0; k < hiddenUnits; ++k) {
hiddenUnitNodeValue[k] = logistic(hiddenUnitNodeValue[k]);
}
and
// Softmax transfer function for output layer.
sum = 0;
for (int j = 0; j < 4; ++j) {
out[j] = logistic(out[j]);
sum += out[j];
}
for (int j = 0; j < 4; ++j) {
out[j] = out[j] / sum;
}
where the logistic function is
public double logistic(double x){
return (1/(1+(Math.exp(-x)));
}
Note that the softmax transfer function gives you outputs that sum to 1, so they can be interpreted as probabilities.
Also, your calculation of the error gradient for the output layer is incorrect. It should simply be
for (int i = 0; i < 4; ++i) {
error[i] = (expectedOutput[i] - out[i]);
}
I haven't tested your code but I am almost certain that you start out with to large weights.
Most of the introductions on the subjects leave it at "init the weights with random values" and leaving out that the algorithm actually diverges (goes to Inf) for some starting values.
Try using smaller starting values, for example between -1/5 and 1/5 and shrink it down.
And additionally do an method for matrix multiplication, you have (only) used that 4 times, much easier to see if there is some problem there.
I had a similar problem with a neural network processing grayscale images. You have 960 input values ranging between 0 and 255. Even with small initial weights, you can end up having inputs to your neurons with a very large magnitude and the backpropagation algorithm gets stuck.
Try dividing each pixel value by 255 before passing it into the neural network. That's what worked for me. Just starting with extremely small initial weights wasn't enough, I believe due to the floating-point precision issue brought up in the comments.
As suggested in another answer, a good way to test your algorithm is to see if your network can learn a simple function like XOR.
And for what it's worth, 3 neurons in the hidden layer was plenty for my purpose (identifying the gender of a facial image)
I wrote an entire new neural-network library and it works. It is sure that in my previous attempt I missed the idea of using transfer functions and their derivatives. Thank you, all!
Related
Now I can get only array of bytes from audio file. But I need frequency of sound. How I can get it. I'm trying the fft. But after that I get very big numbers and it's not frequency. Of course, I can't to mult to i, because this is Java
private static double[] fft(byte[] bytes) {
double[] fft = new double[bytes.length];
for (int k = 0; k < bytes.length; k++) {
for (int n = 0; n < bytes.length; n++) {
fft[k] += bytes[n] * Math.pow(Math.E, -2 * Math.PI * k * n / bytes.length);
}
}
return fft;
}
I would suggest using a complex type for your FFT calculations, it will make everything simpler (or at least more readable) and doesn't add a lot of overhead.
I am not a java person, it doesnt seem like JDK has a built in complex type BUT implementations like this exist:
https://introcs.cs.princeton.edu/java/97data/Complex.java.html
Your FFT could then be something like this (a bit unoptimized pseudo code!):
private static Complex[] fft(byte[] bytes) {
Complex[] fft = new Complex[bytes.length];
for (int k = 0; k < bytes.length; k++) {
for (int n = 0; n < bytes.length; n++) {
Complex temp = new Complex (0,-2 * Math.PI * k * n/bytes.length);
fft[k] += bytes[n] * Complex.exp(temp);
}
}
return fft;
}
You can get the magnitudes with something like
Complex.abs(fft[k])
I would also look at your outer loop (k), this is the size of your FFT and it will currently be the length of the input. This may or may not be what you want, I would suggest looking at signal windowing.
I have a loop like this:
double[][] ED = new double[n][m];
for(int k = 0; k < 20000; k++)
for (int i = 0; i < 200; i++)
for (int j = 0; j < 200; j++)
ED[i][j] = dis(qx[k][i], qy[k][i], dx[k][j], dy[k][j]);
"dis" is a function to calculate the distance between (x1,y1) and (x2,y2). Don't mind it. The problem is when I add another boolean assignment in the loop just like this:
double[][] ED = new double[n][m];
boolean[][] bool = new boolean[n][m];
for(int k = 0; k < 20000; k++)
for (int i = 0; i < 200; i++)
for (int j = 0; j < 200; j++)
{
ED[i][j] = dis(qx[k][i], qy[k][i], dx[k][j], dy[k][j]);
bool[i][j] = ED[i][j] > 5000;
}
The new loop cost 1.5 time over the first one. I think it cost too much. For testing, I break 2 assignment into 2 loop.The strange thing happens, two cost of time are same. Sometimes, code 3 cost less time than code 2
double[][] ED = new double[n][m];
boolean[][] bool = new boolean[n][m];
for(int k = 0; k < 20000; k++)
{
for (int i = 0; i < 200; i++)
for (int j = 0; j < 200; j++)
{
ED[i][j] = dis(qx[k][i], qy[k][i], dx[k][j], dy[k][j]);
}
for (int i = 0; i < 200; i++)
for (int j = 0; j < 200; j++)
{
bool[i][j] = ED[i][j] > 5000;
}
}
My aim is use as less time too calculate bool[i][j], how should I do.
Introducing new, big array bool[][] may have more impact than it seems.
When only single arrayED[i][j] is used, you put less stress on L1 processor cache.
With second array, you have twice as much data, therefore cache will be invalidated more often.
Could you try, instead of using two arrays (bool and arrayED) use single array that holds both double and boolean? There will be significant overhead for array of Objects, but (maybe) compiler will be smart enough to destructure the Object.
With single array, you will have better data locality.
Also, as suggested in comments make sure you do your microbenchmarking properly.
Use http://openjdk.java.net/projects/code-tools/jmh/ and read its documentation how to use it correctly.
Another soltution that would help on multicore system is to use parallel processing. You may create ThreadPoolExecutor (with pool size equal to number of cores you have) then submit each opearation as task to the executor. Operation may be inner-most loop (with counter j) or even two inner-most loops (with counters i and j).
There will be some overhead for coordinating the work but execution time should be much faster if you have 4 or 8 cores.
Yet another idea is to change input data structure. Instead of operating on 4 two-dimensional arrays (qx,qy,dx,dy) could you have single array? it may make dis function faster.
I had a lot of trouble trying to find a solution to this answer because it's hard to search specifically about it. I will try my best to explain it well though.
Currently I have the following code:
for(int x = 0; x < 10000; x++){
for(int z = 0; z < 10000; z++){
if(!exists(x + ";" + z)){
return x + ";" + z;
}
}
}
This is pretty much what I want to do except the way the numbers will "generate" is like so:
0;0
0;1
0;2
0;3
...
0;9999
1;0
1;1
I would like the numbers to generate in the following way "or similar" while still ensuring that every combination exists.
0;0
1;0
1;1
0;1
I created an image to show how the numbers should generate. It can go from bottom to left or left to bottom. (See image)
Thanks!
int len = 3;
for (int i = 0; i < len; i++) {
for (int j = 0; j <= i; j++) //println(i+";"+j);
for (int k = i-1; k >= 0; k--) //println(k+";"+i);
}
In the code above len serves the purpose of defining the length of a square (len is 3 above which represents a 3 x 3). In your own code you picked 10000 but that is too large for printing values
What you want is easiest to do with two internal loops, one that handles vertical and the other horizontal. Other than that just arbitrarily pick which one handles the corner of each i iteration. In the code above the j loop handles the corner case
The algorithm you seem to be looking for is:
for(int distanceFromCorner = 0; distanceFromCorner < 10000; distanceFromCorner++) {
for(int otherAxis = 0; otherAxis <= distanceFromCorner; otherAxis++) {
if(!exists(distanceFromCorner + ";" + otherAxis))
return distanceFromCorner + ";" + otherAxis;
if(!exists(otherAxis + ";" + distanceFromCorner))
return otherAxis + ";" + distanceFromCorner;
}
}
Note that this (like your original algorithm) will be much slower than necessary if you want to generate a large number of IDs. (Optimizing this is a separate question)
I'd really like assessing if any of you could point me towards the most optimized and computetionally quick linear algebra library in terms of Cholesky factorization.
So far I've been using the Apache Commons Math library, but perhaps there are more robust and better-enhanced options already available.
For instance, would PColt, EJML or ojAlgo better choices? The most urgent concerns is mainly one: I need to iteratively calculate (within a 2048 elements for loop generally) the lower triangular Cholesky factor for up to three different matrices; the largest size the matrices will reach is about 2000x2000.
Cholesky factorisation is quite a simple algorithm. Here's the (unoptimised) C# code that I use. C# and Java are quite similar, so should be an easy job for you to convert to Java and make whatever improvements you deem necessary.
public class CholeskyDecomposition {
public static double[,] Do(double[,] input) {
int size = input.GetLength(0);
if (input.GetLength(1) != size)
throw new Exception("Input matrix must be square");
double[] p = new double[size];
double[,] result = new double[size, size];
Array.Copy(input, result, input.Length);
for (int i = 0; i < size; i++) {
for (int j = i; j < size; j++) {
double sum = result[i, j];
for (int k = i - 1; k >= 0; k--)
sum -= result[i, k] * result[j, k];
if (i == j) {
if (sum < 0.0)
throw new Exception("Matrix is not positive definite");
p[i] = System.Math.Sqrt(sum);
} else
result[j, i] = sum / p[i];
}
}
for (int r = 0; r < size; r++) {
result[r, r] = p[r];
for (int c = r + 1; c < size; c++)
result[r, c] = 0;
}
return result;
}
}
Have a look at the Java Matrix Benchmark. The "Inver Symm" case test inverting a matrix using the cholesky decomposition. If you get the source code for the benchmark there is also a pure cholesky decomposition test that you can turn on.
Here's another comparison of various matrix decompositions between ojAlgo and JAMA
I was looking through some old contest questions, and I found this one, it looked fun, http://dwite.ca/old/Problem5Jan2006.pdf , I tried using the floyd warshall algorithm to get the shortest path from any node to any other node, can you guys see what I did wrong? it does not give the desired output set out on the contest question page
import java.io.*;
import java.util.*;
public class DistanceBetween {
public static void main(String[] args) throws FileNotFoundException {
Scanner s = new Scanner(new File("DATA5.txt"));
int n = Integer.parseInt(s.nextLine());
int[][] dist = new int[60][60];
for(int y=0;y<60;++y)for(int x=0;x<60;++x)dist[y][x]=10000000;
Map<Character, Integer> map = new TreeMap<Character, Integer>();
for (int i = 0; i < n; ++i) {
String text[] = s.nextLine().split(" ");
int c = 0;
if (!map.containsKey(text[0].charAt(0))) {
map.put(text[0].charAt(0), c);
c++;
}
if (!map.containsKey(text[0].charAt(1))) {
map.put(text[0].charAt(1), c);
c++;
}
dist[map.get(text[0].charAt(0))][map.get(text[0].charAt(1))] = Integer.parseInt(text[1]);
}
for (int k = 0; k < map.size(); ++k) {
for (int i = 0; i < map.size(); ++i) {
for (int j = 0; j < map.size(); ++j) {
dist[i][j] = Math.min(dist[i][j], dist[i][k] + dist[k][j]);
}
}
}
for (int i = 0; i < 5; ++i) {
String text = s.nextLine();
System.out.println(dist[map.get(text.charAt(0))][map.get(text.charAt(1))]);
}
}}
There are several problems in your code:
Overwritten mapping
Your int c is local variable of the for cycle which means the highest used mapping index doesn't survive to the next iteration, so the reading in next iteration overrides the previous one. So the distance matrix is not properly filled after data loading.
Solution: move the int c = 0; outside from the for loop.
Unidirectional roads
The roads are bidirectional in the instructions, but you register them only as unidirectional. As the consequence of that are higher on non-existent connections between towns.
Solution: add dist[map.get(text[0].charAt(1))][map.get(text[0].charAt(0))] = Integer.parseInt(text[1]); right after the similar one.
Besides these hard issues I have also couple hints for you. You do not have follow them but as if you want to improve your programming skills then you should think about them.
Messy code
Your code is hard to read, there are multiple restated information such as indicies, the solving process is in the single method etc. Such code is not only hard to read but also extremely hard to debug and fix. For your own good I recommend you to write it cleaner.
Algorithm efficiency
Floyd-Warshall's algorithm has a O(n^3) complexity. The size of problem (amount of towns) is A-M = 13. In this complexity it makes 13^3 = 2197 iterations. I know, it might not seem to be a lot, but consider the amount of tasks to solve in a given time limit.
I would recommend you to use Dijkstra's algorithm which has complexity O(|E| + |V|log|V|). In this task the worst case with some simplification is |E| = (|V|^2)/2, |V|=13. It means, that the final number of iterations is 5 (|V|^2 / 2 + |V|log|V|) = 5 (13^2 / 2 + 13 * log13) ~ 5 * 132 = 660. If I am not wrong and made any mistake, this is significantly less, especially when we consider the total amount of tasks.
Input reading
I might be wrong but I attended multiple programming contests and competitions and it never forced attendees to work with files. An input was always redirected from files to a standard input. I guess, that the main reason for this is a security, but the simplification is probably also highly beneficial.
Well that question I got, I am starting to do SPOJ now, and I gotta admit it is pretty difficult later on, but I came across the same kind of question http://www.spoj.com/problems/SHPATH/ , I also used Floyd Warshall
import java.util.*;
public class Floydwarshall {
public static void main(String[] args) {
Scanner s = new Scanner(System.in);
String q = s.nextLine();
for(int t=0;t<Integer.parseInt(q);++t){
int n = Integer.parseInt(s.nextLine());
int[][] cost = new int[n][n];
for (int y = 0; y < n; ++y) {
for (int x = 0; x < n; ++x) {
cost[x][y] = 10000;
}
}
Map<String, Integer> map = new TreeMap<String, Integer>();
int c = 0;
for (int i = 0; i < n; ++i) {
String a = s.nextLine();
if (!map.containsKey(a)) {
map.put(a, c);
c++;
}
int f = Integer.parseInt(s.nextLine());
for (int j = 0; j < f; ++j) {
String text[] = s.nextLine().split(" ");
cost[map.get(a)][Integer.parseInt(text[0]) - 1] =
cost[Integer.parseInt(text[0]) - 1][map.get(a)] = Integer.parseInt(text[1]);
}
}
for (int k = 0; k < map.size(); ++k) {
for (int i = 0; i < map.size(); ++i) {
for (int j = 0; j < map.size(); ++j) {
cost[i][j] = Math.min(cost[i][j], cost[i][k] + cost[k][j]);
}
}
}
int num = Integer.parseInt(s.nextLine());
for (int i = 0; i < num; ++i) {
String text[] = s.nextLine().split(" ");
System.out.println(cost[map.get(text[0])][map.get(text[1])]);
}
}
}}
now it runs alright for the sample input, but when I hand it in, it gives me this
NZEC (non-zero exit code) - this message means that the program exited returning a value different from 0 to the shell. For languages such as C, this probably means you forgot to add "return 0" at the end of the program. For interpreted languages (including JAVA) NZEC will usually mean that your program either crashed or raised an uncaught exception.
Problem is I cannot kind where it crashes or raises an uncaught exception since it
works with the sample input