We Keep Coding

NN don't evaluate well with back propagation algorithm

I've tried to create a basic NN using the book
"Make Your Own Neural Network" by Tariq Rashid
and using the coding train videos:
https://www.youtube.com/watch?v=XJ7HLz9VYz0&list=PLRqwX-V7Uu6aCibgK1PTWWu9by6XFdCfh
and the nn.js class on the coding train git as a reference
https://github.com/shiffman/Neural-Network-p5/blob/master/nn.js
I writing the NN network in java, and I tried just like in the playlist to train the network on XOR after I succeeded in doing a single perceptron.
but for some reason, even though my code is similar to what the book is doing and same in the videos (except in the videos he using JS).
when I train the network for around 500000 times with randomized data set of XOR inputs (total of 4 input [1,0] [0,1] [0,0] [1,1]).
when I giving it to guess after the training the all the 4 options I get results closer to 0.5 than to 1,1,0,0 (the order of the inputs in the test are [1,0] [0,1] [0,0] [1,1])
this is my training function:
public void train(double [] inputs, double[] target) {
//generates the Hidden layer values
this.input = Matrix.fromArrayToMatrix(inputs);
feedForward(inputs);
//convert to matrices
Matrix targets = Matrix.fromArrayToMatrix(target);
//calculate the output error
Matrix outputErrors = Matrix.subtract(targets, output);
//calculate the Gradient
Matrix outputGradient = Matrix.map(output, NeuralNetwork::sigmoidDerivative);
outputGradient = Matrix.matrixMultiplication(outputGradient, outputErrors);
outputGradient.multiply(this.learningRate);
//adjust the output layer bias
this.bias_Output.add(outputGradient);
//calculate the hidden layer weights delta
Matrix hiddenT = Matrix.Transpose(hidden);
Matrix hiddenToOutputDelta = Matrix.matrixMultiplication(outputGradient, hiddenT);
//adjust the hidden layer weights
this.weightsHiddenToOutput.add(hiddenToOutputDelta);
//calculate the hidden layer error
Matrix weightsHiddenToOutputT = Matrix.Transpose(weightsHiddenToOutput);
Matrix hiddenErrors = Matrix.matrixMultiplication(weightsHiddenToOutputT, outputErrors);
//calculate the hidden gradient
Matrix hiddenGradient = Matrix.map(this.hidden, NeuralNetwork::sigmoidDerivative);
hiddenGradient = Matrix.matrixMultiplication(hiddenGradient, hiddenErrors);
hiddenGradient.multiply(this.learningRate);
//adjust the hidden layer bias
this.bias_Hidden.add(hiddenGradient);
//calculate the input layer weights delta
Matrix inputT = Matrix.Transpose(this.input);
Matrix inputToHiddenDelta = Matrix.matrixMultiplication(hiddenGradient, inputT);
//adjust the hidden layer weights
this.weightsInputToHidden.add(inputToHiddenDelta);
}
those are the sigmoid functions:
private static double sigmoid(double x) {
return 1d / (1d+ Math.exp(-x));
}
private static double sigmoidDerivative(double x) {
return (x * (1d - x));
}
I'm using this method to calculate the derivative because the network already getting the sigmoid function on the feed-forward process so all I do is calculate the derivative like that.
and this is my guess/ feedforward function:
public double[] feedForward(double [] inputs) {
double[] guess;
//generates the Hidden layer values
input = Matrix.fromArrayToMatrix(inputs);
hidden = Matrix.matrixMultiplication(weightsInputToHidden, input);
hidden.add(bias_Hidden);
//activation function
hidden.map(NeuralNetwork::sigmoid);
//Generates the output layer values
output = Matrix.matrixMultiplication(weightsHiddenToOutput, hidden);
output.add(bias_Output);
//activation function
output.map(NeuralNetwork::sigmoid);
guess = Matrix.fromMatrixToArray(output);
return guess;
}
this is in the main class the data set I'm giving him:
NeuralNetwork nn = new NeuralNetwork(2,2,1);
double [] label0 = {0};
double [] label1 = {1};
Literal l1 = new Literal(label1,0,1);
Literal l2 = new Literal(label1,1,0);
Literal l3 = new Literal(label0,0,0);
Literal l4 = new Literal(label0,1,1);
Literal[] arr = {l1, l2, l3, l4};
Random random = new Random();
for(int i = 0 ; i<500000 ; i++) {
Literal l = arr[i%4];
nn.train(l.getTruthValue(), l.getLabel());
}
System.out.println(Arrays.toString(nn.feedForward(l1.getTruthValue())));
System.out.println(Arrays.toString(nn.feedForward(l2.getTruthValue())));
System.out.println(Arrays.toString(nn.feedForward(l3.getTruthValue())));
System.out.println(Arrays.toString(nn.feedForward(l4.getTruthValue())));
but for some reason the outputs look like that:
[0.47935468493879807]
[0.5041956026507048]
[0.4575246472403595]
[0.5217568912941623]
I've tried changing it to subtract instead of add-on every bias and weights update (cause you need the negative gradient although both in the book and in the videos they use add instead of subtract) meaning changing those 4 lines to subtract:
this.bias_Output.subtract(outputGradient);
this.weightsHiddenToOutput.subtract(hiddenToOutputDelta);
this.bias_Hidden.subtract(hiddenGradient);
this.weightsInputToHidden.subtract(inputToHiddenDelta);
those are the 2 main outputs I get:
[0.9999779359460259]
[0.9999935716126019]
[0.9999860145346924]
[0.999990155468117]
or
[1.7489664881918983E-5]
[6.205315404676972E-6]
[8.41530873105465E-6]
[1.1853929628341918E-5]
I'm pretty sure the problem isn't in my Matrix class that I've created because I checked it before and all the add, subtract, multiply, transpose and it worked fine.
I would really appreciate if someone could look at this code and help me to figure out the problem

The Mahalanobis distance between a point and the mean vector is always the same

I‘m trying to perform some data cleansing algorithm recently. When I try to calculate the mahalanobis distance between points in the data set and the mean vector, it seems the same.
For example, I have a data set like:
{{2,2,3},{4,5,9},{7,8,9}}
The mean vector is :
{13/3,5,7}
And the covariance matrix is:
{{6.333333333333333,7.5,7.0},{7.5,9.0,9.0},{7.0,9.0,12.0}}
Then the distances between {2,2,3}, {4,5,9}, {7,8,9} and the mean vector are all 8290542, which is quite strange. After calculating on paper, the result is the same.
Does anyone know what's wrong with my code or thought? I'd be more than grateful if someone could help me out. Following is some code I used in dealing with this problem.
import org.apache.commons.math3.linear.RealMatrix;
import org.apache.commons.math3.stat.correlation.Covariance;
import org.apache.mahout.math.*;
import org.apache.mahout.common.distance.MahalanobisDistanceMeasure;
public class Test {
public static void main(String[] args) {
double[] a = {2,2,3};
Vector aVector = new DenseVector(a);
double[] b = {4,5,9};
Vector bVector = new DenseVector(b);
double[] c = {7,8,9};
Vector cVector = new DenseVector(b);
double[] mean = {13/3,5,7};
Vector meanVector = new DenseVector(mean);
MahalanobisDistanceMeasure measure = new MahalanobisDistanceMeasure();
double[][] ma = {{2,2,3},{4,5,9},{7,8,9}};
RealMatrix matrix = new Covariance(ma).getCovarianceMatrix();
Matrix math = new DenseMatrix(matrix.getData());
measure.setCovarianceMatrix(math);
measure.setMeanVector(meanVector);
System.out.println(matrix.toString());
System.out.println(measure.distance(meanVector,cVector));
}
}

You need to use more data.
The mean vector + covariance matrix will otherwise overfit to your data, and give the same distance each.
For 3d data, use at least 20 points.

Java/CGAL verify if a graph is connected (some constraints in description)

it's my first time with CGAL, some of you may argue why do I have to learn CGAL from something like that, but it's a new project that I must do (and... yes, I must use CGAL and Java combined) :/ Long story short... I only have:
Two double arrays, representing x and y coordinates of my vertices. Let's call them double[] x, y;.
Both arrays have S random values.
Two vertices, u and w are connected if distance(x[u], y[u], x[w], y[w]) < CONSTANT (ofc. I do distanceSquared(x[u], y[u], x[w], y[w]) < CONSTANT_SQUARED, so I avoid to call sqrt()).
x and y are filled randomly with values from 0 to UPPER_LIMIT, no other infos are given.
Question, do x and y describes a connected graph?
Right now I have two algoritms:
Algorithm 1:
Build adjacency list (Arraylist<Integer>[] adjLists;) for each vertex (only upper triangular matrix explored). Complexity O(|V|^2) (V = vertices set).
Recursive graph exploration, vertex marking and counting, if visited vertex equals S my graph have only one connected component, my graph is connected. Complexity O(|E|) (E = edges set).
Algorithm 2:
private static boolean algorithmGraph(double[] x, double[] y) {
int unchecked, inside = 0, current = 0;
double switchVar;
while (current <= inside && inside != S - 1) {
unchecked = inside + 1;
while (unchecked < S) {
if ((x[current] - x[unchecked]) * (x[current] - x[unchecked]) + (y[current] - y[unchecked]) * (y[current] - y[unchecked]) <= CONSTANT_SQUARED) {
inside++;
// switch x coordinates | unchecked <-> inside
switchVar = x[unchecked];
x[unchecked] = x[inside];
x[inside] = switchVar;
// switch y coordinates | unchecked <-> inside
switchVar = y[unchecked];
y[unchecked] = y[inside];
y[inside] = switchVar;
}
unchecked++;
}
current++;
}
return inside == S - 1;
}
Funny thing the second one is slower, I do not use data structures, the code is iterative and in-place but the heavy use of switch makes it slow as hell.
The problem spec changed and now I must do it with CGAL and Java, I'll read the whole "https://github.com/CGAL/cgal-swig-bindings" to learn how to use CGAL within Java.... but I'd like some help about this specific instance of CGAL code... Are there faster algorithms already implemented in CGAL?
Thank you for your times guys! Happy coding!

I believe that, without a method of spatial indexing, the best performance you are going to achieve in the worst-case-scenario (all connected) is going to be O(n*(n-1)/2).
If you can afford to build a spatial index (have enough memory to pay for the boost in speed), you may consider R-tree and variants - insertion is O(n) searching is O(log2(n)): this will get your "outlier detection by examining distances" approach for a cost of of O(n*log2(n)) in the worst-case-scenario.
A notable result

Adjacency Matrix In Java

I'm so confused by graphs and adjacency matrices. I'm doing an assignment for a class where I have a text file of nodes and a text file of edges and I have to read each of them and make them a graph onto which I can then perform operations such as determining if the graph is connected, finding a minimal spanning tree, traversals and finding paths. I've never worked with graphs before though, and I'm really confused by the whole thing, and I was wondering if someone could help explain some of this to me.
Firstly, do I build a graph on its own (with node and edges classes perhaps?) and then construct an adjacency matrix from that? Or is the adjacency matrix itself the graph?
And then I'm confused on how to implement the adjacent matrix into the program. The nodes are names things like "ND5" and "NR7" and so I would have to set and read the edges of [ND5][NR7] but I'm not sure how to set up a 2d array like that with strings for the outside and numbers on the inside.
I've been searching all over the internet and read through the whole chapter on graphs in my textbook, and I'm really not understanding just the first basic steps of getting this graph set up. I'd really appreciate the help. Thanks.

Firstly, do I build a graph on its own (with node and edges classes perhaps?) and then construct an adjacency matrix from that? Or is the adjacency matrix itself the graph?
There is no way anyone can answer that question for sure without actually reading the instructions for your assignment. However, unless the assignment specifically mentions Node and Edge classes or something, my guess is that you're just supposed to use the adjacency matrix to represent your graph.
And then I'm confused on how to implement the adjacent matrix into the program. The nodes are names things like "ND5" and "NR7" and so I would have to set and read the edges of [ND5][NR7] but I'm not sure how to set up a 2d array like that with strings for the outside and numbers on the inside.
I can totally understand your confusion here. What you actually want to do is create a bijection (a one-to-one relationship) between your node names and the indices of your matrix. For example, if you have n nodes in your graph, then you need an n×n matrix (i.e. new boolean[n][n]), and each of your nodes would correspond to a single integer in the range 0 until n (not inclusive of n).
I'm not sure what data structures you've covered in your class so far, but the easiest way to do this would probably be to use a Map<String, Integer>, which would let you look up a name like "ND5" and get back an integer (the index).
Another nice alternative might be to use an array. You could put all your node names into an array, sort it with Arrays.sort, and then once it's sorted you can use Arrays.binarySearch to find the index of a particular node name in that array. I think this solution is actually better than using a Map because it lets you do the lookups both ways. You use Arrays.binarySearch to do name-to-index lookups, and you just index into the array to do an index-to-name lookup.
Example: Let's say we have this graph:
Given that graph, here's some sample code of how you could do this: (warning! it's untested)
import java.util.Arrays;
// Add all your node names to an array
String[] nameLookup = new String[4];
nameLookup[0] = "A";
nameLookup[1] = "B";
nameLookup[2] = "C";
nameLookup[3] = "D";
// Our array is already properly sorted,
// but yours might not be, so you should sort it.
// (if it's not sorted then binarySearch won't work)
Arrays.sort(nameLookup);
// I'm assuming your edges are unweighted, so I use boolean.
// If you have weighted edges you should use int or double.
// true => connected, false => not connected
// (entries in boolean arrays default to false)
boolean[][] matrix = new boolean[4];
for (int i=0; i<matrix.length; i++) matrix[i] = new boolean[4];
// I don't want to call Arrays.binarySearch every time I want an index,
// so I'm going to cache the indices here in some named variables.
int nodeA = Arrays.binarySearch(nameLookup, "A");
int nodeB = Arrays.binarySearch(nameLookup, "B");
int nodeC = Arrays.binarySearch(nameLookup, "C");
int nodeD = Arrays.binarySearch(nameLookup, "D");
// I'm assuming your edges are undirected.
// If the edges are directed then the entries needn't be semmetric.
// A is connected to B
matrix[nodeA][nodeB] = true;
matrix[nodeB][nodeA] = true;
// A is connected to D
matrix[nodeA][nodeD] = true;
matrix[nodeD][nodeA] = true;
// B is connected to D
matrix[nodeB][nodeD] = true;
matrix[nodeD][nodeB] = true;
// C is connected to D
matrix[nodeC][nodeD] = true;
matrix[nodeD][nodeC] = true;
// Check if node X is connected to node Y
int nodeX = Arrays.binarySearch(nameLookup, stringNameOfX);
int nodeY = Arrays.binarySearch(nameLookup, stringNameOfY);
if (matrix[nodeX][nodeY]) { /* They're connected */ }
// Print all of node Z's neighbors' names
int nodeZ = Arrays.binarySearch(nameLookup, stringNameOfZ);
for (int i=0; i<matrix.length; i++) {
if (matrix[nodeZ][i]) {
System.out.println(nameLookup[nodeZ] + " is connected to " + nameLookup[i]);
}
}

Visualizing Kmean Clustering

I'm doing a KMean clustering on a 12 dimensional matrix. I managed to get the result in K set of cluster. I want to show the result by plotting it into a 2D graph, but I can't figure it out how can I convert the 12 dimension data into 2 dimension.
Any suggestion on how can I do the conversion or any alternative ways on visualizing the result? I tried Multidimensional Scaling for Java (MDSJ) but it did not work.
The KMean algorithm I'm using was from the Java Machine Learning Library: Clustering basics.

I would do Principal Component Analysis (probably the easiest algorithm from Multidimensional scaling algorithms). (BTW PCA has nothing to do with KMeans, it is a general method for dimensionality reduction)
I assume variables are in columns, observations are in rows.
Standardize the data - convert variables to z-scores. That means: from each cell, subtract the mean of the column and devide the result by the std. deviation of the column. That way you get zero mean and unit variance. The former is obligatory, the latter, I would say, good to do. If you have zero variance, you calculate the eigen-vectors from the covariance matrix, otherwise have to use correlation matrix which kind of standardizes the data automatically. See this for explanation).
Calculate eigen-vectors and eigen-values of the covariance matrix. Sort the eigen-vectors by the eigen-values. (Many libraries already give you eigen-vectors sorted that way).
Use first two columns of the eigen-vector matrix and multiply the original matrix (converted to z-scores), visualize this data.
Using the colt library, you can do the following. It will be similar with other matrix libraries:
import cern.colt.matrix.DoubleMatrix1D;
import cern.colt.matrix.DoubleMatrix2D;
import cern.colt.matrix.doublealgo.Statistic;
import cern.colt.matrix.impl.SparseDoubleMatrix2D;
import cern.colt.matrix.linalg.Algebra;
import cern.colt.matrix.linalg.EigenvalueDecomposition;
import hep.aida.bin.DynamicBin1D;
public class Pca {
// to show matrix creation, it does not make much sense to calculate PCA on random data
public static void main(String[] x) {
double[][] data = {
{2.0,4.0,1.0,4.0,4.0,1.0,5.0,5.0,5.0,2.0,1.0,4.0},
{2.0,6.0,3.0,1.0,1.0,2.0,6.0,4.0,4.0,4.0,1.0,5.0},
{3.0,4.0,4.0,4.0,2.0,3.0,5.0,6.0,3.0,1.0,1.0,1.0},
{3.0,6.0,3.0,3.0,1.0,2.0,4.0,6.0,1.0,2.0,4.0,4.0},
{1.0,6.0,4.0,2.0,2.0,2.0,3.0,4.0,6.0,3.0,4.0,1.0},
{2.0,5.0,5.0,3.0,1.0,1.0,6.0,6.0,3.0,2.0,6.0,1.0}
};
DoubleMatrix2D matrix = new DenseDoubleMatrix2D(data);
DoubleMatrix2D pm = pcaTransform(matrix);
// print the first two dimensions of the transformed matrix - they capture most of the variance of the original data
System.out.println(pm.viewPart(0, 0, pm.rows(), 2).toString());
}
/** Returns a matrix in the space of principal components, take the first n columns */
public static DoubleMatrix2D pcaTransform(DoubleMatrix2D matrix) {
DoubleMatrix2D zScoresMatrix = toZScores(matrix);
final DoubleMatrix2D covarianceMatrix = Statistic.covariance(zScoresMatrix);
// compute eigenvalues and eigenvectors of the covariance matrix (flip needed since it is sorted by ascending).
final EigenvalueDecomposition decomp = new EigenvalueDecomposition(covarianceMatrix);
// Columns of Vs are eigenvectors = principal components = base of the new space; ordered by decreasing variance
final DoubleMatrix2D Vs = decomp.getV().viewColumnFlip();
// eigenvalues: ev(i) / sum(ev) is the percentage of variance captured by i-th column of Vs
// final DoubleMatrix1D ev = decomp.getRealEigenvalues().viewFlip();
// project the original matrix to the pca space
return Algebra.DEFAULT.mult(zScoresMatrix, Vs);
}
/**
* Converts matrix to a matrix of z-scores (by columns)
*/
public static DoubleMatrix2D toZScores(final DoubleMatrix2D matrix) {
final DoubleMatrix2D zMatrix = new SparseDoubleMatrix2D(matrix.rows(), matrix.columns());
for (int c = 0; c < matrix.columns(); c++) {
final DoubleMatrix1D column = matrix.viewColumn(c);
final DynamicBin1D bin = Statistic.bin(column);
if (bin.standardDeviation() == 0) { // use epsilon
for (int r = 0; r < matrix.rows(); r++) {
zMatrix.set(r, c, 0.0);
}
} else {
for (int r = 0; r < matrix.rows(); r++) {
double zScore = (column.get(r) - bin.mean()) / bin.standardDeviation();
zMatrix.set(r, c, zScore);
}
}
}
return zMatrix;
}
}
You could also use weka. I would first load your data into weka, then run PCA using the GUI (under attribute selection). You will see what classes are called with what parameters and then do the same thing from your code. The problem is you will need to convert/wrap your matrix into the data format weka works with.

A similar question has been discussed on CrossValidated2. The basic idea is to find an appropriate projection that separates these clusters (e.g., with discproj in R) and then to plot the projection on the clusters on the new space.

In addition to what the other answers suggest you should probably have a look at multidimensional scaling too.