I have implemented Logistic Regression with Gradient Descent in Java. It doesn't seem to work well (It does not classify records properly; the probability of y=1 is a lot.) I don't know whether my implementation is correct.I have gone through the code several times and i am unable to find any bug. I have been following Andrew Ng's tutorials on Machine learning on Course Era. My Java implementation has 3 classes. namely :
DataSet.java : To read the data set
Instance.java : Has two members : 1. double[ ] x and 2. double label
Logistic.java : This is the main class that implements Logistic Regression with Gradient Descent.
This is my cost function:
J(Θ) = (- 1/m ) [Σmi=1 y(i) log( hΘ( x(i) ) ) + (1 - y(i) ) log(1 - hΘ (x(i)) )]
For the above Cost function, this is my Gradient Descent algorithm:
Repeat ( Θj := Θj - α Σmi=1 ( hΘ( x(i)) - y(i) ) x(i)j
(Simultaneously update all Θj )
)
import java.io.FileNotFoundException;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
public class Logistic {
/** the learning rate */
private double alpha;
/** the weight to learn */
private double[] theta;
/** the number of iterations */
private int ITERATIONS = 3000;
public Logistic(int n) {
this.alpha = 0.0001;
theta = new double[n];
}
private double sigmoid(double z) {
return (1 / (1 + Math.exp(-z)));
}
public void train(List<Instance> instances) {
double[] temp = new double[3];
//Gradient Descent algorithm for minimizing theta
for(int i=1;i<=ITERATIONS;i++)
{
for(int j=0;j<3;j++)
{
temp[j]=theta[j] - (alpha * sum(j,instances));
}
//simulataneous updates of theta
for(int j=0;j<3;j++)
{
theta[j] = temp[j];
}
System.out.println(Arrays.toString(theta));
}
}
private double sum(int j,List<Instance> instances)
{
double[] x;
double prediction,sum=0,y;
for(int i=0;i<instances.size();i++)
{
x = instances.get(i).getX();
y = instances.get(i).getLabel();
prediction = classify(x);
sum+=((prediction - y) * x[j]);
}
return (sum/instances.size());
}
private double classify(double[] x) {
double logit = .0;
for (int i=0; i<theta.length;i++) {
logit += (theta[i] * x[i]);
}
return sigmoid(logit);
}
public static void main(String... args) throws FileNotFoundException {
//DataSet is a class with a static method readDataSet which reads the dataset
// Instance is a class with two members: double[] x, double label y
// x contains the features and y is the label.
List<Instance> instances = DataSet.readDataSet("data.txt");
// 3 : number of theta parameters corresponding to the features x
// x0 is always 1
Logistic logistic = new Logistic(3);
logistic.train(instances);
//Test data
double[]x = new double[3];
x[0]=1;
x[1]=45;
x[2] = 85;
System.out.println("Prob: "+logistic.classify(x));
}
}
Can anyone tell me what am I doing wrong?
Thanks in advance! :)
As I am studying logistic regression, I took the time to review your code in detail.
TLDR
In fact, it appears the algorithm is correct.
The reason you had so much false negatives or false positives is, I think, because of the hyper parameters you choose.
The model was under-trained so the hypothesis was under-fitting.
Details
I had to create the DataSet and Instance classes because you did not publish them, and set up a train data set and a test data set based on the Cryotherapy dataset.
See http://archive.ics.uci.edu/ml/datasets/Cryotherapy+Dataset+.
Then, using your same exact code (for the logistic regression part) and by choosing an alpha rate of 0.001 and a number of iterations of 100000, I got a precision rate of 80.64516129032258 percent on the test data set, which is not so bad.
I tried to get a better precision rate by tweaking manualy those hyper parameters but could not obtain any better result.
At this point, an enhancement would be to implement regularization, I suppose.
Gradient descent formula
In Andrew Ng's video about the the cost function and gradient descent, it is correct that the 1/m term is omitted.
A possible explanation is that the 1/m term is included in the alpha term.
Or maybe it's just an oversight.
See https://www.youtube.com/watch?v=TTdcc21Ko9A&index=36&list=PLLssT5z_DsK-h9vYZkQkYNWcItqhlRJLN&t=6m53s at 6m53s.
But if you watch Andrew Ng's video about regularization and logistic regression you'll notice that the term 1/m is clearly present in the formula.
See https://www.youtube.com/watch?v=IXPgm1e0IOo&index=42&list=PLLssT5z_DsK-h9vYZkQkYNWcItqhlRJLN&t=2m19s at 2m19s.
Related
I have a program that takes in anywhere from 20,000 to 500,000 velocity vectors and must output these vectors multiplied by some scalar. The program allows the user to set a variable accuracy, which is basically just how many decimal places to truncate to in the calculations. The program is quite slow at the moment, and I discovered that it's not because of multiplying a lot of numbers, it's because of the method I'm using to truncate floating point values.
I've already looked at several solutions on here for truncating decimals, like this one, and they mostly recommend DecimalFormat. This works great for formatting decimals once or twice to print nice user output, but is far too slow for hundreds of thousands of truncations that need to happen in a few seconds.
What is the most efficient way to truncate a floating-point value to n number of places, keeping execution time at utmost priority? I do not care whatsoever about resource usage, convention, or use of external libraries. Just whatever gets the job done the fastest.
EDIT: Sorry, I guess I should have been more clear. Here's a very simplified version of what I'm trying to illustrate:
import java.util.*;
import java.lang.*;
import java.text.DecimalFormat;
import java.math.RoundingMode;
public class MyClass {
static class Vector{
float x, y, z;
#Override
public String toString(){
return "[" + x + ", " + y + ", " + z + "]";
}
}
public static ArrayList<Vector> generateRandomVecs(){
ArrayList<Vector> vecs = new ArrayList<>();
Random rand = new Random();
for(int i = 0; i < 500000; i++){
Vector v = new Vector();
v.x = rand.nextFloat() * 10;
v.y = rand.nextFloat() * 10;
v.z = rand.nextFloat() * 10;
vecs.add(v);
}
return vecs;
}
public static void main(String args[]) {
int precision = 2;
float scalarToMultiplyBy = 4.0f;
ArrayList<Vector> velocities = generateRandomVecs();
System.out.println("First 10 raw vectors:");
for(int i = 0; i < 10; i++){
System.out.print(velocities.get(i) + " ");
}
/*
This is the code that I am concerned about
*/
DecimalFormat df = new DecimalFormat("##.##");
df.setRoundingMode(RoundingMode.DOWN);
long start = System.currentTimeMillis();
for(Vector v : velocities){
/* Highly inefficient way of truncating*/
v.x = Float.parseFloat(df.format(v.x * scalarToMultiplyBy));
v.y = Float.parseFloat(df.format(v.y * scalarToMultiplyBy));
v.z = Float.parseFloat(df.format(v.z * scalarToMultiplyBy));
}
long finish = System.currentTimeMillis();
long timeElapsed = finish - start;
System.out.println();
System.out.println("Runtime: " + timeElapsed + " ms");
System.out.println("First 10 multiplied and truncated vectors:");
for(int i = 0; i < 10; i++){
System.out.print(velocities.get(i) + " ");
}
}
}
The reason it is very important to do this is because a different part of the program will store trigonometric values in a lookup table. The lookup table will be generated to n places beforehand, so any velocity vector that has a float value to 7 places (i.e. 5.2387471) must be truncated to n places before lookup. Truncation is needed instead of rounding because in the context of this program, it is OK if a vector is slightly less than its true value, but not greater.
Lookup table for 2 decimal places:
...
8.03 -> -0.17511085919
8.04 -> -0.18494742685
8.05 -> -0.19476549993
8.06 -> -0.20456409661
8.07 -> -0.21434223706
...
Say I wanted to look up the cosines of each element in the vector {8.040844, 8.05813164, 8.065688} in the table above. Obviously, I can't look up these values directly, but I can look up {8.04, 8.05, 8.06} in the table.
What I need is a very fast method to go from {8.040844, 8.05813164, 8.065688} to {8.04, 8.05, 8.06}
The fastest way, which will introduce rounding error, is going to be to multiply by 10^n, call Math.rint, and to divide by 10^n.
That's...not really all that helpful, though, considering the introduced error, and -- more importantly -- that it doesn't actually buy anything. Why drop decimal points if it doesn't improve efficiency or anything? If it's about making the values shorter for display or the like, truncate then, but until then, your program will run as fast as possible if you just use full float precision.
I'm trying to write a java program that will solve any ordinary differential equations using Euler method, but I don't know how to write a code to get any differential equation from the user. I was only able to write the code to solve a predefined ordinary differential equations.
I was able to come with a code to solve some particular ordinary differential equations which were written as functions in the program, I also made research online to look for similar problems but it seem they also wrote it to solve some designated problem not general questions on ordinary differential equations. This was found in most of the article have read online.
Here is my Euler class;
import java.lang.Math;
public class Euler {
private double x0, y0, x1, y1, h, actual;
public Euler (double initialx, double initialy,double stepsize,double finalx1) {
x0 = initialx; y0 = initialy; h=stepsize; x1 = finalx1;
}
public void setEuler (double initialx, double initialy,double stepsize,
double finalx1){
x0 = initialx;y0 = initialy;h =stepsize;x1 = finalx1;
}
public double getinitialx(){
return x0;
}
public double getinitialy(){
return y0;
}
public double getinitialexact(){
return (double) (0.9048*Math.exp(0.1*x0*x0));
}
double func(double x, double y){
return (double) (0.2*x*y);
}
double funct(double x){
return (double) (java.lang.Math.exp(0.1*x*x));
}
public double getinitialerror(){
return (double) Math.abs(actual - y0);
}
public double getEulerResult(){
for (double i = x0 + h; i < x1; i += h){
y0 = y0 + h *(func(x0,y0));
x0 += h;
double actual = (0.9048*funct(x0));
double error = Math.abs(actual - y0);
System.out.printf("%f\t%f\t%f\t%f\n",x0,y0,actual, error);
}
return y0;
}
}
Here is my Driver's class
import java.util.Scanner;
public class EulerTest {
public static void main(String[] args) {
Scanner input = new Scanner(System.in);
Euler myEuler = new Euler(1.0,1.0,0.1,1.5);
System.out.println( "x\t explicit\tactual\t error\t " );
System.out.printf("%f\t%f\t%f\t%f\n", myEuler.getinitialx(),
myEuler.getinitialy(),myEuler.getinitialexact(),
myEuler.getinitialerror());
System.out.printf("my approximated value is %f\n\n",
myEuler.getEulerResult ());
System.out.println("enter another initial value of x: ");
double initialx = input.nextDouble();
System.out.println("enter another initial value of y: ");
double initialy = input.nextDouble();
System.out.println("enter another stepsize value of h: ");
double stepsize = input.nextDouble();
System.out.println("enter another upper bound of x: ");
double finalx1 = input.nextDouble();
myEuler.setEuler(initialx,initialy,stepsize,finalx1);
System.out.println( "x\t explicit\tactual\t error\t " );
System.out.printf("%f\t%f\t%f\t%f\n", myEuler.getinitialx(),
myEuler.getinitialy(),myEuler.getinitialexact(),
myEuler.getinitialerror());
System.out.printf("my approximated value is %f\n\n",
myEuler.getEulerResult ());
}
}
I will be glad if i can en lighted on how to write the java code to collect any ordinary differential equation from the user so as to solve using Euler's method.
What you are looking for is the ability to compile some code at run time, where part of the code is supplied by the user.
There is a package called JOOR that gives you a Reflect class that contains a compile method. The method takes two parameters (a package name:String and the Java code:String).
I've never personally used it, so can not vouch for its robustness, but here is a tutorial and the javadoc:
https://www.jooq.org/products/jOOR/javadoc/latest/org.jooq.joor/org/joor/Reflect.html#compile(java.lang.String,java.lang.String)
https://blog.jooq.org/2018/04/03/how-to-compile-a-class-at-runtime-with-java-8-and-9/
In your case, you would put your user supplied function in place of the following line of code:
return \"Hello World!\";\n"
Beware, you need to be 100% absolutely unconditionally guaranteed that the user can only ever enter a function to be solved. If they are supplying code, remember that unless you take safeguards, the code they enter could very easily be code the removes all of the files on your hard drive (or worse).
For the second part of your question - how do i implement a solution in Java using Euler's method, perhaps check out this link: Euler's Method in java or this https://rosettacode.org/wiki/Euler_method#Java which has it in pretty much every language you can imagine (and probably some you can't).
I've tried to create a basic NN using the book
"Make Your Own Neural Network" by Tariq Rashid
and using the coding train videos:
https://www.youtube.com/watch?v=XJ7HLz9VYz0&list=PLRqwX-V7Uu6aCibgK1PTWWu9by6XFdCfh
and the nn.js class on the coding train git as a reference
https://github.com/shiffman/Neural-Network-p5/blob/master/nn.js
I writing the NN network in java, and I tried just like in the playlist to train the network on XOR after I succeeded in doing a single perceptron.
but for some reason, even though my code is similar to what the book is doing and same in the videos (except in the videos he using JS).
when I train the network for around 500000 times with randomized data set of XOR inputs (total of 4 input [1,0] [0,1] [0,0] [1,1]).
when I giving it to guess after the training the all the 4 options I get results closer to 0.5 than to 1,1,0,0 (the order of the inputs in the test are [1,0] [0,1] [0,0] [1,1])
this is my training function:
public void train(double [] inputs, double[] target) {
//generates the Hidden layer values
this.input = Matrix.fromArrayToMatrix(inputs);
feedForward(inputs);
//convert to matrices
Matrix targets = Matrix.fromArrayToMatrix(target);
//calculate the output error
Matrix outputErrors = Matrix.subtract(targets, output);
//calculate the Gradient
Matrix outputGradient = Matrix.map(output, NeuralNetwork::sigmoidDerivative);
outputGradient = Matrix.matrixMultiplication(outputGradient, outputErrors);
outputGradient.multiply(this.learningRate);
//adjust the output layer bias
this.bias_Output.add(outputGradient);
//calculate the hidden layer weights delta
Matrix hiddenT = Matrix.Transpose(hidden);
Matrix hiddenToOutputDelta = Matrix.matrixMultiplication(outputGradient, hiddenT);
//adjust the hidden layer weights
this.weightsHiddenToOutput.add(hiddenToOutputDelta);
//calculate the hidden layer error
Matrix weightsHiddenToOutputT = Matrix.Transpose(weightsHiddenToOutput);
Matrix hiddenErrors = Matrix.matrixMultiplication(weightsHiddenToOutputT, outputErrors);
//calculate the hidden gradient
Matrix hiddenGradient = Matrix.map(this.hidden, NeuralNetwork::sigmoidDerivative);
hiddenGradient = Matrix.matrixMultiplication(hiddenGradient, hiddenErrors);
hiddenGradient.multiply(this.learningRate);
//adjust the hidden layer bias
this.bias_Hidden.add(hiddenGradient);
//calculate the input layer weights delta
Matrix inputT = Matrix.Transpose(this.input);
Matrix inputToHiddenDelta = Matrix.matrixMultiplication(hiddenGradient, inputT);
//adjust the hidden layer weights
this.weightsInputToHidden.add(inputToHiddenDelta);
}
those are the sigmoid functions:
private static double sigmoid(double x) {
return 1d / (1d+ Math.exp(-x));
}
private static double sigmoidDerivative(double x) {
return (x * (1d - x));
}
I'm using this method to calculate the derivative because the network already getting the sigmoid function on the feed-forward process so all I do is calculate the derivative like that.
and this is my guess/ feedforward function:
public double[] feedForward(double [] inputs) {
double[] guess;
//generates the Hidden layer values
input = Matrix.fromArrayToMatrix(inputs);
hidden = Matrix.matrixMultiplication(weightsInputToHidden, input);
hidden.add(bias_Hidden);
//activation function
hidden.map(NeuralNetwork::sigmoid);
//Generates the output layer values
output = Matrix.matrixMultiplication(weightsHiddenToOutput, hidden);
output.add(bias_Output);
//activation function
output.map(NeuralNetwork::sigmoid);
guess = Matrix.fromMatrixToArray(output);
return guess;
}
this is in the main class the data set I'm giving him:
NeuralNetwork nn = new NeuralNetwork(2,2,1);
double [] label0 = {0};
double [] label1 = {1};
Literal l1 = new Literal(label1,0,1);
Literal l2 = new Literal(label1,1,0);
Literal l3 = new Literal(label0,0,0);
Literal l4 = new Literal(label0,1,1);
Literal[] arr = {l1, l2, l3, l4};
Random random = new Random();
for(int i = 0 ; i<500000 ; i++) {
Literal l = arr[i%4];
nn.train(l.getTruthValue(), l.getLabel());
}
System.out.println(Arrays.toString(nn.feedForward(l1.getTruthValue())));
System.out.println(Arrays.toString(nn.feedForward(l2.getTruthValue())));
System.out.println(Arrays.toString(nn.feedForward(l3.getTruthValue())));
System.out.println(Arrays.toString(nn.feedForward(l4.getTruthValue())));
but for some reason the outputs look like that:
[0.47935468493879807]
[0.5041956026507048]
[0.4575246472403595]
[0.5217568912941623]
I've tried changing it to subtract instead of add-on every bias and weights update (cause you need the negative gradient although both in the book and in the videos they use add instead of subtract) meaning changing those 4 lines to subtract:
this.bias_Output.subtract(outputGradient);
this.weightsHiddenToOutput.subtract(hiddenToOutputDelta);
this.bias_Hidden.subtract(hiddenGradient);
this.weightsInputToHidden.subtract(inputToHiddenDelta);
those are the 2 main outputs I get:
[0.9999779359460259]
[0.9999935716126019]
[0.9999860145346924]
[0.999990155468117]
or
[1.7489664881918983E-5]
[6.205315404676972E-6]
[8.41530873105465E-6]
[1.1853929628341918E-5]
I'm pretty sure the problem isn't in my Matrix class that I've created because I checked it before and all the add, subtract, multiply, transpose and it worked fine.
I would really appreciate if someone could look at this code and help me to figure out the problem
I am having a little issue with formatting returned methods in the main method. I have created the methods and done the calculation, but my issue is if i am calling the other two methods to the main method correctly. I am also having and issue with formatting each method in columns. Do i need to make the columns in the respected methods? or do i need to create them in the main method?
Write a program that analyzes an object falling for 10 seconds. It should contain main and two additional methods. One of the additional methods should return the distance an object falls in meters when passed the current second as an argument. See the formula needed below. The third method should convert meters to feet. You can look up the conversion factor needed online. The main method should use one loop to call the other methods and generate a table as shown below. The table should be displayed in formatted columns with decimals as shown. I believe i am on
SEC METERS FEET
1 4.9 16.1
2 19.6 64.3
3 44.1 144.7
4 78.4 257.2
5 122.5 401.9
6 176.4 578.7
7 240.1 787.7
8 313.6 1028.9
9 396.9 1302.2
10 490.0 1607.6
My code
package week4.yedkois;
public class project3 {
public static void main(String[] args) {
System.out.printf("SEC" + "\n");
meters();
feet();
for (int time = 1; time <= 10; time++) {
System.out.println(time);
}
}
public static void meters() {
double Meters;
double G = 9.8; // meters = .5(9.8)(seconds) ^2
for (int time = 1; time <= 10; time++) {
Meters = (.5 * 9.8 * Math.pow(time, 2));
System.out.printf("%.1f\n", Meters);
}
return;
}
public static void feet() {
double Feet;
double G = 9.8; // meters = .5(9.8)(seconds) ^2
for (int time = 1; time <= 10; time++) {
Feet = (.5 * 9.8 * Math.pow(time, 2) * 3.28084);
System.out.printf("%.1f\n", Feet);
}
return;
}
}
Here is my solution. I use a Tab ("\t") to achieve the same space between the different values. And then I had to redesign your code a little. I use only one if-loop directly in the main-method and hand the current time-value as a parameter into both methods meters() and feet(). That makes it much easier to get all values of one round in line.
Here are some additional remarks:
Java is not C++, so you don't have to use an empty return statement at the end of a method. It's useless there.
In Java variables and method-names always start with a small letter, _ or $. Only class-names and constants start with a capital letter.
Hope this helps for a start.
public class Project3 {
public static void main(String[] args){
System.out.printf("%3s\t%6s\t%6s\n", "SEC", "METERS", "FEET");
for(int time = 1; time <= 10; time++)
{
System.out.print(time + "\t");
meters(time);
feet(time);
System.out.println();
}
}
public static void meters(int time){
double meters;
double g = 9.8; // meters = .5(9.8)(seconds) ^2
meters = (.5 * 9.8 * Math.pow(time, 2));
// the longer the expected maximum length of a result gets
// the higher your reserved number of digits has
// to be, to gain the wanted right bound effect!
System.out.printf("%6.1f\t", meters);
}
public static void feet(int time){
double feet;
double g = 9.8; // meters = .5(9.8)(seconds) ^2
feet = (.5 * 9.8 * Math.pow(time, 2) * 3.28084);
// the longer the expected maximum length of a result gets
// the higher your reserved number of digits has
// to be, to gain the wanted right bound effect!
System.out.printf("%6.1f", feet);
}
}
I tried to make a program (in Java) that calculates pi with the Chudnovsky algorithm but it has the output NaN (Not a Number). Please help me find mistakes in my code, or improve my code. (I don't have a lot of Java programming knowledge)
You can find Chudnovsky's algorithm here:
https://en.wikipedia.org/wiki/Chudnovsky_algorithm
here is my code:
package main;
public class Class1 {
public static void main(String[] args)
{
double nr1=0,nr2=0,nr3=0,pi=0;
int fo1=1, fo2=1, fo3=1;
for(int i=0; i<=20; i++){
for(int fl1=1; fl1<=(6*i); fl1++){fo1 = fo1 * fl1;}
for(int fl2=1; fl2<=(3*i); fl2++){fo2 = fo2 * fl2;}
for(int fl3=1; fl3<=(i); fl3++){fo3 = fo3 * fl3;}
nr1 = ( (Math.pow(-1, i)) * (fo1) * ((545140134*i) + 13591409) );
nr2 = ( (fo2) * (Math.pow(fo3, i)) * ( Math.pow(Math.pow(640320, 3), (i+(1/2)) )) );
nr3 = 12 * (nr1/nr2);
}
pi = 1/nr3;
System.out.println((Math.PI));
System.out.println(pi);
}
}
There are many issues here.
As Andy mentioned, 1/2 is not 0.5.
You are using integers to compute things like 120! which is completely out of bounds for any primitive type.
f01,f02,f03 should be initialized inside each loop, otherwise they grow even bigger
It is not trivial to fix it. You can take a look at
Error calculating pi using the Chudnovsky algorithm - Java
and
http://www.craig-wood.com/nick/articles/pi-chudnovsky/
for some hints, but don't expect built-in primitive types to work with that algorithm.