We Keep Coding

A question about distance calculations in Spatial4J

I'm quite confused about distance calculations.
I'm using this website as a reference:
https://www.omnicalculator.com/other/latitude-longitude-distance
Then I got an implementation of a distance calculation (haversine) from here:
https://tutorialspoint.dev/algorithm/geometric-algorithms/program-distance-two-points-earth
And I want to compare this with various different implementations from Spatial4J.
I'm using these coordinates to test with:
Point 1: 40.688939, -74.04455
Point 2: 40.746853, -73.985633
And I'm getting pretty big differences between different ways of calculating the distance between these two points.
First, what mainly concerns me is that the website (omnicalculator) and the implementation from tutorialspoint agree completely on the distance: 8.132 km
But none of my Spatial4J calculations agree with that number. The one that comes closest to it is the CartesianDistCalc implementation at 8.262 km. The tutorialspoint demo code claims to be using haversine but the output of Spatial4J haversine DistCalc implementation is quite a ways off at 7.313 km.
But can somebody explain to me where these differences are coming from and what the "correct" one is?
Below is my experimental code:
import org.junit.jupiter.api.Test;
import org.locationtech.spatial4j.context.SpatialContext;
import org.locationtech.spatial4j.distance.CartesianDistCalc;
import org.locationtech.spatial4j.distance.GeodesicSphereDistCalc;
class GeodesicCalculationTest {
#Test
void testGeodesicCalculations(){
SpatialContext ctx = SpatialContext.GEO;
var startPoint = ctx.getShapeFactory().pointLatLon(40.688939, -74.04455);
var endPoint = ctx.getShapeFactory().pointLatLon(40.746853, -73.985633);
System.out.println("GEO spatial context: " + ctx.calcDistance(startPoint, endPoint) * 100);
System.out.println("Haversine: " + new GeodesicSphereDistCalc.Haversine().distance(startPoint, endPoint) * 100);
System.out.println("Law of cosine: " + new GeodesicSphereDistCalc.LawOfCosines().distance(startPoint, endPoint) * 100);
System.out.println("Vincenty: " + new GeodesicSphereDistCalc.Vincenty().distance(startPoint, endPoint) * 100);
System.out.println("Cartesian: " + new CartesianDistCalc().distance(startPoint, endPoint) * 100);
System.out.println("Tutorials Point (haversine): " + distance(startPoint.getLat(), endPoint.getLat(), startPoint.getLon(), endPoint.getLon()));
}
public static double distance(double lat1, double lat2, double lon1, double lon2) {
// The math module contains a function
// named toRadians which converts from
// degrees to radians.
lon1 = Math.toRadians(lon1);
lon2 = Math.toRadians(lon2);
lat1 = Math.toRadians(lat1);
lat2 = Math.toRadians(lat2);
// Haversine formula
double dlon = lon2 - lon1;
double dlat = lat2 - lat1;
double a = Math.pow(Math.sin(dlat / 2), 2)
+ Math.cos(lat1) * Math.cos(lat2)
* Math.pow(Math.sin(dlon / 2),2);
double c = 2 * Math.asin(Math.sqrt(a));
// Radius of earth in kilometers. Use 3956
// for miles
double r = 6371;
// calculate the result
return(c * r);
}
}
And the output of running it:
GEO spatial context: 7.31307025220976
Haversine: 7.31307025220976
Law of cosine: 7.313070251733588
Vincenty: 7.3130702522095286
Cartesian: 8.261503667613857
Tutorials Point (haversine): 8.131763102409689
I'm multiplying the Spatial4J calculations by 100 which is also confusing to me... it doesn't really make sense that Spatial4J is giving me answers as 1/100th of a kilometer???
I realise I must be doing something wrong or completely misunderstanding some premises here. I would really appreciate some help in understanding what that is.

The result of SpatialContext.calcDistance(Point p, Point p2) is in degrees. To convert those degrees to km you need to multiply it by the constant DistanceUtils.DEG_TO_KM
SpatialContext ctx = SpatialContext.GEO;
var startPoint = ctx.getShapeFactory().pointLatLon(40.688939, -74.04455);
var endPoint = ctx.getShapeFactory().pointLatLon(40.746853, -73.985633);
double distanceInDegrees = ctx.calcDistance(startPoint, endPoint);
double distanceInKm = distanceInDegrees * DistanceUtils.DEG_TO_KM;
System.out.println("GEO spatial context: " + distanceInKm);
This gives you the output:
GEO spatial context: 8.131774297975046
Which is very close to your example from
Tutorials Point (haversine): 8.131763102409689
The difference is because Spatial4j is using the value 6371.0087714 for the radius of the earth in km, where as in your example you are using 6371.

Getting coordinates in GeoTools

Thanks to Reading ESRI shapefiles from the InputStream in Java I can read my shapefiles and access every GeometryAttribute, but I also need to convert it coordinates in long/lat format, it might be 40°44′55″N, 73 59 11W or best 40.7486, -73.9864.
example of my WKT is
GeometryAttribute sourceGeometry = feature.getDefaultGeometryProperty();
CoordinateReferenceSystem example = sourceGeometry.getDescriptor().getCoordinateReferenceSystem();
String wkt = example.toWKT();
PROJCS["ETRS_1989_Poland_CS92",
GEOGCS["GCS_ETRS_1989",
DATUM["D_ETRS_1989",
SPHEROID["GRS_1980", 6378137.0, 298.257222101]],
PRIMEM["Greenwich", 0.0],
UNIT["degree", 0.017453292519943295],
AXIS["Longitude", EAST],
AXIS["Latitude", NORTH]],
PROJECTION["Transverse_Mercator"],
PARAMETER["central_meridian", 19.0],
PARAMETER["latitude_of_origin", 0.0],
PARAMETER["scale_factor", 0.9993],
PARAMETER["false_easting", 500000.0],
PARAMETER["false_northing", -5300000.0],
UNIT["m", 1.0],
AXIS["x", EAST],
AXIS["y", NORTH]]

GeoTools has a number of ways to reproject your geometries depending on what you want to do after your reproject them.
The simplest is to use a ReprojectingFeatureCollection to provide you with a new collection in your required projection (in this case EPSG:4326) or you can create a JTS.transform and use that on individual geometries.
ReprojectingFeatureCollection rfc = new ReprojectingFeatureCollection(features, CRS.decode("epsg:4326"));
or
CoordinateReferenceSystem source = sourceGeometry.getDescriptor().getCoordinateReferenceSystem();
CoordinateReferenceSystem target = CRS.decode("epsg:4326");
MathTransform transform = CRS.findMathTransform(source, target, lenient);
Geometry geometry2 = JTS.transform(geometry, transform);
Printing the coordinates of those new geometries will give you decimal degrees (3.234545) if you need DMS (1°3'3") then a class like this will help:
public class DMSToDegrees {
static public double convert(int degrees, int minutes, double seconds) {
//to allow for negative (i.e. W or S) values note the sign of the degrees
float sign = Math.signum(degrees);
if(sign==0.0) {
//we'll consider 0 to be positive
sign = 1.0f;
}
//seconds are 1/60th of a minute so express as a fractional minute
double dmins = minutes+seconds/60.0;
//minutes are 1/60th of a degree so express as a fractional degree
double deg = Math.abs(degrees) + dmins/60.0;
// put the sign back on the result
return sign*deg;
}
static public double[] reverse(double degrees){
//to allow for negative (i.e. W or S) values note the sign of the degrees
double sign = Math.signum(degrees);
if(sign==0.0) {
//we'll consider 0 to be positive
sign = 1.0f;
}
double[] ret = new double[3];
degrees = Math.abs(degrees);
ret[0] = Math.floor(degrees);
double mins = degrees - ret[0];
ret[1] = Math.floor(mins*60);
ret[2] = ((mins*60 - ret[1]))*60;
ret[0]*=sign;
return ret;
}
}

Work out the 50 miles radius from London latitude and Longitude coordinates

I have written a Java API, which consumes another API, which is a list of users, with the following properties in JSON format. Users have firstname, lastname, IP address, email, and location coordinates of latitude and longitude.
The Java API written is supposed to get all the users who live in London and/or live in a 50 mile radius.
Mine doesn't because I can't figure out the formula needed to check for the users who live in London, or within a 50 mile radius.
Here is my Java API:
package com.company;
import org.json.JSONArray;
import org.json.JSONObject;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
public class Main
{
public static void main(String[] args)
{
// Using java.net.http.HttpClient
HttpClient client = HttpClient.newHttpClient();
HttpRequest request = HttpRequest.newBuilder().uri(URI.create("https://bpdts-test-app.herokuapp.com/city/London/users")).build();
client.sendAsync(request, HttpResponse.BodyHandlers.ofString())
.thenApply(HttpResponse::body)
.thenApply(Main::parse)
.join();
}
// Parse the returned JSON data
public static String parse(String responseBody)
{
System.out.println("People who live in London");
JSONArray usersLondon = new JSONArray((responseBody));
for (int i = 0; i < usersLondon.length(); i++)
{
JSONObject userLondon = usersLondon.getJSONObject(i);
int id = userLondon.getInt("id");
String first_name = userLondon.getString("first_name");
String last_name = userLondon.getString("last_name");
String email = userLondon.getString("email");
String ip_address = userLondon.getString("ip_address");
int latitude = userLondon.getInt("latitude");
int longitude = userLondon.getInt("longitude");
System.out.println("id: " + id + " " + "first name: " + first_name + " " + "last name: " + last_name + " " + "email: " + email + " "
+ "IP Address: " + ip_address + " " + "latitude: " + latitude + " " + "longtitude: " + longitude);
}
return null;
}
}
So it returns just 6 users, which I know is incorrect. What would the Mathematical formula be, to test whether the coordinates for users in the API are are living in London, and/or living within a 50 mile radius of London?
Appreciate your help.

The way to properly and accurately calculate distance between two points on the earth is with the Inverse/Forward formulas (written in 1975; as far as I know, no one has managed to come up with better formulas since then).
Since your distance is short, you might be able to get away with using sphere-based calculations, but if you want to be truly correct, Inverse/Forward is the way to do it. This is because our planet is not a perfect sphere. The distance from pole to pole is slightly smaller than the equatorial diameter, so the planet is a “squashed” sphere, formally known as an oblate spheroid. This difference is enough to matter when navigating and calculating distances, unless those distances are very small.
Translating the original Inverse/Forward Fortran code is possible (I’ve done it for more than one project), but it’s likely to be easier to make use of the free libraries which do it, such as this one.
The numbers that describe the difference between the planet’s polar and equatorial diameters are known as a “reference ellipsoid.” The one most commonly used is the WGS84 ellipsoid, which is accurate enough for most purposes. It just so happens that the above class has a convenient static WGS84 instance defined.
Before you can calculate anything, first you need to define what “within a 50 mile radius” means. Within 50 miles of what, exactly? The center of London?
Wikipedia says that London is at 51°30′26″N 0°7′39″W, which seems like as reasonable a starting point as any for distance calculations.
Traditionally, latitude and longitude are expressed in decimal degrees when doing calculations, which means a double precision floating point number (that is, a Java double) whose integer part is the number of degrees, and whose decimal part is the minutes and seconds. By convention, positive values are north or east, while negative values are south or west.
Thus, 50°30′N 99°15′W is a latitude of 50.5 and a longitude of -99.25, in decimal degrees.
Your code is obtaining latitude and longitude as ints. I very much doubt that your locations are integer numbers of degrees, since very few locations are at, for example, exactly 49° north. Only you know how those int values are intended to represent minutes and seconds. It is up to you to convert those values to decimal degrees.
Once you have London’s location in decimal degrees, and you know how to convert your user locations into decimal degrees, you can invoke the Inverse method of the Geodesic class I linked to above:
public static List<User> parse(String responseBody)
{
List<User> qualifyingUsers = new ArrayList<>();
// 51 deg 30 min 26 sec N
double londonLat = 51 + (30 / 60.0) + (26 / 60.0 / 60.0);
// 0 deg 7 min 39 sec W
double londonLon = 0 - (7 / 60.0) - (39 / 60.0 / 60.0);
for (int i = 0; i < usersLondon.length(); i++)
{
JSONObject userLondon = usersLondon.getJSONObject(i);
// ...
int latitude = userLondon.getInt("latitude");
int longitude = userLondon.getInt("longitude");
double userLat = convertToDecimalDegrees(latitude);
double userLon = convertToDecimalDegrees(longitude);
GeodesicData result =
Geodesic.WGS84.Inverse(londonLat, londonLon, userLat, userLon);
double distanceInMeters = result.s12;
double distanceInMiles = distanceInMeters / 1609.34;
if (distanceInMiles <= 50)
{
User user = new User();
user.setId(id);
user.setFirstName(first_name);
// etc.
qualifyingUsers.add(user);
}
}
return qualifyingUsers;
}

Implementation of Logistic regression with Gradient Descent in Java

I have implemented Logistic Regression with Gradient Descent in Java. It doesn't seem to work well (It does not classify records properly; the probability of y=1 is a lot.) I don't know whether my implementation is correct.I have gone through the code several times and i am unable to find any bug. I have been following Andrew Ng's tutorials on Machine learning on Course Era. My Java implementation has 3 classes. namely :
DataSet.java : To read the data set
Instance.java : Has two members : 1. double[ ] x and 2. double label
Logistic.java : This is the main class that implements Logistic Regression with Gradient Descent.
This is my cost function:
J(Θ) = (- 1/m ) [Σmi=1 y(i) log( hΘ( x(i) ) ) + (1 - y(i) ) log(1 - hΘ (x(i)) )]
For the above Cost function, this is my Gradient Descent algorithm:
Repeat ( Θj := Θj - α Σmi=1 ( hΘ( x(i)) - y(i) ) x(i)j
(Simultaneously update all Θj )
)
import java.io.FileNotFoundException;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
public class Logistic {
/** the learning rate */
private double alpha;
/** the weight to learn */
private double[] theta;
/** the number of iterations */
private int ITERATIONS = 3000;
public Logistic(int n) {
this.alpha = 0.0001;
theta = new double[n];
}
private double sigmoid(double z) {
return (1 / (1 + Math.exp(-z)));
}
public void train(List<Instance> instances) {
double[] temp = new double[3];
//Gradient Descent algorithm for minimizing theta
for(int i=1;i<=ITERATIONS;i++)
{
for(int j=0;j<3;j++)
{
temp[j]=theta[j] - (alpha * sum(j,instances));
}
//simulataneous updates of theta
for(int j=0;j<3;j++)
{
theta[j] = temp[j];
}
System.out.println(Arrays.toString(theta));
}
}
private double sum(int j,List<Instance> instances)
{
double[] x;
double prediction,sum=0,y;
for(int i=0;i<instances.size();i++)
{
x = instances.get(i).getX();
y = instances.get(i).getLabel();
prediction = classify(x);
sum+=((prediction - y) * x[j]);
}
return (sum/instances.size());
}
private double classify(double[] x) {
double logit = .0;
for (int i=0; i<theta.length;i++) {
logit += (theta[i] * x[i]);
}
return sigmoid(logit);
}
public static void main(String... args) throws FileNotFoundException {
//DataSet is a class with a static method readDataSet which reads the dataset
// Instance is a class with two members: double[] x, double label y
// x contains the features and y is the label.
List<Instance> instances = DataSet.readDataSet("data.txt");
// 3 : number of theta parameters corresponding to the features x
// x0 is always 1
Logistic logistic = new Logistic(3);
logistic.train(instances);
//Test data
double[]x = new double[3];
x[0]=1;
x[1]=45;
x[2] = 85;
System.out.println("Prob: "+logistic.classify(x));
}
}
Can anyone tell me what am I doing wrong?
Thanks in advance! :)

As I am studying logistic regression, I took the time to review your code in detail.
TLDR
In fact, it appears the algorithm is correct.
The reason you had so much false negatives or false positives is, I think, because of the hyper parameters you choose.
The model was under-trained so the hypothesis was under-fitting.
Details
I had to create the DataSet and Instance classes because you did not publish them, and set up a train data set and a test data set based on the Cryotherapy dataset.
See http://archive.ics.uci.edu/ml/datasets/Cryotherapy+Dataset+.
Then, using your same exact code (for the logistic regression part) and by choosing an alpha rate of 0.001 and a number of iterations of 100000, I got a precision rate of 80.64516129032258 percent on the test data set, which is not so bad.
I tried to get a better precision rate by tweaking manualy those hyper parameters but could not obtain any better result.
At this point, an enhancement would be to implement regularization, I suppose.
Gradient descent formula
In Andrew Ng's video about the the cost function and gradient descent, it is correct that the 1/m term is omitted.
A possible explanation is that the 1/m term is included in the alpha term.
Or maybe it's just an oversight.
See https://www.youtube.com/watch?v=TTdcc21Ko9A&index=36&list=PLLssT5z_DsK-h9vYZkQkYNWcItqhlRJLN&t=6m53s at 6m53s.
But if you watch Andrew Ng's video about regularization and logistic regression you'll notice that the term 1/m is clearly present in the formula.
See https://www.youtube.com/watch?v=IXPgm1e0IOo&index=42&list=PLLssT5z_DsK-h9vYZkQkYNWcItqhlRJLN&t=2m19s at 2m19s.

Find average location and remove other locations:

Let's say that I am collecting users location data every 20 minutes. At the end of the day I have an ArrayList of objects that contain lat and long fields and other data.
I have two problems that I am facing and trying to figure out:
Some of the locations are taken from inside of a building so they are not very accurate and could be spread around the actual location where the user was at the time.
Some of the locations are taken at different times but from the same location, as the user didn't moved.
What I want to achieve is to find all the locations that are near one another: lets say 70 meters, find the average location of all those locations and replace them only with this one average location.
So I am coming to the two important questions:
What would be the best way to find all near locations < 70 meter distance (Take in mind that the array contains valid changes in location. So I have to find the groups of near ones and leave the others intact).
Is there a method or a way to find the average location of many near ones?

Regarding near positions I previously answered a similar question here: Android Maps v2 - animate camera to include most markers
Specifically I think you would be able to use this piece of code:
private List<Marker> getSurroundingMarkers(List<Marker> markers,
LatLng origin, int maxDistanceMeters) {
List<Marker> surroundingMarkers = surroundingMarkers = new ArrayList<Marker>();
if (markers == null) return surroundingMarkers ;
for (Marker marker : markers) {
double dist = distBetween(origin, marker.getPosition());
if (dist < maxDistanceMeters) {
surroundingMarkers.add(marker);
}
}
return surroundingMarkers;
}
private float distBetween(LatLng pos1, LatLng pos2) {
return distBetween(pos1.latitude, pos1.longitude, pos2.latitude,
pos2.longitude);
}
/** distance in meters **/
private float distBetween(double lat1, double lng1, double lat2, double lng2) {
double earthRadius = 3958.75;
double dLat = Math.toRadians(lat2 - lat1);
double dLng = Math.toRadians(lng2 - lng1);
double a = Math.sin(dLat / 2) * Math.sin(dLat / 2)
+ Math.cos(Math.toRadians(lat1))
* Math.cos(Math.toRadians(lat2)) * Math.sin(dLng / 2)
* Math.sin(dLng / 2);
double c = 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1 - a));
double dist = earthRadius * c;
int meterConversion = 1609;
return (float) (dist * meterConversion);
}
Possibly rewriting the Marker part to use LatLng instead.
regarding the averaging, it should be a simple matter of (say you have n LatLng's):
lat_avg = (lat1+lat2+...+latn)/n
lng_avg = (lng1+lng2+...+lngn)/n
latlng_avg = new LatLng(lat_avg, lat_lng)

I' not sure how you're getting the user's location - whether your using a traditional LocationManager or play services. I've always modeled my location getting on this approach documented on the android developers blog. If you're flexible in switching between location gathering methods, whether the person is inside or outside should not matter that much. You should be getting the best possible location at any given time.
Assuming you're getting locations on a sufficient time schedule (I don't know when you're checking for updates but since you want everything inside a 70m radius I'm assuming its on a time schedule and not distance change) the basic way to find the average point is:
(1) Convert each lat/long pair into a unit-length 3D vector.
(2) Sum each of those vectors
(3) Normalise the resulting vector
(4) Convert back to spherical coordinates
That approach is documented here as well as in a much earlier SO post on calculating the average of a set of angles
The example code is pretty easy to follow - just plug in the lat long values you get from your location grab and you should be ok.

Well for markers that come from the same location I have created the following method:
public ArrayList<MyLocation> removeSameLocationMarkers(List<ParseObject> objects, int maxDistanceMeters)
{
boolean isLocationExist;
ArrayList<MyLocation> acceptedLocations = new ArrayList<MyLocation>();
if (objects == null) return acceptedLocations;
for (ParseObject location1 : objects)
{
isLocationExist = false;
for (MyLocation location2 : acceptedLocations)
{
if (!location1.equals(location2))
{
float distance = distBetween(location1.getDouble("latitude"), location1.getDouble("longitude"), location2.getLatitude(), location2.getLongitude());
if (distance < maxDistanceMeters)
{
location2.addTimeToLocation(location1.getString("time"));
isLocationExist = true;
}
}
}
if (!isLocationExist)
{
Location newLocation = new Location("");
newLocation.setLatitude(location1.getDouble("latitude"));
newLocation.setLongitude(location1.getDouble("longitude"));
String provider = location1.getString("provider");
if (provider != null)
{
newLocation.setProvider(provider);
}
MyLocation newMyLocation = new MyLocation(newLocation);
newMyLocation.addTimeToLocation(location1.getString("time"));
acceptedLocations.add(newMyLocation);
}
}
return acceptedLocations;
}