We Keep Coding

Really slow Dijkstra algorithm, what am I doing wrong?

I was tasked to perform the Dijkstra Algorithm on big graphs (25 million nodes). These are represented as a 2D array: -each node as a double[] with latitude, longitude and offset (offset meaning index of the first outgoing edge of that node)
-each edge as a int[] with sourceNodeId,targetNodeId and weight of that edge
Below is the code, I used int[] as a tupel for the comparison in the priority queue.
The algorithm is working and gets the right results HOWEVER it is required to be finished in 15s but takes like 8min on my laptop. Is my algorithm fundamentally slow? Am I using the wrong data structures? Am I missing something? I tried my best optimizing as far as I saw fit.
Any help or any ideas would be greatly appreciated <3
public static int[] oneToAllArray(double[][]nodeList, int[][]edgeList,int sourceNodeId) {
int[] distance = new int[nodeList[0].length]; //the array that will be returned
//the priorityQueue will use arrays with the length 2, representing [index, weight] for each node and order them by their weight
PriorityQueue<int[]> prioQueue = new PriorityQueue<>((a, b) -> ((int[])a)[1] - ((int[])b)[1]);
int offset1; //used for determining the amount of outgoing edges
int offset2;
int newWeight; //declared here so we dont need to declare it a lot of times later (not sure if that makes a difference)
//currentSourceNode here means the node that will be looked at for OUTGOING edges
int[] currentSourceNode= {sourceNodeId,0};
prioQueue.add(currentSourceNode);
//at the start we only add the sourceNode, then we start the actual algorithm
while(!prioQueue.isEmpty()) {
if(prioQueue.size() % 55 == 2) {
System.out.println(prioQueue.size());
}
currentSourceNode=prioQueue.poll();
int sourceIndex = currentSourceNode[0];
if(sourceIndex == nodeList[0].length-1) {
offset1= (int) nodeList[2][sourceIndex];
offset2= edgeList[0].length;
} else {
offset1= (int) nodeList[2][sourceIndex];
offset2= (int) nodeList[2][sourceIndex+1];
}
//checking every outgoing edge for the currentNode
for(int i=offset1;i<offset2;i++) {
int targetIndex = edgeList[1][i];
//if the node hasnt been looked at yet, the weight is just the weight of this edge + distance to sourceNode
if(distance[targetIndex]==0&&targetIndex!=sourceNodeId) {
distance[targetIndex] = distance[sourceIndex] + edgeList[2][i];
int[]targetArray = {targetIndex, distance[targetIndex]};
prioQueue.add(targetArray);
} else if(prioQueue.stream().anyMatch(e -> e[0]==targetIndex)) {
//above else if checks if this index is already in the prioQueue
newWeight=distance[sourceIndex]+edgeList[2][i];
//if new weight is better, we have to update the distance + the prio queue
if(newWeight<distance[targetIndex]) {
distance[targetIndex]=newWeight;
int[] targetArray;
targetArray=prioQueue.stream().filter(e->e[0]==targetIndex).toList().get(0);
prioQueue.remove(targetArray);
targetArray[1]=newWeight;
prioQueue.add(targetArray);
}
}
}
}
return distance;
}

For each node that you process, you are doing a linear scan of the priority queue to see if something is already queued, and a second scan to find all the things that are queued if you have to update the distance. Instead, keep a separate multi-set of things that are in the queue.

This is not a proper Dijkstra's implementation.
One of the key elements of Dijkstra is that you mark nodes as "visited" when they have been evaluated and prevent looking at them again because you can't do any better. You are not doing that, so your algorithm is doing many many more computations than necessary. The only place where a priority queue or sort is required is to pick the next node to visit, from amongst the unvisited. You should re-read the algorithm, implement the "visitation tracking" and re-formulate.

DFS (depth first search) sequence of nodes

I want to implement dfs for nodes that are of type long in Java.
My algorithm calculates correctly the number of nodes, and the number
of edges, but not the sequence of nodes. Could you please help me
modify my algorithm so I calculate the order in which the nodes are
visited, correctly?
This is my code:
private int getNumberOfNodes(long firstNode) {
List<Long> marked = new ArrayList<>(); //------------------------------------------->
Stack<Long> stack = new Stack<Long>(); //step 1 Create/declare stack
stack.push(firstNode); //Step 2 Put/push inside the first node
while (!stack.isEmpty()) { //Repeat till stack is empty:
Long node = stack.pop(); //Step 3 Extract the top node in the stack
marked.add(node); //------------------------------------------->
long[] neighbors = xgraph.getNeighborsOf(node); //Get neighbors
if (neighbors.length % 2 == 0) {
} else {
numOfNodesWithOddDegree++;
}
int mnt = 0;
for (long currentNode : neighbors) {
if (!marked.contains(currentNode) && !stack.contains(currentNode) ) { //&& !stack.contains(currentNode)
stack.push(currentNode);
} else {
}
if (!marked.contains(currentNode)) {
numOfEdges++;
}
}
}
return marked.size(); //(int) Arrays.stream(neighbors).count();
}

I guess you exam the marked list for the sequence.
As your graph is undirected, the sequence of traversals could be varied based on which neighbor you pushed into the stack first. which means the logic of your function:
xgraph.getNeighborsOf(node)
could impact your sequence. see Vertex orderings from this wiki https://en.wikipedia.org/wiki/Depth-first_search
so my conclusion is: you may have a different traversal sequence, it does not mean your DFS is wrong, as long as it is Deep first search, it is ok to be a little bit different from the given answer.

Neo4j. Dijkstra. Find paths including a set of nodes (graphalgo)

Now I'm testing dijkstra (org.neo4j.graphalgo.impl.shortestpath.*). The code you can see below:
Dijkstra<Double> dijkstra = new Dijkstra<>(0.0,
startNode,
endNode,
CommonEvaluators.doubleCostEvaluator("weight"),
new DoubleAdder(),
new DoubleComparator(),
Direction.BOTH,
RelationshipTypes.rel);
How I can define nodes which must be included in path? Any ideas?

You will have to brute force all possible paths. As long you dont have to many nodes you want to include you can use this approach. It is basically the traveling sales man problem.
what you can do:
Find all permutations of nodes (so you order the nodes in all possible ways)
Start a shortest path for every permutations going from the first to the second node and so on
Compare the weigth of all paths and take the shortest one
Keep in mind to keep the amount of nodes as small as possible because the TSP is an NP-hard problem. So dont go to close to like 10 nodes to include.
I already requested this feature on github also with some code.
Here is my request.

As Yoshi said, this is a TSP. In your solution, you might run into trouble when going through all possible paths as the amount increases quickly on big graphs.
My idea to possibly improve performance would be to run a Dijkstra between each of the nodes you want to have included in your final path and then create a new graph, just with those nodes and the newly calculated distances as weights in it and run the Dijkstra algorithm and the analyzis of wether all nodes are included in a given path on this (considerably) smaller graph.

So I've found the simple solution. Just made any changes. Not sure that it's correct but it works.
Dijkstra<Double> dijkstra = new Dijkstra<>(0.0,
startPoint,
endPoint,
new CostEvaluator<Double>() {
#Override
public Double getCost(final Relationship relationship, Direction direction) {
if (listIncAll.contains(relationship.getStartNode()) || listIncAll.contains(relationship.getEndNode())) {
return 0.0;
}else {
return relationship.getProperty("weight");
}
}
},
new DoubleAdder(),
new DoubleComparator(),
Direction.BOTH,
RelationshipTypes.rel);
Map<List<PropertyContainer>, Double> pathMap = new HashMap<>();
for (List<PropertyContainer> path : dijkstra.getPaths()) {
if (path.containsAll(listIncAll)) {
double fullWeight = 0;
Iterator<PropertyContainer> i = path.iterator();
while (i.hasNext()){
i.next();
if (i.hasNext()) {
Relationship r = (Relationship) i.next();
fullWeight += r.getProperty("weight");
}
}
pathMap.put(path,fullWeight);
}
}
System.out.println(pathMap.entrySet().stream().sorted((k1, k2) -> -k2.getValue().
compareTo(k1.getValue())).findFirst().get().getKey());
So using CostEvaluator I've found paths which have at least one point from a list of points which must be included. To get paths which have all points I've just added "if (path.containsAll(listIncAll))". In last sout you'll see the found path.

A* Pathfinding Algorithm -- finding a path, but not optimal best path

I'm working on an A* algorithm. This is the code for the pathfinding method. For reference, this is the board I am working with: http://i.imgur.com/xaAzNSw.png?1 Each color tile represents a different heuristic value. For some unknown reason, it finds a path every single time, just not always the correct path. Here is the code for the pathfinding method. If anyone needs any clarifications, I'd be happy to provide them.
public List<Point> findPath(Point start, Point end) {
//declarations and instantiations
List<PathState> closedList = new ArrayList<PathState>(); //the nodes already considered
List<PathState> openList = new ArrayList<PathState>(); //nodes to be considered
openList.add(new PathState(start, end, tm)); //add starting point
PathState current = openList.get(0);
while(!current.isGoal()){
//sort open list to find the pathstate with the best hscore(sorts by hscore)
Collections.sort(openList);
current = openList.get(openList.size() - 1);
closedList.add(current);
openList.remove(current);
//get the valid children of current node
List<PathState> children = current.getChildren();;
if(!current.isGoal()){
for(int i = 0; i < children.size(); i++){
if(!closedList.contains(children.get(i))){
if(openList.contains(children.get(i))){
if(openList.get(openList.indexOf(children.get(i))).getgScore() > children.get(i).getgScore()){
//child is already on the open list, but this node has lower g value
//change g value and parent of node on open list
openList.get(openList.indexOf(children.get(i))).setG(children.get(i).getgScore());
openList.get(openList.indexOf(children.get(i))).changeParent(current);
}
}else{
//child not in closed list
openList.add(children.get(i));
//repaint the terrain panel with shades
tp.addAstarState(children.get(i));
tp.repaint();
try {
Thread.sleep(25);
} catch(Exception e) {
e.printStackTrace();
}
}
}
}
}
}
//returns the path from winning node to start for output
return current.getPath();
}

A* is basically Djikstra's algorithm but you direct your search to the destination by combining your distance function with an estimate of the remaining distance.
At node x, the cost or "score" is (distance_so_far) for djikstra
at A*, it is (distance_so_far + estimate_of_remaining_distance)
To make sure that A* finds the shortest path, the estimate_of_remaining_distance must be a true lower bound. That means it must always be less than the actual remaining distance. That means, you can never overestimate this distance otherwise it becomes an inexact heuristic in which case it won't necessarily find the shortest path.
This is a good reference: http://theory.stanford.edu/~amitp/GameProgramming/AStarComparison.html
It links to this reference which explains more about heuristic functions: http://theory.stanford.edu/~amitp/GameProgramming/Heuristics.html

Setting up an A* search

I am writing a small section of a program in which i have to write up a pathfinding algorithm. The function takes in what will be known as 'routes' that each define a start and end point in 2D space. The algorithm is required to find the shortest and most efficient (to a degree) path (from the origin) to take through these routes, minimising the total distance traveled overall.
I did a bit of research and started down a path that i thought might work. So far i have converted the routes into a directed graph which is all linked up as if it were an idealised road map. I then attempted to perform an A* search on this graph. The heuristic i used calculates the total distance of the 'routes' left to travel and the distance from start (G) value i used was just the cumulative distance traveled to get to the current point. This works for some input but others return no path at all and i cant seem to figure out why.
Is is possible that my heuristic is wrong and this is causing a miscalculation somewhere or is it more likely that the A* procedure itself is wrong? or am i just on completely the wrong track here?
I'll put the getPath function below (written in Java) just in case that helps.
Thanks in advance.
public ArrayList<Vector2> getPath()
{
PriorityQueue<SearchNode> openList = new PriorityQueue<SearchNode>(10, new SearchNodeComparator());
ArrayList<SearchNode> closedList = new ArrayList<SearchNode>();
map.startJobs();
searchDepth = 0;
SearchNode start = searchableGraph.getNode(new Vector2(0, 0));
int goalsLeft = map.getJobCount();
start.setDistanceTraveled(0);
openList.add(start);
while (openList.size() > 0)
{
SearchNode current = openList.peek();
searchDepth++;
if (map.isJobEndPoint(current.getValue()))
{
map.completeJob(current.getValue());
goalsLeft--;
}
if (reachedGoalState(current, searchableGraph.getNodes().size()))
{
return getFinalPath(current);
}
else
{
ArrayList<SearchNode> neighbours = getNeighbours(current);
for (int i = 0; i < neighbours.size(); i++)
{
SearchNode node = neighbours.get(i);
System.out.print("Inspecting node" + node.getValue().toString());
float distanceTraveled = current.getDistanceTraveled() + getDistance(current.getValue(), node.getValue());
float heuristic = heuristic(node);
if (!openList.contains(node) && !closedList.contains(node))
{
node.setDistanceTraveled(distanceTraveled);
node.setDistanceToGoal(distanceTraveled + heuristic);
node.setParent(current);
openList.add(node);
}
else if(openList.contains(node))
{
if (node.getDistanceTraveled() <= distanceTraveled)
{
node.setDistanceToGoal(distanceTraveled + heuristic);
node.setParent(current);
}
}
}
openList.remove(current);
closedList.add(current);
}
}
return new ArrayList<Vector2>();
}

The heuristic i used calculates the total distance of the 'routes' left to travel
The heuristic used by A* must be admissible; that is, it must never over​estimate the distance to the end.
If I understand your description correctly, your heuristic is not admissible, so is not guaranteed to work.