I'm trying to use a PriorityQueue to order objects using a Comparator.
This can be achieved easily, but the objects class variables (with which the comparator calculates priority) may change after the initial insertion. Most people have suggested the simple solution of removing the object, updating the values and reinserting it again, as this is when the priority queue's comparator is put into action.
Is there a better way other than just creating a wrapper class around the PriorityQueue to do this?
You have to remove and re-insert, as the queue works by putting new elements in the appropriate position when they are inserted. This is much faster than the alternative of finding the highest-priority element every time you pull out of the queue. The drawback is that you cannot change the priority after the element has been inserted. A TreeMap has the same limitation (as does a HashMap, which also breaks when the hashcode of its elements changes after insertion).
If you want to write a wrapper, you can move the comparison code from enqueue to dequeue. You would not need to sort at enqueue time anymore (because the order it creates would not be reliable anyway if you allow changes).
But this will perform worse, and you want to synchronize on the queue if you change any of the priorities. Since you need to add synchronization code when updating priorities, you might as well just dequeue and enqueue (you need the reference to the queue in both cases).
I don't know if there is a Java implementation, but if you're changing key values alot, you can use a Fibonnaci heap, which has O(1) amortized cost to decrease a key value of an entry in the heap, rather than O(log(n)) as in an ordinary heap.
One easy solution that you can implement is by just adding that element again into the priority queue. It will not change the way you extract the elements although it will consume more space but that also won't be too much to effect your running time.
To proof this let's consider dijkstra algorithm below
public int[] dijkstra() {
int distance[] = new int[this.vertices];
int previous[] = new int[this.vertices];
for (int i = 0; i < this.vertices; i++) {
distance[i] = Integer.MAX_VALUE;
previous[i] = -1;
}
distance[0] = 0;
previous[0] = 0;
PriorityQueue<Node> pQueue = new PriorityQueue<>(this.vertices, new NodeComparison());
addValues(pQueue, distance);
while (!pQueue.isEmpty()) {
Node n = pQueue.remove();
List<Edge> neighbours = adjacencyList.get(n.position);
for (Edge neighbour : neighbours) {
if (distance[neighbour.destination] > distance[n.position] + neighbour.weight) {
distance[neighbour.destination] = distance[n.position] + neighbour.weight;
previous[neighbour.destination] = n.position;
pQueue.add(new Node(neighbour.destination, distance[neighbour.destination]));
}
}
}
return previous;
}
Here our interest is in line
pQueue.add(new Node(neighbour.destination, distance[neighbour.destination]));
I am not changing priority of the particular node by removing it and adding again rather I am just adding new node with same value but different priority.
Now at the time of extracting I will always get this node first because I have implemented min heap here and the node with value greater than this (less priority) always be extracted afterwards and in this way all neighboring nodes will already be relaxed when less prior element will be extracted.
Without reimplementing the priority queue yourself (so by only using utils.PriorityQueue) you have essentially two main approaches:
1) Remove and put back
Remove element then put it back with new priority. This is explained in the answers above. Removing an element is O(n) so this approach is quite slow.
2) Use a Map and keep stale items in the queue
Keep a HashMap of item -> priority. The keys of the map are the items (without their priority) and the values of the map are the priorities.
Keep it in sync with the PriorityQueue (i.e. every time you add or remove an item from the Queue, update the Map accordingly).
Now when you need to change the priority of an item, simply add the same item to the queue with a different priority (and update the map of course). When you poll an item from the queue, check if its priority is the same than in your map. If not, then ditch it and poll again.
If you don't need to change the priorities too often, this second approach is faster. Your heap will be larger and you might need to poll more times, but you don't need to find your item.
The 'change priority' operation would be O(f(n)log n*), with f(n) the number of 'change priority' operation per item and n* the actual size of your heap (which is n*f(n)).
I believe that if f(n) is O(n/logn)(for example f(n) = O(sqrt(n)), this is faster than the first approach.
Note : in the explanation above, by priority I means all the variables that are used in your Comparator. Also your item need to implement equals and hashcode, and both methods shouldn't use the priority variables.
It depends a lot on whether you have direct control of when the values change.
If you know when the values change, you can either remove and reinsert (which in fact is fairly expensive, as removing requires a linear scan over the heap!).
Furthermore, you can use an UpdatableHeap structure (not in stock java though) for this situation. Essentially, that is a heap that tracks the position of elements in a hashmap. This way, when the priority of an element changes, it can repair the heap. Third, you can look for an Fibonacci heap which does the same.
Depending on your update rate, a linear scan / quicksort / QuickSelect each time might also work. In particular if you have much more updates than pulls, this is the way to go. QuickSelect is probably best if you have batches of update and then batches of pull opertions.
To trigger reheapify try this:
if(!priorityQueue.isEmpty()) {
priorityQueue.add(priorityQueue.remove());
}
Something I've tried and it works so far, is peeking to see if the reference to the object you're changing is the same as the head of the PriorityQueue, if it is, then you poll(), change then re-insert; else you can change without polling because when the head is polled, then the heap is heapified anyways.
DOWNSIDE: This changes the priority for Objects with the same Priority.
Is there a better way other than just creating a wrapper class around the PriorityQueue to do this?
It depends on the definition of "better" and the implementation of the wrapper.
If the implementation of the wrapper is to re-insert the value using the PriorityQueue's .remove(...) and .add(...) methods,
it's important to point out that .remove(...) runs in O(n) time.
Depending on the heap implementation,
updating the priority of a value can be done in O(log n) or even O(1) time,
therefore this wrapper suggestion may fall short of common expectations.
If you want to minimize your effort to implement,
as well as the risk of bugs of any custom solution,
then a wrapper that performs re-insert looks easy and safe.
If you want the implementation to be faster than O(n),
then you have some options:
Implement a heap yourself. The wikipedia entry describes multiple variants with their properties. This approach is likely to get your the best performance, at the same time the more code you write yourself, the greater the risk of bugs.
Implement a different kind of wrapper: handlee updating the priority by marking the entry as removed, and add a new entry with the revised priority.
This is relatively easy to do (less code), see below, though it has its own caveats.
I came across the second idea in Python's documentation,
and applied it to implement a reusable data structure in Java (see caveats at the bottom):
public class UpdatableHeap<T> {
private final PriorityQueue<Node<T>> pq = new PriorityQueue<>(Comparator.comparingInt(node -> node.priority));
private final Map<T, Node<T>> entries = new HashMap<>();
public void addOrUpdate(T value, int priority) {
if (entries.containsKey(value)) {
entries.remove(value).removed = true;
}
Node<T> node = new Node<>(value, priority);
entries.put(value, node);
pq.add(node);
}
public T pop() {
while (!pq.isEmpty()) {
Node<T> node = pq.poll();
if (!node.removed) {
entries.remove(node.value);
return node.value;
}
}
throw new IllegalStateException("pop from empty heap");
}
public boolean isEmpty() {
return entries.isEmpty();
}
private static class Node<T> {
private final T value;
private final int priority;
private boolean removed = false;
private Node(T value, int priority) {
this.value = value;
this.priority = priority;
}
}
}
Note some caveats:
Entries marked removed stay in memory until they are popped
This can be unacceptable in use cases with very frequent updates
The internal Node wrapped around the actual values is an extra memory overhead (constant per entry). There is also an internal Map, mapping all the values currently in the priority queue to their Node wrapper.
Since the values are used in a map, users must be aware of the usual cautions when using a map, and make sure to have appropriate equals and hashCode implementations.
Related
I am having an issue with a simplistic thread-safe implementation of the observer pattern using a ConcurrentSkipListSet to handle keeping track of observer priorities during insertion. The majority of observers will not have any special priority attributed to them, and following this Comparable#compareTo method will show as equal priority when compared (where priority is a value in an enum of five priorities ranging from highest to lowest):
public int compareTo(BaseLink<?> link) {
return this.priority.compareTo(link.getPriority());
}
When I add observers of equal priorities to the ConcurrentSkipListSet , it seems like some of the added objects are simply lost during the insertion process. Changing the priorities of any of the observers I have created while testing this results in those observers being added to the set without issue, though I assume that given enough observers of the same priority the issue will arise again.
I am unsure about what is causing this issue, and of what I should do to help resolve it. Is there anything I can do to resolve this issue? Alternatively if this is an inherent problem with the ConcurrentSkipListSet, are there any other thread-safe data structures that can give me reasonably performant insertion and sorting times for unique objects?
I presume that you are instantiating like this ConcurrentSkipListSet(myComparator) where myComparator implements compareTo as you have shown us.
A ConcurrentSkipListSet is a Set. When you instantiate one using a Comparator, it will use it to:
order the set elements
determine when a new element is already in the set.
In your code, your Comparator.compareTo(...) is saying that every BaseLink with a given priority is the same. That's what is causing your problem.
Solution:
public int compareTo(BaseLink<?> link) {
int res = this.priority.compareTo(link.getPriority());
if (res == 0) {
// tie-breaker for different links with the same priority
res = // compare using some other key / identifier
}
return res;
}
If your BaseLink objects have a natural key or identifier, you could use that as the tie-breaker for objects with the same priority. Even the object's identity hashcode would do ... if you are not worried about "fairness" or "reproducibility" in the ordering of the set.
I need to implement a priority queue where the priority of an item in the queue can change and the queue adjusts itself so that items are always removed in the correct order. I have some ideas of how I could implement this but I'm sure this is quite a common data structure so I'm hoping I can use an implementation by someone smarter than me as a base.
Can anyone tell me the name of this type of priority queue so I know what to search for or, even better, point me to an implementation?
Priority queues such as this are typically implemented using a binary heap data structure as someone else suggested, which usually is represented using an array but could also use a binary tree. It actually is not hard to increase or decrease the priority of an element in the heap. If you know you are changing the priority of many elements before the next element is popped from the queue you can temporarily turn off dynamic reordering, insert all of the elements at the end of the heap, and then reorder the entire heap (at a cost of O(n)) just before the element needs to be popped. The important thing about heaps is that it only costs O(n) to put an array into heap order but O(n log n) to sort it.
I have used this approach successfully in a large project with dynamic priorities.
Here is my implementation of a parameterized priority queue implementation in the Curl programming language.
A standard binary heap supports 5 operations (the example below assume a max heap):
* find-max: return the maximum node of the heap
* delete-max: removing the root node of the heap
* increase-key: updating a key within the heap
* insert: adding a new key to the heap
* merge: joining two heaps to form a valid new heap containing all the elements of both.
As you can see, in a max heap, you can increase an arbitrary key. In a min heap you can decrease an arbitrary key. You can't change keys both ways unfortunately, but will this do? If you need to change keys both ways then you might want to think about using a a min-max-heap.
I would suggest first trying the head-in approach, to update a priority:
delete the item from the queue
re-insert it with the new priority
In C++, this could be done using a std::multi_map, the important thing is that the object must remember where it is stored in the structure to be able to delete itself efficiently. For re-insert, it's difficult since you cannot presume you know anything about the priorities.
class Item;
typedef std::multi_map<int, Item*> priority_queue;
class Item
{
public:
void add(priority_queue& queue);
void remove();
int getPriority() const;
void setPriority(int priority);
std::string& accessData();
const std::string& getData() const;
private:
int mPriority;
std::string mData;
priority_queue* mQueue;
priority_queue::iterator mIterator;
};
void Item::add(priority_queue& queue)
{
mQueue = &queue;
mIterator = queue.insert(std::make_pair(mPriority,this));
}
void Item::remove()
{
mQueue.erase(mIterator);
mQueue = 0;
mIterator = priority_queue::iterator();
}
void Item::setPriority(int priority)
{
mPriority = priority;
if (mQueue)
{
priority_queue& queue = *mQueue;
this->remove();
this->add(queue);
}
}
I am looking for just exactly the same thing!
And here is some of my idea:
Since a priority of an item keeps changing,
it's meaningless to sort the queue before retrieving an item.
So, we should forget using a priority queue. And "partially" sort the
container while retrieving an item.
And choose from the following STL sort algorithms:
a. partition
b. stable_partition
c. nth_element
d. partial_sort
e. partial_sort_copy
f. sort
g. stable_sort
partition, stable_partition and nth_element are linear-time sort algorithms, which should be our 1st choices.
BUT, it seems that there is no those algorithms provided in the official Java library. As a result, I will suggest you to use java.util.Collections.max/min to do what you want.
Google has a number of answers for you, including an implementation of one in Java.
However, this sounds like something that would be a homework problem, so if it is, I'd suggest trying to work through the ideas yourself first, then potentially referencing someone else's implementation if you get stuck somewhere and need a pointer in the right direction. That way, you're less likely to be "biased" towards the precise coding method used by the other programmer and more likely to understand why each piece of code is included and how it works. Sometimes it can be a little too tempting to do the paraphrasing equivalent of "copy and paste".
I need to make a class Persistent queue in which the function enqueue takes an element enqueues it to the current queue and return the new queue. How ever the original queue remains unchanged. Similarly the dequeue function removes the front element an returns the new queue but the original queue remains unchanged. This can of course be done in O(length of queue) but can i do it faster.??
You can use a linked list as queue (not LinkedList, but your own implementation).
To add a new element you only have to create a new instance of your queue class, set its start element to the copied queue's start element, and create a new end element.
Removing an element is similar, but set the end element of the new queue to the second to last element of the copied queue.
The queue class could look like this:
static class Node {
final Node next;
final Object o;
Node(Node next, Object o) {...}
}
final Node start, end;
private Queue(Node start, Node end) {...}
public Queue(Object o) {
start = end = new Node(null, o);
}
public Queue add(Object o) {
return new Queue(start, new Node(end, o));
}
public Queue remove() {
return new Queue(start, end.next);
}
The complexity of this queue's add and remove methods is O(1).
Please note that you can only iterate this queue in reverse order (i.e. newest elements first). Maybe you can come up with something that can be iterated the other way around or even in both directions.
I suggest to have a look to scala implementation. Comment at the top of the class describes the chosen approach (complexity : O(1)).
Queue is implemented as a pair of Lists, one containing the ''in'' elements and the other the ''out'' elements.
Elements are added to the ''in'' list and removed from the ''out'' list. When the ''out'' list runs dry, the
queue is pivoted by replacing the ''out'' list by ''in.reverse'', and ''in'' by ''Nil''.
Adding items to the queue always has cost O(1). Removing items has cost O(1), except in the case
where a pivot is required, in which case, a cost of O(n) is incurred, where n is the number of elements in the queue. When this happens,
n remove operations with O(1) cost are guaranteed. Removing an item is on average O(1).
http://xuwei-k.github.io/scala-library-sxr/scala-library-2.10.0/scala/collection/immutable/Queue.scala.html
What I do is use Java Chronicle (disclaimer: which I wrote). This is an unbounded off heap persisted queue which is stored on disk or tmpfs (shared memory).
In this approach your consumer keeps track of where it is up to in the queue, but no entry is actually removed (except in a daily or weekly maintenance cycle)
This avoids the need to alter the queue except when adding to it, and you would not need to copy it.
As such maintaining multiple references to where each consumer believes is the tail of the queue is O(1) for each consumer.
As Chronicle uses a compact binary form, the limit to how much you can store is limited by your disk space. e.g. a 2 TB drive can store this much data even on a machine with 8 GB before you have to rotate the queue logs.
I'm using the following code for a PriorityQueue<Node<T>>, where Node<T> is not Comparable:
final Map<Node<T>, Double> distances = new HashMap<>();
PriorityQueue<Node<T>> queue = new PriorityQueue<Node<T>>(graph
.getNodes().size(), new Comparator<Node<T>>() {
#Override
public int compare(Node<T> o1, Node<T> o2) {
return distances.get(o1).compareTo(distances.get(o2));
}
});
Later in my code, I modify the distances of the nodes in the map with distances.put(...). How can I ensure that the priority queue is updated correctly to reflect the new sort order?
I've looked at the source for PriorityQueue and see that its peek, poll, and element methods all just get queue[0], but I don't know how to update the order of the queue, as the internal method heapify is private.
As far as I know you can't with the default implementation. However, you can work around it. This depends on what you are using the priority queue for:
For updating only when the priority of a node has increased
For updating both when the priority of a node has increased, and decreased
Case 1 is a much simpler case, because the PriorityQueue already maintains the priority order for you. Simply add the node with the higher priority to the queue, and keep track of whether a node has been popped from the Priority Queue before. (using a boolean array or a hashtable)
When you pop a node, if it hasn't been popped before, you can guarantee that it is the one in the queue with the lowest priority, even if you have added multiple copies to the queue previously.
Case 2 is trickier and will require use of a boolean variable within each node to track if it's "enabled". Also, you need a hashtable or array so you can access the node in the priority queue. When you want to update a node's priority, you have to "disable" the existing node in the queue while adding the new one in. Also keep track of the one you just added as the "new" existing node. When you pop, just ignore nodes that are "disabled".
As for time complexity, amortized cost will still be O(log n) for adding. This is because the cost of updating a node's position in the heap in the "heapify" is O(log n), which is asymptotically equal to calling offer. Time cost for pop will increase depending on the ratio of adding duplicate nodes vs valid nodes.
Alternatively, write your own updating priority queue.
if(!priorityQueue.isEmpty()) {
priorityQueue.add(priorityQueue.remove());
}
should be a simple way to trigger re-heapification.
As seen here
I would like to know what kind of data structure (queue) I should use if I have the following problem:
The queue must have a dynamically assigned length (say, 512).
Every new value is saved at the end of the queue.
When a new value is added, the first is dropped if the queue is already full. If I add 30 new values to a full queue, the 30 first are automatically dropped.
The kind of data stored be arrays or some other simple object.
I need to be able to quickly retrieve the values using a loop, always in order (no random access).
The purpose of this is to have a fixed width data source that a graph will scan to draw its curve.
EDIT: This graph is meant to be shown on an Android custom View. Is there a specific length I could use that would make the looping thru this faster?
EDIT2: Added "When a new value is added, the first is dropped if the queue is already full. If I add 30 new values to a full queue, the 30 first are automatically dropped."
If the capacity of the stack is never going to change, I would use an array. I would keep track of where the "first" node is and keep wrapping it around.
I'd also like to point out that you're behavior seems more like a queue than a stack. Are you sure the stack is what you want?
class RotatingQueue<E> {
Object[] data; // can't do E[]
final int maxSize;
int size = 0; // starts empty
int first = 0; // starts at the front, why not?
RotatingQueue(int size) {
this.maxSize = size;
data = new Object[size];
}
E add(E e) {
E old = (E)(data[first]);
old[first++] = e;
if(size < maxSize) size++;
return old;
}
}
You definitelly need a Circular buffer - array based queue with modular access. You can easily modify the implementation to drop first elements instead of throwing an exception.
See the ArrayBlockingQueue: http://download.oracle.com/javase/1,5.0/docs/api/java/util/concurrent/ArrayBlockingQueue.html
I would recommend LinkedHashSet for unique items or an ArrayList.
It sounds like you need... a stack!
No, I'm just kidding around. "Stack" isn't really the right term here, since a stack is "last-in-first-out", meaning that you add stuff to the top and then take it off in reverse order.
What you really need is a Deque implementation. It's a queue that allows insertion and removal at both ends. I'm gonna assume values only need to be dropped when the queue gets full. Your description didn't make this entirely clear. If you'd always drop values when inserting new ones, you'll end up with a data structure that only has one element in it at any time.
I don't know of an implementation that automatically limits the number of items. ArrayDeque comes pretty close, but you'd still need to check the size on insertions and remove extraneous elements from the start. It offers the typical iterator method of Java collections that allows you to loop over the items. For a collection of this type the order is guaranteed.