We Keep Coding

How to get a random element using Java Stream API? [duplicate]

What is the most effective way to get a random element from a list with Java8 stream api?
Arrays.asList(new Obj1(), new Obj2(), new Obj3());
Thanks.

Why with streams? You just have to get a random number from 0 to the size of the list and then call get on this index:
Random r = new Random();
ElementType e = list.get(r.nextInt(list.size()));
Stream will give you nothing interesting here, but you can try with:
Random r = new Random();
ElementType e = list.stream().skip(r.nextInt(list.size())).findFirst().get();
Idea is to skip an arbitrary number of elements (but not the last one!), then get the first element if it exists. As a result you will have an Optional<ElementType> which will be non empty and then extract its value with get. You have a lot of options here after having skip.
Using streams here is highly inefficient...
Note: that none of these solutions take in account empty lists, but the problem is defined on non-empty lists.

There are much more efficient ways to do it, but if this has to be Stream the easiest way is to create your own Comparator, which returns random result (-1, 0, 1) and sort your stream:
List<String> strings = Arrays.asList("a", "b", "c", "d", "e", "f");
String randomString = strings
.stream()
.sorted((o1, o2) -> ThreadLocalRandom.current().nextInt(-1, 2))
.findAny()
.get();
ThreadLocalRandom has ready "out of the box" method to get random number in your required range for comparator.

While all the given answers work, there is a simple one-liner that does the trick without having to check if the list is empty first:
List<String> list = List.of("a", "b", "c");
list.stream().skip((int) (list.size() * Math.random())).findAny();
For an empty list this will return an Optional.empty.

In the last time I needed to do something like that I did that:
List<String> list = Arrays.asList("a", "b", "c");
Collections.shuffle(list);
String letter = list.stream().findAny().orElse(null);
System.out.println(letter);

If you HAVE to use streams, I wrote an elegant, albeit very inefficient collector that does the job:
/**
* Returns a random item from the stream (or null in case of an empty stream).
* This operation can't be lazy and is inefficient, and therefore shouldn't
* be used on streams with a large number or items or in performance critical sections.
* #return a random item from the stream or null if the stream is empty.
*/
public static <T> Collector<T, List<T>, T> randomItem() {
final Random RANDOM = new Random();
return Collector.of(() -> (List<T>) new ArrayList<T>(),
(acc, elem) -> acc.add(elem),
(list1, list2) -> ListUtils.union(list1, list2), // Using a 3rd party for list union, could be done "purely"
list -> list.isEmpty() ? null : list.get(RANDOM.nextInt(list.size())));
}
Usage:
#Test
public void standardRandomTest() {
assertThat(Stream.of(1, 2, 3, 4).collect(randomItem())).isBetween(1, 4);
}

Another idea would be to implement your own Spliterator and then use it as a source for Stream:
import java.util.List;
import java.util.Random;
import java.util.Spliterator;
import java.util.function.Consumer;
import java.util.function.Supplier;
public class ImprovedRandomSpliterator<T> implements Spliterator<T> {
private final Random random;
private final T[] source;
private int size;
ImprovedRandomSpliterator(List<T> source, Supplier<? extends Random> random) {
if (source.isEmpty()) {
throw new IllegalArgumentException("RandomSpliterator can't be initialized with an empty collection");
}
this.source = (T[]) source.toArray();
this.random = random.get();
this.size = this.source.length;
}
#Override
public boolean tryAdvance(Consumer<? super T> action) {
if (size > 0) {
int nextIdx = random.nextInt(size);
int lastIdx = size - 1;
action.accept(source[nextIdx]);
source[nextIdx] = source[lastIdx];
source[lastIdx] = null; // let object be GCed
size--;
return true;
} else {
return false;
}
}
#Override
public Spliterator<T> trySplit() {
return null;
}
#Override
public long estimateSize() {
return source.length;
}
#Override
public int characteristics() {
return SIZED;
}
}
public static <T> Collector<T, ?, Stream<T>> toShuffledStream() {
return Collectors.collectingAndThen(
toCollection(ArrayList::new),
list -> !list.isEmpty()
? StreamSupport.stream(new ImprovedRandomSpliterator<>(list, Random::new), false)
: Stream.empty());
}
and then simply:
list.stream()
.collect(toShuffledStream())
.findAny();
Details can be found here.
...but it's definitely an overkill, so if you're looking for a pragmatic approach. Definitely go for Jean's solution.

If you don't know in advance the size of the your list, you could do something like that :
yourStream.collect(new RandomListCollector<>(randomSetSize));
I guess that you will have to write your own Collector implementation like this one to have an homogeneous randomization :
public class RandomListCollector<T> implements Collector<T, RandomListCollector.ListAccumulator<T>, List<T>> {
private final Random rand;
private final int size;
public RandomListCollector(Random random , int size) {
super();
this.rand = random;
this.size = size;
}
public RandomListCollector(int size) {
this(new Random(System.nanoTime()), size);
}
#Override
public Supplier<ListAccumulator<T>> supplier() {
return () -> new ListAccumulator<T>();
}
#Override
public BiConsumer<ListAccumulator<T>, T> accumulator() {
return (l, t) -> {
if (l.size() < size) {
l.add(t);
} else if (rand.nextDouble() <= ((double) size) / (l.gSize() + 1)) {
l.add(t);
l.remove(rand.nextInt(size));
} else {
// in any case gSize needs to be incremented
l.gSizeInc();
}
};
}
#Override
public BinaryOperator<ListAccumulator<T>> combiner() {
return (l1, l2) -> {
int lgSize = l1.gSize() + l2.gSize();
ListAccumulator<T> l = new ListAccumulator<>();
if (l1.size() + l2.size()<size) {
l.addAll(l1);
l.addAll(l2);
} else {
while (l.size() < size) {
if (l1.size()==0 || l2.size()>0 && rand.nextDouble() < (double) l2.gSize() / (l1.gSize() + l2.gSize())) {
l.add(l2.remove(rand.nextInt(l2.size()), true));
} else {
l.add(l1.remove(rand.nextInt(l1.size()), true));
}
}
}
// set the gSize of l :
l.gSize(lgSize);
return l;
};
}
#Override
public Function<ListAccumulator<T>, List<T>> finisher() {
return (la) -> la.list;
}
#Override
public Set<Characteristics> characteristics() {
return Collections.singleton(Characteristics.CONCURRENT);
}
static class ListAccumulator<T> implements Iterable<T> {
List<T> list;
volatile int gSize;
public ListAccumulator() {
list = new ArrayList<>();
gSize = 0;
}
public void addAll(ListAccumulator<T> l) {
list.addAll(l.list);
gSize += l.gSize;
}
public T remove(int index) {
return remove(index, false);
}
public T remove(int index, boolean global) {
T t = list.remove(index);
if (t != null && global)
gSize--;
return t;
}
public void add(T t) {
list.add(t);
gSize++;
}
public int gSize() {
return gSize;
}
public void gSize(int gSize) {
this.gSize = gSize;
}
public void gSizeInc() {
gSize++;
}
public int size() {
return list.size();
}
#Override
public Iterator<T> iterator() {
return list.iterator();
}
}
}
If you want something easier and still don't want to load all your list in memory:
public <T> Stream<T> getRandomStreamSubset(Stream<T> stream, int subsetSize) {
int cnt = 0;
Random r = new Random(System.nanoTime());
Object[] tArr = new Object[subsetSize];
Iterator<T> iter = stream.iterator();
while (iter.hasNext() && cnt <subsetSize) {
tArr[cnt++] = iter.next();
}
while (iter.hasNext()) {
cnt++;
T t = iter.next();
if (r.nextDouble() <= (double) subsetSize / cnt) {
tArr[r.nextInt(subsetSize)] = t;
}
}
return Arrays.stream(tArr).map(o -> (T)o );
}
but you are then away from the stream api and could do the same with a basic iterator

The selected answer has errors in its stream solution...
You cannot use Random#nextInt with a non-positive long, "0" in this case.
The stream solution will also never choose the last in the list
Example:
List<Integer> intList = Arrays.asList(0, 1, 2, 3, 4);
// #nextInt is exclusive, so here it means a returned value of 0-3
// if you have a list of size = 1, #next Int will throw an IllegalArgumentException (bound must be positive)
int skipIndex = new Random().nextInt(intList.size()-1);
// randomInt will only ever be 0, 1, 2, or 3. Never 4
int randomInt = intList.stream()
.skip(skipIndex) // max skip of list#size - 2
.findFirst()
.get();
My recommendation would be to go with the non-stream approach that Jean-Baptiste Yunès put forth, but if you must do a stream approach, you could do something like this (but it's a little ugly):
list.stream()
.skip(list.isEmpty ? 0 : new Random().nextInt(list.size()))
.findFirst();

Sometimes you may want to get a random item somewhere in the stream. If you want to get random items even after filtering your list, this code snippet will work for you:
List<String> items = Arrays.asList("A", "B", "C", "D", "E");
List<String> shuffledAndFilteredItems = items.stream()
.filter(value -> value.equals("A") || value.equals("B"))
//filter, map...
.collect(Collectors.collectingAndThen(
Collectors.toCollection(ArrayList::new),
list -> {
Collections.shuffle(list);
return list;
}));
String randomItem = shuffledAndFilteredItems
.stream()
.findFirst()
.orElse(null);
Of course there may be faster / optimized ways, but it allows you to do it all at once.

Loop until using Java 8 streams [duplicate]

Is there a Java 8 stream operation that limits a (potentially infinite) Stream until the first element fails to match a predicate?
In Java 9 we can use takeWhile as in the example below to print all the numbers less than 10.
IntStream
.iterate(1, n -> n + 1)
.takeWhile(n -> n < 10)
.forEach(System.out::println);
As there is no such operation in Java 8, what's the best way of implementing it in a general way?

Operations takeWhile and dropWhile have been added to JDK 9. Your example code
IntStream
.iterate(1, n -> n + 1)
.takeWhile(n -> n < 10)
.forEach(System.out::println);
will behave exactly as you expect it to when compiled and run under JDK 9.
JDK 9 has been released. It is available for download here: JDK 9 Releases.

Such an operation ought to be possible with a Java 8 Stream, but it can't necessarily be done efficiently -- for example, you can't necessarily parallelize such an operation, as you have to look at elements in order.
The API doesn't provide an easy way to do it, but what's probably the simplest way is to take Stream.iterator(), wrap the Iterator to have a "take-while" implementation, and then go back to a Spliterator and then a Stream. Or -- maybe -- wrap the Spliterator, though it can't really be split anymore in this implementation.
Here's an untested implementation of takeWhile on a Spliterator:
static <T> Spliterator<T> takeWhile(
Spliterator<T> splitr, Predicate<? super T> predicate) {
return new Spliterators.AbstractSpliterator<T>(splitr.estimateSize(), 0) {
boolean stillGoing = true;
#Override public boolean tryAdvance(Consumer<? super T> consumer) {
if (stillGoing) {
boolean hadNext = splitr.tryAdvance(elem -> {
if (predicate.test(elem)) {
consumer.accept(elem);
} else {
stillGoing = false;
}
});
return hadNext && stillGoing;
}
return false;
}
};
}
static <T> Stream<T> takeWhile(Stream<T> stream, Predicate<? super T> predicate) {
return StreamSupport.stream(takeWhile(stream.spliterator(), predicate), false);
}

allMatch() is a short-circuiting function, so you can use it to stop processing. The main disadvantage is that you have to do your test twice: once to see if you should process it, and again to see whether to keep going.
IntStream
.iterate(1, n -> n + 1)
.peek(n->{if (n<10) System.out.println(n);})
.allMatch(n->n < 10);

As a follow-up to #StuartMarks answer. My StreamEx library has the takeWhile operation which is compatible with current JDK-9 implementation. When running under JDK-9 it will just delegate to the JDK implementation (via MethodHandle.invokeExact which is really fast). When running under JDK-8, the "polyfill" implementation will be used. So using my library the problem can be solved like this:
IntStreamEx.iterate(1, n -> n + 1)
.takeWhile(n -> n < 10)
.forEach(System.out::println);

takeWhile is one of the functions provided by the protonpack library.
Stream<Integer> infiniteInts = Stream.iterate(0, i -> i + 1);
Stream<Integer> finiteInts = StreamUtils.takeWhile(infiniteInts, i -> i < 10);
assertThat(finiteInts.collect(Collectors.toList()),
hasSize(10));

Update: Java 9 Stream now comes with a takeWhile method.
No needs for hacks or other solutions. Just use that!
I am sure this can be greatly improved upon:
(someone could make it thread-safe maybe)
Stream<Integer> stream = Stream.iterate(0, n -> n + 1);
TakeWhile.stream(stream, n -> n < 10000)
.forEach(n -> System.out.print((n == 0 ? "" + n : "," + n)));
A hack for sure... Not elegant - but it works ~:D
class TakeWhile<T> implements Iterator<T> {
private final Iterator<T> iterator;
private final Predicate<T> predicate;
private volatile T next;
private volatile boolean keepGoing = true;
public TakeWhile(Stream<T> s, Predicate<T> p) {
this.iterator = s.iterator();
this.predicate = p;
}
#Override
public boolean hasNext() {
if (!keepGoing) {
return false;
}
if (next != null) {
return true;
}
if (iterator.hasNext()) {
next = iterator.next();
keepGoing = predicate.test(next);
if (!keepGoing) {
next = null;
}
}
return next != null;
}
#Override
public T next() {
if (next == null) {
if (!hasNext()) {
throw new NoSuchElementException("Sorry. Nothing for you.");
}
}
T temp = next;
next = null;
return temp;
}
public static <T> Stream<T> stream(Stream<T> s, Predicate<T> p) {
TakeWhile tw = new TakeWhile(s, p);
Spliterator split = Spliterators.spliterator(tw, Integer.MAX_VALUE, Spliterator.ORDERED);
return StreamSupport.stream(split, false);
}
}

You can use java8 + rxjava.
import java.util.stream.IntStream;
import rx.Observable;
// Example 1)
IntStream intStream = IntStream.iterate(1, n -> n + 1);
Observable.from(() -> intStream.iterator())
.takeWhile(n ->
{
System.out.println(n);
return n < 10;
}
).subscribe() ;
// Example 2
IntStream intStream = IntStream.iterate(1, n -> n + 1);
Observable.from(() -> intStream.iterator())
.takeWhile(n -> n < 10)
.forEach( n -> System.out.println(n));

Actually there are 2 ways to do it in Java 8 without any extra libraries or using Java 9.
If you want to print numbers from 2 to 20 on the console you can do this:
IntStream.iterate(2, (i) -> i + 2).peek(System.out::println).allMatch(i -> i < 20);
or
IntStream.iterate(2, (i) -> i + 2).peek(System.out::println).anyMatch(i -> i >= 20);
The output is in both cases:
2
4
6
8
10
12
14
16
18
20
No one mentioned anyMatch yet. This is the reason for this post.

This is the source copied from JDK 9 java.util.stream.Stream.takeWhile(Predicate). A little difference in order to work with JDK 8.
static <T> Stream<T> takeWhile(Stream<T> stream, Predicate<? super T> p) {
class Taking extends Spliterators.AbstractSpliterator<T> implements Consumer<T> {
private static final int CANCEL_CHECK_COUNT = 63;
private final Spliterator<T> s;
private int count;
private T t;
private final AtomicBoolean cancel = new AtomicBoolean();
private boolean takeOrDrop = true;
Taking(Spliterator<T> s) {
super(s.estimateSize(), s.characteristics() & ~(Spliterator.SIZED | Spliterator.SUBSIZED));
this.s = s;
}
#Override
public boolean tryAdvance(Consumer<? super T> action) {
boolean test = true;
if (takeOrDrop && // If can take
(count != 0 || !cancel.get()) && // and if not cancelled
s.tryAdvance(this) && // and if advanced one element
(test = p.test(t))) { // and test on element passes
action.accept(t); // then accept element
return true;
} else {
// Taking is finished
takeOrDrop = false;
// Cancel all further traversal and splitting operations
// only if test of element failed (short-circuited)
if (!test)
cancel.set(true);
return false;
}
}
#Override
public Comparator<? super T> getComparator() {
return s.getComparator();
}
#Override
public void accept(T t) {
count = (count + 1) & CANCEL_CHECK_COUNT;
this.t = t;
}
#Override
public Spliterator<T> trySplit() {
return null;
}
}
return StreamSupport.stream(new Taking(stream.spliterator()), stream.isParallel()).onClose(stream::close);
}

Here is a version done on ints - as asked in the question.
Usage:
StreamUtil.takeWhile(IntStream.iterate(1, n -> n + 1), n -> n < 10);
Here's code for StreamUtil:
import java.util.PrimitiveIterator;
import java.util.Spliterators;
import java.util.function.IntConsumer;
import java.util.function.IntPredicate;
import java.util.stream.IntStream;
import java.util.stream.StreamSupport;
public class StreamUtil
{
public static IntStream takeWhile(IntStream stream, IntPredicate predicate)
{
return StreamSupport.intStream(new PredicateIntSpliterator(stream, predicate), false);
}
private static class PredicateIntSpliterator extends Spliterators.AbstractIntSpliterator
{
private final PrimitiveIterator.OfInt iterator;
private final IntPredicate predicate;
public PredicateIntSpliterator(IntStream stream, IntPredicate predicate)
{
super(Long.MAX_VALUE, IMMUTABLE);
this.iterator = stream.iterator();
this.predicate = predicate;
}
#Override
public boolean tryAdvance(IntConsumer action)
{
if (iterator.hasNext()) {
int value = iterator.nextInt();
if (predicate.test(value)) {
action.accept(value);
return true;
}
}
return false;
}
}
}

Go to get library abacus-common. It provides the exact API you want and more:
IntStream.iterate(1, n -> n + 1).takeWhile(n -> n < 10).forEach(System.out::println);
Declaration： I'm the developer of AbacusUtil.

If you know the exact amount of repititions that will be performed, you can do
IntStream
.iterate(1, n -> n + 1)
.limit(10)
.forEach(System.out::println);

IntStream.iterate(1, n -> n + 1)
.peek(System.out::println) //it will be executed 9 times
.filter(n->n>=9)
.findAny();
instead of peak you can use mapToObj to return final object or message
IntStream.iterate(1, n -> n + 1)
.mapToObj(n->{ //it will be executed 9 times
if(n<9)
return "";
return "Loop repeats " + n + " times";});
.filter(message->!message.isEmpty())
.findAny()
.ifPresent(System.out::println);

You can't abort a stream except by a short-circuiting terminal operation, which would leave some stream values unprocessed regardless of their value. But if you just want to avoid operations on a stream you can add a transform and filter to the stream:
import java.util.Objects;
class ThingProcessor
{
static Thing returnNullOnCondition(Thing thing)
{ return( (*** is condition met ***)? null : thing); }
void processThings(Collection<Thing> thingsCollection)
{
thingsCollection.stream()
*** regular stream processing ***
.map(ThingProcessor::returnNullOnCondition)
.filter(Objects::nonNull)
*** continue stream processing ***
}
} // class ThingProcessor
That transforms the stream of things to nulls when the things meet some condition, then filters out nulls. If you're willing to indulge in side effects, you could set the condition value to true once some thing is encountered, so all subsequent things are filtered out regardless of their value. But even if not you can save a lot of (if not quite all) processing by filtering values out of the stream that you don't want to process.

Even I was having a similar requirement -- invoke the web-service, if it fails, retry it 3 times. If it fails even after these many trials, send an email notification. After googling a lot, anyMatch() came as a saviour. My sample code as follows. In the following example, if webServiceCall method returns true in the first iteration itself, stream does not iterate further as we have called anyMatch(). I believe, this is what you are looking for.
import java.util.stream.IntStream;
import io.netty.util.internal.ThreadLocalRandom;
class TrialStreamMatch {
public static void main(String[] args) {
if(!IntStream.range(1,3).anyMatch(integ -> webServiceCall(integ))){
//Code for sending email notifications
}
}
public static boolean webServiceCall(int i){
//For time being, I have written a code for generating boolean randomly
//This whole piece needs to be replaced by actual web-service client code
boolean bool = ThreadLocalRandom.current().nextBoolean();
System.out.println("Iteration index :: "+i+" bool :: "+bool);
//Return success status -- true or false
return bool;
}

If you have different problem, different solution may be needed but for your current problem, I would simply go with:
IntStream
.iterate(1, n -> n + 1)
.limit(10)
.forEach(System.out::println);

Might be a bit off topic but this is what we have for List<T> rather than Stream<T>.
First you need to have a take util method. This methods takes first n elements:
static <T> List<T> take(List<T> l, int n) {
if (n <= 0) {
return newArrayList();
} else {
int takeTo = Math.min(Math.max(n, 0), l.size());
return l.subList(0, takeTo);
}
}
it just works like scala.List.take
assertEquals(newArrayList(1, 2, 3), take(newArrayList(1, 2, 3, 4, 5), 3));
assertEquals(newArrayList(1, 2, 3), take(newArrayList(1, 2, 3), 5));
assertEquals(newArrayList(), take(newArrayList(1, 2, 3), -1));
assertEquals(newArrayList(), take(newArrayList(1, 2, 3), 0));
now it will be fairly simple to write a takeWhile method based on take
static <T> List<T> takeWhile(List<T> l, Predicate<T> p) {
return l.stream().
filter(p.negate()).findFirst(). // find first element when p is false
map(l::indexOf). // find the index of that element
map(i -> take(l, i)). // take up to the index
orElse(l); // return full list if p is true for all elements
}
it works like this:
assertEquals(newArrayList(1, 2, 3), takeWhile(newArrayList(1, 2, 3, 4, 3, 2, 1), i -> i < 4));
this implementation iterate the list partially for a few times but it won't add add O(n^2) operations. Hope that's acceptable.

I have another quick solution by implementing this (which is rly unclean in fact, but you get the idea):
public static void main(String[] args) {
System.out.println(StreamUtil.iterate(1, o -> o + 1).terminateOn(15)
.map(o -> o.toString()).collect(Collectors.joining(", ")));
}
static interface TerminatedStream<T> {
Stream<T> terminateOn(T e);
}
static class StreamUtil {
static <T> TerminatedStream<T> iterate(T seed, UnaryOperator<T> op) {
return new TerminatedStream<T>() {
public Stream<T> terminateOn(T e) {
Builder<T> builder = Stream.<T> builder().add(seed);
T current = seed;
while (!current.equals(e)) {
current = op.apply(current);
builder.add(current);
}
return builder.build();
}
};
}
}

Here is my attempt using just Java Stream library.
IntStream.iterate(0, i -> i + 1)
.filter(n -> {
if (n < 10) {
System.out.println(n);
return false;
} else {
return true;
}
})
.findAny();

How to short-circuit a reduce() operation on a Stream?

This is essentially the same question as How to short-circuit reduce on Stream?. However, since that question focuses on a Stream of boolean values, and its answer cannot be generalized for other types and reduce operations, I'd like to ask the more general question.
How can we make a reduce on a stream so that it short-circuits when it encounters an absorbing element for the reducing operation?
The typical mathematical case would be 0 for multiplication. This Stream :
int product = IntStream.of(2, 3, 4, 5, 0, 7, 8)
.reduce(1, (a, b) -> a * b);
will consume the last two elements (7 and 8) regardless of the fact that once 0 has been encountered the product is known.

Unfortunately the Stream API has limited capabilities to create your own short-circuit operations. Not so clean solution would be to throw a RuntimeException and catch it. Here's the implementation for IntStream, but it can be generalized for other stream types as well:
public static int reduceWithCancelEx(IntStream stream, int identity,
IntBinaryOperator combiner, IntPredicate cancelCondition) {
class CancelException extends RuntimeException {
private final int val;
CancelException(int val) {
this.val = val;
}
}
try {
return stream.reduce(identity, (a, b) -> {
int res = combiner.applyAsInt(a, b);
if(cancelCondition.test(res))
throw new CancelException(res);
return res;
});
} catch (CancelException e) {
return e.val;
}
}
Usage example:
int product = reduceWithCancelEx(
IntStream.of(2, 3, 4, 5, 0, 7, 8).peek(System.out::println),
1, (a, b) -> a * b, val -> val == 0);
System.out.println("Result: "+product);
Output:
2
3
4
5
0
Result: 0
Note that even though it works with parallel streams, it's not guaranteed that other parallel tasks will be finished as soon as one of them throws an exception. The sub-tasks which are already started will likely to run till finish, so you may process more elements than expected.
Update: alternative solution which is much longer, but more parallel-friendly. It's based on custom spliterator which returns at most one element which is result of accumulation of all underlying elements). When you use it in sequential mode, it does all the work in single tryAdvance call. When you split it, each part generates the correspoding single partial result, which are reduced by Stream engine using the combiner function. Here's generic version, but primitive specialization is possible as well.
final static class CancellableReduceSpliterator<T, A> implements Spliterator<A>,
Consumer<T>, Cloneable {
private Spliterator<T> source;
private final BiFunction<A, ? super T, A> accumulator;
private final Predicate<A> cancelPredicate;
private final AtomicBoolean cancelled = new AtomicBoolean();
private A acc;
CancellableReduceSpliterator(Spliterator<T> source, A identity,
BiFunction<A, ? super T, A> accumulator, Predicate<A> cancelPredicate) {
this.source = source;
this.acc = identity;
this.accumulator = accumulator;
this.cancelPredicate = cancelPredicate;
}
#Override
public boolean tryAdvance(Consumer<? super A> action) {
if (source == null || cancelled.get()) {
source = null;
return false;
}
while (!cancelled.get() && source.tryAdvance(this)) {
if (cancelPredicate.test(acc)) {
cancelled.set(true);
break;
}
}
source = null;
action.accept(acc);
return true;
}
#Override
public void forEachRemaining(Consumer<? super A> action) {
tryAdvance(action);
}
#Override
public Spliterator<A> trySplit() {
if(source == null || cancelled.get()) {
source = null;
return null;
}
Spliterator<T> prefix = source.trySplit();
if (prefix == null)
return null;
try {
#SuppressWarnings("unchecked")
CancellableReduceSpliterator<T, A> result =
(CancellableReduceSpliterator<T, A>) this.clone();
result.source = prefix;
return result;
} catch (CloneNotSupportedException e) {
throw new InternalError();
}
}
#Override
public long estimateSize() {
// let's pretend we have the same number of elements
// as the source, so the pipeline engine parallelize it in the same way
return source == null ? 0 : source.estimateSize();
}
#Override
public int characteristics() {
return source == null ? SIZED : source.characteristics() & ORDERED;
}
#Override
public void accept(T t) {
this.acc = accumulator.apply(this.acc, t);
}
}
Methods which are analogous to Stream.reduce(identity, accumulator, combiner) and Stream.reduce(identity, combiner), but with cancelPredicate:
public static <T, U> U reduceWithCancel(Stream<T> stream, U identity,
BiFunction<U, ? super T, U> accumulator, BinaryOperator<U> combiner,
Predicate<U> cancelPredicate) {
return StreamSupport
.stream(new CancellableReduceSpliterator<>(stream.spliterator(), identity,
accumulator, cancelPredicate), stream.isParallel()).reduce(combiner)
.orElse(identity);
}
public static <T> T reduceWithCancel(Stream<T> stream, T identity,
BinaryOperator<T> combiner, Predicate<T> cancelPredicate) {
return reduceWithCancel(stream, identity, combiner, combiner, cancelPredicate);
}
Let's test both versions and count how many elements are actually processed. Let's put the 0 close to end. Exception version:
AtomicInteger count = new AtomicInteger();
int product = reduceWithCancelEx(
IntStream.range(-1000000, 100).filter(x -> x == 0 || x % 2 != 0)
.parallel().peek(i -> count.incrementAndGet()), 1,
(a, b) -> a * b, x -> x == 0);
System.out.println("product: " + product + "/count: " + count);
Thread.sleep(1000);
System.out.println("product: " + product + "/count: " + count);
Typical output:
product: 0/count: 281721
product: 0/count: 500001
So while result is returned when only some elements are processed, the tasks continue working in background and counter is still increasing. Here's spliterator version:
AtomicInteger count = new AtomicInteger();
int product = reduceWithCancel(
IntStream.range(-1000000, 100).filter(x -> x == 0 || x % 2 != 0)
.parallel().peek(i -> count.incrementAndGet()).boxed(),
1, (a, b) -> a * b, x -> x == 0);
System.out.println("product: " + product + "/count: " + count);
Thread.sleep(1000);
System.out.println("product: " + product + "/count: " + count);
Typical output:
product: 0/count: 281353
product: 0/count: 281353
All the tasks are actually finished when the result is returned.

A general short-circuiting static reduce method can be implemented using the spliterator of a stream. It even turned out to be not very complicated! Using spliterators seems to be the way to go a lot of times when one wants to work with steams in a more flexible way.
public static <T> T reduceWithCancel(Stream<T> s, T acc, BinaryOperator<T> op, Predicate<? super T> cancelPred) {
BoxConsumer<T> box = new BoxConsumer<T>();
Spliterator<T> splitr = s.spliterator();
while (!cancelPred.test(acc) && splitr.tryAdvance(box)) {
acc = op.apply(acc, box.value);
}
return acc;
}
public static class BoxConsumer<T> implements Consumer<T> {
T value = null;
public void accept(T t) {
value = t;
}
}
Usage:
int product = reduceWithCancel(
Stream.of(1, 2, 0, 3, 4).peek(System.out::println),
1, (acc, i) -> acc * i, i -> i == 0);
System.out.println("Result: " + product);
Output:
1
2
0
Result: 0
The method could be generalised to perform other kinds of terminal operations.
This is based loosely on this answer about a take-while operation.
I don't know anything about the parallelisation potential of this.

My own take at this is to not use reduce() per se, but use an existing short-circuiting final operation.
noneMatch() or allMatch() can be used for this when using a Predicate with a side effect. Admittedly also not the cleanest solution, but it does achieve the goal :
AtomicInteger product = new AtomicInteger(1);
IntStream.of(2, 3, 4, 5, 0, 7, 8)
.peek(System.out::println)
.noneMatch(i -> {
if (i == 0) {
product.set(0);
return true;
}
int oldValue = product.get();
while (oldValue != 0 && !product.compareAndSet(oldValue, i * oldValue)) {
oldValue = product.get();
}
return oldValue == 0;
});
System.out.println("Result: " + product.get());
It short-circuits and can be made parallel.

this is how it is done after the introduction of takeWhile
since Java 9
int[] last = {1};
int product = IntStream.of(2, 3, 4, 5, 0, 7, 8)
.takeWhile(i -> last[0] != 0).reduce(1, (a, b) -> (last[0] = a) * b);

IntStream.range(0, 1_000_000) stops at 113383 (Project Euler: Longest Collatz Sequence)

I have a strange problem in my code to answer this:
The following iterative sequence is defined for the set of positive integers:
n → n/2 (n is even)
n → 3n + 1 (n is odd)
Using the rule above and starting with 13, we generate the following sequence:
13 → 40 → 20 → 10 → 5 → 16 → 8 → 4 → 2 → 1
It can be seen that this sequence (starting at 13 and finishing at 1) contains 10 terms. Although it has not been proved yet (Collatz Problem), it is thought that all starting numbers finish at 1.
Which starting number, under one million, produces the longest chain?
NOTE: Once the chain starts the terms are allowed to go above one million.
public class Problem14 extends Problem<Integer> {
private final int maximum;
public Problem14(final int maximum) {
this.maximum = maximum;
}
#Override
public void run() {
result = IntStream.range(1, maximum).boxed()
.peek(System.out::println)
.collect(Collectors.toMap(i -> i, i -> (int)Iterators.intStream(new CollatzGenerator(i)).count()))
.entrySet().stream()
.peek(System.out::println)
.max(Comparator.comparingInt(Map.Entry::getValue))
.get().getKey();
}
#Override
public String getName() {
return "Problem 14";
}
}
public abstract class Problem<T> implements Runnable {
protected T result;
public String getResult() {
return String.valueOf(result);
}
abstract public String getName();
}
public class CollatzGenerator implements PrimitiveIterator.OfInt {
private int current;
public CollatzGenerator(final int start) {
if (start < 1) {
throw new IllegalArgumentException("generators.CollatzGenerator: start < 1: start = " + start);
}
this.current = start;
}
#Override
public boolean hasNext() {
return (current != 1);
}
#Override
public int nextInt() {
int returnInt = current;
if (current % 2 == 0) {
current /= 2;
}
else {
current = 3 * current + 1;
}
return returnInt;
}
}
public abstract class Iterators {
//...
public static IntStream intStream(final PrimitiveIterator.OfInt iterator) {
return StreamSupport.intStream(
Spliterators.spliteratorUnknownSize(iterator, 0), false
);
}
//...
}
Calling code is along the lines of:
new Problem14(1_000_000).run();
So to rephrase, the problem is that the program never terminates, what I do see is that it prints all integers from 1 to 113383, presumably from the first .peek(System.out::println) call.
Also an extra question is that, currently I box my IntStream to a Stream<Integer> to be able to do .collect(Collectors.toMap(i -> i, i -> (int)Iterators.intStream(new CollatzGenerator(i)).count()))...
I'd like to get rid of the boxing and use the IntStream::collect method, however I do not understand how to do it.

So to rephrase, the problem is that the program never terminates, what I do see is that it prints all integers from 1 to 113383, presumably from the first .peek(System.out::println) call.
This is happening because you are assuming all the numbers that will be generated in the collatz will be in range of a int type. But it's not. That is why it is failing. Try making everything to long - PrimitiveIterator.OfLong, StreamSupport.longStream, etc., and it will work.
I'd like to get rid of the boxing and use the IntStream::collect method, however I do not understand how to do it.
I don't understand why you would like to do that. Boxing will still be done somewhere, as you can't create a Map of primitive int. Still, if you want, you would in fact need to do some more work. Here's how you do it:
#Override
public void run() {
ObjIntConsumer<Map<Integer, Long>> objIntConsumer =
(map, value) -> map.put(value, Iterators.longStream(new CollatzGenerator(value)).count());
BiConsumer<Map<Integer, Long>, Map<Integer, Long>> biConsumer =
(targetMap, accumulatorMap) -> targetMap.putAll(accumulatorMap);
result = IntStream.range(1, maximum)
.collect(LinkedHashMap::new, objIntConsumer, biConsumer)
.entrySet().stream()
.peek(System.out::println)
.max(Comparator.<Entry<Integer, Long>>comparingLong(entry -> entry.getValue()))
.get().getKey();
}

I have made short code in C that works to "infinite" using big numbers (I have ended at 104282959 where I killed it).. It checks that numbers converges to one (when proved, continue with next unchecked)..
There are few optimizations, it can be littlebit confusing. But I have not read any paper about that, so there can be more optimizations..
There is no waranty that it works well..
//gcc main.c -lgmp
#include <stdio.h>
#include <stdlib.h>
#include <gmp.h>
void collatz_conjucture(mpz_t *res, mpz_t n)
{
mpz_mod_ui(*res, n, 1UL);
if(mpz_cmp_ui(*res, 1UL) == 0) {
mpz_mul_ui(*res, n, 3UL);
mpz_add_ui(*res, *res, 1UL);
//mpz_fdiv_q_ui(*res, *res, 2UL); //Posible to uncomment, skip one step of computation (always odd after even), but it will slow down increment of next
} else {
mpz_fdiv_q_ui(*res, n, 2UL);
}
}
int main(int argc, char **argv)
{
// Init
mpz_t current_b, next_b;
mpz_init_set_str(current_b, "1", 10);
mpz_init_set(next_b, current_b);
//Starts computation - i means step
for (unsigned long i = 0; i < 0xFFFFF; ++i) {
//for (;;) { // replace above for infinite
mpz_set(current_b, next_b);
do {
if(mpz_cmp(current_b, next_b) == 0) {
mpz_add_ui(next_b, next_b, 1UL);
}
collatz_conjucture(&current_b, current_b);
} while (mpz_cmp(current_b, next_b) > 0);
char *result = mpz_get_str(NULL, 10, next_b);
printf("%s\n", result);
free(result);
result = NULL;
}
mpz_clear(current_b);
mpz_clear(next_b);
return 0;
}

Generator functions equivalent in Java

I would like to implement an Iterator in Java that behaves somewhat like the following generator function in Python:
def iterator(array):
for x in array:
if x!= None:
for y in x:
if y!= None:
for z in y:
if z!= None:
yield z
x on the java side can be multi-dimensional array or some form of nested collection. I am not sure how this would work. Ideas?

Had the same need so wrote a little class for it. Here are some examples:
Generator<Integer> simpleGenerator = new Generator<Integer>() {
public void run() throws InterruptedException {
yield(1);
// Some logic here...
yield(2);
}
};
for (Integer element : simpleGenerator)
System.out.println(element);
// Prints "1", then "2".
Infinite generators are also possible:
Generator<Integer> infiniteGenerator = new Generator<Integer>() {
public void run() throws InterruptedException {
while (true)
yield(1);
}
};
The Generator class internally works with a Thread to produce the items. By overriding finalize(), it ensures that no Threads stay around if the corresponding Generator is no longer used.
The performance is obviously not great but not too shabby either. On my machine with a dual core i5 CPU # 2.67 GHz, 1000 items can be produced in < 0.03s.
The code is on GitHub. There, you'll also find instructions on how to include it as a Maven/Gradle dependency.

Indeed Java has no yield, but you can now use Java 8 streams. IMO it's really a complicated iterator since it's backed by an array, not a function. Given it's a loop in a loop in a loop can be expressed as a Stream using filter (to skip the nulls) and flatMap to stream the inner collection. It's also about the size of the Python code. I've converted it to an iterator to use at your leisure and printed to demonstrate, but if all you were doing was printing, you could end the stream sequence with forEach(System.out::println) instead of iterator().
public class ArrayIterate
{
public static void main(String args[])
{
Integer[][][] a = new Integer[][][] { { { 1, 2, null, 3 },
null,
{ 4 }
},
null,
{ { 5 } } };
Iterator<Object> iterator = Arrays.stream(a)
.filter(ax -> ax != null)
.flatMap(ax -> Arrays.stream(ax)
.filter(ay -> ay != null)
.flatMap(ay -> Arrays.stream(ay)
.filter(az -> az != null)))
.iterator();
while (iterator.hasNext())
{
System.out.println(iterator.next());
}
}
}
I'm writing about implementation of generators as part of my blog on Java 8 Functional Programming and Lambda Expressions at http://thecannycoder.wordpress.com/ which might give you some more ideas for converting Python generator functions into Java equivalents.

I wish Java had generator/yield, but since it doesn't using Iterators is probably your best bet.
In this example I stuck with arrays, but in general I would advise using Iterable Collection instead, eg. List. In the example I show how it's pretty easy to get iterators for arrays though:
package example.stackoverflow;
import com.sun.xml.internal.xsom.impl.scd.Iterators;
import java.util.Arrays;
import java.util.Iterator;
public class ArrayGenerator<T> implements Iterable<T> {
private final T[][][] input;
public ArrayGenerator(T[][][] input) {
this.input = input;
}
#Override
public Iterator<T> iterator() {
return new Iter();
}
private class Iter implements Iterator<T> {
private Iterator<T[][]> x;
private Iterator<T[]> y;
private Iterator<T> z;
{
x = Arrays.asList(input).iterator();
y = Iterators.empty();
z = Iterators.empty();
}
#Override
public boolean hasNext() {
return z.hasNext() || y.hasNext() || x.hasNext();
}
#Override
public T next() {
while(! z.hasNext()) {
while(! y.hasNext()) {
y = Arrays.asList(x.next()).iterator();
}
z = Arrays.asList(y.next()).iterator();
}
return z.next();
}
#Override
public void remove() {
throw new UnsupportedOperationException("remove not supported");
}
}
public static void main(String[] args) {
for(Integer i :
new ArrayGenerator<Integer>(
new Integer[][][]{
{
{1, 2, 3},
{4, 5}
},
{
{},
{6}
},
{
},
{
{7, 8, 9, 10, 11}
}
}
)) {
System.out.print(i + ", ");
}
}
}

There is no yield in Java, so you have to do all these things for yourself, ending up with ridiculous code as this one:
for(Integer z : new Iterable<Integer>() {
#Override
public Iterator<Integer> iterator() {
return new Iterator<Integer>() {
final Integer[][][] d3 =
{ { { 1, 2, 3 }, { 4, 5, 6 }, { 7, 8, 9 } },
{ { 10, 11, 12 }, { 13, 14, 15 }, { 16, 17, 18 } },
{ { 19, 20, 21 }, { 22, 23, 24 }, { 25, 26, 27 } } };
int x = 0;
int y = 0;
int z = 0;
#Override
public boolean hasNext() {
return !(x==3 && y == 3 && z == 3);
}
#Override
public Integer next() {
Integer result = d3[z][y][x];
if (++x == 3) {
x = 0;
if (++y == 3) {
y = 0;
++z;
}
}
return result;
}
#Override
public void remove() {
throw new UnsupportedOperationException();
}
};
}
}) {
System.out.println(z);
}
But if your sample would have more than one single yield it would end up even worse.

The translation from Python-style generators to Java-style iterators can be automated. If you're willing to accept code generation in your build process, you might be interested in this prototype tool that does the translation for you:
https://github.com/Calvin-L/gen2it

Assuming the Python data structure you describe in your question can be described using the following Java type:
List<List<List<T>>>;
and you want to use it in an operation like this:
for (T z : iterator(array)) {
// do something with z
}
If so, then one can implement your Python iterator() pretty trivially using Java 8 streams:
public <T> Iterable<T> iterator(List<List<List<T>>> array) {
return array.stream()
.filter(Objects::nonNull) // -> emits stream of non-null `x`s
.flatMap(x -> x.stream()).filter(Objects::nonNull) // -> emits […] `y`s
.flatMap(y -> y.stream()).filter(Objects::nonNull) // -> emits […] `z`s
.collect(Collectors.toList()); // get list of non-null `z`s to iterate on
}
Of course, you can not collect the results and output a stream for further streamed processing (people tell me that it is a good idea):
public <T> Stream<T> streamContent(List<List<List<T>>> array) {
return array.stream()
.filter(Objects::nonNull) // -> emits stream of non-null `x`s
.flatMap(x -> x.stream()).filter(Objects::nonNull) // -> emits […] `y`s
.flatMap(y -> y.stream()).filter(Objects::nonNull); // -> emits […] `z`s
}
// ...
streamContent(array).forEach(z -> {
// do something with z
});

Very late to the game but I wanted to offer my solution as a reference.
https://github.com/tsi-software/JavaGenerator
A Java class that allows you to write "Generator" code as similarly as possible to Python and C#.

No, Java does not have "generators" or "yield" per-se, but the same functionality is available by using the Observer Pattern. This is enhanced when using a modern implementation like RxJava. Your code would subscribe to the Obserable and whenever it tries to read the next value from the Observable it would cause it "generate" it's next value. The Observable can maintain it's own state just like a generator for Python or JavaScript. When there are no new values to be read, the "next()" method will block waiting on new data to be available. A good example of this can be found HERE.

You can use the iterator of a stream to accomplish this.
// Save the iterator of a stream that generates fib sequence
Iterator<Integer> myGenerator = Stream
.iterate(new Integer[]{ 1, 1 }, x -> new Integer[] { x[1], x[0] + x[1] })
.map(x -> x[0]).iterator();
// Print the first 5 elements
for (int i = 0; i < 5; i++) {
System.out.println(myGenerator.next());
}
System.out.println("done with first iteration");
// Print the next 5 elements
for (int i = 0; i < 5; i++) {
System.out.println(myGenerator.next());
}
Output:
1
1
2
3
5
done with first iteration
8
13
21
34
55

Using Seq from a new library, which has implemented generator in Java, you can write your own generator functions just like in Python
public Seq<Integer> generate(List<List<List<Integer>>> array) {
return c -> {
for (List<List<Integer>> x : array) {
if (x != null) {
for (List<Integer> y : x) {
if (y != null) {
for (Integer z : y) {
if (z != null) {
c.accept(z);
}
}
}
}
}
}
};
}
Then you can manipulate/collect the returned seq just like a normal Java stream.