I need to create a custom Flowable with backpressure implemented. I'm trying to achieve some sort of paging. That means when downstream requests 5 items I will "ask the data source" for items 0 - 5. Then when downstream needs another 5, I'll get items 5 - 10 and emit back.
The best thing I've found so far is to use Flowable.generate method but I really don't understand why there is no way (as far as I know) how to get the requested number of items the downstream is requesting. I can use the state property of generator to save the index of last items requested so then I need only the number of newly requested items. The emmiter instance I got in the BiFunction apply is GeneratorSubscription which is extending from AtomicLong. So casting emmiter to AtomicLong can get me the requested number. But I know this can't be the "recommended" way.
On the other hand when you use Flowable.create you get the FlowableEmitter which has long requested() method. Using generate is suiting me more for my use-case, but now I'm also curious what is the "correct" way to use Flowable.generate.
Maybe I'm overthinking the whole thing so please point me in the right direction. :) Thank you.
This is what the actual code looks like (in Kotlin):
Flowable.generate(Callable { 0 }, BiFunction { start /*state*/, emitter ->
val requested = (emitter as AtomicLong).get().toInt() //this is bull*hit
val end = start + requested
//get items [start to end] -> items
emmiter.onNext(items)
end /*return the new state*/
})
Ok, I found out that the apply function of the BiFunction is called that many times as is the request amount (n). So there's no reason to have a getter for it. It's not what I have hoped for but it is apparently how generate works. :)
Related
We have a stateful test for an order system. There is an Arbitrary that will generate an Order object that has a number of LineItem's.
There are actions to:
Create an Order
Cancel a LineItem
The action to create an order takes the order itself, eg:
Arbitraries.defaultFor(Order.class).map(CreateOrderAction::new)
The state for the actions has knowledge about all created orders.
To cancel a LineItem, we need knowledge about what orders are created. Inside CancelLineItemAction is it safe to do the following?
LineItem line = Arbitraries.<Collection<Order>>of(state.orders())
.flatMap(order -> Arbitraries.<Collection<LineItem>>of(order.lineItems()))
.sample();
Based on the javadoc of Arbitrary.sample(), it seems safe, but this construct isn't explicitly mentioned in the documentation on stateful tests, and we don't want to use it extensively only to break the reproducibility of our tests.
TLDR
Arbitrary.sample() is not designed to be used in that way
I recommend to use a random cancel index with modulo over the number of line items
1. Why Arbitrary.sample() is not recommended
Arbitrary.sample() is designed to be used outside of properties, e.g. to experiment with generated values or to use it in other contexts like JUnit Jupiter. There are at least three reasons:
The underlying random seed used for generating values depends on what happens
before sampling. Thus the results are not really reproducible.
Sampling will not consider any added domain contexts that may change what's
being generated.
Values generated by sample() DO NOT PARTICIPATE IN SHRINKING
2. Option 1: Hand in a Random object and use it for generating
Hand in a Random instance when generating a CancelLineItemAction:
Arbitraries.random().map(random -> new CancelLineItemAction(random))
Use the random to invoke a generator:
LineItem line = Arbitraries.of(state.orders())
.flatMap(order -> Arbitraries.of(order.lineItems()))
.generator(100).next(random).value();
But actually that's very involved for what you want to do. Here's a simplification:
3. Option 2: Hand in a Random object and use it for picking a line item
Same as above but don't take a detour with sampling:
List<LineItem> lineItems = state.orders().stream()
.flatMap(order -> order.lineItems().stream())
.collect(Collectors.toList());
int randomIndex = random.nextInt(lineItems.size());
LineItem line = lineItems.get(randomIndex);
Both option 1 and 2 will (hopefully) behave reasonably in jqwik's lifecycle
but they won't attempt any shrinking. That's why I recommend the next option.
4. Option 3: Hand in a cancel index and modulo it over the number of line items
To generate the action:
Arbitraries.integer().between(0, MAX_LINE_ITEMS)
.map(cancelIndex -> new CancelLineItemAction(cancelIndex))
Use it in action:
List<LineItem> lineItems = state.orders().stream()
.flatMap(order -> order.lineItems().stream())
.collect(Collectors.toList());
int randomIndex = cancelIndex % lineItems.size();
LineItem line = lineItems.get(randomIndex);
The approach is described in more detail here: https://blog.johanneslink.net/2020/03/11/model-based-testing/
5. Future Outlook
In some more or less distant future jqwik may allow to hand in the current state when generating actions. This would make stuff like yours a bit simpler. But this feature has not yet been prioritized.
Im trying to understand streams in java better, and im just trying to represent a nested for loop as a stream but im struggling to figure it out.
Ive tried a handful of things, but I cant seem to grasp it.
For example, something like this.
for (Profile profile : profiles) {
for (User user : users) {
if (user.getUserId().equals(profile.getProfileId())) {
profile.setField(false);
}
}
}
How would one do this as a stream in Java?
I think you are trying to understand stream, but let me say that the for version is not worse and the Stream version is not better: performance test and intuition will probably help you there, as well as readability.
That said, you problem here is that your second look require the first one to work:
profiles.stream()
.filter(profile -> users.stream()
.anyMatch(user -> user.getUserId().equals(profile.getProfileId())))
.forEach(profile -> profile.setField(false));
This is strictly not the same as your for loop: you call setField several time per profile, while in the stream version it is only ever done if there is one matching user. I assumed that setField is a setter, and as such since its value is constant, it should not matter if it is called once or several times.
I would advise you not to use the first stream in this case, or limit yourself to forEach:
profiles.forEach(profile -> profile.setField(!users.stream()
.anyMatch(user -> user.getUserId().equals(profile.getProfileId())));
Which could be simplified by first getting all user ids and using contains from the generated set:
var userIds = users.stream().map(User::getUserId).collect(toSet());
profiles.forEach(profile -> profile.setField(!userIds.contains(profile.getProfileId()));
I try to achieve the following:
I want to verify that a specific node in body ("entry") has between a minimum and a maximum number of direct subnodes called "content" (1 to 10 nodes). I don't care about what is in these subnodes, which value they have or how many subnodes they have.
Since I'm new to pact and don't really know how to match something like this I hope someone can help me.
Thanks in advance.
Edit1:
I use a node matcher one of my collegues built like following:
return builder.given("given").uponReceiving("description").path(SERVICEPATH)
.query(query).method(METHOD_GET).headers(ACCEPT, REQ_ACCEPT).willRespondWith().status(200)
.matchHeader(HEADER_CONTENT_TYPE, RESP_HEADER_CONTENT_TYPE_REGEX).body(responseBody)
.matchBody(new PactDslXmlNodeMatcher().node("entry").node("content").value(".*")).toPact();
Don't let that irritate you, that .matchBody just converts that node to the typical $. + path + '[#text]'-notation, where .value adds an regex-matcher rule to body matchers.
I also had a look at: https://github.com/DiUS/pact-jvm/tree/master/pact-jvm-consumer-junit where maxArrayLike / minArrayLike looks promising, but I don't know how I can apply this to my case.
Edit2:
Now I have a very cool PactDslWithProvider as following:
return builder.given("Genereller Test").uponReceiving("Feed soll Eintraege enthalten").path(SERVICEPATH)
.query(query).method(METHOD_GET).headers(ACCEPT, REQ_ACCEPT).willRespondWith().status(200)
.matchHeader(HEADER_CONTENT_TYPE, RESP_HEADER_CONTENT_TYPE_REGEX)
.body(responseBody)
.matchBody(new PactDslXmlNodeMatcher().node("feed").minMaxType(minNumberNodesInFeed, maxNumberNodesInFeed))
.toPact();
The method "minMaxType" adds a MinMaxTypeMatcher to the body-category with the path of the node.
The actual behaviour of this:
It matches type, min and max of the most inner nodes of $.feed. Like: $.feed.0.title, $.feed.1.link, ..., $.feed.6.entry.0.title
But what I actually want is to verify that $.feed has a min and a max number of subnodes. How can I achieve that?
It sounds like you're trying to use Pact to do a functional test, rather than a contract test. Generally speaking, you shouldn't be writing tests that care about how many items there are in an array. Have a read of these two links and let me know how you go.
https://docs.pact.io/best-practices/consumer#use-pact-for-contract-testing-not-functional-testing-of-the-provider
https://docs.pact.io/best-practices/consumer/contract-tests-vs-functional-tests
Also, join us on slack.pact.io if you haven't already.
In a scenario of re-solving a previously solved problem (with some new data, of course), it's typically impossible to re-assign a vehicle's very-first assignment once it was given. The driver is already on its way, and any new solution has to take into account that:
the job must remain his (can't be assigned to another vehicle)
the activity that's been assigned to him as the very-first, must remain so in future solutions
For the sake of simplicity, I'm using a single vehicle scenario, and only trying to impose the second bullet (i.e. ensure that a certain activity will be the first in the solution).
This is how I defined the constraint:
new HardActivityConstraint()
{
#Override
public ConstraintsStatus fulfilled(JobInsertionContext iFacts, TourActivity prevAct, TourActivity newAct, TourActivity nextAct,
double prevActDepTime)
{
String locationId = newAct.getLocation().getId();
// we want to make sure that any solution will have "C1" as its first activity
boolean activityShouldBeFirst = locationId.equals("C1");
boolean attemptingToInsertFirst = (prevAct instanceof Start);
if (activityShouldBeFirst && !attemptingToInsertFirst)
return ConstraintsStatus.NOT_FULFILLED_BREAK;
if (!activityShouldBeFirst && attemptingToInsertFirst)
return ConstraintsStatus.NOT_FULFILLED;
return ConstraintsStatus.FULFILLED;
}
}
This is how I build the algorithm:
VehicleRoutingAlgorithmBuilder vraBuilder;
vraBuilder = new VehicleRoutingAlgorithmBuilder(vrpProblem, "schrimpf.xml");
vraBuilder.addCoreConstraints();
vraBuilder.addDefaultCostCalculators();
StateManager stateManager = new StateManager(vrpProblem);
ConstraintManager constraintManager = new ConstraintManager(vrpProblem, stateManager);
constraintManager.addConstraint(new HardActivityConstraint() { ... }, Priority.HIGH);
vraBuilder.setStateAndConstraintManager(stateManager, constraintManager);
VehicleRoutingAlgorithm algorithm = vraBuilder.build();
The results are not good. I'm only getting solutions with a single job assigned (the one with the required activity). In debug it's clear that the job insertion iterations consider many viable options that appear to solve the problem entirely, but at the bottom line, the best solution returned by the algorithm doesn't include the other jobs.
UPDATE: even more surprising, is that when I use the constraint in scenarios with over 5 vehicles, it works fine (worst results are with 1 vehicle).
I'll gladly attach more information if needed.
Thanks
Zach
First, you can use initial routes to ensure that certain jobs need to be assigned to specific vehicles right from the beginning (see example).
Second, to ensure that no activity will be inserted between start and your initial job(location) (e.g. "C1" in your example), you need to prohibit it the way you defined your HardActConstraint, just modify it so that a newAct can never be between prevAct=Start and nextAct=act(C1).
Third, with regards to your update, just have in mind that the essence of the algorithm is to ruin part of the solution (remove a number of jobs) and recreate the solution again (insert the unassigned jobs). Currently, the schrimpf algorithm ruins a number of jobs relative to the total number of jobs, i.e. noJobs = 0.5 * totalNoJobs for the random ruin and 0.3 * totalNoJobs for the radial ruin. If your problem is very small, the share of jobs to be removed might not sufficiant. This is going to change with next release, where you can use an algorithm out of the box which defines an absolute minimum of jobs that need to be removed. For the time being, modify the shares in your algorithmConfig.xml.
The Title is self explanatory. This was an interview question. In java, List is an interface. So it should be initialized by some collection.
I feel that this is a tricky question to confuse. Am I correct or not? How to answer this question?
Assuming you don't have a copy of the original List, and the randomizing algorithm is truly random, then no, you cannot restore the original List.
The explanation is far more important on this type of question than the answer. To be able to explain it fully, you need to describe it using the mathematical definitions of Function and Map (not the Java class definitions).
A Function is a Map of Elements in one Domain to another Domain. In our example, the first domain is the "order" in the first list, and the second domain is the "order" in the second list. Any way that can get from the first domain to the second domain, where each element in the first domain only goes to one of the elements in the second domain is a Function.
What they want is to know if there is an Inverse Function, or a corresponding function that can "back map" the elements from the second domain to the elements in the first domain. Some functions (squaring a number, or F(x) = x*x ) cannot be reversed because one element in the second domain might map back to multiple (or none) elements in the first domain. In the squaring a number example
F(x) = x * x
F(3) = 9 or ( 3 -> 9)
F(12) = 144 or ( 12 -> 144)
F(-11) = 121 or (-11 -> 121)
F(-3) = 9 or ( -3 -> 9)
attempting the inverse function, we need a function where
9 maps to 3
144 maps to 12
121 maps to -11
9 maps to -3
Since 9 must map to 3 and -3, and a Map must have only one destination for every origin, constructing an inverse function of x*x is not possible; that's why mathematicians fudge with the square root operator and say (plus or minus).
Going back to our randomized list. If you know that the map is truly random, then you know that the output value is truly independent of the input value. Thus if you attempted to create the inverse function, you would run into the delimma. Knowledge that the function is random tells you that the input cannot be calculated from the output, so even though you "know" the function, you cannot make any assumptions about the input even if you have the output.
Unless, it is pseudo-random (just appears to be random) and you can gather enough information to reverse the now-not-truly random function.
If you have not kept some external order information (this includes things like JVM trickery with ghost copies), and the items are not implicitly ordered, you cannot recover the original ordering.
When information is lost, it is lost. If the structure of the list is the only place recording the order you want, and you disturb that order, it's gone for good.
There's a user's view, and there's internals. There's the question as understood and the question as can be interpreted.
The user's view is that list items are blocks of memory, and that the pointer to the next item is a set of (4?8? they keep changing the numbers:) bytes inside this memory. So when the list is randomized and the pointer to the next item is changed, that area of memory is overriden and can't be recovered.
The question as understood is that you are given a list after it had been randomized.
Internals - I'm not a Java or an OS guy, but you should look into situations where the manner in which the process is executed differs from the naive view: Maybe Java randomizes lists by copying all the cells, so the old list is still kept in memory somewhere? Maybe it keeps backup values of pointers? Maybe the pointers are kept at an external table, separate from the list, and can be reconstructed? Maybe. Internals.
Understanding - Who says you haven't got an access to the list before it was randomized? You could have just printed it out! Or maybe you have a trace of the execution? Or who said you're using Java's built it list? Maybe you are using your own version controlled list? Or maybe you're using your own reversable-randomize method?
Edwin Buck's answer is great but it all depends what the interviewer was looking for.