In unit-tests should I write one unit per case or one unit per assert from support cost point of view? I have the following code
void methodUnderTest(Resource resource) {
if(!resource.hasValue()) {
Value value = valueService.getValue(resource);
resource.setValue(value);
}
// resource.setLastUpdateTime(new Date()); // will be added in future
db.persist(resource);
email.send(resource);
}
The commented line will be added in near future and I think what it will cost to update units.
As far as I see there are two ways to test this code.
Write 2 units passing resource with value and without value. In both tests verify that db.persist and email.send were called. When LastUpdateTime is added I'll have to update both tests to verify the property was set.
Write separate unit tests: one checks that db.persist was called, the other checks email.send, third and fourth for resource with and without value. When LastUpdateTime is added I just write new unit.
I like the second way because I like the idea that I won't have to touch working units. But that would be probably a lot of code duplication because actually all 4 tests do the same and only use different asserts.
The first approach looks more correct from 'just one concept per unit test' point of view. But aren't such tests hard to maintain? Adding something new I will always have to revise all existing tests and it doesn't sound good.
Is there some best practise here I should follow?
I propose you put the common stuff of all tests into the setUp() for the tests, the have exactly one assert per unit test, as in your second way of testing.
When you add the new line of code, you just add one more test case with a single assert.
Not modification of existing tests, not code duplication.
There is no general answer for this. You need to balance your needs and constraints:
If a test is small, executes quickly and fails very rarely, there is no reason to split it into several.
If you have many assertions in a single test, then all of them probably provide value towards figuring out why the test fails but only the first one will be executed, keeping you from valuable information. Try to have only a single assert per test (combine the assertions into one big string, for example).
If you test several features in a single test, you have a variant of the "missing valuable information." If each test tests a single feature, the combination of succeeded and failed tests may give you the clue why they fail. Try to aim for a single feature per test.
Lastly, it gives me a good feeling when I see thousands of tests being executed.
Related
On my unit test I want to check if one or another method was called. I can easily verify how many times some methods are called thanks to Mockito, but verify has no verification mode like "OR". Any workarounds?
In my case I want to check if on SharedPreferences.Editor was called .apply() or .commit() because two of this possibilities satisfies me and saves data. Unfortunately if I call verify(mEditor).apply() but someone will change implementation to .commit() in example due to requirement of instant save, the test will fail, but shouldn't because I want to only test from this point of view if data are saved or not. It's the unit test and should be independent from changes like that and checks only scope of what are tested inside.
The work around you ask for would be to catch the underlying MockitoAssertionError (or just AssertionError):
try {
verify(mEditor).apply();
} catch (MockitoAssertionError mae) {
// apply was not called. Let's verify commit instead.
verify(mEditor).commit();
}
Alternatively, if both apply and commit call some (internal) save method you could also try verifying that (assuming it is exposed -- mock-based testing can be at odds with information hiding). Or, if you have control over the code you're testing you could refactor it along these lines.
The better advice, though, would be to avoid the need for this altogether, as argued in the answer by #GhostCat.
I am not aware of a good way of doing that, and honestly, I think the real answer is: do not do that. Yes, the other answer shows a way to achieve what you ask for, but then:
You know what your production code should be doing. Meaning: instead of writing a single piece of verification code that allows "this or that", rather write two independent tests, one for "this", and one for "that".
In other words: you control what goes into your tests. So write one test that should result in apply(), and one that should result in commit(). And then verify() that one case that each test is expected to see!
Unit tests should be straight forward. When something fails, you quickly look at the unit test and you already know where to look in the production code to spot the root cause. Anything that adds complexity to your tests might make that harder. It is better to have two tests that follow a clear "when then verify" path, instead of having one (or multiple) tests that go "when then verify this OR verify that".
I am learner in writing Junit Test Cases. I have seen writing Junit cases Pattern that We usually make test class for each class indivisually by their name and write test cases for each method of that class in its respective class so that maximum code coverage can occur.
What I was thinking If I make test cases for my feature that would be better choice because In future any number of methods Signature changes I don't have to change or create again unnecessary test cases for those modified methods or newly created. Because that moment I would have certain test cases for my developed feature. So my test cases are running fine for particular feature then I can be sure in minimum number of test cases code that everything is fine.
By keeping this I don't have to write test cases for each and every methods of each class. Is it a good way?
Well, test cases are written for a reason. Each and every methods have to be working properly as expected. If you only do test cases for the feature level, how do you find exactly where the error occurred and how confidently you can ship your code to next level?
The better approach will be to do unit test cases for each class and do an integration test to make sure everything works good.
We found success in utilizing both. By default we use one per class. But when particular use-cases come up, e.g. use-cases that involve multiple classes or use cases where the existing boiler plate testing code prevents the use case from being properly tested, then we would create a test class for that scenario.
by keeping this I don't have to write test cases for each and every
methods of each class. Is it a good way?
In this case you write only integration tests and no unit tests.
Writing tests for use cases is really nice but it is not enough because it is hard to cover all cases of all methods invoked in an integration test because there may have a very important number of branches while it is much easier in an unit test.
Besides, a use case test may be successful for bad reasons : thanks to side effects between multiple methods invoked.
By writing an unit test you protect yourself against this kind of issue.
Definitively, unit and integration tests are are not opposed but complementary. So you have to write both to get a robust application.
In my Vaadin GUI application, there are so many methods which look like below.
#Override
protected void loadLayout() {
CssLayout statusLayout = new CssLayout();
statusLayout.addComponent(connectedTextLabel);
statusLayout.addComponent(connectedCountLabel);
statusLayout.addComponent(notConnectedTextLabel);
statusLayout.addComponent(notConnectedCountLabel);
connectionsTable.getCustomHeaderLayout().addComponent(statusLayout);
connectionsTable.getCustomHeaderLayout().addComponent(commandLayout);
connectionsTable.getCustomHeaderLayout().addComponent(historyViewCheckbox);
bodySplitter.addComponent(connectionsTable);
bodySplitter.addComponent(connectionHistoryTable);
bodySplitter.setSplitPosition(75, Sizeable.Unit.PERCENTAGE);
bodySplitter.setSizeFull();
bodyLayout.addComponent(bodySplitter);
if (connectionDef.getConnectionHistoryDef() == null) {
historyViewCheckbox.setVisible(false);
}
if (connectionDef.getConnectionStatusField() == null || connectionDef.getConnectedStatusValue() == null || connectionDef.getConnectedStatusValue().isEmpty()) {
connectedTextLabel.setVisible(false);
connectedCountLabel.setVisible(false);
notConnectedTextLabel.setVisible(false);
notConnectedCountLabel.setVisible(false);
}
}
protected void setStyleNamesAndControlIds() {
mainLayout.setId("mainLayout");
header.setId("header");
footer.setId("footer");
propertyEditorLayout.setId("propertyEditorLayout");
propertyEditor.setId("propertyEditor");
mainLayout.setStyleName("mainLayout");
propertyEditorLayout.setStyleName("ui_action_edit");
header.setStyleName("TopPane");
footer.setStyleName("footer");
}
These methods are used for setting up the layout of GUIs. They do not produce a single distinct output. Almost every line in these methods is doing a separate job, which is not almost relevant to other lines.
Usually, when unit testing a method, I check the return value of the method, or validate calls on a limited number of external objects such as database connections.
But, for methods like above, there is no such single output. If I wrote unit tests for such methods, My test code checks for each method call happens in every line in the method, and in the end, it looks almost like the method itself.
If someone altered the code in any way, the test will break and they will have to update the test to match the change. But, there is no assurance that the change didn't actually break anything since test doesn't check the actual UI drawn in the browser.
For an example, if someone changed a style name of a control, he will have to update the test code with the new style name and the test will pass. But, for things to actually work without any issue, he has to change the relevant scss style files too. But the test didn't make any contribution to detect this issue. Same applies to layout setup code as well.
Is there any advantage of writing unit tests like above, other than keeping the code coverage rating at a higher level? For me, it feels useless and writing a test to compare the decompiled bytecode of the method to the original decompiled bytecode kept as a string in the test looks much better than these kinds of tests.
Is there any advantage of writing unit tests like above, other than
keeping the code coverage rating at a higher level?
Yes, if you take a sensible approach. It might not make sense, as you say, to test that a control has a particular style. So focus your tests on the parts of your code that are likely to break. If there is any conditional logic that goes into producing your UI, test that logic. The test will then protect your code from future changes that could break your logic.
As for you comment about testing methods that don't return a value, you can address that several ways.
It's your code, so you can restructure it to be more testable. Think about breaking it down into smaller methods. Isolate your logic into individual methods that can be called in a test.
Indirect verification - Rather than focusing on return values, focus on the effect your method has on other objects in the system.
Finally consider if unit testing of the UI is right for you and your organization. UIs are often difficult to unit test (as you have pointed out). Many companies write functional tests for their UIs. These are tests that drive the UI of the actual product. This is very different from unit tests which do not require the full product and are targeted at very small units of functionality.
Here's one simple example you could look to see how to fly over your application and try what is needed. This is vaadin 8, CDI & Wildfly Swarm example and in no way the only way to test UI's of Vaadin application.
https://github.com/wildfly-swarm/wildfly-swarm-examples/blob/master/vaadin/src/it/java/org/wildfly/swarm/it/vaadin/VaadinApplicationIT.java
Is there a way to create a custom unit test result in TestNG/JUnit (or any other Java testing framework)? I understand that unit tests can either pass, or fail (or ignored), but currently I really would like to have the third option.
The company I'm working with right now has adapted the testing style of cleverly comparing screenshots of their application and so the test can either pass, fail, or diff, when the screenshots does not match with predetermined tolerance. In addition, they have their in house test "framework" and runners. This was done long before I joined.
What I would like to do is to migrate test framework to the one of the standard ones, but this process should be very gradual.
The approach I was thinking about was to create a special exception (e.g. DiffTolleranceExcededException), fail the test and then customize test result in the report.
Maybe you already mean the following with
The approach I was thinking about was to create a special exception
(e.g. DiffTolleranceExcededException), fail the test and then
customize test result in the report.
but just in case: You certainly can use the possibility to give a pre-defined message string to the assertions. In your case, if the screenshots are identical, the tests pass. If they are too different, the tests just fail. If they are within tolerance, you make them fail with a message like "DIFFERENT BUT WITHIN-TOLERANCE" or whatever - these failures are then easily distinguishable. Certainly, you could also invert the logic: Add a message to the failures that are not within the tolerance, to make these visually prominent.
You should follow this approach for customize our test reports, adding a new column on test report and create your test report (with screenshot for example).
I'm coming from a Perl background where I used Test::More to handle unit testing. Using that framework, I knew the order in which the tests took place and could rely on that, which I understand is not encouraged with the JUnit framework. I've seen several ways to get around this, but I want to understand the proper/intended way of doing things.
In my Perl unit testing I would build up tests, knowing that if test #3 passed, I could make some assumptions in further tests. I don't quite see how to structure that in the JUnit world so that I can make every test completely independent.
For example, suppose I have a class that parses a date from a string. Methods include:
parse a simple date (YYYY-MM-DD)
parse a simple date with alternate separator (YYYY_MM_DD or YYYY/MM/DD)
parse a date with a string for a month name (YYYY-MON-DD)
parse a date with a string month name in a different language
and so on
I usually write my code to focus as many of the externally-accessible methods into as few core methods as possible, re-using as much code as possible (which is what most of us would do, I'm sure). So, let's say I have 18 different tests for the first method, 9 that are expected to pass and 9 that throw an exception. For the second method, I only have 3 tests, one each with the separators that work ('_' & '/') and one with a separator that doesn't work ('*') which is expected to fail. I can limit myself to the new code being introduced because I already know that the code properly handles the standard boundary conditions and common errors, because the first 18 tests already passed.
In the Perl world, if test #20 fails, I know that it's probably something to do with the specific separator, and is not a general date parsing error because all of those tests have already passed. In the JUnit world, where tests run in a random order, if test #20 fails, I don't know if it's because of a general date parsing issue or because of the separator. I'd have to go and see which other ones failed and then make some assumptions there. That's not too hard to do, of course, but maybe in a bigger, more complex class, it would be more difficult to do.
How do other people deal with building up a set of tests? Should I put each and every test in a separate class and use a test suite? That seems tedious. (And before someone suggests that I put the first18 in one class and the second 3 in another, and use a test suite for just those groupings, let's pretend that all 18 of the early tests build on each other, too).
And, again, I know there are ways around this (FixedMethodOrder in JUnit 4.11+ or JUnit-HierarchicalContextRunner) but I want to understand the paradigm as its intended to be used.
In the JUnit world, where tests run in a random order, if test #20 fails, I don't know if it's because of a general date parsing issue or because of the separator. I'd have to go and see which other ones failed and then make some assumptions there.
Yes that is correct. If something in your code is broken then multiple tests may fail. That is a good thing. Use intent revealing test method names and possibly use the optional String message parameter in the JUnit assertions to explain what exactly failed the test.
How do other people deal with building up a set of tests? Should I put each and every test in a separate class and use a test suite?
The general convention is one test class per source class. Depending on what build tool you are using, you may or may not need to use test suites. If you are using Ant, you probably need to collect the tests into test suites, but if you are using Maven, the test plugins for maven will find all your test classes for you so you don't need suites.
I also want to point out that you should be coding to Java interfaces as much as possible. If you are testing class C that depends on an implementation of interface I, then you should mock your I implementation in your C test class so that C is tested in isolation. Your mock I should follow what the interface is supposed to do. This also keeps the number of failing tests down. If there is a bug in your real I implementation, then only your I tests should fail, the C tests should still all pass (since you are testing it against a fake but working I implementation)
Don't worry about suites yet. You'll know when you need them. I've only had to use them a handful of times, and I'm not entirely sold on their usefulness...but I leave that decision up to you.
To the meat of your question - the conventional way with JUnit tests is to neither know nor depend on the order of execution of your tests; this ensures that your tests are not run-order dependent, and if they are, something is wrong with your tests* and validation.
The main core concept behind unit tests is that they test a unit of code - as simple as a single function. If you're attempting to test five different things at once, your test is far too large, and should be broken out. If the method you're testing is monolithic in nature, and difficult to test, it should be refactored and broken out into different slices of responsibility.
Tests that exercise a larger flow are better suited for integration-style tests, which tend to be written as unit tests, but aren't actually unit tests.
I've not run into a scenario in which, if I knew that if a certain test failed, I could expect different behavior in the other tests. I've never thought that such a thing was necessary to be noted, since the only thing I care about in my unit test is how that unit of code behaves given a certain input.
Keep your tests small and simple to understand; the test should only make one assertion about the result (or a general assertion of the state of your result).
*: That's not to say that it's completely broken, but those sorts of tests should be fixed sooner rather than later.