I am running tests with uiautomator. When I get to the end of my test, I need to test my results. My problem is of one test fails, the others will not be tested. I need them all to be tested regardless of the results of other test. This is my attempts:
public void testSomeUI() {
////lots of stuff
assertEquals(///assertion///);
assertEquals(///assertion///);
assertEquals(///assertion///);
....and so on
}
Also I tried:
public void testSomeUI() {
////lots of stuff
testValue1();
testValue2();
testValue3();
....and so on
}
private void testValue1(){
assertEquals(///assertion///);
}
private void testValue2(){
assertEquals(///assertion///);
}
private void testValue3(){
assertEquals(///assertion///);
}
..and so on
If one fails, the last three won't run. Any suggestions? Thanks.
Problem is that once an assert fails, it breaks out of the method. That's why the rest don't get run.
Try using a test framework like JUnit (which UIAutomator appears to be built on). Then write one method per assert. That way will you not only get all asserts to run every time, you also break down the tests into suitably small sizes. If they're named properly, you may not need to debug at all, since you can tell by name of the failing test where the problem really is.
Here's a link to a tutorial for example.
Related
When debugging a test I realized that something was interfering with Mockito well functioning. Somehow, the inclusion of breakpoints in specific classes leads to a different output.
I try to illustrate it with a simple example.
public class MockitoTrial {
#Test
public void simpleTest() {
var func = Mockito.mock(Function.class);
Entry<String, Integer> entry = new SimpleEntry<>("one", 1);
when(func.apply(eq(Entry.class))).thenReturn(entry);
assertThat(func.apply(Entry.class)).isEqualTo(entry);
}
}
If I set a breakpoint for instance in org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.interceptAbstract, and rerun it in debug-mode, the test fails.
It seems apparently unrelated to the IDE as it happens when debugging remotely as well.
The library versions I am using:
assertj-core-3.22.0
junit-jupiter-api-5.8.2
mockito-core-4.5.1
The question pointed out by Jonasz, gives a good clue.
If I instead set the breakpoint in org.mockito.internal.handler.MockHandlerImpl.handle, the test does already pass even in debug-mode.
Now, I add a watcher for the expression invocation.getMock().toString() and rerun it in debug-mode. Doing that, I got the test to fail again.
Mockito is very sensitive to calls to the mock methods in between its processing.
I'm executing a few hundred tests in test classes consisting of a singular beforeMethod test, followed by a variable amount of primary tests and occasionally an afterMethod.
The purpose of the beforMethod test, is to populate the test environment with data used in the primary tests while separating logging and recording from the primary tests, which we report on.
We have set up an automatic issue creation tool using a listener. We've found that it would give great value to add execution time to this tool, so that it can show us how long it would take to reproduce the errors in said issues.
To this end, I have made a simple addition to this code, that uses ITestResult.getEndMillis() and getStartMillis() to get the execution time.
The problem we're experiencing with this approach, is that if the test encounters a failure during the primary tests, ITestResult.getStartMillis() will not account for the start time of the before method, but only the primary method.
How would we go about determining the start time of the test class itself (always the beforeMethod), rather than just the current method?
Since we're running hundreds of tests in a massive setup, a solution that allows this without changing each separate test class, would definitely be preferable.
The setup of the java test classes look something like this (scrubbed of business specifics):
package foobar;
import foobar
#UsingTunnel
#Test
public class FLOWNAME_TESTNAME extends TestBase {
private final Value<String> parameter;
public FLOWNAME_TESTNAME(Value<String> parameter) {
super(PropertyProviderImpl.get());
this.parameter = parameter;
}
#StoryCreating(test = "TESTNAME")
#BeforeMethod
public void CONDITIONS() throws Throwable {
new TESTNAME_CONDITIONS(parameter).executeTest();
}
#TestCoverage(test = "TESTNAME")
public void PRIMARYTESTS() throws Throwable {
TESTCASE1 testcase1 = new TESTCASE1(parameter.get());
testcase1.executeTest();
testcase1.throwSoftAsserts();
TESTCASE2 testcase2 = new TESTCASE2(parameter.get());
testcase2.executeTest();
testcase2.throwSoftAsserts();
}
}
So in this case, the problem arises when the listener detects a failure in either TESTCASE1 or TESTCASE2, because these will not include the execution time of TESTNAME_CONDITIONS because that test is inside a different method, yet practically speaking, they are part of the same test flow, aka the same test class.
I found a solution to the issue.
It is possible to use ITestResult.getTestContext().getStartDate().getTime() to obtain the time of which the test class itself is run, rather than the current test method.
The final solution was quite simply:
result.getEndMillis() - result.getTestContext().getStartDate().getTime()) / 60000
Where "result" corresponds to ITestResult.
This outputs the time between the start of the test and the end of the last executed method.
When writing code that interacts with external resources (such as using a web service or other network operation), I often structure the classes so that it can also be "stubbed" using a file or some other input method. So then I end up using the stubbed implementation to test other parts of the system and then one or two tests that specifically test calling the web service.
The problem is I don't want to be calling these external services either from Jenkins or when I run all of the tests for my project (e.g. "gradle test"). Some of the services have side effects, or may not be accessible to all developers.
Right now I just uncomment and then re-comment the #Test annotation on these particular test methods to enable and disable them. Enable it, run it manually to check it, then remember to comment it out again.
// Uncomment to test external service manually
//#Test
public void testSomethingExternal() {
Is there is a better way of doing this?
EDIT: For manual unit testing, I use Eclipse and am able to just right-click on the test method and do Run As -> JUnit test. But that doesn't work without the (uncommented) annotation.
I recommend using junit categories. See this blog for details : https://community.oracle.com/blogs/johnsmart/2010/04/25/grouping-tests-using-junit-categories-0.
Basically, you can annotate some tests as being in a special category and then you can set up a two test suites : one that runs the tests of that category and one that ignores tests in that category (but runs everything else)
#Category(IntegrationTests.class)
public class AccountIntegrationTest {
#Test
public void thisTestWillTakeSomeTime() {
...
}
#Test
public void thisTestWillTakeEvenLonger() {
....
}
}
you can even annotate individual tests"
public class AccountTest {
#Test
#Category(IntegrationTests.class)
public void thisTestWillTakeSomeTime() {
...
}
Anytime I see something manually getting turned on or off I cringe.
As far as I can see you use gradle and API for JUnit says that annotation #Ignore disables test. I will add gradle task which will add #Ignore for those tests.
If you're just wanting to disable tests for functionality that hasn't been written yet or otherwise manually disable some tests temporarily, you can use #Ignore; the tests will be skipped but still noted in the report.
If you are wanting something like Spring Profiles, where you can define rulesets for which tests get run when, you should either split up your tests into separate test cases or use a Filter.
You can use #Ignore annotation to prevent them from running automatically during test. If required, you may trigger such Ignored tests manually.
#Test
public void wantedTest() {
return checkMyFunction(10);
}
#Ignore
#Test
public void unwantedTest() {
return checkMyFunction(11);
}
In the above example, unwantedTest will be excluded.
I'm using Spock to write tests for a Java application. The test includes a JMockit MockUp for the class. When debugging a test (using IntelliJ) I would like to be able to step into the Java code. I realize using F5 to step into the code won't work, because of all the Groovy magic with reflection that goes on. Unfortunately, even if I set a breakpoint in the Java code, it still will not be hit, even though it runs through the code.
Here is the code to test:
public class TestClass {
private static void initializeProperties() {
// Code that will not run during test
}
public static boolean getValue() {
return true; // <=== BREAK POINT SET HERE WILL NEVER BE HIT
}
}
Here is the test code:
class TestClassSpec extends Specification {
void setup() {
new MockUp<TestClass>() {
#Mock
public static void initializeProperties() {}
}
}
void "run test"() {
when:
boolean result = TestClass.getValue()
then:
result == true
}
}
The test passes and everything works well, but I'm not able to debug into the Java code.
Note: I can debug if I do not use the JMockit MockUp, but that is required to test a private static method in my production code.
Any ideas on how to get the debugger to stop in the Java code from a Spock test that uses JMockit?
This answer has plagued me for days, and I have searched and searched for an answer. Fifteen minutes after posting for help, I find the answer . So for anyone else who runs into this, here's my solution - this is all thanks to this answer: https://stackoverflow.com/a/4852727/2601060
In short, the breakpoint is removed when the class is redefined by JMockit. The solution is to set a break point in the test, and once the debugger has stopped in the test THEN set the breakpoint in the Java code.
And there was much rejoicing...
I'm having a strange problem with Easymock 3.0 and JUnit 4.8.2.
The problem only occurs when executing the tests from Maven and not from Eclipse.
This is the unit test (very simple):
...
protected ValueExtractorRetriever mockedRetriever;
...
#Before
public void before() {
mockedRetriever = createStrictMock(ValueExtractorRetriever.class);
}
#After
public void after() {
reset(mockedRetriever);
}
#Test
public void testNullValueExtractor() {
expect(mockedRetriever.retrieve("PROP")).andReturn(null).once();
replay(mockedRetriever);
ValueExtractor retriever = mockedRetriever.retrieve("PROP");
assertNull(retriever);
assertTrue(true);
}
And I get:
java.lang.IllegalStateException: 1 matchers expected, 2 recorded.
The weird thing is that I'm not even using an argument matcher. And that is the only method of the test! and to make it even worst it works from Eclipse and fails from Maven!
I found a few links which didn't provide me with an answer:
Another StackOverflow post
Expected Exceptions in JUnit
If I change the unit test and add one more method (which does use an argument matcher):
#Test
public void testIsBeforeDateOk() {
expect(mockedRetriever.retrieve((String)anyObject())).andReturn(new PofExtractor()).anyTimes();
replay(this.mockedRetriever);
FilterBuilder fb = new FilterBuilder();
assertNotNull(fb);
CriteriaFilter cf = new CriteriaFilter();
assertNotNull(cf);
cf.getValues().add("2010-12-29T14:45:23");
cf.setType(CriteriaType.DATE);
cf.setClause(Clause.IS_BEFORE_THE_DATE);
CriteriaQueryClause clause = CriteriaQueryClause.fromValue(cf.getClause());
assertNotNull(clause);
assertEquals(CriteriaQueryClause.IS_BEFORE_THE_DATE, clause);
clause.buildFilter(fb, cf, mockedRetriever);
assertNotNull(fb);
Filter[] filters = fb.getFilters();
assertNotNull(filters);
assertEquals(filters.length, 1);
verify(mockedRetriever);
logger.info("OK");
}
this last method passes the test but not the other one. How is this possible!?!?!
Regards,
Nico
More links:
"bartling.blogspot.com/2009/11/using-argument-matchers-in-easymock-and.html"
"www.springone2gx.com/blog/scott_leberknight/2008/09/the_n_matchers_expected_m_recorded_problem_in_easymock"
"stackoverflow.com/questions/4605997/3-matchers-expected-4-recorded"
I had a very similar problem and wrote my findings in the link below.
http://www.flyingtomoon.com/2011/04/unclosed-record-state-problem-in.html (just updated)
I believe the problem in on another test that affects your current test. The problem is on another test class and it affects you test. In order to find the place of the real problem, I advice to disable the problematic tests one by one till you notify the failing test.
Actually this is what I did. I disabled the failing tests one by one till I found the problematic test. I found a test that throws an exception and catches by "#extected" annotation without stopping the recording.
We had this problem recently, and it only reared its head when we ran the entire test suite (1100+ test cases). Eventually, I found that I could put a breakpoint on the test that was blowing up, and then step back in the list of tests that Eclipse had already executed, looking for the previous test case that had set up a mock incorrectly.
Our problem turned out to be somebody using EasyMock.anyString() outside of an EasyMock.expect(...) statement. Sure enough, it was done two tests before the one that was failing.
So essentially, what was happening is that the misuse of a matcher outside of an expect statement was poisoning EasyMock's state, and the next time we tried to create a mock, EasyMock would blow up.
I believe the first error message
java.lang.IllegalStateException: 1
matchers expected, 2 recorded.
means your mockedRetriever methods called twice but test expects it was called once. So your Eclipse and Maven's configuration differs.
And I have no reason to reset mock after test. Just keep in mind JUnit creates new class instance for every single test method.
EDITED:
What about the reason why the last test method passed the answer is:
expect(mockedRetriever.retrieve((String)anyObject())).andReturn(new PofExtractor()).anyTimes();
But in your first test method it is:
expect(mockedRetriever.retrieve("PROP")).andReturn(null).once();
as equivalent of:
expect(mockedRetriever.retrieve("PROP")).andReturn(null);