I have a jar file that I reflect from it test classes (junit test).
I created TestSuite instance and added the tests to it.
In order to check my code, I've tried to add only one example test class to the test suite.
TestSuite suite = new TestSuite();
suite.addTest(new JUnit4TestAdapter(ExampleTest.class));
Then I called suite.run() in order to run the test:
TestResult result = new TestResult();
suite.run(result);
The problem is that when the test is done, the failures list that should be in result is empty.
Enumeration<TestFailure> failures = result.failures(); // = empty list
How can I get the failures using testSuite? If I use JUnitCore.runclasses , I do get the failures list but it can't be used with an instance of TestSuite, which I have to use since I get the test classes as input.
Edit-1:
If it is possible to create a Class that extends TestSuite and add classes to the suite dynamically, it will be good for my needs to.
Any suggestion will be great.
Edit-2:
From more searching in the web, I saw there us a difference between a failure and a testFailure. How can I fail a test with testFailure and not a failure?
After some more debugging, I found out that the TestResult object did catch the failures but reported them as Errors (Assertion Errors).
I also found out that TestResult.errors() returned an enumerator of TestFailure. Each TestFailure contain information about the thrown exception, so I can distinguish now between errors and failures by checking which of the errors are Assertion Errors and which are not.
Related
I have a couple of classes which contain Tests.
I have a main method that uses JUnitCore in order to run all tests.
What can I do in order to run specific tests in each class?
Currently I use something like this to run all my tests :
Result result = JUnitCore.runClasses(TestJunit.class, TestJunit2.class);
Maybe there is a possibility to categorize the relevant tests and then run them using JUnitCore ?? Thanks !
You can build a org.junit.runner.Request by providing class and method name and pass it to run method of JUnitCoreclass. This will execute given test of the specified class.
Request request = Request.method(TestClass.class, "methodName");
Result result = new JUnitCore().run(request);
You can check the result of test by invoking wasSuccessful() method available in Result class .
I am using extentreports to add reports for my test which is written with Java and Selenium.
I notice that if a ExtentTest has two logs, "INFO" and "PASSED", if the pass log is not catched it will be considered as PASSED. How can I change it in a way that if a test is not passed it will automatically be singed as Failed?
Not possible to change the behavior to fail the test case by default. Its a design decision, every test framework follows (TestNG, Junit, NUnit etc) not just Extent Reports.
The assumption is to start the test, that it will always pass. So even you have only Info logs, test case is marked as passed.
However, if by default, you assume the test is failed, until you encounter PASS, the test is still considered FAILED, because, even a single fail status in the entire test fails the test, regardless of how many pass logs you have.
You don't provide detail of your code and version you are using for extent report.
Assume u are using extent report version 2.40 and use this code :
#AfterMethod
public void tearDown(ITestResult result)
{
if(result.getStatus()==ITestResult.FAILURE)
{
//hope u know how to create ExtenTest and ExtentReport instance
//logger is extent test instance
logger.log(LogStatus.FAIL, "Title verification", image);
}
//report is ExtentReport instance
report.endTest(logger);
report.flush();
}
I wrote a eclipse plugin that as part of its job, it reads all test cases of the current active projects and run them. The problem is when it wants to run test cases I get the error initializationError : No runnable methods.
Extra information that may help:
Unit test code is running JUnit4.11. All test cases are annotated with #Test. Test are all public and not static. And tests are working correctly when I run them directly. Therefore, there is no problem with tests, and it is probably because of below code that I am using to load test class and then run that.
The code that I used to load class and then run test cases: (Suppose the active project open in eclipse is ExampleProject).
//Load class contains test cases
File root = new File("c:\ExampleProject\bin")
classLoader = URLClassLoader.newInstance(new URL[] {root.toURI().toURL()});
cls = Class.forName(testClassName, true, classLoader);
//Above lines work and I have test class determined with testClassName
//create a request
Request request = Request.aClass(cls);
//run the request
Result result = new JUnitCore().run(request);
for (Failure failure : result.getFailures()) {
System.out.println(failure.toString());
}
It prints initializationError(testClassName): No runnable methods.
Please let me know if any one knows what is the problem and how can I fix that.
PS: Note that when I run a test case from my plugin (C:\myplugin\bin), then above code works fine and it runs the test case exist in plugin, but when I want to run a test from current project then get the above run time error. Probably, some tings is wrong about classPath?
Using JUnit 4, my ant build script that invokes junit with the task always reports the test class names as "unknown". Is there a way of fixing this that does not involve deriving from TestCase?
I declare my test methods with the #Test annotation, e.g.:
SomeTest.java:
public class SomeTest {
#Test
public void doSomething() { .. }
}
and collect them together in a TestSuite:
AllTests.java:
#RunWith(Suite.class)
#Suite.SuiteClasses( {
SomeTest.class
})
public class AllTests {
}
I then invoke the ant task via NetBeans, which has nice integrated support for printing results of a JUnit run. Everything seems find except the test names are always "unknown", e.g.:
6 tests passed, 2 tests failed, 2 tests caused an error (8.4s)
com.mystuff.AllTests FAILED
Unknown passed (0.0s)
Unknown passed (0.0s)
Unknown passed (0.0s)
Unknown passed (0.0s)
Unknown passed (0.0s)
Unknown passed (0.0s)
Unknown FAILED (0.0s)
Unknown caused an ERROR: at org.hibernate.somethingorother.java:1234 (0.0s)
..etc..
The tests operate correctly, and I can usually figure out the point of failure from the stack trace that's logged, but everything being reported as 'Unknown' is simply annoying and obtuse.
Having read up on a few SO posts, I discovered there are two ways to set your tests up, the old JUnit 3 way of deriving from TestCase, and the new Junit 4 way of using #Test annotations.
Not able to execute tests with #Test annotation when my test extends TestCase(Junit) in Eclipse
JUnit confusion: use 'extends TestCase' or '#Test'?
And this post mentions "In JUnit 4 the test classes no longer extend a common framework class. So there's no inherited method getName any more"
How to obtain test case name in JUnit 4 at runtime?
I tried extending my test case from TestCase, and this did make the name be reported correctly, but this single change screwed up some of its lifecycle and the tests failed in bizzarre ways. My understanding is that when the JUnit runner sees your tests class is derived from TestCase is runs it as a JUnit 3 test case and would therefore ignore all my #Before, #After etc. annotations. I don't see the point in stepping back into JUnit 3 test mechanisms.
Well this is embarrassing, but it deserves to be said. I think I just answered my own question.
The answer seems to be one of the tasks attributes, enableTestListenerEvents, described here: http://ant.apache.org/manual/Tasks/junit.html
I had ignored it because the docs say "since Ant 1.8.2 - Ant 1.7.0 to 1.8.1 behave as if this attribute was true by default." and I'm running 1.8.2, but I'd misread it, it's actually saying that in 1.8.2 it's off by default.
So the fix is to change my task in build.xml:
<junit haltonfailure="true" printsummary="on" fork="yes" dir=".">
to
<junit haltonfailure="true" printsummary="on" fork="yes" dir="."
enableTestListenerEvents="true">
NetBeans now prints up-to-the-moment information in the Test Results pane, including test names, as the tests progress. The whole feedback experience is far richer. Tell your friends :)
I'm having a strange problem with Easymock 3.0 and JUnit 4.8.2.
The problem only occurs when executing the tests from Maven and not from Eclipse.
This is the unit test (very simple):
...
protected ValueExtractorRetriever mockedRetriever;
...
#Before
public void before() {
mockedRetriever = createStrictMock(ValueExtractorRetriever.class);
}
#After
public void after() {
reset(mockedRetriever);
}
#Test
public void testNullValueExtractor() {
expect(mockedRetriever.retrieve("PROP")).andReturn(null).once();
replay(mockedRetriever);
ValueExtractor retriever = mockedRetriever.retrieve("PROP");
assertNull(retriever);
assertTrue(true);
}
And I get:
java.lang.IllegalStateException: 1 matchers expected, 2 recorded.
The weird thing is that I'm not even using an argument matcher. And that is the only method of the test! and to make it even worst it works from Eclipse and fails from Maven!
I found a few links which didn't provide me with an answer:
Another StackOverflow post
Expected Exceptions in JUnit
If I change the unit test and add one more method (which does use an argument matcher):
#Test
public void testIsBeforeDateOk() {
expect(mockedRetriever.retrieve((String)anyObject())).andReturn(new PofExtractor()).anyTimes();
replay(this.mockedRetriever);
FilterBuilder fb = new FilterBuilder();
assertNotNull(fb);
CriteriaFilter cf = new CriteriaFilter();
assertNotNull(cf);
cf.getValues().add("2010-12-29T14:45:23");
cf.setType(CriteriaType.DATE);
cf.setClause(Clause.IS_BEFORE_THE_DATE);
CriteriaQueryClause clause = CriteriaQueryClause.fromValue(cf.getClause());
assertNotNull(clause);
assertEquals(CriteriaQueryClause.IS_BEFORE_THE_DATE, clause);
clause.buildFilter(fb, cf, mockedRetriever);
assertNotNull(fb);
Filter[] filters = fb.getFilters();
assertNotNull(filters);
assertEquals(filters.length, 1);
verify(mockedRetriever);
logger.info("OK");
}
this last method passes the test but not the other one. How is this possible!?!?!
Regards,
Nico
More links:
"bartling.blogspot.com/2009/11/using-argument-matchers-in-easymock-and.html"
"www.springone2gx.com/blog/scott_leberknight/2008/09/the_n_matchers_expected_m_recorded_problem_in_easymock"
"stackoverflow.com/questions/4605997/3-matchers-expected-4-recorded"
I had a very similar problem and wrote my findings in the link below.
http://www.flyingtomoon.com/2011/04/unclosed-record-state-problem-in.html (just updated)
I believe the problem in on another test that affects your current test. The problem is on another test class and it affects you test. In order to find the place of the real problem, I advice to disable the problematic tests one by one till you notify the failing test.
Actually this is what I did. I disabled the failing tests one by one till I found the problematic test. I found a test that throws an exception and catches by "#extected" annotation without stopping the recording.
We had this problem recently, and it only reared its head when we ran the entire test suite (1100+ test cases). Eventually, I found that I could put a breakpoint on the test that was blowing up, and then step back in the list of tests that Eclipse had already executed, looking for the previous test case that had set up a mock incorrectly.
Our problem turned out to be somebody using EasyMock.anyString() outside of an EasyMock.expect(...) statement. Sure enough, it was done two tests before the one that was failing.
So essentially, what was happening is that the misuse of a matcher outside of an expect statement was poisoning EasyMock's state, and the next time we tried to create a mock, EasyMock would blow up.
I believe the first error message
java.lang.IllegalStateException: 1
matchers expected, 2 recorded.
means your mockedRetriever methods called twice but test expects it was called once. So your Eclipse and Maven's configuration differs.
And I have no reason to reset mock after test. Just keep in mind JUnit creates new class instance for every single test method.
EDITED:
What about the reason why the last test method passed the answer is:
expect(mockedRetriever.retrieve((String)anyObject())).andReturn(new PofExtractor()).anyTimes();
But in your first test method it is:
expect(mockedRetriever.retrieve("PROP")).andReturn(null).once();
as equivalent of:
expect(mockedRetriever.retrieve("PROP")).andReturn(null);