I am having difficulty understanding the difference between these 3. Aren't all of these testing frameworks, and not libraries? If so, how is it that we are able to use these in combination? Shouldn't we be able to only use one testing framework, since the framework dictates the flow of control of code unlike in the case of libraries where we call the library code from our code. How can 2 independent frameworks work together on the same piece of code? I've observed that Mockito is used in conjunction with TestNG a lot. Isn't TestNG sufficient for most testing scenarios?
Both Testng and Junit are Testing framework used for Unit Testing. TestNG is similar to JUnit. Few more functionalities are added to it that makes TestNG more powerful than JUnit.
Annotations
Both JUnit and TestNG uses annotations and almost all the annotations looks similar.
TestNG uses #BeforeMethod ,#AfterMethod similar to #Before ,#After in JUnit4.
.........................
Mockito is a java based mocking framework, used in conjunction with other testing frameworks such as JUnit and TestNG.
It internally uses Java Reflection API and allows to create objects of a service. A mock object returns a dummy data and avoids external dependencies. It simplifies the development of tests by mocking external dependencies and apply the mocks into the code under test.
Summary
So as its name suggest Mockito is used for mocking the data or fake data which is act as real object to test, its acts like an stub or driver while still we need junit or testng with it. while Junit and TestNG is a unit testing framework of java
Explanation Tutorial
https://www.youtube.com/watch?v=eILy4p99ac8
Article on Mockito
https://www.journaldev.com/21816/mockito-tutorial
I work on a project which has existed for many years. The time it takes to build the project with all tests is almost sensational (not in a good way). This is mainly due to a lot of modules, as well as heaps of unit tests which uses a repository to set up test data rather then to mock the desired behaviour. Unit tests using a repository use a lot of time for test setup, and they run quite slowly. This adds up to a lot of time as the system is quite large.
We write all new unit tests by using Mockito to mock the repository (except when we are actually testing the repository obviously). We also try to rewrite all existing unit tests to using mocks of the repository instead of an actual repository whenever we have the time and opportunity. Completely eliminating the use of repo's in our tests has a huge effect on how much time it takes to run the tests.
A lot of the legacy code sets up its test data by using builders and test-utilities which in turn uses the repository. As the domain is quite complex, this often involves setting up a fair amount of objects and how they are related to each other. Re-writing a class of tests (say ~15 tests) to using only mocked object can therefore be quite time-consuming. And as everywhere else, time is not an infinite resource.
If we are adding some new functionality to a class, it would be far easier to just write one new repository test (in addition to the existing 15) than to find out exactly how the test data needs to be set up by using different mock objects.
I have tried to find some information on how and to what extent the test setup affects the actual time it takes to run the tests, but I have failed to find any useful information. My only "facts" are the observations I make when running a test class. The test setup for a repo test may easily take ~10 seconds, while the test setup for a mocked test class starts in less than a second.
NOTE: I am aware that I can use JUnit Stopwatch to benchmark a single or a set of tests, but my question is more concerned with best practices than exactly how long it takes me to run my tests.
I have two questions:
Say I encounter a test class which already has 15 unit tests where none of them mocks any behaviour. I write a test and do a small fix in the class. I do not have the time to re-write the whole test class to mock objects. Should I just add a new test without mocking any behaviour and follow the existing (and bad) pattern? Does it really matter whether I have 15 non-mocked tests and 1 mocked test or if I have 16 non-mocked tests?
In my test class with 15 unit tests, some of the tests are easier to refactor than others. Is it worth it to re-write only five of the tests to using mocked objects? Is it against best practice or in any other way not good to have a test class where some of the tests uses mocks and some don't?
Your question is really subjective but I'll try to suggest few options you can explore. It all depends upon how much you're willing to spend.
Should I just add a new test without mocking any behavior and follow the existing (and bad) pattern? Does it really matter whether I have 15 non-mocked tests and 1 mocked test or if I have 16 non-mocked tests?
Its not about just one new test. If you're still writing in bad/slow pattern, you're just increasing technical debt. You've to lay out the best practices for writing new unit tests yourself.
In my test class with 15 unit tests, some of the tests are easier to
refactor than others. Is it worth it to re-write only five of the
tests to using mocked objects?
Absolutely. Why not? You saying for yourself the improvements you're getting by following newer style of code.
Is it against best practice or in any other way not good to have a
test class where some of the tests uses mocks and some don't?
Well one best practice is to have consistent code everywhere. Mix of old styled repositories and newer one with mocks does not go too well as far as best practices are concerned. But I'd be more concerned if the code you write is not well covered with unit tests, whatever style if may be.
At the end of the day, you're the one to decide and look at all the trade offs like how much build time improvements can you achieve by newer mocked repositories, what is the frequency of your builds, and can this be achieved using hardware improvements and other factors.
As written by #ShanuGupta there is not general answer to your question.
But here is my thought:
After correctness the readability is the second most desirable value of code, especially test code (since it is executable specification)
On the other hand there is no rule that a unit cannot have more than one Test class. Therefore I'd separate "old" test methods not using mocks from the "mocking" tests by placing them in separate test classes.
I have a 4 rest api for crud operations. So when I am writing a junit for those rest api's,
1.Should I write a single test case which do all the crud operation or it should be different test cases for each rest api?
2.I'f I write separate test case, then is it ok to use the record created in create test case in the update or get test cases. In that case there is a dependency between each test cases.
3.How can I enforce to Junit to run test cases on implement order given that my test case method are not in alphabetical order.
DISCLAIMER: there is no "one right answer".
SUGGESTIONS:
If at all possible, you should code your JUnit tests so that each runs independently. There should not be any "ordering" dependency between any specific test.
All things being equal, I would recommend a different test for each operation.
I would also recommend liberal use of fixtures, or a Mock object library like Mockito.
Failing all else, Junit 4.11 and higher offers #FixMethodOrder annotation
I am doing some Junit testing and I need to know how to run a Test class only if a specific test from another class was passed.
There is 'Categories' feature in JUnit. (See: https://github.com/junit-team/junit/wiki/Categories)
This question has already been replied in this post
In the #Before function you need to retreive a runtime value from your system and check that it matches your requirement. Everything will stop if it doesn't
IMHO, This is bad practice, even in a well-designed Integration test-suite. I would encourage you to rethink your overall test class design.
If these tests are truly meant to be unit tests, they should be atomic and independent of each other. See this post for a good read.
Having said that, I have often used JUnit 4.x to build & run rather large suites of Integration tests (backend functional tests that test a RESTful services responses). If this is your use case, I recommend restructuring your tests such that you never have one test in TestClassA depend on a test in TestClassB. That is a bad idea, as it will making your tests more fragile. And difficult for other devs to understand the intention of your tests, taken together as a whole.
When I have found that I have dependencies across multiple test classes, I have factored out a "test-superclass" to both test classes and do my "setup work" in that superclass. Or you can factor out a utility class that contains static methods for creating somewhat complex test conditions to start with.
But, even using JUnit as a vehicle to run these kind of "integration" tests should be done with caution and careful intent.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 3 years ago.
Improve this question
At work we are currently still using JUnit 3 to run our tests. We have been considering switching over to JUnit 4 for new tests being written but I have been keeping an eye on TestNG for a while now. What experiences have you all had with either JUnit 4 or TestNG, and which seems to work better for very large numbers of tests? Having flexibility in writing tests is also important to us since our functional tests cover a wide aspect and need to be written in a variety of ways to get results.
Old tests will not be re-written as they do their job just fine. What I would like to see in new tests though is flexibility in the way the test can be written, natural assertions, grouping, and easily distributed test executions.
I've used both, but I have to agree with Justin Standard that you shouldn't really consider rewriting your existing tests to any new format. Regardless of the decision, it is pretty trivial to run both. TestNG strives to be much more configurable than JUnit, but in the end they both work equally well.
TestNG has a neat feature where you can mark tests as a particular group, and then easily run all tests of a specific group, or exclude tests of a particular group. Thus you can mark tests that run slowly as in the "slow" group and then ignore them when you want quick results. A suggestion from their documentation is to mark some subset as "checkin" tests which should be run whenever you check new files in. I never saw such a feature in JUnit, but then again, if you don't have it, you don't REALLY miss it.
For all its claims of high configuration, I did run into a corner case the a couple weeks ago where I couldn't do what I wanted to do... I wish I could remember what it is, but I wanted to bring it up so you know that it's not perfect.
The biggest advantage TestNG has is annotations... which JUnit added in version 4 anyways.
First I would say, don't rewrite all your tests just to suit the latest fad. Junit3 works perfectly well, and the introduction of annotations in 4 doesn't buy you very much (in my opinion). It is much more important that you guys write tests, and it sounds like you do.
Use whatever seems most natural and helps you get your work done.
I can't comment on TestNG b/c I haven't used it. But I would recommend unitils, a great wrapper for JUnit/TestNG/DBUnit/EasyMock, regardless of which route you take. (It supports all the flavors mentioned above)
TestNG's biggest draw cards for me include its support test groups, and more importantly - test group dependencies (marking a test as being dependent of a group causes the tests to simply skip running when the dependent group fails).
TestNG's other big draw cards for me include test parameters, data providers, annotation transformers, and more than anything - the vibrant and responsive user community.
Whilst on the surface one might not think all of TestNGs features above might not be needed, once you start to understand the flexibility bring to your tests, you'll wonder how you coped with JUnit.
(disclaimer - I've not used JUnit 4.x at all, so am unable to really comment on advances or new features there).
About a year ago, we had the same problem. I spent sometime considering which move was better, and eventually we realized that TestNG has no 'killer features'. It's nice, and has some features JUnit 4 doesn't have, but we don't need them.
We didn't want people to feel uncomfortable writing tests while getting to know TestNG because we wanted them to keep writing a lot of tests.
Also, JUnit is pretty much the de-facto standard in the Java world. There's no decent tool that doesn't support it from the box, you can find a lot of help on the web and they added a lot of new features in the past year which shows it's alive.
We decided to stick with JUnit and never looked back.
Cheers to all the above. Some other things I've personally found I like more in TestNG are:
The #BeforeClass for TestNG takes place after class creation, so you aren't constrained by only being able to call static methods of your class in it.
Parallel and parameterized tests, maybe I just don't have enough of a life... but I just get a kick writing one set of Selenium tests, accepting a driver name as a parameter. Then defining 3 parallel test groups, 1 each for the IE, FF and Chrome drivers, and watching the race! I originally did 4, but way too many of the pages I've worked on break the HtmlUnit driver for one reason or another.
Yeah, probably need to find that life. ;)
I wanted to share the one I encountered today. I found built-in Parameterized runner is quite crude in Junit4 as compare to TestNG (I know each framework has its strengths but still). The Junit4 annotation #parameters is restricted to one set of parameters. I encountered this problem while testing the valid and invalid behavior for functionality in same test class. So the first public, static annotated method that it finds will be used, but it may find them in any order. This causes us to write different classes unnecessarily. However TestNG provides clean way to provide different kind of data providers for each and every method. So we can test the same unit of code with valid and invalid way in same test class putting the valid/invalid data separately. I will go with TestNG.
Also one more advantage of TestNG is supporting of parallel testing. In our era of multicores it's important, i think.
I also used both frameworks. But i using hamcrest for assertations. Hamcrest allows you easily write your own assert method. So instead of
assertEquals(operation.getStatus(), Operation.Status.Active);
You can write
assertThat(operation, isActive());
That gives you opportunity to use higher level of abstraction in your tests. And this makes your tests more robust.
JUnit 4 Vs TestNG – Comparison by mkyong.com ( updated on 2013).
Conclusion: I suggest to use TestNG as core unit test framework for Java project, because TestNG is more advance in parameterize testing, dependency testing and suite testing (Grouping concept).
TestNG is meant for functional, high-level testing and complex integration test. Its flexibility is especially useful with large test suites.
In addition, TestNG also cover the entire core JUnit4 functionality. It’s just no reason for me to use JUnit anymore.
In simple terms, TestNG = JUnit + lot more. So, Why debate ? go and
grab TestNG :-)
You can find more detailed comparison here.
Why we use TestNG instead of JUnit?
The declaration of #BeforeClass and #AfterClass method has to be static in JUnit whereas, there is more flexibility in TestNG in the method declaration, it does not have these constraints.
In TestNG, we can parametrize tests using 2 ways. #Parameter or #DataProvider annotation.
i) #Parameter for simple cases, where key value mapping is required.(data is provided through xml file)
ii) #DataProvider for complex cases. Using 2 dimensional array, It can provide data.
In TestNG, since #DataProvider method need not be static, we can use multiple data provider methods in the same test class.
Dependency Testing: In TestNG, if the initial test fails, then all subsequent dependent tests will be skipped, not marked as failed. But JUnit marked it failed.
Grouping: Single tests can belong to multiple groups and then run in different contexts (like slow or fast tests). A similar feature exists in JUnit Categories but lacks the #BeforeGroups / #AfterGroups TestNG annotations that allow initializing the test / tearing it down.
Parallelism: If you’d like to run the same test in parallel on multiple threads, TestNG has you covered with a simple to use annotation while JUnit doesn’t offer a simple way to do so out of the box.
TestNG #DataProvider can also support XML for feeding in data, CSVs, or even plain text files.
TestNG allows you to declare dependencies between tests, and skip them if the dependency test didn’t pass.
#Test(dependsOnMethods = { "dependOnSomething" })
This functionality doesn’t exist in JUnit
Reporting:
TestNG reports are generated by default to a test-output folder that includes HTML reports with all of the test data, passed/failed/skipped, how long did they run, which input was used and the complete test logs. In addition, it also exports everything to an XML file which can be used to construct your own report template.
On the JUnit front, all of this data is also available via XML, but there’s no out of the box report and you need to rely on plugins.
Resource Link:
A Quick JUnit vs TestNG Comparison
JUnit vs. TestNG: Which Testing Framework Should You Choose?
A good difference is given in this tutorial side by side: TestNG Vs JUnit: What's the Difference?
A couple of additions to Mike Stone's reply:
1) The most frequent thing I use TestNG's groups for is when I want to run a single test method in a test suite. I simply add this test to the group "phil" and then run this group. When I was using JUnit 3, I would comment out the entries for all methods but the one I wanted to run in the "suite" method, but then would commonly forget to uncomment them before checkin. With the groups, I no longer have this problem.
2) Depending on the complexity of the tests, migrating tests from JUnit3 to TestNG can be done somewhat automatically with sed and creating a base class to replace TestCase that static imports all of the TestNG assert methods.
I have info on my migration from JUnit to TestNG here and here.
My opinion about what makes TestNG truly far more powerful:
1. JUnit still requires the before/after class methods to be static, which limits
what you can do prior to the running of tests, TestNG never has this issue.
2. TestNG #Configuration methods can all take an optional argument to their
annotated methods in the form of a ITestResult, XmlTest, Method, or
ITestContext. This allows you to pass things around that JUnit wouldn't
provide you. JUnit only does this in listeners and it is limited in use.
3. TestNG comes with some pre-made report generation classes that you can copy
and edit and make into your own beautiful test output with very little
effort. Just copy the report class into your project and add a listener
to run it. Also, ReportNG is available.
4. TestNG has a handful of nice listeners that you can hook onto so you can do
additional AOP style magic at certain phases during testing.
Your question seems two folded to me. On one had you would like to compare two test frameworks, on the other hand you would like to implement tests easily, have natural assertions, etc...
Ok, firstly JUnit has been playing catchup with TestNG in terms of functionality, they have bridged the gap some what with v4, but not well enough in my opinion. Things like annotations and dataproviders are still much better in TestNG. Also they are more flexible in terms of test execution, since TestNG has test dependency, grouping and ordering.
JUnit still requires certain before/after methods to be static, which limits what you can do prior to the running of tests, TestNG never has this issue.
TBH, mostly the differences between the two frameworks don't mean much, unless your focusing on integration/automation testing. JUnit from my experience is built from the ground up for unit testing and is now being pushed towards higher levels of testing, which IMO makes it the wrong tool for the job. TestNG does well at unit testing and due to its robust dataproviding and great test execution abilities, works even better at integration/automation test level.
Now for what I believe is a separate issue, how to write well structured, readable and maintainable tests. Most of this I am sure you know, but things like Factory Pattern, Command Pattern and PageObjects (if your testing websites) are vital, it is very important to have a layer of abstraction between what your testing (SUT) and what the actual test is (assertions of business logic). In order to have much nicer assertions, you can use Hamcrest. Make use of javas inheritance/interfaces to reduce repetition and enforce commonality.
Almost forgot, also use the Test Data Builder Pattern, this coupled with TestNG's dataprovider annotation is very useful.