I am trying to get java source code coverage after testing with a GUI based functional test using selenium.
what all source (actual application) code( NOT selenium test code) was executed or covered after running the selenium functional tests in java.
Is this can be achieved using Emma or jacoco or any other?
Anyone who has done this could please give me an idea to start this ?
Our Java Test Coverage tool can do this.
The process is straightfoward:
Apply the Test Coverage tool instrumenter to your source code and compile.
Run the application, exercising it any way you desire (including running selenium, or hand interaction, or unit tests)
Cause the application to exit from main.
The instrumenter's code inserted in main will write coverage data
Display/explore the test coverage with the tools GUI interface
You may have a long-running (server) application that doesn't naturally exit; in that case you can't get the test coverage dumped by exiting main because you didn't exit. In this case, you add a call to the tool's DumpTestCoverage() method, at some place in your application that you can cause to be executed by some interaction with the application. Then when you want a snapshot, simply exercise that feature.
I always use Jacoco plugin with Gradle. It cool in a way that it generates report and makes everything simple. You can configure it in build.gradle file.
Related
I am a beginner at automation testing.
I am trying to execute tests that were written in java using Selenium WebDriver, Cucumber with Gherkin annotations in JMeter.
When I run my tests without JMeter, I just use the opportunity to run them as Junit test using the runner class (in Eclipse).
To run my tests in JMeter, I created a jar file, uploaded it to Junit sub-folder, all dependencies-jars were uploaded to lib sub-folder.
I tried to create a JUnit request in JMeter, but it didn't help as when we use Cucumber, we don't mark methods with #Test. I have just an empty constructor for my Runner class (as you can see from the picture above).
I tried to add my jar to Test Plan and create an instance of my class in BeanShell, but it is not executed as JUnit tests.
My question is how to run java test scripts that were written based on Cucumber with Gherkin (based on Selenium WebDriver)?
Maybe I have to use other Samplers...
Any examples are greatly appreciated.
I believe you need to run another class, in particular io.cucumber.core.cli.Main, something like:
io.cucumber.core.cli.Main.main(new String[]{
"--glue",
"the package which contains the glue classes",
"/your/feature/file"});
Check out Running Cucumber -> From the command line for all the available arguments explained. By the way, you can use OS Process Sampler for this, it will be way easier.
Depending on what you're trying to achieve it might be faster and easier to convert your Selenium tests into "native" JMeter ones as if you're going to use your Selenium tests for creating the load it will require immense hardware resources and you won't get performance metrics on the HTTP protocol level.
I have just inherited a old java codebase (around 10 - 15 years old). It does not have any automated testing coverage, or at least the contemporary world knows about it. I am planning to write some Fitnesse scripts around it, to begin with.
I know about Concordion etc. And I have my reason to pick Fitnesse. I will keep away from that since that is not the topic of this question.
My question is, I don't know of a quick way to measure the code coverage done by the Fitnesse tests as I write them. I know that jacoco (or similar libraries) should be able to report them but I just can't figure out exactly how.
So, if anyone of you have worked on Fitnesse test scripts and have managed to have Jenkins report on the coverage achieved by the scripts, please help.
Thanks.
I have not done this myself, because I tend to use FitNesse to test deployed applications (and I cannot measure code coverage in the 'real installation'). But I believe this should be fairly straightforward if you run your FitNesse tests as part of a Jenkins (or any other build server's) jUnit run, which measures code coverage.
To have your FitNesse tests be executed as part of a jUnit run: create a Java class annotated with #RunWith(FitNesseRunner.class) and give it a #Suite("MyPageOrSuite.That.IWantToRun") annotation to indicate which tests to run. This will execute the specified page(s) in the same Java process that the jUnit process is using, so if that is instrumented in some way to determine code coverage the coverage of your FitNesse tests will be included in the report.
Sample jUnit test class, running FitNesse.SuiteAcceptanceTests.SuiteSlimTests.TestScriptTable:
#RunWith(FitNesseRunner.class)
#FitNesseRunner.Suite("FitNesse.SuiteAcceptanceTests.SuiteSlimTests.TestScriptTable")
#FitNesseRunner.FitnesseDir(".")
#FitNesseRunner.OutputDir("../target/fitnesse-results")
public class FitNesseRunnerTest {
}
Is such a thing possible?
To explain: I'm on a project where the Acceptance Tests are being run in a bespoke way from an actual webpage rather than as part of one of Maven's test phases. It's a long story why so please consider criticisms of this to be off-topic!
Would like to see the coverage after clicking a button that runs the tests. On-the-fly coverage while the tests are running isn't a necessity but would at least like to be able to see the line-by-line Java code coverage in Eclipse after the tests have finished - ideally with the page still up and JVM still running.
Would be grateful if any replies could include any needed tools, pom fragments and setup information.
EDIT: Forgot to mention the customer is only interested in using open source tools.
Yes, this is possible and easy. Our (Semantic Designs) Java Test Coverage tools do this just fine.
You instrument the application and compile it. As it runs (anything: units tests, ad hoc interactive exercises, etc.) it collects test coverage data. At any moment, you can cause the test coverage data to be dumped without disturbing the application. That dump coverage data can be immediately imported and viewed.
I'm writing some system tests in Groovy and piggybacking on its unit testing infrastructure. It works pretty well except that I don't like the default JUnit test runner, which displays . for each test in the suite waiting until the end to report details about the errors and failures. The system tests can take a long time to run so it's useful to be able to interrupt them in the middle once you know that a failure or error exists but you can only do that if you know which testcase failed and what the failure was while the suite is running.
Are there any alternate JUnit test runners which I could use that provide this functionality out of the box?
I'm not exactly sure what you mean by "textui", but IntelliJ IDEA 11 Community Edition has an integrated JUnit test runner that works with Groovy exactly as you describe (but it has a graphical user interface only, as far as I know).
Is there a tool for Java which, given a set of JUnit tests, and a class to test, will tell you which lines of the class are tested by the tests? ie. required to be present for the tests to run successfully. I don't mean "code coverage", which only tells you whether a line is executed, but something stronger than that: Is the line required for the test to pass?
I often comment out a line of code and run a test to see if the test really is testing that line of code. I reckon this could be done automatically by a semi-smart tool (eg. something like an IDE that can work out what can be removed from a method whilst keeping it compilable).
There's an open source mutation-testing tool called Jester that changes the lines of your source code, then runs your tests, and reports whether your tests passed anyway. Sounds closer to what you're looking for than code coverage tools.
Jester is a test tester for testing your java JUnit tests (Pester is for Python PyUnit tests). It modifies your source code, runs the tests and reports if the tests pass despite the changes to the code. This can indicate missing tests or redundant code.
WRT the discussion about whether these tools are needed in a pure TDD project, there is a link on the Jester project webpage to a posting about the benefits of using Jester on code written during a TDD session (Uncle Bob's infamous bowling TDD example).
What you are looking for might be referred to as mutation testing. While mutation testing won't tell you which lines of code are required to pass, per se. What mutation testing does is modify your source code looking for changes it can make to your code but your test still passes. E.g. changing
if (a < b)
to
if (a >= b)
and seeing if the test still passes. This will highlight weaknesses in your test.
Another java library for mutation testing is jumble.
I use emma for most of my projects. i included it in my ant build file and it generates html files for the reports
two other coverage projects i read about but haven't tried yet are clover or cobertura
I love cobertura, because the generated reports are IMHO the most beautiful. And it has its own ant target!
In comparison to emma, it has also branch coverage, not only line coverage, which is misleading very often.