Related
I am trying to create an index of unused Java methods in the form of a json file.
There are a couple different ways in which the methods can be referenced. I have already checked for all the other ways and have a relatively small list of possibly unused java methods.
The final way in which a method can be used is in other java files. They would be called with a basic class.method(args,args2,etc...) syntax somewhere in the java source code.
My question is, is there an easy way to just check my list of possible unused methods to see if any of them are not used in the java code. It would be ideal if this could be done at runtime, but it would also work if I could create a file that I could then read in at runtime.
I have tried using pre-built software like UCDetector, but the source code is huge, and running UCDetector takes hours and often doesn't even finish. It also checks all methods to see if they are used which is a waste of time since I have narrowed it down to a small number of possible methods to check.
You should use your IDE (eclipse, intelliJ), or some static code analysis tool such as findbugs, pmd, checkstyle.
It seems like you are trying to reinvent the wheel.
One option might be to use "coverage analysis" tools to see what is not used (https://en.m.wikipedia.org/wiki/Java_Code_Coverage_Tools). If you have good branch coverage with your unit tests, simply running the tests with coverage will yield the result you're looking for. If you don't have good tests coverage, you might run the application itself with code instrumented for coverage calculation, but as with unit tests - the quality of the result will depend on the amount of code executed with your unit tetst or manual test.
Some examples of the coverage tools you might use are : JaCoCo (http://www.eclemma.org/jacoco/) and Cobertura (http://cobertura.github.io/cobertura).
Alternatively you might instrument your code yourself in order to log methods usage, as it might be more lightweight than calculating full line coverage. This is however indeed reinventing the wheel.
This SO question has similar solutions: How to find unused/dead code in java projects
I am working on a number of projects and we are using Java, Springs, Maven and Jenkins for CI but I am running into a issues that some of the programmers are not adding real junit test cases to the projects. I want maven and jenkins to run the test before deploying to the server. Some of the programers made a blank test so it starts and stops and will pass the tests.
Can someone please tell me how can I automat this check so maven and jenkins can see if the test put out some output.
I have not found any good solution to this issue, other than reviewing the code.
Code coverage fails to detect the worst unit tests I ever saw
Looking at the number of tests, fails there too. Looking at the test names, you bet that fails.
If you have developers like the "Kevin" who writes tests like those, you'll only catch those tests by code review.
The summary of how "Kevin" defeats the checks:
Write a test called smokes. In this test you invoke every method of the class under test with differing combinations of parameters, each call wrapped in try { ... } catch (Throwable t) {/* ignore */}. This gives you great coverage, and the test never fails
Write a load of empty tests with names that sound like you have thought up fancy test scenarios, eg widgetsTurnRedWhenFlangeIsOff, widgetsCounterrotateIfFangeGreaterThan50. These are empty tests, so will never fail, and a manager inspection the CI system will see lots of detailed test cases.
Code review is the only way to catch "Kevin".
Hope your developers are not that bad
Update
I had a shower moment this morning. There is a type of automated analysis that can catch "Kevin", unfortunately it can still be cheated around, so while it is not a solution to people writing bad tests, it does make it harder to write bad tests.
Mutation Testing
This is an old project, and won't work on recent code, and I am not suggesting you use this. But I am suggesting that it hints at a type of automated analysis that would stop "Kevin"
If I were implementing this, what I would do is write a "JestingClassLoader" that uses, e.g. ASM, to rewrite the bytecode with one little "jest" at a time. Then run the test suite against your classes when loaded with this classloader. If the tests don't fail, you are in "Kevin" land. The issue is that you need to run all the tests against every branch point in your code. You could use automatic coverage analysis and test time profiling to speed things up, though. In other words, you know what code paths each test executes, so when you make a "jest" against one specific path, you only run the tests that hit that path, and you start with the fastest test. If none of those tests fail, you have found a weakness in your test coverage.
So if somebody were to "modernize" Jester, you'd have a way to find "Kevin" out.
But that will not stop people writing bad tests. Because you can pass that check by writing tests that verify the code behaves as it currently behaves, bugs and all. Heck there are even companies selling software that will "write the tests for you". I will not give them the Google Page Rank by linking to them from here, but my point is if they get their hands on such software you will have loads of tests that straight-jacket your codebase and don't find any bugs (because as soon as you change anything the "generated" tests will fail, so now making a change requires arguing over the change itself as well as the changes to all the unit tests that the change broke, increasing the business cost to make a change, even if that change is fixing a real bug)
I would recommend using Sonar which has a very useful build breaker plugin.
Within the Sonar quality profile you can set alerts on any combination of metrics, so, for example you could mandate that your java projects should have
"Unit tests" > 1
"Coverage" > 20
Forcing developers to have at least 1 unit test that covers a minimum of 20% of their codebase. (Pretty low quality bar, but I suppose that's your point!)
Setting up an additional server may appear like extra work, but the solution scales when you have multiple Maven projects. The Jenkins plugin for Sonar is all you'll need to configure.
Jacoco is the default code coverage tool, and Sonar will also automatically run other tools like Checkstyle, PMD and Findbugs.
Finally Stephen is completely correct about code review. Sonar has some basic, but useful, code review features.
You need to add a code coverage plugin such as JaCoCo, EMMA, Cobertura, or the likes. Then you need to define in the plugin's configuration the percent of code coverage (basically "code covered by the tests") that you would like to have, in order for the build to pass. If it's below that number, you can have the build failing. And, if the build is failing, Jenkins (or whatever your CI is) won't deploy.
As others have pointed out, if your programmers are already out to cheat coding practices, using better coverage tools won't solve your problem. They can be out cheated as well.
You need to sit down with your team and have an honest talk with them about professionalism and what software engineering is supposed to be.
In my experience, code reviews are great but they need to happen before the code is committed. But for that to work in a project where people are 'cheating', you'll need to at least have a reviewer you can trust.
http://pitest.org/ is a good solution for so called "mutation testing":
Faults (or mutations) are automatically seeded into your code, then your tests are run. If your tests fail then the mutation is killed, if your tests pass then the mutation lived.
The quality of your tests can be gauged from the percentage of mutations killed.
The good thing is that you can easyli use it in conjunction with maven, jenkins and ... SonarQube!
We're adding unit tests to previously untested code, as the need arises to modify that code. It's difficult to get useful coverage metrics since the majority of code in any package is known to be untested.
Are there any tools available to measure differential code coverage, that is, the percent of code modified in a given changeset which was covered by a unit test?
Use pycobertura. It's a command-line tool to prevent code coverage regression by diffing two coverage reports. It tells you whether your new code is better or worse than the previous version, coverage-wise.
$ pycobertura diff ./master/coverage.xml ./myfeature/coverage.xml
It's language agnostic since it just relies on the Cobertura report (XML file) generated by your testing/coverage tool.
Pycobertura can also generate HTML reports which fit nicely in CI/CD tools such as Jenkins.
https://github.com/aconrad/pycobertura
Continuous integration tools like Jenkins let you keep a history of test coverage and show you a graph that includes a coverage trend compared to previous builds. Example: Cobertura Jenkins Plugin
There is a gradle plugin that computes diff code coverage
https://github.com/form-com/diff-coverage-gradle
All you need is to provide diff file to the plugin or you can use git diff tool that is shown in the example
Take a look into Sonar, really good tool to analyze entire application quality and coverage.
I recently did exactly that, using JaCoCo and the ConQAT framework for code analysis. The approach is as follows:
Load the source code from the baseline version and the head version (and potentially intermediary ones, if you also have done tests in-between)
Compare the program history method by method to determine where changes happened
Annotate coverage information to each tested revision
Search for methods that have not been covered since they last changed
There is also a blog post containing a more detailed description including visualizations and more advanced topics like refactoring detection to only identify changes worth testing.
You can use diff-test-coverage for this. diff-test-coverage is a Node.js commandline tool which filters test coverage based on a (source control) diff.
It works by filtering the coverage report information, so it can report the coverage information for new code.
It supports both Git and Mercurial and all common coverage report formats.
Is there a tool for Java which, given a set of JUnit tests, and a class to test, will tell you which lines of the class are tested by the tests? ie. required to be present for the tests to run successfully. I don't mean "code coverage", which only tells you whether a line is executed, but something stronger than that: Is the line required for the test to pass?
I often comment out a line of code and run a test to see if the test really is testing that line of code. I reckon this could be done automatically by a semi-smart tool (eg. something like an IDE that can work out what can be removed from a method whilst keeping it compilable).
There's an open source mutation-testing tool called Jester that changes the lines of your source code, then runs your tests, and reports whether your tests passed anyway. Sounds closer to what you're looking for than code coverage tools.
Jester is a test tester for testing your java JUnit tests (Pester is for Python PyUnit tests). It modifies your source code, runs the tests and reports if the tests pass despite the changes to the code. This can indicate missing tests or redundant code.
WRT the discussion about whether these tools are needed in a pure TDD project, there is a link on the Jester project webpage to a posting about the benefits of using Jester on code written during a TDD session (Uncle Bob's infamous bowling TDD example).
What you are looking for might be referred to as mutation testing. While mutation testing won't tell you which lines of code are required to pass, per se. What mutation testing does is modify your source code looking for changes it can make to your code but your test still passes. E.g. changing
if (a < b)
to
if (a >= b)
and seeing if the test still passes. This will highlight weaknesses in your test.
Another java library for mutation testing is jumble.
I use emma for most of my projects. i included it in my ant build file and it generates html files for the reports
two other coverage projects i read about but haven't tried yet are clover or cobertura
I love cobertura, because the generated reports are IMHO the most beautiful. And it has its own ant target!
In comparison to emma, it has also branch coverage, not only line coverage, which is misleading very often.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
What code analysis tools do you use on your Java projects?
I am interested in all kinds
static code analysis tools (FindBugs, PMD, and any others)
code coverage tools (Cobertura, Emma, and any others)
any other instrumentation-based tools
anything else, if I'm missing something
If applicable, also state what build tools you use and how well these tools integrate with both your IDEs and build tools.
If a tool is only available a specific way (as an IDE plugin, or, say, a build tool plugin) that information is also worth noting.
For static analysis tools I often use CPD, PMD, FindBugs, and Checkstyle.
CPD is the PMD "Copy/Paste Detector" tool. I was using PMD for a little while before I noticed the "Finding Duplicated Code" link on the PMD web page.
I'd like to point out that these tools can sometimes be extended beyond their "out-of-the-box" set of rules. And not just because they're open source so that you can rewrite them. Some of these tools come with applications or "hooks" that allow them to be extended. For example, PMD comes with the "designer" tool that allows you to create new rules. Also, Checkstyle has the DescendantToken check that has properties that allow for substantial customization.
I integrate these tools with an Ant-based build. You can follow the link to see my commented configuration.
In addition to the simple integration into the build, I find it helpful to configure the tools to be somewhat "integrated" in a couple of other ways. Namely, report generation and warning suppression uniformity. I'd like to add these aspects to this discussion (which should probably have the "static-analysis" tag also): how are folks configuring these tools to create a "unified" solution? (I've asked this question separately here)
First, for warning reports, I transform the output so that each warning has the simple format:
/absolute-path/filename:line-number:column-number: warning(tool-name): message
This is often called the "Emacs format," but even if you aren't using Emacs, it's a reasonable format for homogenizing reports. For example:
/project/src/com/example/Foo.java:425:9: warning(Checkstyle):Missing a Javadoc comment.
My warning format transformations are done by my Ant script with Ant filterchains.
The second "integration" that I do is for warning suppression. By default, each tool supports comments or an annotation (or both) that you can place in your code to silence a warning that you want to ignore. But these various warning suppression requests do not have a consistent look which seems somewhat silly. When you're suppressing a warning, you're suppressing a warning, so why not always write "SuppressWarning?"
For example, PMD's default configuration suppresses warning generation on lines of code with the string "NOPMD" in a comment. Also, PMD supports Java's #SuppressWarnings annotation. I configure PMD to use comments containing "SuppressWarning(PMD." instead of NOPMD so that PMD suppressions look alike. I fill in the particular rule that is violated when using the comment style suppression:
// SuppressWarnings(PMD.PreserveStackTrace) justification: (false positive) exceptions are chained
Only the "SuppressWarnings(PMD." part is significant for a comment, but it is consistent with PMD's support for the #SuppressWarning annotation which does recognize individual rule violations by name:
#SuppressWarnings("PMD.CompareObjectsWithEquals") // justification: identity comparision intended
Similarly, Checkstyle suppresses warning generation between pairs of comments (no annotation support is provided). By default, comments to turn Checkstyle off and on contain the strings CHECKSTYLE:OFF and CHECKSTYLE:ON, respectively. Changing this configuration (with Checkstyle's "SuppressionCommentFilter") to use the strings "BEGIN SuppressWarnings(CheckStyle." and "END SuppressWarnings(CheckStyle." makes the controls look more like PMD:
// BEGIN SuppressWarnings(Checkstyle.HiddenField) justification: "Effective Java," 2nd ed., Bloch, Item 2
// END SuppressWarnings(Checkstyle.HiddenField)
With Checkstyle comments, the particular check violation (HiddenField) is significant because each check has its own "BEGIN/END" comment pair.
FindBugs also supports warning generation suppression with a #SuppressWarnings annotation, so no further configuration is required to achieve some level of uniformity with other tools. Unfortunately, Findbugs has to support a custom #SuppressWarnings annotation because the built-in Java #SuppressWarnings annotation has a SOURCE retention policy which is not strong enough to retain the annotation in the class file where FindBugs needs it. I fully qualify FindBugs warnings suppressions to avoid clashing with Java's #SuppressWarnings annotation:
#edu.umd.cs.findbugs.annotations.SuppressWarnings("UWF_FIELD_NOT_INITIALIZED_IN_CONSTRUCTOR")
These techniques makes things look reasonably consistent across tools. Note that having each warning suppression contain the string "SuppressWarnings" makes it easy to run a simple search to find all instances for all tools over an entire code base.
I use a combination of Cobertura, Checkstyle, (Ecl)Emma and Findbugs.
EclEmma is an awesome Eclipse plugin that shows the code coverage by coloring the java source in the editor (screenshot) - the coverage is generated by running a JUnit test. This is really useful when you are trying to figure out which lines are covered in a particular class, or if you want to see just which lines are covered by a single test. This is much more user friendly and useful than generating a report and then looking through the report to see which classes have low coverage.
The Checkstyle and Findbugs Eclipse plugins are also useful, they generate warnings in the editor as you type.
Maven2 has report plugins that work with the above tools to generate reports at build time. We use this to get overall project reports, which are more useful when you want aggregate numbers. These are generated by our CI builds, which run using Continuum.
All of the following we use and integrate easiy in both our Maven 2.x builds and Eclipse/RAD 7:
Testing - JUnit/TestNG
Code analysis - FindBugs, PMD
Code coverage - Clover
In addition, in our Maven builds we have:
JDepend
Tag checker (TODO, FIXME, etc)
Furthermore, if you're using Maven 2.x, CodeHaus has a collection of handy Maven plugins in their Mojo project.
Note: Clover has out-of-the-box integration with the Bamboo CI server (since they're both Atlassian products). There are also Bamboo plugins for FindBugs, PMD, and CheckStyle but, as noted, the free Hudson CI server has those too.
I use the static analysis built into IntelliJ IDEA. Perfect integration.
I use the code coverage built into Intellij IDEA (based on EMMA). Again, perfect integration.
This integrated solution is reliable, powerful, and easy-to-use compared to piecing together tools from various vendors.
Checkstyle is another one I've used at a previous company... it's mainly for style checking, but it can do some static analysis too. Also, Clover for code coverage, though be aware it is not a free tool.
We are using FindBugs and Checkstyle as well as Clover for Code Coverage.
I think it's important to have some kind of static analysis, supporting your development. Unfortunately it's still not widely spread that these tools are important.
We use FindBugs and JDepend integrated with Ant. We use JUnit but we're not using any coverage tool.
I'm not using it integrated to Rational Application Developer (the IDE I'm using to develop J2EE applications) because I like how neat it looks when you run javac in the Windows console. :P
I've had good luck with Cobertura. It's a code coverage tool which can be executed via your ant script as part of your normal build and can be integrated into Hudson.
Our team use PMD and Cobertura, actually our projects are maven projects and there is very simple to include plug ins for code analysis. The real question would be for specific project which analysis you need to use, my opinion is that it's you couldn't use the same plugins for each project.
in our project we use Sonar in front of checkstyle, pmd.... together with the CI (Bamboo, Hudson) we get also a nice history of our source quality and what directing we go. I do like Sonar, because you one central tool in the CI Stack that does it for you, and you can easy customize the rules for each project.
Structure 101 is good at code analysis and finding the cyclic package dependencies.
I am looking for many answers to learn about new tools and consolidate this knowledge in a one question/thread, so I doubt there will be 1 true answer to this question.
My answer to my own question is that we use:
Findbugs to look for common errors bad/coding - run from maven, and also integrates easily into Eclipse
Cobertura for our coverage reports - run from maven
Hudson also has a task-scanner plugin that will display a count of your TODO and FIXMEs, as well as show where they are in the source files.
All are integrated with Maven 1.x in our case and tied into Hudson, which runs our builds on check-in as well as extra things nightly and weekly. Hudson trend graphs our JUnit tests, coverage, findbugs, as well as open tasks. There is also a Hudson plugin that reports and graphs our compile warnings. We also have several performance tests with their own graphs of performance and memory use over time using the Hudson plots plugin as well.