We have big-old Java project with a lot of legacy code. Now we have code conventions and Checkstyle rules to check it.
We want to configure Jenkins job, which runs after every commit to SVN, to check, if modified/added lines are not violate any our Checkstyle rule.
It's not an option to check whole project, because of legacy code. Also we can't just reformat all the code, because then it will be difficult to determine who changed a particular line and why.
The approach we consider - to make diff between previous and current Checkstyle report, and see, if there are new violations.
The question is - how we can get access to the previous Checkstyle report in Jenkins?
Or how to configure checkstyle to fail build only if there are new violations?
And may be there are ready-made solutions for such check?
This is a bit round-about, but if you set up a Sonar instance to analyze your project, you can query violations data programmatically through it's remote access API. You know the violations count in the legacy code (presumably, that number won't change frequently). That's your baseline - you don't want the count to go higher than that.
So inside your CI job, you could add a script at the end that calls Sonar to get the count of violations at each of the severity levels you care about (Blocker, Critical, etc), and fail the build if the current count exceeds the threshold/benchmark.
Sonar is pretty awesome overall, especially for projects with lots of legacy code, 'cuz you can drill down and figure out where your riskiest areas are. They've got a public instance running if you haven't checked it out yet.
The solution was to make a bash script which make diff with previous checkstyle report and fails build if there are new violations.
Have you checked on Checkstyle Plugin that is available for jenkins? It will generate reports for each commit and you could compare the results for each change. Each commit will trigger a new build and a new report will be generated at the end.
Have you checked on Checkstyle Plugin that is available for jenkins? It will generate reports for each commit and you could compare the results for each change. Each commit will trigger a new build and a new report will be generated at the end.Shiva Kumar
Of course I checked it, but the question is - how we can get access to the Checkstyle report from previous biuld?
May I also suggest the Static Code Analysis Plug-ins. This will show you the trend line of Checkstyle issues as a nice graph. You can also use this with the CI Game plugin that will award points for fixing Checkstyle issues, and deduct points for causing more checkstyle issues.
I've found the CI Game plugin to work really well -- especially if you get your developers involved in a bit of a contest:
We're adding a little something to this month's contest. As you all know, first prize is a Cadillac Eldorado. Anybody want to see second prize? (Holds up prize) Second prize is a set of steak knives. Third prize is you're fired.
Alec Baldwin as Blake in Glengarry Glen Ross
It's amazing how fast those Checkstyle issues get fixed when you turn it into a fun game.
Related
We are using Sonar Qube 6.7.3 and sonar-java-plugin 5.3
We have made below changes to our sonar configuration recently
Enabled new rules
Changed configuration to include byte code(changed from 'clean sonar:sonar' to 'clean package sonar:sonar')
We are using sonar svn plugin and provide valid credentials to it.
I understand providing byte code to sonar will help it identify more issues but, I expect Sonar to flag new issues based on svn code commit date and last analysis date, but it is not.
Please let me know why it is flagging issues in old code as new?
Sonar Sanner always scans the entire code base. If somebody has decided that some code structures are wrong or dangerous (the ruleset have been changed) then SonarQube has to notify about all occurrences of that code. Why? Let's think about the following example:
After a plugin upgrade, SonarQube provides a new very important security rule which forbids the use of a dangerous cipher algorithm. Now is the question:
is it only dangerous in new code?
is it always dangerous?
Of course, it is always dangerous. SonarQube doesn't force you to fix everything (usage of the quality gates is optional). Its main goal is to let you know how many problems (code smells/bugs/vulnerabilities) exist in the whole code base.
I'm evaluating SonarQube 5.4 with SonarLint eclipse plugin.
SonarQube as well as the plugin are set up and are running. But now I'm pretty confused how SonarLint is supposed to run in 'connected mode':
SonarLint is connected with SonarQube and is bound to the corresponding project. But some issues are only shown in SonarQube. It was my understanding SonarLint should be able to identify issues like Malicious code vulnerability - May expose internal representation by incorporating reference to mutable object. But it does not. SonarQube does.
When analysing a single file with SonarLint, there are a lot of debug messages in the SonarLint Console like Class not found in resource cache : org/company/project/CommonSuperClass. But even worse: Class not found in resource cache : java/lang/Class. Is it supposed to do that?
We are specifically interested in highlighting the issues introduced by developer. SonarQube is connected our repo and does a nice job in blaming the committer. But it seems there is no way of showing my own issues in sonarlint.
I'd like to run the SonarLint analysis at a time of my choice, so I decided to deactivated "Run SonarLint automatically". But it seems I can only analyze files manually, not packages or projects. Am I missing something again? I do not want to click on every one of my ~2000 files and analyze it by hand.
SonarLint and SonarQube are 2 different products:
You want very fast feedback on the code you are working on to make sure you don't inject issues => SonarLint analyses the files as you open them to write or review code
You want a 360° vision of the quality of your code => SonarQube analyses all the files of your project
The "connected mode" is the bridge between the 2 worlds, and its development is still underway. For instance, we plan to make it possible to see inside SonarLint all the issues found on the project by SonarQube (see and vote for SLE-54).
I am working on a number of projects and we are using Java, Springs, Maven and Jenkins for CI but I am running into a issues that some of the programmers are not adding real junit test cases to the projects. I want maven and jenkins to run the test before deploying to the server. Some of the programers made a blank test so it starts and stops and will pass the tests.
Can someone please tell me how can I automat this check so maven and jenkins can see if the test put out some output.
I have not found any good solution to this issue, other than reviewing the code.
Code coverage fails to detect the worst unit tests I ever saw
Looking at the number of tests, fails there too. Looking at the test names, you bet that fails.
If you have developers like the "Kevin" who writes tests like those, you'll only catch those tests by code review.
The summary of how "Kevin" defeats the checks:
Write a test called smokes. In this test you invoke every method of the class under test with differing combinations of parameters, each call wrapped in try { ... } catch (Throwable t) {/* ignore */}. This gives you great coverage, and the test never fails
Write a load of empty tests with names that sound like you have thought up fancy test scenarios, eg widgetsTurnRedWhenFlangeIsOff, widgetsCounterrotateIfFangeGreaterThan50. These are empty tests, so will never fail, and a manager inspection the CI system will see lots of detailed test cases.
Code review is the only way to catch "Kevin".
Hope your developers are not that bad
Update
I had a shower moment this morning. There is a type of automated analysis that can catch "Kevin", unfortunately it can still be cheated around, so while it is not a solution to people writing bad tests, it does make it harder to write bad tests.
Mutation Testing
This is an old project, and won't work on recent code, and I am not suggesting you use this. But I am suggesting that it hints at a type of automated analysis that would stop "Kevin"
If I were implementing this, what I would do is write a "JestingClassLoader" that uses, e.g. ASM, to rewrite the bytecode with one little "jest" at a time. Then run the test suite against your classes when loaded with this classloader. If the tests don't fail, you are in "Kevin" land. The issue is that you need to run all the tests against every branch point in your code. You could use automatic coverage analysis and test time profiling to speed things up, though. In other words, you know what code paths each test executes, so when you make a "jest" against one specific path, you only run the tests that hit that path, and you start with the fastest test. If none of those tests fail, you have found a weakness in your test coverage.
So if somebody were to "modernize" Jester, you'd have a way to find "Kevin" out.
But that will not stop people writing bad tests. Because you can pass that check by writing tests that verify the code behaves as it currently behaves, bugs and all. Heck there are even companies selling software that will "write the tests for you". I will not give them the Google Page Rank by linking to them from here, but my point is if they get their hands on such software you will have loads of tests that straight-jacket your codebase and don't find any bugs (because as soon as you change anything the "generated" tests will fail, so now making a change requires arguing over the change itself as well as the changes to all the unit tests that the change broke, increasing the business cost to make a change, even if that change is fixing a real bug)
I would recommend using Sonar which has a very useful build breaker plugin.
Within the Sonar quality profile you can set alerts on any combination of metrics, so, for example you could mandate that your java projects should have
"Unit tests" > 1
"Coverage" > 20
Forcing developers to have at least 1 unit test that covers a minimum of 20% of their codebase. (Pretty low quality bar, but I suppose that's your point!)
Setting up an additional server may appear like extra work, but the solution scales when you have multiple Maven projects. The Jenkins plugin for Sonar is all you'll need to configure.
Jacoco is the default code coverage tool, and Sonar will also automatically run other tools like Checkstyle, PMD and Findbugs.
Finally Stephen is completely correct about code review. Sonar has some basic, but useful, code review features.
You need to add a code coverage plugin such as JaCoCo, EMMA, Cobertura, or the likes. Then you need to define in the plugin's configuration the percent of code coverage (basically "code covered by the tests") that you would like to have, in order for the build to pass. If it's below that number, you can have the build failing. And, if the build is failing, Jenkins (or whatever your CI is) won't deploy.
As others have pointed out, if your programmers are already out to cheat coding practices, using better coverage tools won't solve your problem. They can be out cheated as well.
You need to sit down with your team and have an honest talk with them about professionalism and what software engineering is supposed to be.
In my experience, code reviews are great but they need to happen before the code is committed. But for that to work in a project where people are 'cheating', you'll need to at least have a reviewer you can trust.
http://pitest.org/ is a good solution for so called "mutation testing":
Faults (or mutations) are automatically seeded into your code, then your tests are run. If your tests fail then the mutation is killed, if your tests pass then the mutation lived.
The quality of your tests can be gauged from the percentage of mutations killed.
The good thing is that you can easyli use it in conjunction with maven, jenkins and ... SonarQube!
I am currently working on a rather large project with a team distributed across the United States. Developers regular commit code to the source repository. We have the following application builds (all are managed by an application, no manual processes):
Continuous Integration: a monitor checks to see if the code repository has been updated, if so it does a build and runs our unit test suite. On errors, the team receive email notifications
Daily Build: Developers use this build to verify their bug fixes or new code on an actual application server, and if "things" succeed, the developer may resolve the task.
Weekly Build: Testers verify the resolved issue queue on this build. It is a more stable testing environment.
Current Release build: used for demoing and an open testing platform for potential new users.
Each build refreshes the database associated with it. This cleans data and verifies any databases changes that go along with the new code are pulled in. One concern I hear from our testers is that we need to pre-populate the weekly build database with some expected testing data, as opposed to more generic data that developers work with. This seems like a legitimate concern/need and is something we are working on.
I am tossing what we are doing out to see if the SO community sees any gap with what we are doing, or have any concerns. Things seems to be working well, but it FEELS like it could be better. Your thoughts?
An additional step that is followed is that once the release build passes tests (say smoke test) then it is qualified as a good build (say a golden build) and you use some kind of labeling mechanism to label all the artefacts (code, install scripts, makefiles, installable etc.) that went into the creation of the golden image. The golden build may become a release candidate later or not.
Probably you are already doing this, since you don't mention I added what I had observed.
this is pretty much the way we do it.
The DB of the testers themselves is only reset on demand. If we would refresh this automatically every week then
we would lose the references to bug symptoms; if a bug is found but a developer only looks at it a few weeks later (or simply after the weekend) then all eveidence of that bug may have dissapeared
testers might be in the middle of a big test case (taking more than 1 day for instance)
we have tons of unit tests which are running against a DB which is refreshed (automatically of course) each time an integration build is executed
regards,
Stijn
I think you have a good, comprehensive process, as long as it fits in with when your customers want to see updates. One possible gap I can see is that it looks like you wouldn't be able to get a critical customer bug fix into production in less than a week, since your test builds are weekly and then you'd need time for the testers to verify the fix.
If you fancy thinking about things a different way, have a look at this article on continuous deployment - it can be a bit hard to accept the concept at first, but it definitely has some potential.
How does your team handle Builds?
We use Cruise Control, but (due to lack of knowledge) we are facing some problems - Code freeze in SVN - Build management
Specifically, how do you make available a particular release when code is constantly being checked in?
Generally, can you discuss what best practices you use in release management?
I'm positively astonished that this isn't a duplicate, but I can't find another one.
Okay, here's the deal. They are two separate, but related questions.
For build management, the essential point is that you should have an automatic, repeatable build that rebuilds the entire collection of software from scratch, and goes all the way to your deliverable configuration. in other words, you should build effectively a release candidate every time. Many projects don't really do this, but I've seen it burn people (read "been burned by it") too many times.
Continuous integration says that this build process should be repeated every time there is a significant change event to the code (like a check in) if at all possible. I've done several projects in which this turned into a build every night because the code was large enough that it took several hours to build, but the ideal is to set up your build process so that some automatic mechanism --- like an ant script or make file --- only rebuilds the pieces affected by a change.
You handle the issue of providing a specific release by in some fashion preserving the exact configuration of all affected artifacts for each build, so you can apply your repeatable build process to the exact configuration you had. (That's why it's called "configuration management.") The usual version control tools, like git or subversion, provide ways to identify and name configurations so they can be recovered; in svn, for example, you might construct a tag for a particular build. You simply need to keep a little bit of metadata around so you know which configuration you used.
You might want to read one of the "Pragmatic Version Control" books, and of course the stuff on CI and Cruise Control on Martin Fowler's site is essential.
Look at continuous integration: best pratices, from Martin Fowler.
Well, I have managed to find a related thread, I participated in, a year ago. You might find it useful, as well.
And here is how we do it.
[Edited]
We are using Cruise Control as integration tool. We just deal with the trunk, which is the main Subversion repository in our case. We seldom pull out a new branch for doing new story cards, when there is a chance of complex conflicts. Normally, we pull out a branch for a version release and create the build from that and deliver that to our test team. Meanwhile we continue the work in trunk and wait for the feedback from test team. Once all tested we create a tag from the branch, which is immutable logically in our case. So, we can release any version any time to any client in case. In case of bugs in the release we don't create tag, we fix the things there in the branch. After getting everything fixed and approved by test team, we merge the changes back to trunk and create a new tag from the branch specific to that release.
So, the idea is our branches and tags are not really participating in continuous integration, directly. Merging branch code back to the trunk automatically make that code becomes the part CI (Continuous Integration). We normally do just bugfixes, for the specific release, in branches, so it doesn't really participate into CI process, I believe. To the contrary, if we start doing new story cards, for some reasons, in a branch, then we don't keep that branch apart too long. We try to merge it back to trunk as soon as possible.
Precisely,
We create branches manually, when we plan a next release
We create a branch for the release and fix bugs in that branch in case
After getting everything good, we make a tag from that branch, which is logically immutable
At last we merge the branch back to trunk if has some fixes/modifications
Release Management goes well beyond continuous integration.
In your case, you should use Cruise Control to automatically make a tag, which allows developers to go on coding while your incremental build can take place.
If your build is incremental, that means you can trigger it every x minutes (and not for every commit, because if they are too frequent, and if your build is too long, it may not have time to finish before the next build tries to take place). The 'x' should be tailored to be longer that a compilation/unit test cycle.
A continuous integration should include automatic launch of unit tests as well.
Beyond that, a full release management process will involve:
a series of deployment on homologation servers
a full cycle of homologation / UAT (User Acceptance Test)
non-regression tests
performance / stress tests
pre-production (and parallel run tests)
before finally releasing into production.
Again "release management" is much more complex than just "continuous integration" ;)
Long story short: Create a branch copied from trunk and checkout/build your release on that branch on the build server.
However, to get to that point in a completely automated fashion using cc.net is not an easy task. I could go into details about our build process if you like, but it's probably too fine grained for this discussion.
I agree with Charlie about having an automatic, repeatable build from scratch. But we don't do everything for the "Continuous" build, only for Nightly, Beta, Weekly or Omega (GA/RTM/Gold) release builds. Simply because some things, like generating documentation, can take a long time, and for the continuous build you want to provide developer with rapid feedback on a build result.
I totally agree with preserving exact configuration, which is why branching a release or tagging is a must. If you have to maintain a release, i.e. you can't just release another copy of trunk, then a branch on release approach is the way to go, but you will need to get comfortable with merging.
You can use Team Foundation Server 2008 and Microsoft Studio Team System to accomplish your source control, branching, and releases.