We rely heavily on logic in our Junit runner and currently call into it with mvn test
The logic mainly consists of
Instantiating a RuntimeHook and binding it to Runner.Builder().hook if an appropriate JVM switch is specified on the command line.
Calling Runner.Builder().clientFactory() with a mock, again if an appropriate JVM switch is specified on the command line.
Last setting minor things like Runner.Builder().tags and Runner.Builder().path
All of this works perfect today. Our main gripe however is Maven is slow.
Is there a way to accomplish the above logic and run without the expensive build time of Maven?
Since you are mixing a bit of Java code, I think you are going to depend on Maven for build + dependencies. That said, there are ways to ask Maven to dump all JAR dependencies into a text file. For example:
mvn dependency:build-classpath -Dmdep.includeScope=test -Dmdep.outputFile=classpath.txt
And then you may be able to shape a command that uses the java binary directly and you probably already know that com.intuit.karate.Main supports all the extension things. The bad news is I don't think we support things like the custom HTTP Client yet, you can see this PR where someone did this for the RuntimeHook. Perhaps you can contribute. Or maybe you have logic in Java code anyway, so calling the Karate Runner directly may be the way to go.
I think maybe the solution you will land on is mvnd - I haven't used it yet, but I'm hearing very good things on Twitter from some experienced Java folks I follow.
Related
We are experiencing a problem with our Jenkins CI server.
Our CI implementation relies on several Groovy scripts, which we execute in Jenkins as "System Groovy scripts". This has been this way for years, and the scripts have undergone no recent modifications, and implement build flows, business logic steps such as version checking, etc.
Yesterday we started experiencing an exception in every Jenkins job that we tried to lauch that, one way or another, tried to execute Groovy scripts. The exception is:
java.lang.StackOverflowError
at org.codehaus.groovy.antlr.parser.GroovyRecognizer.additiveExpression(GroovyRecognizer.java:12478)
at org.codehaus.groovy.antlr.parser.GroovyRecognizer.shiftExpression(GroovyRecognizer.java:9695)
at org.codehaus.groovy.antlr.parser.GroovyRecognizer.relationalExpression(GroovyRecognizer.java:12383)
at org.codehaus.groovy.antlr.parser.GroovyRecognizer.equalityExpression(GroovyRecognizer.java:12307)
at org.codehaus.groovy.antlr.parser.GroovyRecognizer.regexExpression(GroovyRecognizer.java:12255)
at org.codehaus.groovy.antlr.parser.GroovyRecognizer.andExpression(GroovyRecognizer.java:12223)
at org.codehaus.groovy.antlr.parser.GroovyRecognizer.exclusiveOrExpression(GroovyRecognizer.java:12191)
hundreds of similar lines
at org.codehaus.groovy.antlr.parser.GroovyRecognizer.compoundStatement(GroovyRecognizer.java:7510)
at org.codehaus.groovy.antlr.parser.GroovyRecognizer.compatibleBodyStatement(GroovyRecognizer.java:8834)
at org.codehaus.groovy.antlr.parser.GroovyRecognizer.statement(GroovyRecognizer.java:899)
at org.codehaus.groovy.antlr.parser.GroovyRecognizer.compilationUnit(GroovyRecognizer.java:757)
at org.codehaus.groovy.antlr.AntlrParserPlugin.transformCSTIntoAST(AntlrParserPlugin.java:131)
at org.codehaus.groovy.antlr.AntlrParserPlugin.parseCST(AntlrParserPlugin.java:108)
at org.codehaus.groovy.control.SourceUnit.parse(SourceUnit.java:236)
at org.codehaus.groovy.control.CompilationUnit$1.call(CompilationUnit.java:161)
at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:846)
at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:550)
at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:526)
at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:503)
at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:302)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:281)
at groovy.lang.GroovyShell.parseClass(GroovyShell.java:731)
at groovy.lang.GroovyShell.parse(GroovyShell.java:743)
at groovy.lang.GroovyShell.parse(GroovyShell.java:770)
at groovy.lang.GroovyShell.parse(GroovyShell.java:761)
at groovy.lang.GroovyShell$parse.call(Unknown Source)
at com.cloudbees.plugins.flow.FlowDSL.executeFlowScript(FlowDSL.groovy:80)
at com.cloudbees.plugins.flow.FlowRun$FlyweightTaskRunnerImpl.run(FlowRun.java:219)
at hudson.model.Run.execute(Run.java:1759)
at com.cloudbees.plugins.flow.FlowRun.run(FlowRun.java:155)
at hudson.model.ResourceController.execute(ResourceController.java:89)
at hudson.model.Executor.run(Executor.java:240)
at hudson.model.OneOffExecutor.run(OneOffExecutor.java:43)
This looks like that the Groovy parser inside Jenkins is reaching the top of the stack while trying to parse the groovy script (as I have said, this abruptly started to happen with many scripts that worked perfectly before and had undergone no recent modification).
Currently our Jenkins installation (v1.594) runs on a Websphere 8.5.5.2 application server on AIX v7.1 (don't know exactly the fix pack level and / or if it has recently suffered any kind of update, still trying to gather the info).
After a restart, we returned to normal behavior (all the scripts were working as usual again without any modification to them).
Does anyone know about some incompatibility of any underlying library with Jenkins Groovy parsing?
There is a problem with the groovy code; causing the parser to go nuts:
java.lang.StackOverflowError
at org.codehaus.groovy.antlr.parser.GroovyRecognizer.additiveExpression(GroovyRecognizer.java:12478)
Based on a similar ticket:
https://issues.apache.org/jira/browse/GROOVY-1783,
it is possible that your code has circular references; or creating too many functions on the fly. You can take the approach of analyzing your code and trying to put anything that is going to make allocations outside of loops; in particular complex inline functions.
Another approach is to go look at the Build Flow plugin and scroll down the documentation and see how you could write an extension point rather than use groovy. This may not be easy to do and requires effort; but you can write a lot of tests for your code that way. You would still use groovy for the glue; but use java directly for the hot spots.
A third approach would be to file a ticket on the Groovy issue tracker; and see what the experts find out.
I am looking for a solution that can be run on a continuous integration server (TeamCity) which seems to rule out IDE specific solutions like the externalization that is built into eclipse.
I need to generate warnings when strings are found in the java source files, unless they have been somehow flagged (by comment or annotation) as not requiring internationalization, eg the // $NON-NLS-1$ comment for eclipse.
The code is built using maven in teamcity.
I have not been able to find existing solutions or any ideas on how a custom solution might be hooked into either maven or teamcity (note I am not hugely familiar with either so could be overlooking something fairly obvious).
It seems that you need some specifically crafted Static Code Analysis tool.
I am afraid that no such thing exists yet (or I should say no such thing is available for public use...). I am not sure if you would be able to create warnings from the tool though. I know it would be possible with Gradle, but integrating Gradle task with Java code is not an easiest thing to do (BTDT).
Maybe some simple HTML report would do? In this case, you can simply use Scanner with regular expressions that matches all the strings literals... That is unless you'll find exclusions (another RegExp) in the same line. That's pretty basic thing to do.
What is the simplest way to add a compile-time step to analyse and modify the source code before it is compiled to byte code?
Can I write this in Java?
Would it be best written as an IDE plugin?
Can I write this in Java?
Yes, definitely. There are numerous Java based libraries for manipulating bytecode:
Commons BCEL
ASM
Javassist
Would it be best written as an IDE plugin?
In my opinion, no. You didn't mention which IDE you're using, but from my own experience, writing an IDE plugin has a steeper learning curve than adding a custom step to a build tool like Ant/Maven/Gradle. Even if you aren't currently using one of these build tools, in my personal opinion, it would be easier to adopt one of these tools rather than write an IDE plugin.
Also, tying a build step to a particular IDE makes your build less portable. Two things to consider before going this route:
1) How you would run your build on a continuous integration server like Jenkins or Bamboo. It's not impossible to invoke a headless Eclipse/Netbeans build that uses custom plugins on a build server, but it's not nearly as straightforward as running a build that uses "standard" tools like Ant/Maven/Gradle.
2) How would it impact other members of your team? You'd need to find a way to distribute to the plugin to each developer, deal with versioning and updates of the plugin, etc. Is everyone on your team using the same IDE?
I don't know anything about your project, your team (if you're working on a team), or the type of software you're developing so these considerations may not apply to you. I've only mentioned them as food for thought based on my own experiences.
What is the simplest way to add a compile-time step to analyse and
modify the source code before it is compiled to byte code?
What are you using for your builds? Ant? Maven? Gradle? The exact steps you'd follow are highly dependent on your build tool.
Depending on what you're trying to accomplish, you may not need to write anything at all.
For example, analysing parts of the code and splitting the work into
multiple threads where necessary –
Check out AspectJ. You can probably write an aspect that intercepts calls to certain methods and submits them to an ExecutorService. There are off the shelf plugins to invoke the AspectJ compiler from most common build systems.
If you do want to write something on your own, I think your best bet would be to write a custom Ant task. I suggest an Ant task because it's the lowest common denominator. It can of course be run using Ant, but both Maven and Gradle can invoke Ant tasks as well.
Write a new class that extends Task and do your thing in there.
public class MyTask extends Task {
public void execute() {
// do your bytecode manipulation here...
}
}
You'd invoke it like this from your Ant script:
<taskdef name="mytask" classname="MyTask" classpath="classes"/>
<mytask/>
Check out the Apache Axis2 code generation task for an example of doing build time code generation and how to deal with classpath issues/accessing your code.
I am currently developing a simple plugin that retrieves results from a Jenkins build. I am extending Notifier and using build.getResults() to get the information. However, when I upload my plugin, I can't set it as a post-build action.
When I run my builds, they break on build.getResults() since I am trying to get the results while the build is still running.
What can I do to properly get the build result ?
Best thing is to look at existing plugins which use Notifier extension point (click to expand implementing plugins list).
Check that you have the Descriptor implemenation (inner) class, as well as config.jelly. Also check jenkins.out and jenkins.err logs for any exceptions (such as malformed config.jelly).
Edit: Actually, Notifier subclass of this plugin looks really simple as Notifiers go: https://wiki.jenkins-ci.org/display/JENKINS/The+Continuous+Integration+Game+plugin , see especially its GamePublisher.java and corresponding config.jelly, and it's GameDescriptor.java, which has been made a full outer class (often descriptor is inner class). Also if you want options into Jenkins' Global configuration, you need a global.jelly, but if you don't have such options, that is something you can just leave out (unlike config.jelly, which you must have for Notifier even if it is empty, like here).
As a general note, it can be really annoying when things do not work, and you do not get any error, your stuff simply is just not displayed by Jenkins... If you just want to get things to work for you, using Groovy build step might be easier, but if you want to get things to work for others, then doing a decent full plugin reduces support requests.
Since this sounds so simple, are you sure you need a plugin ? Take a look at using a Groovy Postbuild step instead; they're much easier to write. There are some good usage examples in the link. If you decide you really need a plugin, see if you can extend an existing one rather than writing your own; it's an easier way to understand the ins and outs of Jenkins plugin writing.
This question came to me when developing using Eclipse.
I use JBoss Application Server and use hot code replacement. But this option requires that the 'build automatically' option to be enabled. This makes Eclipse build the workspace automatically (periodically or when a file is saved?) and for a large code base this takes too much time and processing which makes the machine freeze for a while. Also sometimes an error message is shown saying that hot code replacement failed.
The question that I have is: is there a better way to see the result of a code change?
Currently I have the following two suggestions:
Have unit tests - this will allow to run a single test and see the result of a code change. ( But for a JavaEE application that uses EJBs is it easy to setup unit tests?)
Use OSGi - which allows to add jars to the running system without bringing down the JVM.
Any ideas on above suggestions or any other suggestion or a framework that allows to do this is welcome.
Did you take a look at http://zeroturnaround.com/jrebel/?
I tell you how I work. I hope it is useful. First of all, I disable "Build Automatically". Maybe it is better if you do a simple ANT script to compile and see errors/exceptions. Also, I make jar file with the same script. Afterwards, I use OSGi to check the application. Yo do not need to stop server, only you need to change versions in deployed bundles. Intead of JBoss Server I use Equinox which includes Jetty Web Server.
May you have a nice day!
With JRebel, you wouldn't have to build your project (skip the build totally). Plus, only the changed resources will be reloaded so the update happens instantly. Plus, session is preserved so you do not have re-login to the application after the update was performed.
Even though the question I asked was quite specific to Java, I thought that mentioning using an interpreted programming language that avoids the compilation step is anther way of seeing result of a code change faster.