I'm trying to make the integration between JUnit, Jenkins and TM4J (Jira) using this tutorial: https://support.smartbear.com/tm4j-cloud/docs/api-and-test-automation/junit-integration.html.
The problem is that the tm4j_result.json file is NOT generated, although the result from the Cucumber integration is being generated successfully.
Is it possible to generate BOTH Cucumber and JUnit reports?
This is the Jenkins reference, which contains the exact same file pattern example and the bitbucket code in the reference shows only
adaptivist library and surefire listener
To answer your question : "Is it possible to generate BOTH Cucumber and JUnit reports?"
I would answer there is no asking this, since from your automated tests project, you will "either" work with BDD, hence use the cucumber file, or, work with non-BDD; which will in this case resort to the Junit file.
Should you have both, a solution using both will be needed at a pipeline level and a double test execution command will be deemed necessary to keep things reasonably simple.
There is no mix of BDD and non BDD all in the same test execution command, to put it simply !
Related
I am a beginner at automation testing.
I am trying to execute tests that were written in java using Selenium WebDriver, Cucumber with Gherkin annotations in JMeter.
When I run my tests without JMeter, I just use the opportunity to run them as Junit test using the runner class (in Eclipse).
To run my tests in JMeter, I created a jar file, uploaded it to Junit sub-folder, all dependencies-jars were uploaded to lib sub-folder.
I tried to create a JUnit request in JMeter, but it didn't help as when we use Cucumber, we don't mark methods with #Test. I have just an empty constructor for my Runner class (as you can see from the picture above).
I tried to add my jar to Test Plan and create an instance of my class in BeanShell, but it is not executed as JUnit tests.
My question is how to run java test scripts that were written based on Cucumber with Gherkin (based on Selenium WebDriver)?
Maybe I have to use other Samplers...
Any examples are greatly appreciated.
I believe you need to run another class, in particular io.cucumber.core.cli.Main, something like:
io.cucumber.core.cli.Main.main(new String[]{
"--glue",
"the package which contains the glue classes",
"/your/feature/file"});
Check out Running Cucumber -> From the command line for all the available arguments explained. By the way, you can use OS Process Sampler for this, it will be way easier.
Depending on what you're trying to achieve it might be faster and easier to convert your Selenium tests into "native" JMeter ones as if you're going to use your Selenium tests for creating the load it will require immense hardware resources and you won't get performance metrics on the HTTP protocol level.
I am trying to create the following setup :
A Selenium (Java) project that has a set of 10 automated test cases.
When this project is executed, it generates an HTML test execution report.
This project should be 'hosted' on an internal network.
Anyone who has access to the network should be able to 'invoke' this project, which in turn executes the test cases and passes the HTML report to the person who invoked it.
The project should be accessible ONLY for execution and the code should NOT be accessible.
My goal is that this implementation should be executable by any framework irrespective of the technology that the framework uses. I was thinking of creating the project as a WebService using Java (servlet).
My question is:
Can this implementation be accessed by any external automation framework ?
Are there any limitations to this implementation?
Is there a better way to implement this requirement?
Thanks in advance.
You can create a maven project and have your automated tests under maven test folder.Configure your tests to run through POM.xml(use maven surefire plugin).Configure a jenkins job to run the maven test.Anybody with access the jenkins can build/run this task!
Below link should give you a headstart
http://learn-automation.com/selenium-integration-with-jenkins/
As a matter of fact, it is something we did on one of our projects. As I cannot share specifics, I will give you overall architectural view of the project.
The core of all things was a service that could run JUnit tests on requests. It was a Soap web-service, but nothing stops you from making it REST. To implement this you need to implement your version of JUnit test runners (see for example: http://www.mscharhag.com/java/understanding-junits-runner-architecture or https://github.com/junit-team/junit4/wiki/Test-runners)
If you use JUnit as test framework for running your Selenuim tests this may be a great solution for you - JUnit will generate HTML reports for you if you configure it properly, it will hide actual test suite implementation from users and run test suite on demand. This solution is also great because it operates on JUnit level and does not care about what kind of tests it actually runs, so it can be also reused for any other kind of automated tests.
So to answer all your questions:
Can this implementation be accessed by any external automation
framework ? -> yes, it can be accessed by anybody who able send http
requests
Are there any limitations to this implementation? -> none that I am
aware of
Is there a better way to implement this requirement? -> well, I
didn't actually work with TestNG much so I don't know if it is
easier or more difficult to do it on Junit level. You can use
Jenkins or other CI tool as well to achieve same results - they can
run JUnit tests for you and almost always have API ready for this,
although those APIs may be not perfect.
So I'd say that if you need this only for one thing you can use CI tools for this purpose, if you don't have CI tools available then choice has been made for you. However, from our experience, having this kind of service was a great asset for a company and I really wonder why there's no such products available elsewhere yet.
I am really new to Fit/Fitnesse and, in general, to test automation.
I am trying to use them from Eclipse.
I have several question about it:
is there a way to obtain the html tables that Fitnesse pass to Fit?
once I write several tests with Fitnesse, is there a way to call them several times from Java without clicking on the Test button of the wiki?
About passing objects from one table to another in a flow. I read about symbols but it seems that, in java, they works only with ColumnFixturewhile I would like to use DoFixture. how to do this?
Finally,is if there is any plugin for eclipse you suggest to use with Fit/Fitnesse?
Regarding you question 2: I would recommend using the JUnit integration (#RunWith(FitNesseRunner.class) to run the test page (or a suite) as a unit test from Eclipse. This also gives you the ability to debug inside your fixture code.
It takes a bit of configuration to get it running 'just right'. In my pre-packaged FitNesse I provide a unit test FixtureDebug where you only have to enter the test name (and you can also use that to run your tests on a build/continuous integration server).
I have used Cucumber with Jenkins in the past and jenkins-cucumber-jvm-reports-plugin-java is fantastic. All I need to do is to generate a cucumber json output to get a beautiful report.
My new project uses Bamboo and I could not find anything similar or atleast something closer. Has any one have experience using Cucumber and Bamboo? Could use some helpful pointers. As this exercise is POC, either Ruby or Java would be great.
I can see that this was asked a long time ago. Not sure whether it is still valid but here is my experience with the same problem.
In my project we are using cucumber with bamboo and report generationn is also vital for us.
We found that the Cucumber Report Plugin for bamboo is not sufficient for us so we moved to the Cucumber Reporting tool from MasterThought. As I see this is the exact same thing that the jenkins-cucumber-jvm-reports-plugin-java is using.
To utilize this in Bamboo we are calling it directly through maven after we have executed our tests during our build but there is also a maven-cucumber-reporting-mojo which can be used if you want to configure it through maven (we haven't used this because at that point it supported only one json input and we had hundreds of jsons but it's possible that this functionality was added in the mean time).
We are setting the generated html based output as artifact for our build so we can access the nice reports in bamboo for every build.
I hope it helps.
I'm interested in generating reports for a certain group of (non-software) engineers in my company using JUnit. Unlike typical JUnit testing where you are ensuring objects, methods, etc are bug free, I want to use JUnit to confirm if results in large log files are within range and give reports on it.
My background: Experienced with Java. I am familiar with JUnit 4, I have spent the last hour or 2 looking through ANT documentation and I think I have a strong idea how it works (I just don't know all the XML commands), and I use Eclipse religiously. Specifically Eclipse June (latest).
My goal is to create a slew of testcases that assert if the values in the logs are within the right range. The test class can then be run over and over on every single log file. After each run, the results will be added to an HTML file.
I am aware of the JUnit Report features in Ant but I am having a lot of trouble putting all the pieces of the puzzle together properly. I know I can hack my way through this project quite easily, but I want to do this the right way.
Can any point me in the right direction in how to:
Make the right build.xml files
Set up JUnit classes/suites/runners so that it runs over and over for a given list of logs
Set up ANT so it does output HTML/XML and how to point it to the right style-sheet so it can be opened in IE/Firefox
How to make the output file
Override features so the reports are custom.
What to override for post-processing. (So I can convert the output HTML into a PDF, etc)
How to create a standalone executable that will do all this automatically - Perhaps incorporating a little Swing if the user doesn't supply an input manually.
How to properly loop through many tests (currently I am just making a main function and doing:
code:
public static void main(String[] args) {
JUnitCore junit = new JUnitCore();
RunListener listener = new RunListener();
junit.addListener(listener);
Result result = junit.run(GearAndBrakeFaultLogReports.class);
for (Failure f : result.getFailures()) {
System.out.println(f.toString());
System.out.println(f.getDescription());
System.out.println(f.getTestHeader());
System.out.println(f.getException());
System.out.println(f.getMessage());
}
System.out.println("Done.");
}
I have a feeling that is the "hacky" way of doing it and if I knew how to plug into Ant properly, it will run my tests, generate the XML/HTML and I would have the output desired.
I imagine I will have more questions as this project develops, but I think getting over the initial hump will be very good for me! I know there are a lot of questions, so any help and I'll gladly point bump :) My hope is, someone familiar with these tools can answer all these with a matter of pointing me to web sites, functions, or an example project that does a similar thing.
For creating a PDF from JUnit test results:
http://junitpdfreport.sourceforge.net/managedcontent/
As a unit testing frameword, JUnit reports are very specific to software engineering. Although, you might be able to customize it for a non-unit testing scenario, it will take a lot of customizing. Unlike other testing frameworks (such as TestNG) JUnit doesn't provide the ability to make your own report generator. TestNG, however, can be used to run JUnit tests and even produces a JUnit report in addition to its own reports. TestNG can run your JUnit tests and then create custom reports by providing a custom implementation of the org.testng.IReporter interface.
http://testng.org/doc/documentation-main.html
Having written reports in Java for many years now, I would strongly recommend trying out a reporting tool such as JasperReports and the iReport tool. Jasper Reports can process custom data sources such as your log files and produce reports in many formats, including XML, HTML, and PDF.
http://community.jaspersoft.com/project/ireport-designer