We Keep Coding

How to create a XML report by executing a JAR

I want to develop a stand-alone test-solution delivered as a jar that can be used in a CI/CD environment without being recompiled all the time. Therefore I packed a fat-jar from a multi-maven-module containing a few libaries, a Spring Boot application and a submodule called test-runner.
Executing the fat-jar from within GitLab CI/CD works, but I think that was only the first half of it. I want to produce a JUnit XML report to output the test-results. What I understood from my research is that I would have to implement my own reporter. Is there a more complete example out there?
The test runner
public class Runner {
SummaryGeneratingListener listener = new SummaryGeneratingListener();
public void runOne() {
LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request()
.selectors(selectClass(MyTest.class)).build();
Launcher launcher = LauncherFactory.create();
TestPlan testPlan = launcher.discover(request);
launcher.registerTestExecutionListeners(listener);
launcher.execute(testPlan);
}
public static void resultReport(Result result) {
System.out.println("Finished. Result: Failures: " + result.getFailureCount() + ". Ignored: "
+ result.getIgnoreCount() + ". Tests run: " + result.getRunCount() + ". Time: "
+ result.getRunTime() + "ms.");
}
public static void main(String[] args) {
Runner runner = new Runner();
runner.runOne();
TestExecutionSummary summary = runner.listener.getSummary();
summary.printTo(new PrintWriter(System.out));
}
}
Background:
My test-solution is generic and uses a configuration file to parameterize the tests. All tests run in parallel versus a system-under-test. So before this attempt all gitlab-jobs called mvn test to execute the tests and generate the reports, but it recompiled everything every run. I thought about speeding things up.

To generate XML reports, you can use the LegacyXmlReportGeneratingListener with a path to save your reports to as first argument:
LegacyXmlReportGeneratingListener xmlListener = new LegacyXmlReportGeneratingListener(Paths.get("reports"), new PrintWriter(System.out));
In your runOne() method, you need to register your listener accordingly:
public void runOne() {
LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request()
.selectors(selectClass(MyTest.class)).build();
Launcher launcher = LauncherFactory.create();
TestPlan testPlan = launcher.discover(request);
launcher.registerTestExecutionListeners(listener);
launcher.registerTestExecutionListeners(xmlListener);
launcher.execute(testPlan);
}
This will generate one XML file per test root in the folder you passed to the listener during initialization.
More information can be found in the JavaDoc

You can use Console launcher to geneate Junit5 xml reports
java -jar junit-platform-console-standalone-1.6.2.jar #junitArgs.txt --reports-dir=reports
junitArgs.txt file has following info:
-classpath fat jar path
--scan-classpath

Getting error "Class name "org.apache.jmeter.report.processor.graph.impl.TotalTPSGraphConsumer" in JAVA

While trying to generate Summary Report graphically in Jmeter in Java(Not in GUI) I am getiing below error.
ERROR 2020-03-10 14:10:48.284 [jmeter.r] (): Class name "org.apache.jmeter.report.processor.graph.impl.TotalTPSGraphConsumer" is not valid.
Full code just in case:
import org.apache.jmeter.engine.StandardJMeterEngine;
import org.apache.jmeter.report.dashboard.ReportGenerator;
import org.apache.jmeter.reporters.ResultCollector;
import org.apache.jmeter.reporters.Summariser;
import org.apache.jmeter.save.SaveService;
import org.apache.jmeter.util.JMeterUtils;
import org.apache.jorphan.collections.HashTree;
import java.io.File;
import static org.apache.jmeter.JMeter.JMETER_REPORT_OUTPUT_DIR_PROPERTY;
public class RunJMeterTest {
public static void main(String[] args) throws Exception {
StandardJMeterEngine jmeter = new StandardJMeterEngine();
JMeterUtils.loadJMeterProperties("/path/to/your/jmeter/bin/jmeter.properties");
JMeterUtils.setJMeterHome("/path/to/your/jmeter");
JMeterUtils.initLocale();
SaveService.loadProperties();
HashTree testPlanTree = SaveService.loadTree(new File("/path/to/your/jmeter/bin/test.jmx"));
Summariser summer = null;
String summariserName = JMeterUtils.getPropDefault("summariser.name", "summary");
if (summariserName.length() > 0) {
summer = new Summariser(summariserName);
}
String logFile = "/path/to/your/jmeter/bin/result.jtl";
ResultCollector logger = new ResultCollector(summer);
logger.setFilename(logFile);
testPlanTree.add(testPlanTree.getArray()[0], logger);
jmeter.configure(testPlanTree);
jmeter.run();
JMeterUtils.setProperty(JMETER_REPORT_OUTPUT_DIR_PROPERTY, "/path/to/dashboard");
ReportGenerator generator = new ReportGenerator(logFile, null);
generator.generate();
}
}
Do anyone knows the reason behind? Highly appreciate any suggestions.
Thank You.

Looking into TotalTPSGraphConsumer.java source code
/**
* The class TotalTPSGraphConsumer provides a graph to visualize transactions
* rate per second.
*
* #since 5.0
*/
So make sure to use the same JMeter version for:
Creating the .jmx test plan
For running the test from the Java code, your Project Classpath must use the same versions of the dependencies libraries as the JMeter from the point 1.
As per 9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure article you should always be using the latest version of JMeter so make sure you use JMeter 5.2.1 (or whatever is the latest stable JMeter version which is available at JMeter Downloads page)
In particular to get rid of the above error you need to have ApacheJMeter_core version 5.2.1 in your project classpath.

When running JMeter script from Java functions and properties do not work

I am trying to execute JMeter script from Java code using
String jmeterHome = "D:/tmp/apache-jmeter-3.3";
StandardJMeterEngine jmeter = new StandardJMeterEngine();
JMeterUtils.loadJMeterProperties(jmeterHome + "/jmeter.properties");
JMeterUtils.setJMeterHome(jmeterHome);
JMeterUtils.initLocale();
SaveService.loadProperties();
File script = new File(jmeterHome + "/http_localhost.jmx");
HashTree testPlanTree = SaveService.loadTree(script);
Summariser summer = null;
String summariserName = JMeterUtils.getPropDefault("summariser.name", "summary");
if (summariserName.length() > 0) {
summer = new Summariser(summariserName);
}
String logFile = jmeterHome + "/file.jtl";
ResultCollector logger = new ResultCollector(summer);
logger.setFilename(logFile);
testPlanTree.add(testPlanTree.getArray()[0], logger);
jmeter.configure(testPlanTree);
jmeter.run();
In JMeter GUI in "Thread Group" configuration I am setting "Number Of Threads" to be "${__P(xxx,20)}". It works fine from the GUI - I can execute script with default value of "20". But the code above does not start any threads. Java code prints that it is trying to start zero threads.
I have seen Jmeter functions don't executing when calling from java code and I do have following dependencies in my project
<dependency>
<groupId>org.apache.jmeter</groupId>
<artifactId>ApacheJMeter_java</artifactId>
<version>3.3</version>
</dependency>
<dependency>
<groupId>org.apache.jmeter</groupId>
<artifactId>ApacheJMeter_http</artifactId>
<version>3.3</version>
</dependency>
<dependency>
<groupId>org.apache.jmeter</groupId>
<artifactId>ApacheJMeter_functions</artifactId>
<version>3.3</version>
</dependency>

Your code is wrong, you are not referencing the correct jmeter.properties path.
Working example:
String jmeterHome = "/data/jmeter/jmeters/apache-jmeter-3.3/";
StandardJMeterEngine jmeter = new StandardJMeterEngine();
JMeterUtils.setJMeterHome(jmeterHome);
JMeterUtils.loadJMeterProperties(jmeterHome + "bin/jmeter.properties");
JMeterUtils.initLocale();
SaveService.loadProperties();
File script = new File("/data/jmeter/workspace/JMeterFromJava/scenario/localhost.jmx");
HashTree testPlanTree = SaveService.loadTree(script);
Summariser summer = null;
String summariserName = JMeterUtils.getPropDefault("summariser.name", "summary");
if (summariserName.length() > 0) {
summer = new Summariser(summariserName);
}
String logFile = "/data/jmeter/workspace/JMeterFromJava/results/file.jtl";
ResultCollector logger = new ResultCollector(summer);
logger.setFilename(logFile);
testPlanTree.add(testPlanTree.getArray()[0], logger);
jmeter.configure(testPlanTree);
jmeter.run();
EDIT 29 september 2017:
You submitted to bugzilla a demo project showing the problem. It helped understand your problem.
Status:
- Code works if App is ran as a Main application
- Code works if AppTest is ran from Eclipse
- But code fails when ran through mvn clean install
The failure is due to this code:
https://github.com/apache/jmeter/blob/trunk/src/jorphan/org/apache/jorphan/reflect/ClassFinder.java#L335

Small workaround:
final List<String> classPathUrls = new ArrayList<>();
for (final URL url : URLClassLoader.class.cast(this.getClass().getClassLoader()).getURLs()) {
classPathUrls.add(url.getPath());
}
final String classPath = StringUtils.join(classPathUrls, ";");
JMeterUtils.setProperty("search_paths", classPath);

When Jmeter functions are used in the Java code, Jmeter tries to compare the function related classes from java class path with classes from the 'search_path' (Reads classes from the jars). So Jmeter function works only if required function class is present in both the path (Jmeter has seperate class for each function).
This is why we need to make sure the 'ApacheJMeter_functions' jar added in the pom (which will be added in class path ) and the path to jmeter functions jar is set to 'search_path'.Both should have same version.
You can refer this link for more details

The problem with JMeter is that it wants to have a library in classpath with exact name of "ApacheJMeter_functions.jar" it can't be "ApacheJMeter_functions-3.3.jar" even if the files are binary equal

Using JunitCore for tests in different projects

Background:
I am currently working on a project in eclipse that programatically executes JUnit tests that are pushed to a server.
So far everything works but I would like to know the results of the tests (specifically any failures) so I can push them out to an email. Right now the tests just output to the console but that doesn't seem to give me much output to actually use.
Right now I use the Runtime class to call the tests but that doesn't seem to have the functionality I need for getting results.
I have looked into the JUnitCore class but can't call any tests outside of the current java project.
So my main question would be how can I use JUnitCore to run junit tests in a specific JAR file? Or is there an easier way to approach this problem using a different class?
This is the only thing I've been able to get to work:
RunTests()
{
junitCore = new JUnitCore();
junitCore.run(AllTests.class);
}
But I would like to do something along the lines of this:
RunTests()
{
junitCore = new JUnitCore();
junitCore.run("C:\\$batch\\test\\hil research\\201507071307\\CommsTestRunner\\plugins\\TestSuite\\US35644.class");
}
I would appreciate any suggestions to this problem I am having. I'm an EE and was just introduced to java last month so this has been quite the challenge for me.

JUnitCore expects to read loaded classes, not class files in a JAR. Your real question is likely how to load the JAR (or directory of .class files) so it can be run.
Poke around with URLClassLoader; once you've amended the classpath appropriately, you can get a Class out of findClass and pass it into the JUnitCore methods you've found.

Since the tests might have classes that are also used by your server (but not necessarily at the same version) I would suggest not having your server directly run the tests. Instead, you can have your server start a new JVM that runs the tests. This is how IDEs like Eclipse run tests. You simply need to write a main class that has JUnit run the tests, and serializes the results on disk.
Your main class would look something like this:
public class MyRunner {
public static void main(String... args) throws IOException {
String path = System.getProperty("resultPath");
if (path == null) {
throw new NullPointerException("must specify resultPath property");
}
// Possibly install a security manager to prevent calls to System.exit()
Result result = new JUnitCore().runMain(new RealSystem(), args);
try (ObjectOutputStream out = new ObjectOutputStream(new FileOutputStream(path)) {
out.writeObject(result);
}
System.exit(result.wasSuccessful() ? 0 : 1);
}
}
Then your server simply needs to construct a java command line with the jars that include the tests, the JUnit jar file, and a jar that contains MyRunner.

Remotely running automated test AutoitX cannot move the mouse or send keys

I have a weird specific question.
In my testing framework I have done the following:
1) Using Java and Ant/TestNG to run the tests.
2) Using Selenium to actually run the browser GUI Automation
3) Using AutoitX4Java to able to automate basic mouse movement clicks and send keys
4) Using Windows operating system.
5) Kick off the automated test with an "ant run" command. My build.xml file will compile the java source code and run the test.
So I am actually able to run my automation code successfully. When I am physically in front of my windows machine, and logged in as the local system user. I notice my Java test with AutoitX4Java can move the mouse around and click.
But the problem is when I use another Windows machine and remote connect in using "psexec" or since I have installed ssh on my windows machine, I ssh it runs the Java program and compiles it and runs the test. But it can't move the mouse around. It is almost as if AutoitX4Java just stopped working.
There are time pauses I notice when it runs those lines of code that requires AutoitX4Java. So I know it is doing something. Just appears to not be viewable on the screen (I cannot see mouse movements, right clicks, or send keystrokes) Despite everything else is still viewable (selenium web browser actions.)
I have a monitor that is connected to the physical machine, so I can view what is going on, when I remote in from the other machine.
Here is the Java code I used for the test:
package installFFExtension;
import java.lang.*;
import org.testng.annotations.Test;
import org.testng.annotations.*;
import com.thoughtworks.selenium.DefaultSelenium;
import com.thoughtworks.selenium.SeleneseTestBase;
import com.thoughtworks.selenium.Selenium;
//imports for AutoIT
import autoitx4java.AutoItX;
import com.jacob.com.LibraryLoader;
import java.io.File;
public class firefox extends SeleneseTestBase {
public Selenium selenium;
#BeforeTest
public void beforeMethod() {
selenium = new DefaultSelenium("localhost", 4444, "firefox", "https://mytestsite.com");
selenium.start();
}
#Test
public void extensionInstallation(){
//this part initializes the AutoIT integration into this Selenium Java test
File file = new File("lib", "jacob-1.17-x86.dll"); //path to the jacob dll
System.setProperty(LibraryLoader.JACOB_DLL_PATH, file.getAbsolutePath());
AutoItX x = new AutoItX();
try{
//dashboard log in
selenium.windowMaximize();
selenium.open("/login/");
selenium.click("id=email");
selenium.type("id=email", "testuser");
selenium.type("id=pass", "testpassword");
selenium.click("id=submitInfo");
selenium.waitForPageToLoad("30000");
selenium.click("link=Install Page");
selenium.waitForPageToLoad("30000");
selenium.click("Install");
selenium.waitForPageToLoad("30000");
selenium.click("id=firefox");
// need to slow selenium to allow page load adding a sleep timer
Thread.sleep(1000);
x.mouseClick("left", 140, 408, 1, 15);
x.send("installer");
selenium.type("id=name", "installer");
x.mouseClick("left", 140, 469, 1, 15);
x.send("password");
selenium.type("id=password", "password");
selenium.click("Install");
x.mouseClick("left", 222, 372, 1, 15);//this was added to make install bar to go away in Firefox.
x.mouseClick("left", 150, 523, 1, 15);
}
catch(Exception e){
System.err.println("Exception Caught: "+e);
}
}
#AfterTest
public void afterMethod() {
selenium.stop();
// selenium.shutDownSeleniumServer();
}
}

AutoIT will only work when the browser is open on the same machine as the test is running.
AutoIT cannot be used when running distributed testing although I have heard claims otherwise using a compiled exe on the remote PC, it sounds complex and unstable.
The question may be is to investigate why you need to use AutoIT at all. What are you trying to achieve which cannot be done via Selenium? What are you trying to test?