We Keep Coding

TestNG - Getting start time of before method

I'm executing a few hundred tests in test classes consisting of a singular beforeMethod test, followed by a variable amount of primary tests and occasionally an afterMethod.
The purpose of the beforMethod test, is to populate the test environment with data used in the primary tests while separating logging and recording from the primary tests, which we report on.
We have set up an automatic issue creation tool using a listener. We've found that it would give great value to add execution time to this tool, so that it can show us how long it would take to reproduce the errors in said issues.
To this end, I have made a simple addition to this code, that uses ITestResult.getEndMillis() and getStartMillis() to get the execution time.
The problem we're experiencing with this approach, is that if the test encounters a failure during the primary tests, ITestResult.getStartMillis() will not account for the start time of the before method, but only the primary method.
How would we go about determining the start time of the test class itself (always the beforeMethod), rather than just the current method?
Since we're running hundreds of tests in a massive setup, a solution that allows this without changing each separate test class, would definitely be preferable.
The setup of the java test classes look something like this (scrubbed of business specifics):
package foobar;
import foobar
#UsingTunnel
#Test
public class FLOWNAME_TESTNAME extends TestBase {
private final Value<String> parameter;
public FLOWNAME_TESTNAME(Value<String> parameter) {
super(PropertyProviderImpl.get());
this.parameter = parameter;
}
#StoryCreating(test = "TESTNAME")
#BeforeMethod
public void CONDITIONS() throws Throwable {
new TESTNAME_CONDITIONS(parameter).executeTest();
}
#TestCoverage(test = "TESTNAME")
public void PRIMARYTESTS() throws Throwable {
TESTCASE1 testcase1 = new TESTCASE1(parameter.get());
testcase1.executeTest();
testcase1.throwSoftAsserts();
TESTCASE2 testcase2 = new TESTCASE2(parameter.get());
testcase2.executeTest();
testcase2.throwSoftAsserts();
}
}
So in this case, the problem arises when the listener detects a failure in either TESTCASE1 or TESTCASE2, because these will not include the execution time of TESTNAME_CONDITIONS because that test is inside a different method, yet practically speaking, they are part of the same test flow, aka the same test class.

I found a solution to the issue.
It is possible to use ITestResult.getTestContext().getStartDate().getTime() to obtain the time of which the test class itself is run, rather than the current test method.
The final solution was quite simply:
result.getEndMillis() - result.getTestContext().getStartDate().getTime()) / 60000
Where "result" corresponds to ITestResult.
This outputs the time between the start of the test and the end of the last executed method.

Pass values between TestNG tests

Background: I'm executing tests with TestNG and I have a class annotated with #Test that generates a number, or ID if you will, and that same number is the input value of my second test. Is it possible to pass values between TestNG tests?

Sure. For example if you have two tests that is related you can pass the values from one test to another via test context attributes:
#Test
public void test1(ITestContext context) { //Will be injected by testNG
/* Do the test here */
context.setAttribute("myOwnAttribute", "someTestResult");
}
#Test(dependsOnMethods = "test1")
public void test2(ITestContext context) { //Will be injected by testNG
String prevResult = (String) context.getAttribute("myOwnAttribute");
}

You should create one test that handles whole case. Tests can't depend on each other, it's considered as bad practise. If you are using maven order of tests execution can be different in different environments.

Bad practice or not, it can be accomplished by simply using class fields. Just make sure your cases are executed in predictable order (eg. using #Test(priority) or dependsOn TestNG feature).

How to run categorized tests with JUnitCore?

I need to run certain tests depending using JUnitCore and Categories but I can't find a way to make it work, can you please take a look and let me know if this is valid?
I have the following TestSuite called:
#RunWith(Categories.class)
#IncludeCategory(FeatureA.class) //this is the interface required as per categories documenation
#SuiteClasses( { AllTests.class } ) //imagine that AllTests contains all my tests
public class FeatureASuite {
} //if I'm not mistaken this configuration
// will go over all my tests and
// pick up only the ones with category FeatureA
And then I have a main class that will handle the execution as follows:
public static void main(String[] args) {
List<Class<?>> classes = new ArrayList<Class<?>>(); //classes collection
boolean featureA= true; //as this is an example featureA is always enabled
if(featureA) { //if feature A enabled then..
classes.add(FeatureASuite.class); //...add the feature A suite.
}
JUnitCore jUnitCore = new JUnitCore(); //create the facade
jUnitCore.runClasses(classes.toArray(new Class[classes.size()])); //run the classes specified
}
After executing the code the tests are not run. I have tried this with a different runner (instead of using Categories.class I have tried Suite.class) and tests are executed, however I need to specify categories per test method and Suite.class is not hitting that mark.

I have found why my approach was not working, the implementation above is actually correct, the issue (what I consider a junit bug) is in how Junit reacts to RunWith, if any of the classes under SuiteClasses contains RunWith annotation for any reason the execution will stop before even starting to run a first test.

Junit test can't pass all test case once

I have a very strange problem, when i try to run a JUnit test with multiple test case, it will only pass the first test case and shown IndexOut of Bound error
public class ABCTest {
#Test
public void basicTest1(){...}
#Test
public void basicTest2(){...}
...
but if i commend the rest test case, test them one by one, it will pass all of them.
public class ABCTest {
#Test
public void basicTest1(){...}
//#Test
//public void basicTest2(){...}
//...

Since you do not provide the complete testcase and implementation class, I have to make some assumptions.
Most likely you are mutating the state of the tested object by the testcase.
Usually you try to get a clean test fixture for each unit test. This works by having a method with the #Before annotation which creates a new instance of the class under test. (This was called 'setUp()' in older versions of junit.)
This ensures that the order of test method execution as well as the number of executions does not matter and each method is working isolated.

Look at what you are doing inside of the test case and see if you are changing data that may be used by the other test cases and not restoring it to the original state. For example you have a text file that you read and write to in basicTest1 that you then read again in basicTest2 but assume the file is the same as it was before you ran basicTest1.
This is just one possible problem. would need to see the code for more insight

Junit test that creates other tests

Normally I would have one junit test that shows up in my integration server of choice as one test that passes or fails (in this case I use teamcity). What I need for this specific test is the ability to loop through a directory structure testing that our data files can all be parsed without throwing an exception.
Because we have 30,000+ files that that 1-5 seconds each to parse this test will be run in its own suite. The problem is that I need a way to have one piece of code run as one junit test per file so that if 12 files out of 30,000 files fail I can see which 12 failed not just that one failed, threw a runtimeexception and stopped the test.
I realize that this is not a true "unit" test way of doing things but this simulation is very important to make sure that our content providers are kept in check and do not check in invalid files.
Any suggestions?

I think what you want is parameterized tests. It's available if you're using JUnit4 (or TestNG). Since you mention JUnit, you'll want to look at the #RunWith(Parameterized.class)
and #Parameters annotations' documentation.

I'd write one test that read all the files, either in a loop or some other means, and collected all the failed files in a collection of some kind for reporting.
Maybe a better solution would be a TestNG test with a DataProvider to pass along the list of file paths to read. TestNG will create and run one test for each file path parameter passed in.

A Junit3 answer: Create a TestSuite, that creates the instances of the TestCases that you need, with each TestCase initialized according to your dynamic data. The suite will run as a whole within a single JVM instance, but the individual TestCases are independent of each other (setUp, tearDown get called, the error handling is correct, reporting gives what you asked for, etc).
The actual implementation can be a bit clumsy, because TestCase conflates the Name of the test with the METHOD to be run, but that can be worked around.
We normally just combine the suite with the dynamic testcases in the same class, and use the suite() method to get the TestSuite. Ant's JUnit task is smart enough to notice this, for example.
public class DynamicTest extends TestCase {
String filename ;
public DynamicTest ( String crntFile ) {
super("testMethod");
filename = crntFile ;
}
// This is gross, but necessary if you want to be able to
// distinguish which test failed - otherwise they all share
// the name DynamicTest.testMethod.
public String getName() {
return this.getClass().getName() + " : " + filename ;
}
// Here's the actual test
public void testMethod() {
File f = new File( filename ) ;
assertTrue( f.exists() ) ;
}
// Here's the magic
public static TestSuite suite() {
TestSuite s = new TestSuite() ;
for ( String crntFile : getListOfFiles() ) {
s.addTest( new DynamicTest(crntFile ) ) ;
}
return s ;
}
}
You can, of course, separate the TestSuite from the TestCase if you prefer. The TestCase doesn't hold up well stand alone, though, so you'll need to have some care with your naming conventions if your tests are being auto-detected.