We Keep Coding

How should I manage dependencies among test cases [duplicate]

When I run go test, my output:
--- FAIL: TestGETSearchSuccess (0.00s)
Location: drivers_api_test.go:283
Error: Not equal: 200 (expected)
!= 204 (actual)
--- FAIL: TestGETCOSearchSuccess (0.00s)
Location: drivers_api_test.go:391
Error: Not equal: 200 (expected)
!= 204 (actual)
But after I run go test again, all my tests pass.
Tests fail only when I reset my mysql database, and then run go test for the first time.
For every GET request, I do a POST request before to ensure that there is data created in the DB.
Could anyone help me with how to make sure that tests are run sequentially? That is the POST requests are run before the GET requests?

You can't / shouldn't rely on test execution order. The order in which tests are executed is not defined, and with the use of testing flags it is possible to exclude tests from running, so you have no guarantee that they will run at all.
For example the following command will only run tests whose name contains a 'W' letter:
go test -run W
Also note that if some test functions mark themselves eligible for parallel execution using the T.Parallel() method, the go tool will reorder the tests to first run non-parallel tests, and then run parallel tests in parallel under certain circumstances (controlled by test flags like -p). You can see examples of this in this answer: Are tests executed in parallel in Go or one by one?
Tests should be independent from each other. If a test function has prerequisites, that cannot be done/implemented in another test function.
Options to do additional tasks before a test function is run:
You may put it in the test function itself
You may put it in a package init() function, in the _test.go file itself. This will run once before execution of test functions begins.
You may choose to implement a TestMain() function which will be called first and in which you may do additional setup before you call M.Run() to trigger the execution of test functions.
You may mix the above options.
In your case in package init() or TestMain() you should check if your DB is initialized (there are test records inserted), and if not, insert the test records.
Note that starting with Go 1.7, you may use subtests in which you define execution order of subtests. For details see blog post: Using Subtests and Sub-benchmarks, and the package doc of the testing package.

For those who as I am is getting problems because of multiple concurring tests running simultaneously. I found a way to limit the maximum number of test running in parallel:
go test -p 1
With this, your test will run sequentially one by one.
Source

Apart for 3rd party libraries like Convey and Ginkgo, with plain Golang 1.7 you can run tests sequentially. You can read more here
func TestFoo(t *testing.T) {
// <setup code>
t.Run("A=1", func(t *testing.T) { ... })
t.Run("A=2", func(t *testing.T) { ... })
t.Run("B=1", func(t *testing.T) { ... })
// <tear-down code>
}
And you can run them conditionally with:
go test -run '' # Run all tests.
go test -run Foo # Run top-level tests matching "Foo", such as "TestFooBar".
go test -run Foo/A= # For top-level tests matching "Foo", run subtests matching "A=".
go test -run /A=1 # For all top-level tests, run subtests matching "A=1".
So lets say you got an user package from a REST api that you want to test. You need to test the create handler in order to be able to test the login handler. Usually I would have this on the user_test.go
type UserTests struct { Test *testing.T}
func TestRunner(t *testing.T) {
t.Run("A=create", func(t *testing.T) {
test:= UserTests{Test: t}
test.TestCreateRegularUser()
test.TestCreateConfirmedUser()
test.TestCreateMasterUser()
test.TestCreateUserTwice()
})
t.Run("A=login", func(t *testing.T) {
test:= UserTests{Test: t}
test.TestLoginRegularUser()
test.TestLoginConfirmedUser()
test.TestLoginMasterUser()
})
}
Then I can append methods to the UserTest type that wont be executed by the go test command in any _test.go file
func (t *UserTests) TestCreateRegularUser() {
registerRegularUser := util.TableTest{
Method: "POST",
Path: "/iot/users",
Status: http.StatusOK,
Name: "registerRegularUser",
Description: "register Regular User has to return 200",
Body: SerializeUser(RegularUser),
}
response := util.SpinSingleTableTests(t.Test, registerRegularUser)
util.LogIfVerbose(color.BgCyan, "IOT/USERS/TEST", response)
}

The best way to achieve that is to create a TestMain, as presented here.
import (
"testing"
"os"
)
func TestMain(m *testing.M) {
// Do your stuff here
os.Exit(m.Run())
}

It's also possible to synchronize the test using wait groups:
awaitRootElement := sync.WaitGroup{}
awaitRootElement.Add(1)
t.Run("it should create root element", func(t0 *testing.T) {
// do the test which creates root element
awaitRootElement.Done()
})
awaitRootElement.Wait()
t.Run("it should act on root element somehow", func(t0 *testing.T) {
// do tests which depend on root element
})
Note that you should wait before scheduling the tests, since the asynchronous test execution otherwise might deadlock (the test routine is awaiting another test which never gets to run).

Is it possible to run a test over a complete data set passing the complete data set in #dataprovider and report the result of the whole?

I have a testNg class where I have to run the same test over a complete set of data. I could provide each data line in the #dataprovider, however, the result for each data line would be displayed in the report file. In my case, I have to test millions of data. This would cause a very cluttered report. Below is an example.
TestData:
My Test data is a file containing multiple JSON lines where each JSON object should have a name and occupation.
{"name":"Jim", "occupation":"racing"}
{"name":,}
My Test NG class will look like:
public class TestJSon {
#DataProvider("lineProvider")
public Object[][] lineProvider() {
// return JSON lines
}
#Test (dataProvider = "lineProvider")
public void testJsonLines(String name, String occupation) {
// Test name and occupation
}
}
Now the output report will contain:
testJsonLines passed for line with name Jim
testJsonLines failed for line {"name":,}
testJsonLines failed for line {}
testJsonLines failed for line {"name":"Jack","occupation":"}
I would like a report where it is simply printed:
testJSON inputs failed for inputs in lines 2,3 and 4
To summarize, I am trying to make my report group the results of a run over all data ra rather than print each. Is there a way to do this ?

I had a similar issue. We test millions of data - dataprovider approach either leads to outofmemory error or the report is not something which is easy to look at.
I ended up having my own implementation of IReporter which writes the result to an excel sheet which allows me to have filters or do any customization i want. Ofcourse this has to be in conjunction with other listeners (ITestListener/IInvokedMethodListener) to capture the results the way you want it to.
In my case, I am using my own parallelism as well to get over the outofmemory error with so much data - I usually used to hit it at 90000 testdata.

JUnit test case failed

I have a simple test case:
public class FileManagerTest {
String dirPath = “/myDir/”
#Before
public void setUp() {
mFileManager = MyFileManager.getInstance();
}
#Test
private void testPersistFiles() {
System.out.println(“testPersistFiles()…”);
//it deletes old files & persists new files to /myDir/ directory
boolean successful =mFileManager.persistFiles();
Assert.assertTrue(successful);
}
#Test
public void testGetFiles() {
System.out.println(“testGetFiles()…”);
mFileManager.persistFiles();
//I double checked, the persistFiles() works, the files are persisted.
List<File> files = mFileManager.getFilesAtPath(dirPath);
Assert.assertNotNull(files); //Failure here!!!!
}
#Test
public void testGetFilesMap() {
System.out.println(“testGetFilesMap()…”);
mFileManager.persistFiles();
Map<String, File> filesMap = mFileManager.getFilesMapAtPath(dirPath);
Assert.assertNotNull(files);
}
}
The persistFiles() function in FileManager delete all files under /myDir/ then persist files again.
As you see above, I have a System.out.println(…) in each test function. When I run it , I can see all the prints in the following order:
testGetFilesMap()…
testGetFiles()…
testPersistFiles()…
However, test is failed at testGetFiles(). Two things I don't understand:
I don’t understand, it is failed at testGetFiles() why I can still see the print testPersistFiles() which sounds like even it is failed, it doesn't stop running, but continues to run the next test testPersistFiles()? What is happening behind the scene in JUnit test case??
Another thing I don’t understand is why testGetFiles() is failed? I can see log that the persistFiles() has persisted files. Why it got null after that?

I don’t understand, it is failed at testGetFiles() why I can still see the print testPersistFiles() which sounds like even it is failed, i
That is how unit testing works. Each test should be isolated and working using only its set of data. Unit test frameworks run every test so you can see which parts of the system work and which do not, they do not stop on the first failure.

mFileManager.getFilesAtPath(dirPath);
You are not searching the files in the right place
String dirPath = “/myDir/”
Are you sure that this path is ok? with a slash before the directory name?

For each of your tests, JUnit creates a separate instance of that class and runs it. Since you seem to have 3 tests, JUnit will create 3 instances of your class, execute #Before on each of them to initialize state, and then run them.
The order in which they are run is typically the order in which the tests are written but this is not guaranteed.
Now about the print statement - you see that it's the first statement in your test so it will be executed. Then mFileManager.persistFiles(); is executed. For some reason it returns a false and hence the test fails.
As to why it returns false, you can run a local debugger, put a break point at the beginning of that method, single-step and see.

How make JUnit print assertion results

How can I get the results of my JUnit assertions to be printed [to standard output]?
I have some tests like this:
#Test
public void test01()
{
Position p = getPositionAt('a', 1);
assertNotNull("a1 exists", p);
assertNotNull("figure exists a1", p.getFigure());
p = getPositionAt('a', 2);
assertNotNull("exists a2", p);
assertNull("figure exists a2", p.getFigure());
p = getPositionAt('b', 1);
assertNotNull("exists b1", p);
assertNull("figure exists b1", p.getFigure());
}
This is the printed output format I am hoping to get:
a1 exists -success
figure exists a1 -success
exists a2 -success
figure exists a2 -succcess
exists b1 -succcess
figure exists b1 -failed
Is there way to do this using runners and suites? Or does there exist any assertSuccess(), assertFailed() methods?

First, you have two issues not one. When an assertion fails, an AssertionError exception is thrown. This prevents any assertion past this point from being checked. To address this you need to use an ErrorCollector.
Second, I do not believe there is any way built in to JUnit to do this. However, you could implement your own methods that wrap the assertions:
public static void assertNotNull(String description, Object object){
try{
Assert.assertNotNull(description, object);
System.out.println(description + " - passed");
}catch(AssertionError e){
System.out.println(description + " - failed");
throw e;
}
}

All the assertXXX methods have a form that allows for displaying a String on error:
assertNotNull("exists a2", p); // prints "exists a2" if p is null
There is no particular value in printing a message on success.
EDIT
Junit typically provides 2 forms of an assert. To follow the example above, you can test for a null value in 1 of 2 ways:
assertNotNull(p)
or
assertNotNull("my message on failure", p)
The framework will print the error messages with no other effort required by you (it's provided by the framework).
To test for exceptions you would use the following pattern:
try{
someCall();
catch(Exception e){
fail(): // exception shouldn't happen, use assertTrue(true) if it should
}
Again, there are versions of these methods for adding a message
Check the API

One last resort option is to pair each assert with a corresponding System.out.println, though obviously that is less than ideal. Still, it will solve the problem if all else fails.

Existing Answers/Comments here contain enough info to understand how to print something based on JUnit assertions - but they also explain how doing so is probably not what you actually want to do, and is probably missing the point of running unit tests in the first place.
You should be viewing the results of the tests themselves, instead of trying to print something while you don't understand how/where to view test results themselves.
Now then how/where to view results themselves depends on how you are running your tests - you need to understand how you are running your tests, and then research how to view test results according to how you are running them. Here are a few (but not limited to) examples:
Running tests in IntelliJ
Running tests in Eclipse
Running tests on command line
Running tests in Jenkins

WebDriver + TestNG - How to handle test results

I'm quite new to WebDriver and TestNG framework. I've started with a project that does a regression test of an e-commerce website. I'm done with the login and registration and so on. But there is something that I don't quite understand.
Example, I have this easy code that searches for a product.
driver.get(url + "/k/k.aspx");
driver.findElement(By.id("q")).clear();
driver.findElement(By.id("q")).sendKeys("xxxx"); //TODO: Make this dynamic
driver.findElement(By.cssSelector("input.submit")).click();
Now I want to check if xxxx is represented on the page. This can be done with
webdriver.findElement(By.cssSelector("BODY")).getText().matches("^[\\s\\S]*xxxxxx[\\s\\S]*$")
I store this in a Boolean and check if its true or false.
Now to the question, based on this Boolean value I want to say that the test result is success or fail. How can I do that? What triggers a testNG test to fail?

TestNG or any other testing tool decides success or failure of a test based on assertion.
Assert.assertEquals(actualVal, expectedVal);
So if actualVal and expectedVal are same then test will pass else it will fail.
Similarly you will find other assertion options if you using any IDE like Eclipse.

If you want to stop your test execution based on the verification of that text value, then you can use Asserts. However, if you want to log the outcome of the test as a failure and carry on, you should try using soft assertions, which log the verification as passed or failed and continue with the test. Latest Testng comes equipped to handle this - info at Cedric's blog

write this code where your if condition fails
throw new RuntimeException("XXXX not found: ");

u can use throw exception, and each method which will cal this meth should also throw Excetion after method name or you can use try catch. sample:
protected Boolean AssertIsCorrectURL(String exedctedURL) throws Exception {
String errMsg = String.format("Actual URL page: '%s'. Expected URL page: '%s'",
this.driver.getCurrentUrl(), exedctedURL);
throw new Exception(errMsg);
}

You can do this.
boolean result = webdriver.findElement(By.cssSelector("BODY")).getText().matches("^[\s\S]xxxxxx[\s\S]$")
Assert.assertTrue(result);