I'm currently using Selenium for testing our web application's interface. However, it isn't very reliable (it's hard to set a good waiting timeout, absolutely a nightmare to work with in any webpages involving frames) and lacks many features (popups/downloads).
I took a look at http://sikuli.csail.mit.edu/ and quite like it, but again, it isn't very reliable as in very often it fails to identify the correct buttons/links.
What is a good approach for testing web applications?
PS: I'm after the ones that would allow for testing even if the tester has ZERO knowledge about the internal of the codes (not even the method signatures).
Unfortunately, UI testing is not very reliable in general. Having to use XPath, because IDs aren't set on elements on the page. Having to use frames (I hate frames and glad its being deprecated in HTML5).
It's the nature of the beast unfortunately and a major issue I find with Selenium, which is my prefered UI testing tool for browsers, is that people start on the Selenium IDE which tries to record everything you do, but it can only do so much making tests unreliable.
Record/Replay tools should be called Record/Tweak/Replay, so that it can handle tests correctly.
Writing your tests for Selenium RC or WebDriver (these are currently being merged to create Selenium 2) allows you to handle the issues that you may see in Selenium IDE. It makes you write tests using programming languages like Java, Python, C# and many more. Since you can add conditionals to your tests.
The way that I have got less technical people to use Selenium is to create a DSL for them to use, so that it is a more natural language when writing tests.
Have a look at Sahi.
It does not need to wait for Ajax or page loads.
It does not use XPaths, using DOM relations (_near, _in) instead.
It traverses frames/iframes automatically
The recorder works for frames, iframes, and popup windows
It handles auto generated IDs well
The recorder and playback works on almost all browsers and operating systems.
It does not use special privileges to run tests.
Over the last year, Sahi has had seven public releases and the support is prompt on the forums.
Disclaimer: I am the author of Sahi.
Instead of using selenium.waitForPageToLoad("30000"); use like this
selenium.waitForPageToLoad(Timeout);
and don't forget to set the timeout using settimeout option
selenium.setTimeout(Timeout);
I too faced the same issues with the timeout. After setting the timeout, things worked well.
Related
I'm looking for a way to test the different functionality of my JavaEE server application. I first tried to do it with Junit and TestNG, but building the different scenario was too tedious.
What i'm searching is a tools that will simulate an user using my server application trough his browser, and then once this scenario is established i could check the output and verify if it's working.
Here an exemple:
An user connects to the server, the server application interface is displayed.
He executes the functionality to modify his personal datas
He modifies the different that he want to change.
He saves the modifications.
Then what i want to check is the actual state of the data in the database.
What i need is something that will simulate the action he did with his browser like i said above...
I've read different article about different tools that could do the works but i'm not sure because i don't really know what to type in google.
I discovered Jmeter (that is not working with my application because of web socket) and the Grinder.
The Grinder seems to be interesting but most of the things i've read about refers to it as load testing tools, which is not what i'm looking for.
Can someone experienced tells me if i can do what i want with the Grinder ?
You can use scripting tools like: Sikuli(Record Playback & Scripting) or Automa (Component Identification Scripting)
The most popular tool currently is Selenium. It will certainly do the job. I would also mention Geb because it provides more convenient API when compared to Web Driver.
I'll also let myself to give an advice, although it doesn't relate directly to your question.
If your project is big enough (4-5 teams over a couple of years can deliver a lot of code), you should think what to automate.
These tests can be very heavy both in terms of CPU load and in terms of time.
So if you'll rely ultimately on these tests, your build will run ages and will be potentially unstable.
So these tools should be used only to complement unit/component and integration testing that should exist anyway and they will use completely different tools.
Also in UI, consider using various testing techniques that would test only the UI side (mock the server endpoints and so on).
Folks - I am looking for an automation tool to automate windows 8 metro application and an iOS application for functional automation
I have researched some tools and found about
Ranorex UI automation tool (for metro and iOS applications)
UI automation (by Microsoft) (for metro apps)
UI Instruments (for iOS application)
The real problem I am facing is with all these UI automation tools, if there is a small change in the UI, the scripts are failing ......
I want to test the functionality first and need more robust scripts/tools to handle UI changes.
How can I identify metro app objects uniquely? I see that the UI code is XAML
Also, I have the source code of the app available and don't mind using variables or resources from it in my automation code....
Can anyone please suggest any good tools or How I can handle this?
My manager wants a real customer experience !! Also, would be great if scripting can be done using Java, Javascript
No record and playblack tools which identify based on the static text please !!
This is unfortunately one of the pit falls of front end automation. The reality is a small change can make a very large impact on your automated scripts. This is why most people suggest that you do not spend an inordinate amount of time on the front end and stick to unit/integration tests, and only automate ui functionality if you have no other choice.
I can't speak as to UI automation, or UI Instruments, but as far as Ranorex is concerned to make a more robust test the key is the xPath that Ranorex uses to track your elements. The obvious answer is to track on the objects unique id, but this isn't always feasible. What you can do is find an object above it that will be there, for instance instead of finding a specific table element, find the table, and programmatically create the adapters for the element on that table, then find the element you are looking for from one of it's properties, inner text/tag value/any available property. You will still run into situations where a ui change will cause script failures, that is unfortunately part of the process of front end automation.
I don't personally recommend using your source code for your front end testing, it can cause unrealistic scenarios that can actually cause more problems than they actually help find. The true answer here is to think of your front end automation as a last resort for testing, and if you can test something in another way to use that other way.
Given a site with an existing extensive set of test cases written in Java using Selenium for a site, and this site is being re-written in AngularJs, how does one go about integrating the Selenium tests into the new app?
I have looked into Protractor, the new recommended end-to-end test runner that uses Selenium, however, this appears to suggest writing test in Javascript. This article goes into a little more detail about how to get this set up working, but again, the tests are written in Javascript.
As much as I would like to write the tests in Javascript, I would like to avoid a rewrite, so a means to get the existing ones integrated would be nice. Is there a configuration in Protractor that would enable this to happen?
I am afraid there is no good solution for you here. There are numerous obstacles in your way.
Protractor is written in JavaScript and is a wrapper around Selenium's WebDriverJS implementation. Given that WebDriverJS uses a very different API than the Java WebDriver (due to node's asynchronous nature), even if it is technically possible to get a node library to wrap a Java library, it would not work in this case.
Protractor and Selenium do not support the same API, so even if there were a Java based Protractor, you would find yourself doing a bit of retooling anyway.
If you are leveraging Angular then while the overall behavior of your page may be the same the underlying DOM structure, etc will not be. Either this or you are significantly failing to leverage Angular. Therefore unless you have a really good Page Object abstraction layer in place, you are going to have to rewrite your tests anyway. If you do have a good Page Object layer in place, you will need to rewrite that layer in any event.
In my opinion your best best are as follows:
If your specs themselves are written in a higher level spec language like Cucumber, then those , in theory at least, can be ported to javascript using cucumber-js and you can simply re-implement the underlying definitions (no small task) on top of protractor.
You could get really ambitious and port protractor to Java if it seems like that is less effort than rewriting your tests in javascript. You would then be at liberty to minimize issue #2 above but I still think #3 would lead you to conclude that porting your tests to javascript has the lower LOE.
I am currently using HtmlUnit and Selenium to drive it (WebDriver) within my production code.
I am scaping and interacting with various websites programmatically with these libraries and am having some success and not experiencing memory issues (ensuring sessions are always cleaned up).
I am wondering if these libraries are okay for a production environment or recommended against. This is difficult to find via Google due to the enormous amount of information about automated testing rather than how I am using them.
I realise this is a fairly generic question, but I am seeking advice on these libraries and potentially better alternatives.
WebDriver and Selenium are perfectly suited for production environnement. I use them quite extensively for 2 years now on a multi-machines/multi-datacenters distributed grid and had absolutely no performance nor stability problems we couldn't have coped with.
Our preferred driver is the Firefox one (heavier than HTMLUnit, and harder to configure), and we had to tweak the grid to understand how many instances we can run. Our maximum for stability was 1 per core
Our selenium/webdriver instances have run 24/7 for 2 years now (1 year with selenium 1, and the other migrating selenium 2/ WebDriver incrementally) and with an appropriate monitoring (you should monitor Memory Usage/CPU Usage) and a bunch of load testing, we had reached the good level where we have experienced several monthes without restarting a process
We've used HTMLUnit extensively too, and are equally satisfied with this library
The essential point of my post is : YES, these library are production-ready. But, as all production software, you'll have to benchmark their use to find the appropriate configuration for the optimal stability. I recommend you to use the Selenium Grid in production, which is a great way to parallelize process
I'm using HtmlUnit for something similar in production and have had quite a bit of issues - mostly performance related. Currently I switched to snapshot version of HtmlUnit 2.10 where some important for me performance improvements were implemented (e.g. replacing ArrayList.contains() with HashSet.contains() on DomNode.addDomChangeListener()).
Still, the CPU load is quite high on JavaScript-heavy pages. Typically, I can't run more than 10 of them simultaneously on dual core Linux box. I believe HtmlUnit using Rhino (JavaScript engine) in interpreter mode only, which is pretty slow. Also, you need to be careful with releasing all resources used by HtmlUnit to avoid memory leaks.
All in all, it certainly noticeable that HtmlUnit was designed to run relatively short lived test cases and not long running server applications. It's possible to tweak it enough so it's manageable but certainly it could have been better.
Another approach I found promising is phantom-js, which is headless version of WebKit browser, native app which is much faster on running JavaScript.
Generally, use your testing "gut feeling" about that. What WebDriver and HTMLUnit does is, that it simulates real user performing some actions in the webpage.
My personal gut feeling says, that I should do as less production testing, as possible. So I personally would use these tools only for verification, if my webapp is still alive.
Yes, its generic answer for generic question, but try this:
Gather around people responsible for the webapp and ask them:
Should be it tested on production? (so there is always slight chance, that some customers will see those test data)
If yes, what should be tested on production?
If yes, should it be automated?
And then you have answer ;)
I'm not trying duplicate questions such as this one:
Unit testing framework for a Swing UI
What I'd like to know is, does anyone have any good comparisons for the various Swing Unit testing libraries such as:
WindowTester Pro
FEST
etc...
We've never done any GUI testing so we're not familiar with the gotchas that may lie ahead.
Thanks in advance.
I have had some rather good experience with Abbot and FEST, both open source libraries for Swing UI testing.
Abbot seems not supported anymore; it was a bit hard to get into, because the recorder wasn't generating scripts "good enough". Actually, I've used the recorder to "learn" the script language (XML tags), and I finally wrote the scripts myself directly with a simple text editor. This worked quite well.
FEST takes another approach where you have to code (in Java) your UI tests. That makes it a tool reserved for Java developers, whereas Abbot could be used by other people (e.g. QA Team testers).
The main issues with both tools, and probably with any UI testing tool, are:
to find a way to identify components uniquely without using their position or text content (which can change from one revision to another or makes it difficult to test the same application in a different Locale)
to use correct timing in scripts: those testing tools can run your UI much faster than a human user, thus your UI may not be fast enough for them (e.g. it may take several dozens ms to open a dialog, even more to populate a table from a database)
For both issues, there is a solution though.
For components identification, I strongly advise to name all Swing Components (using Component.setName()) in the UI and use a naming strategy for that, which can ensure that there are never 2 components with the same name visible at the same time. In the guts-gui library, I have even developed a strategy that automatically names Swing components that are stored as fields in panels, this helps adding component names after the application has been coded.
For script timing, both frameworks accept a timeout value while waiting for a dialog to appear; it's up to you to choose the best value, considering the facts that your tests may run on different kinds of machine with more or less available power. You should use a timeout that is large enough to ensure that the script won't report false negatives (e.g. a dialog that appears after 1sec, whereas the script waits for only 500ms), but also not too long so that when there is a real error (e.g. an expected dialog never appears). I suggest to use timeouts ranging from 2 to 5 seconds, that should fit most testing platforms and most applications.
Hope this helps.
Jemmy provides a reasonably good capabilities for UI testing. Although, it wouldn't be a staight out of the box solution for JUnit testing, it could be easily extended to suit your purposes.
I am not sure about other UI testing tools, but when compared with RFT it provides you a handle of the actual UI object (RFT returns a proxy object). This could be handy from my experience.
It is an open-source project (licensed under CDDL) and is actively under development.
I think other popular (or used to be??) was jfcUnit. Though I don't think this is under active development.
There are many factors to consider. Record /Replay, Unit test support, Nature of code change, Licensing, Cost, multi-platform support, Testing with multiple look and feel, support for i18n testing ... what is your list look like?
Some comments on the tools we used.
IBM Rational Functional Tester :
This has ability to record scripts, and play back. It supports verification points. One of the biggest plus point is no code change is required. RFT modifies the JVM and uses java accessibility extensions to record and test. We use it mainly for Java (swing/awt with lot of 2D and dialog operations). It works with Browsers as well.
RFT expose two mechanisms to identify GUI elements. One uses object map. This very weak and has trouble in long term maintainability. Using "find" API is more programmer friendly, though it requires code change. Having all objects with proper name helps too.
Not at all suitable for unit testing.
Works with Windows and Linux.
Very costly floating license is in the range of 12000USD, fixed licenses will cost half of that. All nodes (recording the tests and running the tests) need license. Pricing is approximate and old, but it will give an idea.
Needs real GUI session, on windows. (It may be OK on linux with VNC)
Jemmy:
We shifted to jemmy for new testing. Supports windows, linux. It used to be free, not sure what Oracle's plan on it. We added our test layer on top of jemmy - for assertions and other verification mechanisms. This presentation on 'jemmy testing toolkit' has some more details on jemmy.
In 2012 #AlexRuiz, the main developer behind FEST decided to stop development of FEST
The FEST codebase was later forked into AssertJ-Swing.
Skip FEST and start with AssertJ-Swing.
You may want to try Swing Testing Toolkit. It use a different approach: It does not record all events. Test instructions are reduced to the bare minimum.