Selenium View Mouse/Pointer - java

Is there any way to actually see the selenium mouse when it's running the tests? Either with a windows cursor image or some kind of dot or cross hair or anything at all!
I'm trying to get a drag and drop function working with selenium and java in an HTML5 web app, and being able to see the cursor to see what it's actually doing would be really useful...

In the end I had to use the Java robot to get this working. Not only to see the mouse, but also because for an HTML5 Web App dragging and dropping is broken in selenium as two movements are needed to the drag and drop to register. Selenium only does one.
My method drags from the centre of each object and allows for an offset if you want to drag past the element you're dragging to.
public void dragAndDropElement(WebElement dragFrom, WebElement dragTo, int xOffset) throws Exception {
//Setup robot
Robot robot = new Robot();
robot.setAutoDelay(50);
//Fullscreen page so selenium coordinates are same as robot coordinates
robot.keyPress(KeyEvent.VK_F11);
Thread.sleep(2000);
//Get size of elements
Dimension fromSize = dragFrom.getSize();
Dimension toSize = dragTo.getSize();
//Get centre distance
int xCentreFrom = fromSize.width / 2;
int yCentreFrom = fromSize.height / 2;
int xCentreTo = toSize.width / 2;
int yCentreTo = toSize.height / 2;
//Get x and y of WebElement to drag to
Point toLocation = dragTo.getLocation();
Point fromLocation = dragFrom.getLocation();
//Make Mouse coordinate centre of element and account for offset
toLocation.x += xOffset + xCentreTo;
toLocation.y += yCentreTo;
fromLocation.x += xCentreFrom;
fromLocation.y += yCentreFrom;
//Move mouse to drag from location
robot.mouseMove(fromLocation.x, fromLocation.y);
//Click and drag
robot.mousePress(InputEvent.BUTTON1_MASK);
//Drag events require more than one movement to register
//Just appearing at destination doesn't work so move halfway first
robot.mouseMove(((toLocation.x - fromLocation.x) / 2) + fromLocation.x, ((toLocation.y - fromLocation.y) / 2) + fromLocation.y);
//Move to final position
robot.mouseMove(toLocation.x, toLocation.y);
//Drop
robot.mouseRelease(InputEvent.BUTTON1_MASK);
}

You can use Selenium "dragAndDrop" and "dragAndDropToObject" commands to drag and drop an element.
"mouseDown", "mouseMoveAt" and "mouseUp" commands are also very good alternatives.
Here is very good examples of both ways in selenium IDE. You can convert that code in to java to use.

Related

What is the X and Y axis format on an iPhone screen/Android screen?

I've been researching the X and Y axis set up on an iPhone screen because I'm trying to use Appium to run automated tests on an application. Part of what I'm working on is swiping but I'm running into issues with moving points, X and Y points.
I don't know what the limits of the X and Y axis are on the iPhone screen or the Android screen. Due to that I'm not sure sure where my cursor is swiping from or to.
So far I've been able to do one swipe via the code below,
TouchAction action1 = new TouchAction((PerformsTouchActions) driver);
action1.press(PointOption.point(250,200))
.waitAction(new WaitOptions().withDuration(Duration.ofMillis(250))) //you can change wait durations as per your requirement
.moveTo(PointOption.point(50, 250))
.release()
.perform();
Thread.sleep(5000);
This runs one successful swipe to the left but the 2nd swipe, which runs successfully, but doesn't swipe the screen to the left. As in, the swipes it, but it doesn't move it past the required threshold to move it to the next page.
Let it be known that the above code is placed within a for loop as shown below.
for(int i = 0; i <= 3; i++){
TouchAction action1 = new TouchAction((PerformsTouchActions) driver);
action1.press(PointOption.point(250,200))
.waitAction(new WaitOptions().withDuration(Duration.ofMillis(250))) //you can change wait durations as per your requirement
.moveTo(PointOption.point(50, 250))
.release()
.perform();
Thread.sleep(5000);
}
Is there any documentation for iPhones or Androids that shows the set up of the X and Y axis, the limitations and so on etc
Edit:
The first piece of code is placed outside of the for loop. I believe that was the piece of code that swiped the screen properly. Once the for loop is entered, the screen is unable to swipe properly. I believe I'm not using the correct syntax in the for loop.
If anyone has any advice on how to properly incorporate the swipe feature into the for that would be of great help.
X, Y coordinate depend on the device/emulator you are using for
automation if device has large display X,Y value will be high.
I will recommend never use hardcoded value of X, Y during swipe or scrolling with the help of coordinate as it can throw exception points are out of device whenever you will be shifted on small screen Device .
Please have a look how you can get dynamic X,Y value with the help of Device’s height and width while scrolling to bottom
public void scrollToBottom() {
// we are scrolling to bottom so X will be constant so we are taking mid point in width.
int x = driver.manage().window().getSize().width / 2;
// starting of Y is from 20% of height as we have one bar in all device for showing network and battery status
int start_y = (int) (driver.manage().window().getSize().height * 0.2);
// end of Y is 80% of height
int end_y = (int) (driver.manage().window().getSize().height * 0.8);
// here scrolling length is (80% -20%) 60%.
TouchAction dragNDrop = new TouchAction(driver)
.press(PointOption.point(x,start_y)).waitAction(WaitOptions.waitOptions(Duration.ofMillis(500)))
.moveTo(PointOption.point(x, end_y))
.release();
dragNDrop.perform();
}
If you want to get X,Y coordinate of any Mobile Element you can get as mentioned below
MobileElement element = (MobileElement) driver.findElementByAccessibilityId("SomeAccessibilityID");
Point location = element.getLocation();
OR
int leftX = element.getLocation().getX();
int rightX = leftX + element.getSize().getWidth();
int middleX = (rightX + leftX) / 2;
int upperY = element.getLocation().getY();
int lowerY = upperY+element.getSize().getHeight();
int middleY = (upperY + lowerY) / 2;

How to scroll screen in android native app having recyclerview using appium

I am trying to automate an app which is built using recyclerview. There are totally 10 tiles and in one screen 1st four tiles will be visible and to get other tiles I need to move the screen upward. I tried to move the screen by finding co-ordinates and "(AndroidElement)driver.findElement(MobileBy.AndroidUIAutomator("new UiScrollable(new UiSelector().resourceIdMatches(\".*id/type_text\")).setMaxSearchSwipes(5).scrollIntoView("new UiSelector().text(\"" + text + "\"))"))" this but there is slightly movement in the screen and couldn't get the remaining tiles. Is there any way to scroll to the bottom of the screen so that I can get last tile also.
Please try this. I think there's some issue in your UiScrollable..
MobileElement listItem=(MobileElement)driver.findElement(MobileBy.AndroidUIAutomator("new UiScrollable(new UiSelector()"
+ ".scrollable(true)).scrollIntoView("
+ "new UiSelector().text(\"<Mention your element text value here>\"))"));
#Sammar Ahmad, yes you were right. I was using wrong element. I constantly tried and finally worked after using co-ordinates.Code looks something like below. Created scroll() in homepage class and called the same from my test class
public void scroll(int x, int y) {
int startY = (int) (driver.manage().window().getSize().getHeight() * 0.90);
int endY = (int) (driver.manage().window().getSize().getHeight() * 0.10);
TouchAction action = new TouchAction(driver);
action.press(point(x, startY)).waitAction(waitOptions(ofSeconds(3))).moveTo(point(x, endY)).release().perform();
}
MobileElement startElement = (MobileElement) driver.findElementById("mention parent element here");
Point location = startElement.getLocation();
homepage.scroll(location.x,location.y);
MobileElement listItem=(MobileElement)driver.findElement(MobileBy.AndroidUIAutomator("new UiScrollable(new UiSelector()).scrollIntoView(text(\"<Mention your element text value here>\"))"));
You can modify #sammar's code like this to scroll to the element.

How to Automate Maps with Selenium Java? or with Page Co-Ordinates. Pleas show me is there any other way to Automate Maps

I'm Trying to capture the Co-ordinates of Maps to do some action on Maps.
wait = new WebDriverWait(driver, 15);
wait.until(ExpectedConditions.elementToBeClickable(By.`path`("//button[contains(text(),'Add Tract')]"))).click();
Utils.scrollUp();
Thread.sleep(10000);
wait.until(ExpectedConditions.visibilityOfElementLocated(By.id("TimeZoneId")));
//Timezone is the area which I'm trying to capture the Co-Ordinates
Point point1 = timezone.getLocation();
SOP("Element's Position from left side is: "+point1.getX()+" pixels.");
SOP("Element's Position from top is: "+point1.getY()+" pixels.");
}
if your map has <canvas> tag, then try
To perform using Actions chains, below is an example C# code similar to Java
IWebElement canvas = driver.FindElement(By.Id("TimeZoneId"));
int xCo = canvas.Location.X;
int yCo = canvas.Location.Y;Actions action = new Actions(driver);
action.MoveToElement(canvas, 1 + xCo, 2 + yCo).Click().Build().Perform();
Try OpenCV
If you are testing overlays on map. use JavaScriptExecutor and by adding hooks to your code in order to perform actions in your Map.
Try with Sikuli (personally, I have not used this. Needs some research)

Javafx limiting drag within window

I am making an app that has a draggable background, everything is work correctly, I can drag, but I have noticed that, I can drag past my window and my scene will re-structure and allow me to drag as far as I want in a given direction, here is a snippet to show what I mean Normal before dragging
After dragging to far bottom right As noticed the scene has allowed the imageview to go as far as the mouse takes it, how can i limit it to stay within the window?
Here is my code
this.imgAdvertising.setOnMousePressed(e -> {
dragDelta.x = this.imgAdvertising.getLayoutX() - e.getSceneX();
dragDelta.y = this.imgAdvertising.getLayoutY() - e.getSceneY();
});
this.imgAdvertising.setOnMouseDragged(e -> {
double x = e.getSceneX() + dragDelta.x;
double y = e.getSceneY() + dragDelta.y;
this.imgAdvertising.setLayoutX(e.getSceneX() + dragDelta.x);
this.imgAdvertising.setLayoutY(e.getSceneY() + dragDelta.y);
});

Java Robot class mouse move to position of a specific pixel (Mouse click a color)

How to find the position of a color that is changing coordinates and needs to be click after being identified.
Purpose of the program complete tasks in a game, requiring the clicking of different colors which aren't always in the same position.
Code currently gets color of mouse's coordinates after 5 seconds of executing program
public class RobotColorClick
{
public RobotColorClick () throws AWTException, IOException, InterruptedException
{
Robot robot = new Robot();
//Delay 5 seconds
robot.delay(5000);
//Gets color (value of red,green,blue) from the mouse position after 5 seconds
Color color = robot.getPixelColor( MouseInfo.getPointerInfo().getLocation().x
, MouseInfo.getPointerInfo().getLocation().y);
//Delay 3 seconds
robot.delay(3000);
//Mouse moves to X and Y then right click
//Problem! How to set X and Y to position color coordinates, position will change
robot.mouseMove(x, y);
robot.mousePress(InputEvent.BUTTON1_MASK);
robot.mouseRelease(InputEvent.BUTTON1_MASK);
}
public static void main(String[] args) throws AWTException, IOException,
InterruptedException
{
new RobotColorClick ();
}
}
if the color is taking continues path and not jumping around read Kevin Mangold answer otherwise if it is just a color appearing anywhere you I think you have 2 options(in case of the background being constant color):
The First one: you can take screen shot iterate over it and get coordinates of any appearing color (or a specific color)then press on it using Robot lib,this may help and this for taking screenshots
The second one: if you don't want to take screenshots you can use the robot lib to iterate over your screen with a 2 nested for loops iterating over all screen pixels THis may help
In case of background image is not constant you can take screenshot of the screen and compare it with the previous one ,use robot lib to press on the difference.
Extra I have read somewhere before that when pressing on a button using robot lib ,it is better to do this
robot.mousePress(InputEvent.BUTTON1_MASK);
robot.delay(1);
robot.mouseRelease(InputEvent.BUTTON1_MASK);
instead of this
robot.mousePress(InputEvent.BUTTON1_MASK);
robot.mouseRelease(InputEvent.BUTTON1_MASK);
Extra thing you can read the game memory from you software and get the color coordinates you want
You'll most likely need to take a screen shot image then spiral out (assuming the "color" is taking a continuous path and not jumping around) from the original location comparing the color of that pixel with the color you are looking for. Once that has been identified, do mouseMove(newX, newY) and then the mousePress()/mouseRelease() methods.

Categories