i am facing an issue while using sikuli through java, if there are 2 elements of same kind(or similar image) it fails to click on the correct element. so i wanted to know if it is possible to make sikuli just work inside a particular region and can some one please explain how can it be done ??
Yes sikuli can work within a particular region. The challenge is defining a region that only contains one of your two elements. You define a region by x,y coordinates. You can also increase the size of a region based on the location of a unique pattern (image) on your display.
while exists("foo.png"):
hover("bar.png")
ClickMeRegion = find("bar.png").nearby(5).right()
ClickMeRegion.click("baz.png")
So in the above I look for image foo.png/bar.png/baz.png image pairs that are being displayed. First I hover on bar.png so that visually I can see which pair the script is looking at. Then I create a region extending 5 pixels around the center of bar.png and extend this to the right of the display. This highlights a single baz.png image. I can then click on the one baz.png that I am interested in.
For more info on regions see: http://doc.sikuli.org/region.html
Related
I have this picture:
It contains circles inside columns. I want to get:
The height of each circle relative to its column (the distance
between the column bottom and the circle)
The height of the whole column
Classify every column so I can use the data from every column
differently according to its rank from left to right using java in
android studio.
I know I'm asking a lot but is that possible?! and if it is how can I do that?
Thanks in advance
Yes it must be possible. I have used this concept in Python using OpenCV but not in Java. The idea is to recognise your target inside an image by image recognition and then calculating pixel ratios. Use the same logic and code in Java as OpenCV is there for Java too.
Helper link below,
https://www.pyimagesearch.com/2016/03/28/measuring-size-of-objects-in-an-image-with-opencv/
If you have already started any other implementation and obtaining errors, post it too.
I'm developing a program with Java and Sikuli and I want to click on a red image with a specific shape that is located on screen.
The problem is that on the screen there is another image with that same shape but different color, blue.
import org.sikuli.script.Screen;
this.screen.type("C:\\Images\\TestImage.png", "a"); // this is what I'm using.
My mouse keeps moving between the two images because it can't tell the difference in color.
There is no way Sikuli can make the right choice for you. It can only locate a matching are based on your pattern (color in this case). To work around this issue you should provide some reference points that are unique and can be used to "help" Sikuli find the right match. For example, if the pattern you are interested in is located at the left side of the screen, then you can limit the search to the left side of the screen only. Or if you have a unique visual object in the are of interested you can use it as a pivot and look only around it.
On top of that, if you have few similar items appear in some ordered fashion (one under another for example), you can let Sikuli find all of them, calculate their coordinates and select the object you need based on these coordinates.
Searching using color is possible with Brobot, a state-based, testable, automation framework for Java. Brobot wraps Sikuli methods and uses Sikuli variables such as Match in its core functionality. Its color methods depend heavily on matrix operations with OpenCV. The Brobot documentation gives more detailed information on the framework. Additionally, there is a page in the documentation specifically for color searches.
Color Selection Methods
There are 3 methods used to identify which colors to search for.
Color averaging. The average color of all pixels is determined from all image files (a single Brobot image can contain multiple image files). The range of acceptable colors around the target color can be adjusted, as well as the required size of the color region.
K-Means. Images that contain a variety of colors are not suitable to color averaging. The k-Means method from OpenCV finds the salient colors in an image, which are then used in the search. Adjustable options include the number of k-Means, acceptable color range, and required size.
Histogram. Brobot divides images into 5 regions (4 corners + the middle region) in order to preserve some spatial color characteristics (for example, blue sky on the top of an image will not match blue sea on the bottom of an image). The search can be adjusted by color range, as well as the number of histogram bins for hue, saturation, and value.
Chained Find Operations
Each of these color methods can be used independently or combined with the traditional Sikuli pattern searches by chaining find operations together. There are two main flavors of chained find operations:
Nested finds. Matches are searched for inside the matches from the previous find operation. The last find operation is responsible for providing the function’s return value.
Confirmed finds. The matches from the first find operation are filtered by the results of following find operations. The later find operations act as confirmation or rejection of the initial matches. Confirmed matches from the first find operation are returned.
Documentation for nested finds and confirmed finds
Finding the Red Image
For your specific question, it would be best to use a chain of two find operations, set up as a confirmed find. The first operation would be a pattern search (equivalent to findAll in Sikuli) and the second operation, the confirmation operation, would be a color search.
Actions in Brobot are built with:
The action configuration (an ActionOptions variable)
The objects to act on (images, regions, locations, etc.)
Brobot encourages defining states with a collection of images that belong together. The code below assumes you have a state StateWithRedImage that contains the image RedImage. The results are returned in a Matches object, which contains information about the matches and the action performed.
public class RedImageFinder {
private final Action action;
private final StateWithRedImage stateWithRedImage;
public RedImageFinder(Action action, StateWithRedImage stateWithRedImage) {
this.action = action;
this.stateWithRedImage = stateWithRedImage;
}
public Matches find() {
ActionOptions actionOptions = new ActionOptions.Builder()
.setAction(ActionOptions.Action.FIND)
.setFind(ActionOptions.Find.ALL)
.addFind(ActionOptions.Find.COLOR)
.keepLargerMatches(true)
.build();
return action.perform(actionOptions, stateWithRedImage.getRedImage());
}
}
Disclaimer: I'm the developer of Brobot. It's free and open source.
Here's something that might help. Create a Region and try to find the image in that region as in example in the link
http://seleniumqg.blogspot.com/2017/06/findfailed-exception-in-sikuili-script.html?m=1
I am trying to use the Java Robot class to create a bot to automate some tedious tasks for me, I have never used the Robot class. I have looked up the Class in the Java docs, usage seems straightforward but I have an issue of finding a certain image(I say image, I mean a certain part of the screen) effectively. Is there any other way other than loading 'x' ammount of pixels, checking them, checking the next ammount etc until I find the image I am looking for? Also is there any list of the Button and MouseButton identifiers needed for the Java Robot class as I cna not find any.
For the mouse button identifiers, you are supposed to use BUTTON1_MASK and other button mask constants from java.awt.event.MouseEvent. For example, to click the mouse you would do something like:
Robot r = new Robot();
r.mousePress(MouseEvent.BUTTON1_MASK);
r.mouseRelease(MouseEvent.BUTTON1_MASK);
I believe BUTTON1_MASK is the left mouse button, BUTTON2_MASK is the middle mouse button, and BUTTON3_MASK is the right mouse button, but it has been a month or so since I have used Robot.
As for the checking for an image, I have no idea how that is normally done. But the way you specified in your question where you just check every group of pixels shouldn't be too computationally expensive because you can get the screen image as an array of primitives, then just access the desired pixel with a bit of math. So when checking the "rectangle" of pixels that you are searching for your image in, only keep checking the pixels as long as the pixels keep matching. The moment you find a pixel that does not match, move onto the next "rectangle" of pixels. The probability that you will find a bunch of pixels that match the image that end up not being the image is extremely low, meaning that each rectangle will only need to check about 5 or fewer pixels on average. Any software that performs this task would have to check every pixel on the screen at least once (unless it makes a few shortcuts/assumptions based on probabilities of image variations occurring), and the algorithm I described would check each pixel about 5 times, so it is not that bad to implement, unless you have a huge image to check.
Hope this helps!
I would like to check the position of the elements on a web application on which I am working on. It's a stable application so the positions won't change so much. After searching on the internet I've found this method:
Point getLocation()
Where on the page is the top left-hand corner of the rendered element?
Returns: A point, containing the location of the top left-hand corner of the element
After playing a little bit with it I realized that the position depends on the window size and screen resolution (though I read it somewhere I still checked it just to make sure it's true).
Is there another way? I have a lot of elements so it will be tricky to just store some values depending on a set of resolutions and window dimensions.That would be a lot of work.
In the end nothing beats manual testing when it comes to visual stuff :D
It has to depend on the window size, because the layout of the page changes with the window size.
What you're asking for is essentially:
Can I get the position of elements on the page as if the window was
infinitely wide and infinitely high so that the result would be
static?
No, sorry, you can't do that, because you're probably the first person wanting to do such thing. The positions you're asking for are meaningless. Also, these numbers will most likely be different across the browsers.
I believe you have two options:
Set a fixed window size for this test. If you'll run the test at 1024*768 px every time, it usually won't break. In this case, your positions shouldn't change and you'll get your results.
Bonus idea: you can get a screenshot of the page and compare the images. If the page's content is more or less static, your screenshot will always be the same, so you can compare it to a reference screenshot.
Don't check the absolute numbers, try to check for example whether a WebElement contains the right text, images (and whether the images have been loaded) or even whether a WebElement is visually contained in another WebElement.
I want to combine two images by overlapping them at some fixed x-axis displacement. Suppose There are two images and both are the left and right snaps of the same panoramic view ,
Now the problem is to find the X -axis displacement of the right
image overlapping the left one so that both pixels match and they
form the panoramic view.
The next issue is how they can be merged so that they feel like a single image without any defect.
I have to do this using android , but please tell me using java /android how this can be done.
The problem is complex. Note that each picture is a projection of word on plane. For each photo you can assume that point of projection is same but plane of projection is different. This means that common parts on each photo are distorted (bended) in different way.
This means that transition between photos is not linear and simple transition is not enough!
here you can find better description of the problem.