Here, we will Focus on Handling Automation using Sikuli tool to Handle Flash-based applications, Windows applications, Desktop applications:
Sikuli follows an image-based technique for interacting with elements on the screen.
Sikuli is a third-party tool that uses image recognition to identify and control UI components.
This is useful when there is no source code/internal code available for the UI/Web Element in the application. The core of Sikuli Script is written in Java. So, we can use Sikuli Script as a standard JAVA library in our project.
=> Take A Look At The Sikuli Beginners Guide Here.
Table of Contents:
Overview Of Sikuli Tool For Handling Automation
Here is a video tutorial:
Features Of Sikuli
- It is an open-source tool like Selenium.
- We can integrate it with Selenium and with other tools as well.
- It can identify elements using images/screenshots. So, we can capture images for automation and can perform our operations on it.
- Supports Flash-based Application Testing.
- Supports Mobile Testing.
- Supports Windows/Desktop Application Testing.
Sikuli Setup For Selenium
- In order to integrate Sikuli with Selenium, we need the jar file. You can download it from this link.
- Create a Java project in Eclipse and add the listed external jars to your project using Build Path.
- Create a Java class inside the project, say SikuliTest.java.
Steps To Automate A Sample Flash Application Using Sikuli
- Take snapshots of all elements to interact.
- Use Class Screen to focus on the screen and perform actions.
- Use Class Pattern to define images captured.
Let us take an example. We are going to automate a Flash game using Sikuli. The above-listed steps are explained in detail below:
#1) Take snapshots of all elements to interact with.
Take Snapshots of all the elements we would be interacting with and maintain in a folder.
Generally, to interact with an element in Selenium, we use attributes like id, className or XPath to identify. But in cases like Flash applications, those properties are not mentioned in the HTML document since the flash application as a whole is considered as an element, as shown in the image.
In our case, the application is GameGirly and we will select a flash game called ‘Audrey’s Mood Swing’ to automate.
So, we will be using images in this case in order to recognize elements to perform actions on them. Since we are using the image recognition technique, so in order to identify elements in the applications, snapshots are required.
Snapshots of all elements that we would be interacting with are saved in a folder as below:
#2) Use Class ‘Screen’ to focus on the screen and perform actions
Screen class is the main class for all the functions provided by Sikuli. The Screen is the class that holds all predefined methods for common actions like click, double-click, providing input to a text box, hover, etc.
Syntax:
Screen screen = new Screen();
Object ‘screen’ – holds the instance of the entire active screen.
Using the object, I can call its methods to perform actions on that current screen.
screen.click(“image.jpg”); – This method is used to click on an element on the screen using the image name as the parameter.
screen.doubleClick(“image.jpg “); – This method is used to double click on an element. It accepts the image name as the parameter.
screen.type(“image.jpg “,”abc”); – This method is used to provide input value to an element. It accepts the image name and text to be sent as parameters.
screen.hover(“image.jpg “); – This method is used to hover over an element. It accepts the image name as the parameter.
screen.exists(“image.jpg”) – This method verifies if the match of the image is present on the screen.
screen.dragDrop (“image1.jpg”,”image2.jpg”) – This method drags and drops the image from one place to another.
#3) Use Class ‘Pattern’ to define the image
Pattern class is used to match the image file with additional attributes to uniquely identify the element on the screen. It takes the path of the image as a parameter.
Syntax:
Pattern playButton = new Pattern(“path of image”);
Object ‘playButton’ holds the match of the image which we would be looking for in the application.
Sample Sikuli Script
Let us automate the Flash-based Game application.
The test steps are to :
- Click on the Play button
- Verify the Mood Meter element is present or not
- Wait for some time and play the advertisement
- Skip the advertisement
- Click on Food icon
- Click on Cereal Bowl
- Click on Cupboard
- Drag and drop the cup to the table
The first action is a click, then our line of code will be as below:
Screen screen = new Screen(); screen.click()
This performs the click action on the currently active screen.
But we are required to click on an element. So, it is necessary to pass the element’s reference in the parenthesis of click () as below:
Pattern playbutton = new Pattern(“path of image”); screen.click(playButton);
This means that the click action is performed on the screen where the image ‘playButton’ is recognized/ a match of that image ‘playButton’ is found
Likewise, each image has to be defined using Pattern class and can be used for various actions.
SikuliTest.java:
public class SikuliTest {
WebDriver driver;
WebDriverWait wait;
String path = System.getProperty("user.dir”);
//refers to current project workspace
@BeforeTest
public void SetDriver(){
System.setProperty("webdriver.chrome.driver",path+"\\Drivers\\chromedriver.exe");
driver = new ChromeDriver();// Object is created - Chrome browser is opened
wait = new WebDriverWait(driver, 60);
//instance of wait is defined
}
@Test
public void playGame() {
driver.get("http://www.gamegirly.com/"); //URL is invoked
WebElement selectGame=
driver.findElement(By.xpath("//div[contains(@class,'game_thumb')]/a[contains
(@href,'Swing')]"));
selectGame.click(); //Game is selected from home page
String Images = path+"\\Images";
//Path where all images are present
Screen screen =new Screen();
//current instance of screen is created
Pattern pattern = new Pattern( Images+"\\PlayButton.PNG" );
//Image is defined
screen.click(pattern);
//Performs click action on the screen where the image match of ‘pattern’
is present
System.out.println("Play button clicked");
Pattern text = new Pattern(Images+"\\Text.PNG");
//image is recognized from path OR image is defined
screen.exists(text);
//performs verfification on screen to check if image match of ‘text’ is present
Pattern PlayAd = new Pattern(Images+"\\PlayAd.PNG");
//Image is defined
screen.wait(PlayAd, 10);
//waits for until image match of ‘PlayAd’ is present. Max wait time is 10
screen.click(PlayAd);
//clicks on the image
Pattern Skip = new Pattern(Images+"\\Ad.PNG");
//Image is defined
screen.wait(Skip, 2);
//Waits until image match of ‘Skip’ is displayed on screen. Max wait time
is 2 seconds
screen.click(Skip);
//Performs click action on the screen where the image match of ‘Skip’
is present
Pattern Food = new Pattern(Images+"\\Food.PNG");
//image is defined
screen.click(Food);
//Performs click action on the screen where the image match of ‘Food’ is
present
Thread.sleep(5000);
System.out.println("Food Option is selected");
Pattern CerealBowl = new Pattern(Images+"\\Cereal Bowl.PNG");
//image is defined
screen.click(CerealBowl);
//Performs click action on the screen where the image match of ‘Cereal
Bowl’ is present
System.out.println("Food Option - CEREAL BOWL is selected");
Pattern Step1 = new Pattern(Images+"\\Step1.PNG");
//image is defined
screen.click(Step1);
Thread.sleep(3000);
System.out.println("Cupboard opened");
Pattern drag1 = new Pattern(Images+"\\drag1.PNG");
Pattern drag2 = new Pattern(Images+"\\drag2.PNG");
//images are defined
screen.dragDrop(drag1, drag2);
//performs drag action on the screen where the image match of ‘drag1’ is
present and drops it on the screen where the image match of ‘drag2’ is
present
System.out.println("Bowl is dragged and dropped");
}
}
Conclusion
Any application can be automated by capturing images and interacting with them. This tool can automate almost anything that you view on your desktop/mobile device.
We hope you enjoyed the tutorial and are ready to try the Sikuli tool.
Happy Reading!!
=> Watch Out The Simple Sikuli Series Here.