As always, we try to bring in new things to learn for our readers. Today, let’s explore an interesting GUI automation tool – Sikuli.
“Automate anything you see” using the Sikuli Graphical User Interface (GUI) automation tool – Complete beginner’s guide to quickly set up and start using the Sikuli Script tool with these in-depth Sikuli Tutorials.
Sikuli Automates anything you see on the screen using the image recognition method to identify GUI elements. Sikuli script allows users to automate GUI interaction by using screenshots.
Table of Contents:
GUI Automation Tool – Sikuli
List Of Tutorials In This Sikuli Series
We have divided this series into 3 parts:
Tutorial #1: How It Works, How To Create A Simple Sikuli Project.
Tutorial #2: How Sikuli Can Be Used With Selenium Web Driver To Automate Webpages.
Tutorial #3: Automating Flash Based Applications Using Sikuli Tool
Sikuli GUI Automation Tool
Let’s start with the 1st part of this series.
Sikuli is a tool that automates Graphical User Interfaces (GUI) using the “Visual Image Match” method. In Sikuli, all the web elements should be taken as an image and stored inside the project. Sikuli will trigger GUI interactions based on the image visual match, the image that we have passed as the parameter, along with all methods.
Sikuli can be very useful for automating flash objects (which do not have ID or name). It can be useful in the situation, where we have a stable GUI (i.e. GUI components not changing).
Even Window-based applications can also be automated using Sikuli. Sikuli provides a friendly Sikuli-script.jar that users can easily use together with Selenium WebDriver. We can even automate Adobe Video/Audio player and Flash Games on the website using Sikuli. With simple API, it makes coding easier.
Practical Uses
- Sikuli can automate Flash Objects/Flash Websites.
- It’s useful to automate the Windows application. We can automate what we see on the screen.
- It provides a simple API. i.e. all methods can be accessed using screen class objects.
- Integration with Selenium and all other tools is easy.
- Using Sikuli, we can automate desktop applications.
- Most of the automation testing tools will not support flash-object automation (E.g. Selenium). Sikuli provides extensive support for automating flash objects.
- It uses a powerful “Visual Match” mechanism to automate desktop & flash objects.
Benefits
- Open-source Tool.
- One of the biggest advantages of Sikuli is that it can easily automate flash objects.
- It makes it easy to automate Windows applications.
- When you’re testing an application under development and you don’t know the ID/name of the elements, then you can go with Sikuli. Sikuli checks the appearance of the image and interacts with it accordingly if a match is found.
Prerequisites:
Before getting started, we need to download and install the following software:
- Any screenshot-capturing tool (For Example, DuckCapture, or qSnap)
- JDK
- Eclipse (detailed steps here to install JDK and Eclipse)
Steps To Create The Sikuli Java Project
Step #1: Sikuli Download – Download Sikuli from here.
Step #2: Extract the zip file which you’ve downloaded. It will contain the Sikuli-script.jar file. Save this extracted file in your local file system.
Step #3: Open Eclipse.
Step #4: Create a Java project File -> New -> Java Project
Step #5:
- Right Click on the project
- Go to Build Path-> Configure Build Path
- Switch to the Libraries tab
- Click the “Add External Jars” button and Add Sikuli-Script.jar in the Build Path.
- Click “OK”
Sikuli-script.jar will be added to your project build path. You’re done. Now you can start writing Sikuli scripts for this project.
Some Sikuli Methods
#1) Creating Object for Screen Class
The screen is a base class provided by Sikuli. We need to create an object for this screen class first, then only we can access all the methods provided by Sikuli.
Syntax:
Screen s=new Screen();
#2) Click On An Element
This method used to click on the specific image present on the screen.
Syntax:
s.click(“<<image name>>”);
For Example,
s.click(“test.png”);
#3) Right Click On An Element
This method is used to right-click on the specific image present on the screen.
Syntax:
s.rightClick(“<<image name>>”);
For Example,
s.rightClick(“test.png”);
#4) Find An Element
This method is used to find a specific element present on the screen.
Syntax:
s.find(“<<image name>>”);
For Example,
s.find(“test.png”);
#5) Double Click on An Element
This method is used to trigger a double-click event on a specific image present on the screen.
Syntax:
s.doubleClick(“<<image name>>”);
For Example,
s.doubleClick(“test.png”);
#6) Check whether an Element present on the Screen
This method is used to check whether the specified element is present on the screen.
Syntax:
s.exists(“<<image name>>”);
For Example,
s.exists(“test.png”);
#7) Type a string on a Textbox
This method is used to enter the specified text in the Text box.
Syntax:
s.type(“<<image name>>”,”String to be typed”);
For Example,
s.type(“test.png”,”HI!!”);
#8) Wheeling on a particular image
This method is used to perform wheeling action on the element image.
Syntax:
s.wheel(“<<image name>>”,<<int position>>,<<int direction>>);
For Example,
s.wheel(“test.png”,25,0);
#9) Drag and Drop a Image/Element
This method is used to drag and drop a specified image from the source position to the target position.
Syntax:
s.dragDrop(“<<source image name>>”,”<<target image name>>”);
For Example,
s.dragDrop(“test.png”,”test1.png”);
#10) Roll Hover on a particular image
This method is used to perform a roll hover event on the specified image.
Syntax:
s.hover(“<<image name>>”);
For Example,
s.hover(“test.png”);
#11) Paste Copied String
This method is used to paste text on the specified textbox.
Syntax:
s.paste(“<<image name>>”,”test”);
For Example,
s.paste(“test.png”,”test”);
Sikuli Examples
#1) YouTube Video – Pause And Play A Video
Step #1: Open a YouTube video link and Capture play and pause element images using the screen capture tool.
Pause button (Note: filename is pause.png)
Play button (Note: filename is play.png)
Copy these images inside the project.
Step #2: Create a package inside the Sikuli java project created and within that create a class named “Youtube”.
Step #3: Type the following code inside that class.
package com.test;
import org.sikuli.script.FindFailed;
import org.sikuli.script.Screen;
public class Youtube {
public static void main(String[] args) throws FindFailed, InterruptedException {
// TODO Auto-generated method stub
Screen s=new Screen();
s.find("pause.png"); //identify pause button
s.click("pause.png"); //click pause button
System.out.println("pause button clicked");
s.find("play.png"); //identify play button
s.click("play.png"); //click play button
}
}
Step #4: Right-click on the class select Run As -> Java Application.
#2) Open Notepad And Type Some Text
Step #1: Capture the notepad icon on the desktop on the screen.
notepad_icon.png
notepad.png
Step #2: Copy these images inside your project.
Step #3: Create a class named “NotepadExample” inside your project and type the following code.
package com.test;
import org.sikuli.script.FindFailed;
import org.sikuli.script.Screen;
public class NotepadExample {
public static void main(String[] args) throws FindFailed {
// TODO Auto-generated method stub
Screen s=new Screen();
s.click("notepad_icon.png");
s.find("notepad.png");
s.type("notepad.png","This is Nice Sikuli Tutorial!!!!");
}
}
Step #4: Open the screen to be tested before executing the code.
Execute this file by Right click Run As -> Java Application.
#3) Drag And Drop
Step #1: Take the screenshot of the required items on the screen, and put it inside your Sikuli project.
[Note: here, downloads icon is “source.png” and flower image is “destination.png”]
Step #2: Put these pictures inside your project.
Step #3: Create a class with the name “DragAndDrop” and write the following code.
package com.test;
import org.sikuli.script.FindFailed;
import org.sikuli.script.Screen;
public class DragAndDrop {
public static void main(String[] args) throws FindFailed, InterruptedException {
// TODO Auto-generated method stub
Screen s=new Screen();
s.find("source.png");
System.out.println("Source image found");
s.find("target.png");
System.out.println("target image found");
s.dragDrop("source.png", "target.png");
}
}
Step #4: Execute this script by right click Run As -> Java Application.
After the execution of this script, the download icon will be dragged and dropped on the image, indicated as a target.
Before Execution:
After Execution:
Drawbacks Of This Tool
- We cannot assure you that the image match will be always accurate. Sometimes, if two or more similar images are available on the screen, Sikuli will attempt to select the wrong image.
- If the image appearance varies in pixel size, it will also result in the “Find Failed ” exception.
- Overhead of taking too many screenshots.
- If anyone of the screenshot is missing, it will affect the execution of the program.
Conclusion
Sikuli is very much useful in automating flash objects. It can be used to automate window-based applications. It is a great tool to play with elements on a screen, based on their visuals.
About the author: This is a guest post by Anitha Eswari. She is currently working as a senior test engineer having sound knowledge of manual and automation testing and various test management tools.
Next Tutorial: In the next part of this series let’s have a deep look at creating the Sikuli maven project and how to integrate Selenium with Sikuli.
Already using this tool? Please share your experience and tips. If you want to get started but have queries let us know.
Was this helpful?
Recommended Reading
-
Rational Robot is an automated functional, regression testing tool for automating Windows, Java, IE and ERP applications under windows platform. This article should be a good start for those who wants to learn Rational Robot test automation tool. Read on for Rational robot tutorial with trial version download and resources.
-
Here is an in-depth tutorial on the process of browser automation through Geb Tool for your benefit: Geb (pronounced "jeb") is the answer to any kind of browser automation challenges. It is a very effective tool to perform automation testing over the web. Geb originated out of the need to…
-
Here is the Ranorex Studio Hands-on Tutorial #1 (Click here for Tutorial #2). Ranorex Studio is a powerful test automation tool for everyone, including non-coders, to record automation tests step-by-step for desktop, web and mobile applications. Maintaining software quality is an important goal for any software system. In today’s world,…
-
Overview of SeeTest Automation: Owing to heavy competition and quick marketing of Mobile Apps, Mobile automation is being quite popular nowadays. There are several Mobile Automation tools that are available in the market as Open Source and Propitiatory ones. Mobile automation tools can be categorized as Image-Based and Object-Based. In…
Hi Anitha, its nice information about mentioned methods in Sikuli and could you help me on below:
1. how to write code for “keydown” characters/alphabets like D, C, V, B, N, S, X.. etc
2. how to write code to select option after doing right click
3. if we use robot and sikuli code then am getting error. so how can i use robot and sikuli combination
1. how to write code for “keydown” characters/alphabets like D, C, V, B, N, S, X.. etc
– To use Alphabets ex (Ctrl+c) >> type(Key.modifiers+’c’)
– or you just want to use arrow keys >> type(Key.DOWN)
– For just typing >> type(“hello world”)
2. how to write code to select option after doing right click
– ex Right click on desktop and click on Refresh
>> for this we need to capture image of mini window which opens after rightclick
rightclick(desktop.png)
click(refresh.png)
Sorry in 1st example syntax would be >> type(“c”+KeyModifier.CTRL)
You can find my findings in below links regarding Sikuli.
http://www.slideshare.net/sgorripotu/sikuli-17934666?related=1
http://www.slideshare.net/sgorripotu/sikuli-satish-gorripotu?related=2
its good ..
ar
Has anyone ever tried to use this tool in a Citrix interface application? It is quite difficult to find a tool that works with Citrix.
Its very good one to know. I am curious to know its next step , that is, how to use it with selenium web driver.
*****Solution*****
This might help.
Open https://launchpad.net/sikuli/+download and download sikuli-setup.jar.
Execute this and this jar will generate a runSetUp.bat file.
Execute bat file and you will get Sikuli-Script.jar.
Hi Guys,
I have an issue when I want to get value of combobox.
I took picture at combobox field and set TargetOffset for Combobox pattern. But I print out result, it doesn’t get value of combobox. Anyone here gets any idea for this issue. I’m using sikuli for implementation and sikuliX for capture pic
But I don’t know how to launch Youtube in the youtube example
all those tests are after youtube is launched
but who launches youtube app?
SIKULI is an open-source GUI based test automation tool. It is mainly used for interacting with elements of web pages and handling windows based popups. Sikuli uses the technique of “Image Recognition” and “Control GUI” to interact with elements of web pages and windows popups. In Sikuli, all the web elements are taken as images and stored inside the project.
subbrao.spi@gmail.com
@sujhata
you can download the sikuli
https://launchpad.net/sikuli/+download
Use this link to download sikuli zip file.
Page not found error come when click on following link
Information:this link is about tutorial1 of sikuli tutorial
As a beginner in using sikuli IDE, have launched the IDE and tried taking screen shot manually by clicking the ‘camera button’. it is taking me always to desktop page and not the one i would like to capture (chrome or any other screens).
I want to capture some images from chrome browser for test. whenever i click image capture button in sikuli IDE (2.0.4) – java version 8, it is taking me to my desktop screen everytime. is there any setting i need to configure to make sure i can capture any screen i want?
@eswai: I’m using Sikuli to test my webpage… I also wrote a little BDD addon to Sikuli to enable behavior driven testing… I run it mainly on 6 Windows 7/8 VMs, couldn’t get it to work reliable on Linux with VMs… Works quite reliable and I’m very satisified with it… /Alex
Hi !
I’m working with sikuli for testing windows app.
I need to test how fast gui call back of the app
Any suggestions ?
Edward, you apparently didn’t use Region selection and similarity threshold(which is rather lax at default settings). with those two, it is realistic to make sikuli see what you meant it to.
I am getting this error
Dec 14, 2015 3:31:06 PM java.util.prefs.WindowsPreferences
WARNING: Could not open/create prefs root node Software\JavaSoft\Prefs at root 0x80000002. Windows RegCreateKeyEx(…) returned error code 5.
[error] ResourceLoaderBasic: checkLibsDir: libs dir is not on system path: E:\sikuli\libs
[action] ResourceLoaderBasic: checkLibsDir: Please wait! Trying to add it to user’s path
[info] runcmd: reg QUERY HKCU
[info] runcmd: reg QUERY HKEY_CURRENT_USER\Environment /v PATH
[error] ResourceLoaderBasic: checkLibsDir: Logout and Login again! (Since libs folder is in user’s path, but not activated)
[error] Terminating SikuliX after a fatal error! Sorry, but it makes no sense to continue!
If you do not have any idea about the error cause or solution, run again
with a Debug level of 3. You might paste the output to the Q&A board.
very good introduction
Hi, can someone please clarify if the comments entered here are reviewed and moderated? if so, I will enter the issue that I am facing in sikuli.? thx
Hi,
Could you please post a tutorial on how to log in to Citrix where I do not want my credentials to be written down in a JAVA code,instead the user is asked to input the same during the execution of the code.
Thanks in advance.
You have to install package 2 after running the runSetup.cmd (windows) to get the sikuli-script.jar . It took me awhile to figure that out
While running the program its is giving “..\tmplib\VisionProxy.dll Can’t find dependent libraries”. Can anyone help
*** classpath dump
0: /C:/Users/ronak/workspace/sikuliFirstTest/bin/
1: /C:/Users/ronak/Desktop/sikuli/sikulixsetup-1.1.1.jar
*** classpath dump end
[error] RunTimeINIT: *** terminating: libs to export not found on above classpath: /sikulixlibs/windows/libs64
Display this error when run sikuli script in eclipse
is there Any Automation Testing tool that support win ce 7 application
how to automate WIN CE application
Hi,
Machine is not useable in parallel while SikuliX scripts or programs are running. Is there any better way to run the Sikuli script in backend? Please help me on this.
Rambabu this is a GUI automation, in GUI keyboard & mouse will be under control of Sikuli driver.
Excellent info about the sikuli Tool
hi,
i am getting below error on creating object of screen
Exception in thread “main” java.lang.ExceptionInInitializerError
at org.sikuli.script.Region.(Region.java:60)
at org.sikuli.script.Screen.(Screen.java:132)
at sachin.demo.main(demo.java:8)
Caused by: java.lang.StringIndexOutOfBoundsException: begin 2, end 3, length 2
at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3756)
at java.base/java.lang.String.substring(String.java:1902)
at org.sikuli.basics.Settings.(Settings.java:115)
… 3 more
Hi,
Can we automate the desktop applications using sikuli. any suggestions ?
@Alex – yeah… it is really a very good tool to automate web pages as well as window based applications.
Not able to find Sikuli-Script.jar. Can someone please guide me ?
How to take handle of tools like Seetest or any other tool using selenium with sikuli?
Please help me!
Very nice tool. But still way behind QTP
Its very good one to know. I am curious to know its next step , that is, how to use it with selenium web driver.
Hi,
Can we automate mobile apps with this sikuli tool..
A new tool ExtensiveTesting http://www.extensivetesting.org/ based on sikuli and selenium (best of the two world in one tool) : it’s a generic and open source test framework with a test recorder and testcase modelisation.
Take a look
I don’t think this tool is trying to emulate UFT. This tool is meant to be an image recognition tool. So basically, it can recognize anything that you want. Also, it can do mobile testing. Both things which UFT cannot do.
Yes we can automate desktop application also.
Hi,
How can we run the sikuli script created on one machine to other machine.
Hi,
The article was helpful to understand how to kick start Sikuli. Will Sikuli fit for automating Oracle EBusiness Suite application R12 application?
Actually, I am finding an alternate open source tool to OATS. Can you pls explain me?
Thanks!
@krishna- uee the above mentioned link to download sikuli zip file,extract it. Inside that folder you can find sikuli-script jar.
Hi,
I’m unable to find Sikuli-Script.jar file
Regards,
Sher Hassan
I am new at this everyone one else seems to be happy with your tutorial but I am not as I try to follow it and its not helping rather make me doubt if this is a great tool for the automation .Your first example YouTube example You are not showing us how to get to you tube and the fact that this automates things you see on screen. I have got the eclipse project open that’s what I see so when I run it ,it does not see the pictures could you please revise that .I would like to know the following:
How to work with sikuli if you have two screens?
Where to put the pictures to be compared?
Navigating to the page that needs testing and finding the pictures
And you have the notepad example: tried to follow that but that’s not clear as well the icon is meant to be on my desk top to click and take a picture of that put it in my bin directory reference it to be compared I am confused can’t get to it or see it my eclipse environment is what I see .
The same goes to the drag and drop example what am I dragging and dropping it where please explain what the source is and the destination.
My understanding the source is where the picture to be dragged is and the destination where I want it to end up so please help clarify this
Good information- will sikuli work with AngularJS/html/Silverlight application?
Hi Anitha,
Can we automate the desktop applications using sikuli. any suggestions ?
Hi Anita,
where can I download Sikuli 64 bit for window and tell me the procedure for testing Desktop Application through Selenium.
Hi,
Some one please help me to find out how to work with “combo box” in sikuli. I don’t test “combo box” value
Very nice info….
Is data driven testing possible in Sikuli ?
Hi, i wanted to know if sikuli can automate all Microsoft applications along with selenium
can we test excel sheet cell data in sikuli?
Very good article.I’m facing an issue it is,whenever there is a firefox browser upgrades the image buttons sizes are increased the increased button image size does not match with the stored image size ,hence i have to replace the stored image buttons with the latest all the time.Is there anyway to handle this to avoid manual replacement when ever firefox upgrade happens
sikuli-script.jar is not there in folder after extracting the zip file
can you say what version you used in this tutorial?
*** classpath dump
0: /C:/Users/ronak/workspace/sikuliFirstTest/bin/
1: /C:/Users/ronak/Desktop/sikuli/sikulixsetup-1.1.1.jar
*** classpath dump end
[error] RunTimeINIT: *** terminating: libs to export not found on above classpath: /sikulixlibs/windows/libs64
Display this error when run sikuli script in eclipse
Screen s=new Screen();
I think this needs to be updated. I added
org.sikuli
sikuli-api
1.2.0
to my project.
The above line needs to implement a couple of methods before it can be used, it seems.
Screen screen = new Screen() {
@Override
public BufferedImage getScreenshot(int i, int i1, int i2, int i3) {
return null;
}
@Override
public Dimension getSize() {
return null;
}
};
I am not able to access any others, such as s.click or any others.
i am using sikuli to automate the flash objests .it is going good but in one place i have to 2 very similar images ,so anybody tell how i can differentiate them
If the required image appears in the same position on the screen, then you can use Region.
Hi,
I trying to learn sikuli.
– i am unable to find the sikuli-script.jar file to download.
– How i can integrate sikuli with eclipse
– How can i write scripts in eclipse using sikuli.
I wanted to use Sikuli to automate something really simple.
The ‘image recognition’ apparently REALLY BLOWS.
If it sees even 1 part of the target image, itll trigger.
Like asking it to look for a frog and it goes “heres the frog” and points at a tree.
Very frustrating waste of time.
This is a very good, quick tutorial. Nicely done. I want to open minds of anyone ons this page looking. I also use the sikuli jar for java, as well a selenium jars.
I write all of my code in java, in a testng framework, data driven, and drive everything through jenkins. Sikuli is actually quite scalable and also quite powerful.
You may however, have to have some creativity to get it to work for you the way you hope.
Hi would like to know if there is any log4 issue with 2.0.4 version.
Is sikuli-api-1.1.0-sources.jar sufficient to run sample program in sikuli. I m getting import statement Error.
I there are any more jar files need to be import plz let me Know.
@subbrao
After extracting,you will get the main folder of sikuli in that you will get SikuliX-IDE—>sikulix.jar
How to open Skype or Windows Default Calculator using Java Code configured with Sikuli jar in Eclipse.
I have tried this and below is the error am getting
Can’t load IA 32-bit .dll on a AMD 64-bit platform”
And my Java code is as below:
package com.test.sikuli;
import org.sikuli.script.*;
public class TestSikuli {
public static void main (String args [])
{
Screen s = new Screen();
App myapp = new App(“application-identifier”);
myapp.open(“C:\\Program Files (x86)\\Online Services\\Skype\\SkypeLauncher.exe”);
}
}
Please help me out to solve this
Thanks in advance!
*** classpath dump
0: /C:/Users/ronak/workspace/sikuliFirstTest/bin/
1: /C:/Users/ronak/Desktop/sikuli/sikulixsetup-1.1.1.jar
*** classpath dump end
[error] RunTimeINIT: *** terminating: libs to export not found on above classpath: /sikulixlibs/windows/libs64
Display this error when run sikuli script in eclipse
Hallo,
I did today both exercise Youtube and DragAndDrop
but I got errors !! Please help.
Youtube.java
Eror at: s.click(“pause.png”)
Error Message: The method dragDrop(PSRML, PSRML, int) in the type Region is not applicable for the arguments (String, String)
DragAndDrop.java
Error: s.dragDrop(“source.png”, “target.png”);
Error Message:
The method dragDrop(PSRML, PSRML, int) in the type Region is not applicable for the arguments (String, String)
can u share a wich dependency should be use for maven project
Hi,
Now, Sikuli can run directly on Android.
AnkuLua = Android + Sikuli + Lua
We are the developer of AnkuLua.
Welcome to try AnkuLua and give us feedbacks.
Hi ..
is there any other open source tool available for UI automation testing?
which of the Sikuli jar files we need to download from the link given? I have macOS
ok . but this is different script.
Some attempted to compare QTP with Sikuli?
QTP doesn’t compare to Silktest and RFT, let alone open source products like Selenium Webdriver and Sikuli Webdriver.
Last I knew, QTP doesn’t support dynamic object recognition. Silk and RFT had that back in 2007. QTP, like Test Complete, needed one to maintain an object hierarchy tree.
Sikuli is for pictorial recognition,typically mixed in with Selenium, when Selenium just can’t seem to recognize some pesky object. Sikuli is also used for Flash/Flex based testing.
Eventually, HP will have a very tough time selling QTP.
How can I open sikuli script that I have created earlier in sikuli IDE?