Introduction to Sikuli GUI Automation Tool (Automate Anything You See on Screen) – Sikuli Tutorial #1

As always we try to bring in new things to learn for our readers. Today let's explore an interesting GUI automation tool – Sikuli.

“Automate anything you see” using Sikuli graphical user interface (GUI) automation tool – Complete beginners guide to quickly set up and start using Sikuli script tool with these in-depth Sikuli Tutorials.

Sikuli automates anything you see on the screen using the image recognition method to identify GUI elements. Sikuli script allows users to automate GUI interaction by using screenshots.

We have divided this series into 2 parts:

Sikuli tutorial #1How It works, How to create a simple Sikuli project.
Sikuli tutorial #2How to create Sikuli maven project, How Sikuli can be used with selenium web driver to automate webpages.

Sikuli GUI Automation tool

Let’s start with a 1st part in this series.

What is covered in this tutorial:

  • Introduction to Sikuli
  • Practical use
  • Benefits
  • Prerequisites
  • Creating a Sikuli Java Project
  • Some Sikuli Methods
  • Examples
  • Drawbacks of Sikuli


Sikuli is a tool to automate graphical user interfaces (GUI) using “Visual Image Match” method. In Sikuli, all the web elements should be taken as an image and stored inside the project. Sikuli will trigger GUI interactions based on the image visual match, the image which we have passed as the parameter along with all methods.

Sikuli can be very much useful to automate flash objects (which do not have ID or name). It can be useful in the situation, where we have a stable GUI (i.e. GUI components not changing).

Even Window based applications can also be automated using Sikuli. Sikuli provides very friendly Sikuli-script.jar, which can be easily used together with Selenium WebDriver. We can even automate Adobe Video/Audio player, Flash Games on the website using Sikuli. With simple API, it makes coding easier.

Practical Uses

  1. Sikuli can be used to automate Flash Objects / Flash Websites.
  2. It can be useful to automate Window based application. We can automate, what we are seeing on the screen.
  3. It provides, simple API. i.e. all methods can be accessed using screen class object.
  4. It can be easily integrated with Selenium and all other tools.
  5. Using Sikuli we can automate desktop applications.
  6. Most of the automation testing tools will not support flash object automation (E.g. Selenium). Sikuli provides extensive support to automate flash objects.
  7. It uses powerful “Visual Match” mechanism to automate desktop & flash objects.


  • Open source Tool.
  • One of the biggest advantages of Sikuli is that it can easily automate Flash objects.
  • It makes easy to automate windows application.
  • When you're testing an application under development and you don't know the ID/name of the elements, then you can go with Sikuli. It will check the appearance of the image and if match found, it will interact with the image accordingly.


Before getting started, we need to download and install the following software:

  • Any screenshot capturing tool (E.g. DuckCapture, or qSnap)
  • JDK
  • Eclipse (detailed steps here to install JDK and Eclipse)

Steps to Create Sikuli Java Project:

Step #1: Sikuli Download – Download Sikuli from here.

Step #2: Extract the zip file which you've downloaded. It will contain the sikuli-script.jar file. Save this extracted file in your local file system.

Step #3: Open Eclipse.

Step #4: Create a java project File -> New  -> Java Project

Step #5:

  1. Right Click on the project
  2. Go to Build Path   -> Configure Build Path
  3. Switch to Libraries tab
  4. Click “Add External Jars” button and Add Sikuli-Script.jar in the Build Path.
  5. Click “Ok”

(Click on image to enlarge)


Sikuli-script.jar will be added to your project build path. You're done. Now you can start writing Sikuli scripts inside this project.

Some Sikuli Methods:

#1: Creating Object for Screen Class
The screen is a base class provided by Sikuli. We need to create an object for this screen class first, then only we can access all the methods provided by Sikuli.

Screen s=new Screen();

#2: Click On An Element
This method used to Click on the specific image present on the screen.

Syntax:“<<image name>>”);


#3: Right Click On An Element
This method used to right click on the specific image present on the screen.

s.rightClick(“<<image name>>”);


#4:  find An Element
This method used to find a specific element present on the screen.

s.find(“<<image name>>”);


 #5: Double Click on An Element
This method used to trigger a double click event on a specific image present on the screen.

s.doubleClick(“<<image name>>”);


#6: Check whether an Element present on the Screen
This Method is used to check whether the specified element is present on the screen.

s.exists(“<<image name>>”);


#7: Type a string on a Textbox
This Method is used to enter the specified text on the Text box.

s.type(“<<image name>>”,”String to be typed”);


#8: Wheeling on a particular image
This method is used to perform wheeling action on the element image.

s.wheel(“<<image name>>”,<<int position>>,<<int direction>>);


 #9: Drag and Drop a Image/Element
This method is used to drag and drop a specified image from source position to target position.

s.dragDrop(“<<source image name>>”,”<<target image name>>”);


#10: Roll Hover on a particular image
This method is used to perform roll hover event on the specified image.

s.hover(“<<image name>>”);


#11: Paste Copied String
This method used to paste text on the specified textbox.

s.paste(“<<image name>>”,”test”);


Sikuli Example #1: YouTube video – Pause and Play a video

Step #1:
Open a YouTube video link and Capture play and pause element images using screen capture tool.

Pause button (note: file name is pause.png)


Play button (note: file name is play.png)


Copy these images inside the project.

Step #2:
Create a package inside the Sikuli java project created and within that create a class named “Youtube”.

Step #3:
Type the following code inside that class.

 package com.test;

import org.sikuli.script.FindFailed;
 import org.sikuli.script.Screen;

public class Youtube {

public static void main(String[] args) throws FindFailed, InterruptedException {
 // TODO Auto-generated method stub

Screen s=new Screen();
 s.find("pause.png"); //identify pause button"pause.png"); //click pause button
 System.out.println("pause button clicked");

s.find("play.png"); //identify play button"play.png"); //click play button


Step #4:
Right click on the class select Run As -> Java Application.

Sikuli Example #2: Open Notepad and type some text

Step #1:
Capture notepad icon on the desktop on the screen.





Step #2:
Copy these images inside your project.

Step #3:
Create a class named “NotepadExample” inside your project and type the following code.

 package com.test;

import org.sikuli.script.FindFailed;
 import org.sikuli.script.Screen;

public class NotepadExample {

public static void main(String[] args) throws FindFailed {
 // TODO Auto-generated method stub

Screen s=new Screen();"notepad_icon.png");
 s.type("notepad.png","This is Nice Sikuli Tutorial!!!!");


Step #4:
Open the screen to be tested before executing the code.
Execute this file by Right click Run As  -> Java Application.

Sikuli Example #3: Drag and drop

Step #1:
Take the screen shot of required items on the screen, and put it them inside your sikuli project.


[Note: here, downloads icon is “source.png” and flower image is “destination.png”]

Step #2:
Put these pictures inside your project.

Step #3:
Create a class with name “DragAndDrop” and write the following code.

 package com.test;

import org.sikuli.script.FindFailed;
 import org.sikuli.script.Screen;

public class DragAndDrop {

public static void main(String[] args) throws FindFailed, InterruptedException {
 // TODO Auto-generated method stub
 Screen s=new Screen();
 System.out.println("Source image found");
 System.out.println("target image found");
 s.dragDrop("source.png", "target.png");


Step #4:
Execute this script by right click Run As  -> Java Application.
After execution of this script, downloads icon will be dragged and dropped on the image, indicated as a target.

Before Execution:

(Click on images to enlarge)


After Execution:


Drawbacks of this tool

  • We cannot assure that image match will be always accurate. Sometimes, if two or more similar images are available on the screen, Sikuli will attempt to select the wrong image.
  • And if image appearance varies in pixel size, will also result in “Find Failed ” exception.
  • Overhead of taking too many screenshots.
  • If anyone of the screenshot is missing, it will affect the execution of the program.

More resources:

  • Simple Sikuli example here.
  • More video tutorials here.
  • Detailed Sikuli documentation here.


Sikuli is very much useful in automating flash objects. It can be used to automate window based applications. It is a great tool to play with elements on a screen, based on their visual.

About the author: This is a guest post by Anitha Eswari. She is currently working as a senior test engineer having sound knowledge on manual and automation testing and various test management tools.

Next Tutorial: In next part of this series let's have a deep look at creating Sikuli maven project and how to integrate Selenium with Sikuli.

Already using this tool? Please share your experience and tips. If you want to get started but have queries let us know.

Recommended Reading

67 thoughts on “Introduction to Sikuli GUI Automation Tool (Automate Anything You See on Screen) – Sikuli Tutorial #1”

  1. I don’t think this tool is trying to emulate UFT. This tool is meant to be an image recognition tool. So basically, it can recognize anything that you want. Also, it can do mobile testing. Both things which UFT cannot do.

  2. Hi,
    I trying to learn sikuli.
    – i am unable to find the sikuli-script.jar file to download.
    – How i can integrate sikuli with eclipse
    – How can i write scripts in eclipse using sikuli.

  3. @eswai: I’m using Sikuli to test my webpage… I also wrote a little BDD addon to Sikuli to enable behavior driven testing… I run it mainly on 6 Windows 7/8 VMs, couldn’t get it to work reliable on Linux with VMs… Works quite reliable and I’m very satisified with it… /Alex

  4. You have to install package 2 after running the runSetup.cmd (windows) to get the sikuli-script.jar . It took me awhile to figure that out

  5. Some attempted to compare QTP with Sikuli?

    QTP doesn’t compare to Silktest and RFT, let alone open source products like Selenium Webdriver and Sikuli Webdriver.

    Last I knew, QTP doesn’t support dynamic object recognition. Silk and RFT had that back in 2007. QTP, like Test Complete, needed one to maintain an object hierarchy tree.

    Sikuli is for pictorial recognition,typically mixed in with Selenium, when Selenium just can’t seem to recognize some pesky object. Sikuli is also used for Flash/Flex based testing.

    Eventually, HP will have a very tough time selling QTP.

  6. Hi,

    Some one please help me to find out how to work with “combo box” in sikuli. I don’t test “combo box” value

  7. Hallo,

    I did today both exercise Youtube and DragAndDrop
    but I got errors !! Please help.
    Eror at:“pause.png”)

    Error Message: The method dragDrop(PSRML, PSRML, int) in the type Region is not applicable for the arguments (String, String)
    Error: s.dragDrop(“source.png”, “target.png”);

    Error Message:
    The method dragDrop(PSRML, PSRML, int) in the type Region is not applicable for the arguments (String, String)

  8. I wanted to use Sikuli to automate something really simple.

    The ‘image recognition’ apparently REALLY BLOWS.
    If it sees even 1 part of the target image, itll trigger.

    Like asking it to look for a frog and it goes “heres the frog” and points at a tree.

    Very frustrating waste of time.

  9. Edward, you apparently didn’t use Region selection and similarity threshold(which is rather lax at default settings). with those two, it is realistic to make sikuli see what you meant it to.

  10. is there Any Automation Testing tool that support win ce 7 application

    how to automate WIN CE application

  11. Hi,
    Now, Sikuli can run directly on Android.
    AnkuLua = Android + Sikuli + Lua
    We are the developer of AnkuLua.
    Welcome to try AnkuLua and give us feedbacks.

  12. I am getting this error
    Dec 14, 2015 3:31:06 PM java.util.prefs.WindowsPreferences
    WARNING: Could not open/create prefs root node Software\JavaSoft\Prefs at root 0x80000002. Windows RegCreateKeyEx(…) returned error code 5.
    [error] ResourceLoaderBasic: checkLibsDir: libs dir is not on system path: E:\sikuli\libs
    [action] ResourceLoaderBasic: checkLibsDir: Please wait! Trying to add it to user’s path
    [info] runcmd: reg QUERY HKCU
    [info] runcmd: reg QUERY HKEY_CURRENT_USER\Environment /v PATH
    [error] ResourceLoaderBasic: checkLibsDir: Logout and Login again! (Since libs folder is in user’s path, but not activated)
    [error] Terminating SikuliX after a fatal error! Sorry, but it makes no sense to continue!
    If you do not have any idea about the error cause or solution, run again
    with a Debug level of 3. You might paste the output to the Q&A board.

  13. This is a very good, quick tutorial. Nicely done. I want to open minds of anyone ons this page looking. I also use the sikuli jar for java, as well a selenium jars.

    I write all of my code in java, in a testng framework, data driven, and drive everything through jenkins. Sikuli is actually quite scalable and also quite powerful.

    You may however, have to have some creativity to get it to work for you the way you hope.

  14. Hi !
    I’m working with sikuli for testing windows app.
    I need to test how fast gui call back of the app
    Any suggestions ?

  15. How to open Skype or Windows Default Calculator using Java Code configured with Sikuli jar in Eclipse.

    I have tried this and below is the error am getting

    Can’t load IA 32-bit .dll on a AMD 64-bit platform”

    And my Java code is as below:

    package com.test.sikuli;

    import org.sikuli.script.*;

    public class TestSikuli {

    public static void main (String args [])
    Screen s = new Screen();
    App myapp = new App(“application-identifier”);“C:\\Program Files (x86)\\Online Services\\Skype\\SkypeLauncher.exe”);


    Please help me out to solve this
    Thanks in advance!

  16. Hi,

    The article was helpful to understand how to kick start Sikuli. Will Sikuli fit for automating Oracle EBusiness Suite application R12 application?

    Actually, I am finding an alternate open source tool to OATS. Can you pls explain me?


  17. *** classpath dump
    0: /C:/Users/ronak/workspace/sikuliFirstTest/bin/
    1: /C:/Users/ronak/Desktop/sikuli/sikulixsetup-1.1.1.jar
    *** classpath dump end
    [error] RunTimeINIT: *** terminating: libs to export not found on above classpath: /sikulixlibs/windows/libs64

    Display this error when run sikuli script in eclipse

Leave a Comment