What is Sikuli? How Sikuli can be integrated with Selenium WebDriver?

Sikuli is an Open Source GUI based Automation tool. This tool helps us to recognize the web objects if webdriver is unable to locate the web element based on the XPATH or CSS locators. 

Sikuli uses the technique of "Image Recognition" & "Control GUI" to interact with Web Elements. All the Web Elements are taken as images and stored inside the project.

To integrate Sikuli with Selenium Web Driver first we need to download the latest JAR from the following url.

https://raiman.github.io/SikuliX1/downloads.html

The above jar file will help us to integrate the Web Driver with Sikuli. So, please add this jar dependency in your IDE either Eclipse or Intellj. In the below screen you can see that I have added or integrated the Sikuli jar with the project.

In Sikuli we have 3 major classes which it supports 'Region', 'Match' & 'Screen'. In this tutorial we will focus on Screen class and its methods. Below are the methods which Screen class supports.

Here, I am automating the login screen of facebook and assume that login button is not identifying by WebDriver using locators like Xpath or CSS. So, I will use here Sikuli to perform click event on the login button.

Please save the image of login button in c: drive in .PNG format. After that give that image reference to Pattern Class like below.

Screen s=new Screen();

Pattern loginButton=new Pattern("c:/fblogin.png");
s.click(loginButton);

Below is the code which I have written in Intellj IDE to perform click even on login button of facebook page.

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.sikuli.script.FindFailed;
import org.sikuli.script.Pattern;
import org.testng.annotations.Test;
import org.sikuli.script.Screen;


import java.util.concurrent.TimeUnit;

public class FBLogin
{
    @Test
    public void login() throws FindFailed
    {

        System.setProperty("webdriver.chrome.driver","c:/Browser Executables/chromedriver.exe");
        Screen s=new Screen();
        Pattern loginButton=new Pattern("c:/Images/fblogin.PNG");

        WebDriver driver;
        driver=new ChromeDriver();

        driver.manage().window().maximize();
        driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);

        driver.get("https://www.facebook.com");

        driver.findElement(By.xpath("//*[@id='email']")).sendKeys("Vikas4903");
        driver.findElement(By.xpath("//*[@id='pass']")).sendKeys("Vikas4903");
        s.click(loginButton);

        driver.close();
    }
}

I will keep adding more automation concepts in my upcoming posts. Please keep me posted if you have any questions and concerns.



Best of Luck.