Scaling Selenium Testing at Yelp – What is Selenium? – Webdriver Examples

What is Selenium?

• Test Framework For Web Applications
• Simulate User interactions
• Code in any language
• Test every browser

Selenium Server

• →
• →

A Few Terms

• Selenium RC (2004)
• Selenium IDE
• Selenium Webdriver (2009)

Selenium RC

• Executes JavaScript to drive the browser
• Still a lot of examples out there
• Don't use it

Selenium IDE

• Test Generation
• Flow Recording
• Feels really fragile

Selenium Webdriver

• Replaced Selenium RC in 2009
• Still uses parts of Selenium RC
• Addressed a lot of the design issues
• Object based design - WebDriver + WebElement

Webdriver Examples

Using the Python Library

WebDriver

    def do_selenium_stuff():
            driver = WebDriver(
                command_executor='http://localhost:4444/wd/hub',
                desired_capabilities=DesiredCapabilities.FIREFOX
            )
            driver.implicitly_wait(10)

            driver.get('http://yelp.com')

            for handle in driver.window_handles:
                driver.switch_to_window(handle)

            driver.refresh()
            img = driver.get_screenshot_as_png()
            element = driver.find_element_by_css_selector('.foo .bar')
                            

WebElement

    element = driver.find_element_by_css_selector('.foo .bar')

    element.click()
    element.send_keys('foo bar')

    sub_element = element.find_element_by_id('sub-element')
                        

Selenese API Requests

    POST /session
        desiredCapabilities: {"browser": "firefox", "version": "23.0"}

    POST /session/2c34ac26-2d8c-4bfb-9210-3038f7c92252/element
        using: "css selector"
        value: ".foo input.bar"

    GET /session/2c34ac26-2d8c-4bfb-9210-3038f7c92252/element/0/value
        value: "Text to enter"
                        

Remote Webdriver

• A Simple Proxy
• Starts browsers
• Forwards selenese commands to browsers
• Browsers plugins manage execution of commands

How we write Selenium Tests at Yelp

Our Workflow

• Start a sandboxed environment
• Create fixtures
• Execute a flow
• Confirm side effects

Login & Write a Review

Sandboxing

    $ sandbox testify yelp.tests.write_a_review

• Starts a new mysql instance and creates tables
• Starts starts apache, memcached, etc
• Executes command (runs our test)
• Shutdowns the sandbox

Create Fixtures

    def create_fixtures(email, password):
        user = User.create(email, password)
        business = Business.create()
        return user, business
                    

Create Flows

    def perform_login(driver, user):
        driver.get('http://dev.yelp.com/login')
        driver.find_element_by_id('email').send_keys(user.email)
        driver.find_element_by_id('password').send_keys(user.password)
        driver.find_element_by_css_selector('button[type=submit]').click()

    def perform_write_a_review(driver, rating, review_text):
        driver
            .find_element_by_css_selector(
                'li:nth-child('+ rating +') input[type=radio]')
            .click()
        driver
            .find_element_by_css_selector('input[name=review]')
            .send_keys(review_text)
        driver.find_element_by_id('submit').click()
                    

Create Assertions

    def assert_user_wrote_review(user, business, review_text, rating):
        review = business.reviews[0]
        assert review.text == review_text
        assert review.rating == rating
        assert review.user == user
                    

Write The Test

    def test_write_a_review(driver):
        email = 'kstruys@yelp.com'
        password = 'password'
        review_text = 'This place is awesome!'
        rating = 5
        user, business = create_fixtures(email, password)
        perform_login(driver, user)
        assert_user_logged_in(driver, user)
        driver.get('http://dev.yelp.com/writeareview/' + business.id)
        perform_write_a_review(driver, rating, review_text)
        assert 'Your Review has been posted' in
            driver.find_element_by_id('alert-message')
        assert_user_wrote_review(user, business, review_text, rating)
                    

Selenium Components

API To Widgets

    form = driver.find_element_by_css_selector('.signup-form')

    signup_form = SignupForm(form)
    signup_form.update(
        first_name="Ken",
        last_name="Struys"...
    )
    signup_form.submit()
                        

Page Objects

API To Pages

    class LoginPage {
        public LoginPage(WebDriver driver) {
            this.driver = driver;

            if (!"Login".equals(driver.getTitle())) {
                throw new IllegalStateException("Not the login page");
            }
        }

        public Homepage login(String username, String password) {
            this.fill_form(username, password);
            this.submit();
            return new Homepage(this.driver);
        }
    }
                    

Why is Selenium Important?

Better Coverage

• Unit testing is necessary but not sufficient
• Unacceptable to ever break the important flows

More Clientside Code

• Serverside
  • You control the software/hardware that produces HTML
  • Page refreshes allow flows to restart
• Clientside
  • Different JS engines performing rendering
  • A minor bug could result in a blank/broken page
  • Logging errors is more difficult

Deployment

• Dev
• →Stage
• →Prod

Stage Testing is Tedious, Boring and Error Prone

Common Complaints

Selenium is inherently flaky

• Selenium RC was really flaky
• Requires education - Dos and Don't
• Requires Practice!

Ain't nobody got time for that!

• You're manually testing anyways
• Integration tests take less time to write
• It's really not that complicated

Emulate the User

Write tests as a QA employee, not an engineer

JavaScript Injection

• Avoid it! (as much as possible)
• Events in javascript are not consistent across browsers/versions
• More security restrictions
• Webdriver fires events at the OS Level (when possible)

Adding Selenium To Your Stack

Minimal Setup

• Every web startup should get basic selenium coverage
• Run a Docker container for firefox+selenium server
• Something working in a couple days

Selenium at Yelp

• ~3-4 Production pushes a day
• Python Library + Firefox (we want to add more browsers)

4 Years ago

• Infrastructure setup by an intern (yup, that easy!)
• Only test major flows - Write a Review, Signup, Login
• ~ 40 Tests
• ~ 10-15 minutes to run our selenium suite

Today

• We learned how to fix and avoid flaky tests (still learning)
• ~ 1000 selenium tests
• ~ 30 minutes to run our selenium suite

Flaky Tests

• Understand flakes vs failures
• Avoid bad features in selenium (e.g. code injection)
• Enable implicit wait
• Selenium RC -> Webdriver

Saucelabs

• Free for opensource
• 350 browser/OS combinations version
• Video Recording/Logging
• Tunnel to your internal dev environment
• saucelabs.com

The Future

• Appium at Yelp
• Opensource Python Libraries
• Continuous Testing/Alerting