piece together an automation solution – Agenda – A Little Background

Agenda

This slide is mainly about how to make a automation solution in the following situation

• Can not find a existing solution / Framework to use
• No enough time / resources to build things from scratch

• Introduction of RHEV-H
• RHEV-H automation Difficulties
• List of used Resources and Technology
  • Languages
    • Python
    • Javascript
    • Golang
  • Framework, Tools
    • Bottle, Cobbler
    • uinput
    • Aria2
    • Mongodb, Meteor
    • Ansible

A Little Background

• About RHEV-H
• Why need a customized automation solution

RHEV-H

• Dedicated Hypervisor
• Built from RHEL components
  • Supports same hardware as RHEL
• A TUI interface for installation and configuration
• Provided in ISO format
• Minimal state information persisted to a Config partition
  • Read-Only FS
• Deployment Modes
  • CD-ROM
  • Flash Memory
  • Network (PXE) Boot

not good for automation test

• Can not use beaker to Provision
• Hard to manipulate (no useful API)

• automation test can not be "stateless"

first thing first

• A Provision System Cobbler
• Use uinput simulate keyboard action
  • the user level input subsystem, create and to handle input devices from an application

structure 01

How to run the test script

Some Troubles

• script need run right after the network service enabled
• need find way to persist state on a read-only system

Solution

• inject test script as system service use `livecd-tools`
• should be a server - client structure

structure 02

How to trigger the test run

• All RHEVH builds can be downloaded from brewweb
• Simple HTTP link address

Need a web-crawler to monitor brewweb

Scrapy

An open source and collaborative framework for extracting the data you need from websites

A real instance Scrapyd a scrapy server

Need download the builds to local

Aria2

aria2 is a lightweight multi-protocol, multi-source command-line download utility. Can be manipulated via built-in JSON-RPC and XML-RPC interfaces, and support state hook so can be used to trigger next step action

A aria2 instance

Structure 03

Befor Aria2

some other attempts

• wget, curl
• httplib of python

Now need a helper controller

• Glue the components together
• Receive and store test-results
• To organize test-cases

A web framework

Bottle

Bottle is a fast, simple and lightweight WSGI micro web-framework for Python

plus extra functions and API, author named it Igor

Structure 04

Data Management

• Fetched urls from Scrapy
• Test results from Igor
• Aria2 need to know the new build's url
• Igor need to know which test machine is available

Need a database

Mongodb

MongoDB (from humongous) is a cross-platform document-oriented database. Classified as a NoSQL database

A MongoDB instance

Structure 05

Before MongoDB

some other attempts

• csv
• sqlite

Machine Management

• Many rhevh test-cases require specific hardware configuration
  • network, e.g.: VLAN, Bond
  • storage, e.g.: iscsi, fc
  • other, e.g.: register with RHEVM
• The configuration of test-machine often changed
• need to track all these information above

Use Ansbile manage configuration

• Can easily get detailed configuration from multi-machines
• Don't need to install anything, will not effect test environment
• All data from ansbile are json, can easily export to Mongodb

Structure 06

configuration Management

Need 10+ latest RHEVM test instance

• Setup a rhevm for Ansible is a `Configuration`
• Can use Ansible simultaneously setup 10+ rhevm

Structure 07

Last Step - Add a UI

• Need find a way to visualize all data in database
• A simple dashboard will be convenient to show raw data from Ansbile

Meteor + MongoDB

• Meteor, or MeteorJS, is an open-source JavaScript web application framework, written using Node.js. Meteor allows for rapid prototyping and produces cross-platform (web, Android, iOS) code. It integrates with MongoDB and uses the DDP and a publish–subscribe pattern to automatically propagate data changes to clients without requiring the developer to write any synchronization code
• From personal opinion, it's very good to make a small web app, especially give a simple UI for automation project.

A simple machine configuration management site that built use Meteor, MongoDB and Ansible. Link

DDP Protocol

• Distributed Data Protocol
• A modified version of SockJS
• just a idea, use ddp instead of http, maybe can make automation also responsive, real-time
• especially useful in semi-automation test.