CrowdDB

Presentation ofFranklin et al. CrowdDB: Answering Queries with Crowdsourcing

The problems

Crowdsourcing

• Distribute workload to a large group of people
• Small tasks with no training requirements
• Replica tasks

Amazon Mechanical Turk

• Human Intelligent Task(HIT) The smallest amount of work a worker can accept
• Assignment A replica of a HIT
• HIT group A automaticly created group of HITs

Example HIT

CrowdDB

• Combines a database and AMT
• Uses a declerative query languageas a interface for AMT

Design consideration

• Performance and variability
• Task design and ambiguity
• Affinity and learning
• Relativly small worker pool
• Open versus closed world

Architecture

User interface generation

• User interface has a large impact on performance.
• CrowdDB automaticly creates UI for tasks.
• Options are leading

CrowdSQL

A superset of SQL

CrowdSQL

• CROWD keyword
• CNULL
• CROWDEQUAL
• CROWDORDER

CrowdSQL - examples

    CREATE TABLE Department (
      university STRING,
      name STRING,
      url CROWD STRING,
      phone STRING,
      PRIMARY KEY (university, name)
    );

    CREATE CROWD TABLE Professor (
      name STRING PRIMARY KEY,
      email STRING UNIQUE,
      university STRING,
      department STRING,
      url STRING,
      FOREIGN KEY (university, department)
        REF Department(university, name)
    );

CrowdSQL - examples

SELECT p FROM picture
  WHERE subject = "Golden Gate Bridge"
  ORDER BY CROWDORDER(p, "Which picture visualizes better %subject");

Experiments

Tuning parameters

• Response time
• Responsiveness
• Worker quality

Tuning parameters

Complex queries

• Entity resolution
• Ordering pictures
• Joining on unknown values