hbase-datatypes-wip

HBase Data Types (WIP)

Nick Dimiduk Member of Technical Staff, HBase HBase Birds of a Feather, Hadoop Summit 2013

motivation

• byte[] is necessary but not sufficient for building applications
• every application must (re-)solve it's own subset of the "serialization problem"
• ecosystem tools must agree in order to interoperate
• application code full of pre/post client API-call serialization litter

design

tricky bits

• encoding format => data on disk => painful to change
• respect encoding performance overhead in tight loops
• user extensible, user usable API
  • realistic API that real apps can build against
  • don't impost on POJOs (no forced interfaces, &c)
  • avoid magic (no ASM, no AOP, avoid ORM scorn)
• community agreement on what data types to ship out of the box
• more stuff I haven't {thought of,encountered} yet

inspiration

• order-preserving encoding strategies: SQLite4, Orderly, Phoenix
• user extensible data types: PostgreSQL
• (SQL) data types and API supporting execution engines Hive, Pig, Phoenix, Impala
• Experiments with client APIs: Typo

implementation

• HBASE-8089: Add type support (parent ticket)
• HBASE-8201: Implement serialization strategies (patch available)
• HBASE-8694: Performance evaluation of serialization strategies (unassigned)
• HBASE-8693: Implement extensible type API based on serialization primitives (WIP)
• HBASE-7941: Provide client API with support for primitive types (unassigned)
• HBASE-8593: Type support in ImportTSV tool (WIP)

future directions

• consider data types anywhere else users touch HBase
  • bundled MR jobs
  • Filters
• extend RegionServer API with type support (?!?)
  • Coprocessors?
  • StoreFile formats?
• non-Java implementation for non-Java clients

thanks!