Project Status Report – What is MapServer? – MapServer History

Project Status Report

FOSS4G-Europe 2015 - Como - 2015-07-15

Stephan Meißl stephan.meissl@eox.at @Schpidi

Thomas Bonfort tbonfort@terriscope.fr @tbonfort

 This work is licensed under a Creative Commons Attribution 4.0 International License.

What is MapServer?

• Open Source platform for publishing spatial data to the Web
• MIT-style license
• Runs on all major platforms (Windows, Linux, Mac OS X)
• OSGeo project
• mapserver.org

MapServer History

• 1994: Steve Lime implements Web-based Arc/Info AML generator
• 1994-95?: Frank Warmerdam releases Shapelib
• 1997: v1.0 (limited sharing)
• 1998: v2.0 via Fornet website
• 1999: v3.0 via UMN MapServer website
• 2001: PostGIS support by Paul Ramsey in v3.6
• 2002: GRASS Users Conference - Trento
• 2003-05: Yearly MapServer Users Meeting
• 2006: Foundation of OSGeo with MapServer as founding project
• 2006+: FOSS4G Conferences
• 2009+: C Tribe Code Sprints
• 2012: Addition of TinyOWS & MapCache ⇒ MapServer Suite (v6.2)
• 2013: v6.4
• 2014: v7.0beta & 20th anniversary
• 2015: v7.0 (beta2 released 07/2015)

Statistics

• ~150k Lines Of Code
• ~12k commits
• ~100 contributors
• Mailing lists
  • -users ~1800
  • -dev ~400
• International PSC
  • 14 members
  • NA: 9 (US: 5, CA: 4)
  • EU: 5 (FR: 2, FI: 1, AT: 1, HU: 1)
• 113 RFCs

MapServer 7.0

• Main new features
  • UTFGrids
  • WFS 2.0
  • Heatmaps/Density Maps
  • Javascript/V8 StyleItem/Geomtransform
  • Unification of attribute/geometry filtering
  • Layer Level Compositing / Blending
• Others
  • Text rendering re-factoring
  • GD Removal
  • Removed BITMAP labels
  • Layer Level Encoding
  • WCS 2.0

UTFGrids

• Specification by MapBox for vector tiles
• Implemented in a "Google Summer of Code" project
• Interactivity important (rollover, events)
• Supported by OpenLayers

WFS 2.0

• Download Services for INSPIRE

Javascript (V8) StyleItem/GeomTransfom

• Integration of the V8 JavaScript engine
• Style geometries/features programatically rather than through class expressions
• "Experimental mode": backward compatibility not guaranteed

LAYER
    ...
    STYLEITEM "javascript:///path/to/my/file.js"
    CLASS # empty class required
    END
END

//Make symbol size 14 or 7
  var size = shape.attributes.NAME.length > 10 ? 14:7;

  var style = "STYLE SIZE " + size;
  style += " SYMBOL 'circle'";
  style += " COLOR 255 0 0 END";

  //Return style to MapServer
  style;

Unification of attribute/geometry filtering

• Currently filtering done by MapServer
• PostGIS, Oracle, OGR support native spatial filtering e.g.

where st_intersects(the_geom, st_geomfromtext('wkt...'))

WCS 2.0

• Implements available extensions
• Passes all available CITE tests
• Official OGC WCS 2.0 Reference Implementation

HeatMaps

• Vector to raster processing pipeline
• Interpolation and rendering parameters
  • Automatic scaling
  • Interpolation in color spaces RGB/HSL
  • Weighting per attribute/expression
• Supports tile mode

Interpolation in HSL color space

Interpolation in HSL color space

Interpolation in RGB color space

Pure density

Setting a fixed scale

Layer Blending

Layer Filters

Text rendering re-factoring

• Before naive implementation duplicated for each back-end
• Integration of Harfbuzz library
• Supports complex text layout e.g. for Arabic text
  • Unprocessed characters
  • With bi-directional processing
  • With glyph shaping rendering letters according to context
• Barely visible to the end user, but
  • Exact label placement (alignment/centering/letter spacing of text)
  • Possibility of using one font per language

Improving performance (notice the log scales)

Malayalam Example

Tibéto-Birman Example

GD Removal

• Added back in 1995
• Too high maintenance costs despite its usefulness in certain niches
• Automatic fall-back to AGG + 8bit

Removed BITMAP labels

• Truetype font included by default now
• Supports many glyphs without external dependencies
• A few use cases are not supported
• Automatic fall-back in place

Layer Level Encoding

• Solves encoding handling problems e.g. data sources in multiple encodings in a single mapfile
• Data sources internally converted to UTF8
• The mapfile has to be encoded in UTF8 now
• All output generated by MapServer in UTF8

MapCache 1.4

• Tiling Library with Apache, FastCGI, Nginx and Node frontends
• Versatile
  • Multiple Cache Backends
  • Multiple Protocols
  • Tile Management Features: seeding, recompression, interpolation
• Native code: Fast and Efficient

Standard Tile Addressing

• TMS
• WMTS (RestFull + KVP)
• VirtualEarth QuadKeys
• MapGuide
• Google x,y,z

WMS

• WMS-Tiled
• Responds to untiled requests by assembling tiles
• Can mirror a full WMS service: layer-by-layer, layer lists
• Forwards GetFeatureInfo requests

(OGC) Proxy

• Intercepts requests that can be served from cached tiles
• Forwards other requests to other upstream server(s)

Miscellaneous

• Caching HTTP Headers
• "Auto" Expiry of old tiles
• Error Reporting
  • Error Message
  • Empty Image
  • Status Code
• Meta-Tiling
• Watermarking
• Max-Cached-Zoom: Upsample lower level tiles

Image Operations

• Image data can be recompressed/optimized before being stored into a tile
  • Avoid double JPEG compression, i.e. request PNG from source WMS
  • Be agressive on compression when storing a tile (e.g. quantized PNG with max compression)
  • CPU time for dynamic recompressions can be lowered, at the cost of more bandwidth

Image Formats

• PNG
  • Compression level
  • Quantization (color reduction to 8bit palette)
• JPEG
  • Compression level
  • RGB/YCbCr
• Mixed (JPEG + PNG)
  • Use for raster coverages on limited areas
  • PNG for tiles with transparency
  • JPEG otherwise

Handling Empty/Uniform Tiles

• Strategy dependent on the cache backend
• • Symbolic linking
  • Reconstruct Paletted 1bit PNG from uniform color
  • For fully seeded caches, treat absent tile as fully transparent

Grids

• Multiple Grids per Tileset (e.g. WGS84 + Mercator)
• Handles Grid Aliasing
  • EPSG:3857/EPSG:900913
  • Non Standard Codes
• Grid Subsets
  • min/max resolution
  • Restricted extent (with tolerance!)

Caches

• Backend able to store tile data for given x,y,z
• Extensible: Exists(), Get(), Put(), Delete()
• Some backend specific hacks: symlinking, blank detection

Disk Cache

• Mainly for dev, testing, or small tilesets
• Pros
  • Simple, just provide a base directory
  • Fast
  • Supports blank tile detection
  • (Re-)Use custom layouts: TileCache, ArcGis, Worldwind, Template
• Cons
  • Difficult to manage large number of files
  • Hits File System Limits
  • Wasted space due to File Sytem block size

Sqlite Cache

• Stores tile data as blobs in an sqlite file (/path/to/{grid}/{tileset}/{z}.db)
• Can be split into multiple db files (per dimension, grid, tileset. Configurable max number of tiles per db)
• Pros
  • Single File
  • Support any schema (MBtiles, custom...)
  • Efficient Space (blank tile detection possible)
• Cons
  • Some tweaking needed for very large caches (more than 1 Tb)
  • Concurrent Insertions are slow

MemCache

• Ideal for Temporary Data: Forecasts, Sensors, ...
• Pros
  • Easily distributed
  • Automatic pruning
• Cons
  • Limited Storage

REST Caches

• REST Back-end
  • Support GET/PUT/DELETE for storing tiles
  • Authentication/authorization plug-ins for S3, Azure, and Google Cloud Storage
  • Costs highly depend on the use case (number of applications, bandwidth)

(Geo)TIFF Cache

• Specialized storage for satellite imagery basemaps
• JPEG tiles are read directly from a constrained (Geo)TIFF file hierarchy
• Configurable number of tiles per TIFF file
• Templated access to file name
• Pros
  • Efficient on disk usage (TIFF container)
  • TIFF files can be shared with other applications
• Cons
  • Limited to JPEG data (no transparency)
  • No Concurrent writing to a single TIFF file
  • Updates/deletion of individual tiles does not reclaim storage space
• Use-Case

Misc Caches

• Riak (distributed, extensible, fault tolerant)
• Berkeley DB
• Couchbase (soon)
• Redis (soon)

Composing Caches

• Composite Caches Select different caches based on rules (zoom level, grid)
• Fallback Caches
• Multitier Caches mix caches with different performance/storage characteristics. If a tile is not found in the fast/small/expensive/transient cache, it will be looked for in the slow/large/cheap/permanent cache and moved into the first one for subsequent accesses.

Seeder: Pre-generating tiles

• Multi thread/process seeding
• "Drill-Down" mode to take advantage of caching on source WMS
• Seed particular subset
  • Given Dimension
  • Regenerate old tiles
  • Restricted min/max zoom and/or extent
  • Restrict to geometry (e.g. seed to level 18 all areas where population density is over a given value)
• Pruning mode

Dimensions

• Multiple Dimensions per Tileset
• E.g. Forecast Time, Elevation, Client ID
• Multiple Types
  • List of Values
  • Intervals
  • Regular Expression
• Use-Case: BoM MetEye
• Coming soon: Tile assembling when multiple dimension values passed

Time Dimension

• Each timestamp corresponds to a limited extent
• Dynamic Mosaic given a requested time interval
• Create Animated GIF from selected tiles (soon)

Documentation translation

• Started to use transifex for simple collaboration
• Attract further translations

How to Contribute?

• Users
  • Beta and release candidate testing
  • Bug reports on Github, including test cases
  • Documentation (wiki, transifex, Github pull requests)
  • Answers on the mapserver-users list and IRC
• Developers
  • Bug fixes/patches
  • New features
  • Binary builds