Project Status Report – What is MapServer? – MapServer History



Project Status Report – What is MapServer? – MapServer History

1 0


foss4g-e_2015_mapserver_status

MapServer Project Status Report presentation at FOSS4G-Europe 2015

On Github Schpidi / foss4g-e_2015_mapserver_status

Project Status Report

FOSS4G-Europe 2015 - Como - 2015-07-15

Stephan Meißl stephan.meissl@eox.at @Schpidi

Thomas Bonfort tbonfort@terriscope.fr @tbonfort

 This work is licensed under a Creative Commons Attribution 4.0 International License.

What is MapServer?

  • Open Source platform for publishing spatial data to the Web
  • MIT-style license
  • Runs on all major platforms (Windows, Linux, Mac OS X)
  • OSGeo project
  • mapserver.org

MapServer History

  • 1994: Steve Lime implements Web-based Arc/Info AML generator
  • 1994-95?: Frank Warmerdam releases Shapelib
  • 1997: v1.0 (limited sharing)
  • 1998: v2.0 via Fornet website
  • 1999: v3.0 via UMN MapServer website
  • 2001: PostGIS support by Paul Ramsey in v3.6
  • 2002: GRASS Users Conference - Trento
  • 2003-05: Yearly MapServer Users Meeting
  • 2006: Foundation of OSGeo with MapServer as founding project
  • 2006+: FOSS4G Conferences
  • 2009+: C Tribe Code Sprints
  • 2012: Addition of TinyOWS & MapCache ⇒ MapServer Suite (v6.2)
  • 2013: v6.4
  • 2014: v7.0beta & 20th anniversary
  • 2015: v7.0 (beta2 released 07/2015)

Statistics

  • ~150k Lines Of Code
  • ~12k commits
  • ~100 contributors
  • Mailing lists
    • -users ~1800
    • -dev ~400
  • International PSC
    • 14 members
    • NA: 9 (US: 5, CA: 4)
    • EU: 5 (FR: 2, FI: 1, AT: 1, HU: 1)
  • 113 RFCs

MapServer 7.0

  • Main new features
    • UTFGrids
    • WFS 2.0
    • Heatmaps/Density Maps
    • Javascript/V8 StyleItem/Geomtransform
    • Unification of attribute/geometry filtering
    • Layer Level Compositing / Blending
  • Others
    • Text rendering re-factoring
    • GD Removal
    • Removed BITMAP labels
    • Layer Level Encoding
    • WCS 2.0

UTFGrids

  • Specification by MapBox for vector tiles
  • Implemented in a "Google Summer of Code" project
  • Interactivity important (rollover, events)
  • Supported by OpenLayers

WFS 2.0

  • Download Services for INSPIRE

Javascript (V8) StyleItem/GeomTransfom

  • Integration of the V8 JavaScript engine
  • Style geometries/features programatically rather than through class expressions
  • "Experimental mode": backward compatibility not guaranteed
LAYER
    ...
    STYLEITEM "javascript:///path/to/my/file.js"
    CLASS # empty class required
    END
END
//Make symbol size 14 or 7
  var size = shape.attributes.NAME.length > 10 ? 14:7;

  var style = "STYLE SIZE " + size;
  style += " SYMBOL 'circle'";
  style += " COLOR 255 0 0 END";

  //Return style to MapServer
  style;

Unification of attribute/geometry filtering

  • Currently filtering done by MapServer
  • PostGIS, Oracle, OGR support native spatial filtering e.g.
where st_intersects(the_geom, st_geomfromtext('wkt...'))
Significant performance gain particularly in WFS queries Simpler to use Was blocking release of version 7.0

WCS 2.0

  • Implements available extensions
  • Passes all available CITE tests
  • Official OGC WCS 2.0 Reference Implementation

HeatMaps

  • Vector to raster processing pipeline
  • Interpolation and rendering parameters
    • Automatic scaling
    • Interpolation in color spaces RGB/HSL
    • Weighting per attribute/expression
  • Supports tile mode

Interpolation in HSL color space

Interpolation in HSL color space

Interpolation in RGB color space

Pure density

Setting a fixed scale

Layer Blending

Layer Filters

Text rendering re-factoring

  • Before naive implementation duplicated for each back-end
  • Integration of Harfbuzz library
  • Supports complex text layout e.g. for Arabic text
    • Unprocessed characters
    • With bi-directional processing
    • With glyph shaping rendering letters according to context
  • Barely visible to the end user, but
    • Exact label placement (alignment/centering/letter spacing of text)
    • Possibility of using one font per language

Improving performance (notice the log scales)

Malayalam Example

Tibéto-Birman Example

GD Removal

  • Added back in 1995
  • Too high maintenance costs despite its usefulness in certain niches
  • Automatic fall-back to AGG + 8bit

Removed BITMAP labels

  • Truetype font included by default now
  • Supports many glyphs without external dependencies
  • A few use cases are not supported
  • Automatic fall-back in place

Layer Level Encoding

  • Solves encoding handling problems e.g. data sources in multiple encodings in a single mapfile
  • Data sources internally converted to UTF8
  • The mapfile has to be encoded in UTF8 now
  • All output generated by MapServer in UTF8

MapCache 1.4

  • Tiling Library with Apache, FastCGI, Nginx and Node frontends
  • Versatile
    • Multiple Cache Backends
    • Multiple Protocols
    • Tile Management Features: seeding, recompression, interpolation
  • Native code: Fast and Efficient

Standard Tile Addressing

  • TMS
  • WMTS (RestFull + KVP)
  • VirtualEarth QuadKeys
  • MapGuide
  • Google x,y,z

WMS

  • WMS-Tiled
  • Responds to untiled requests by assembling tiles
  • Can mirror a full WMS service: layer-by-layer, layer lists
  • Forwards GetFeatureInfo requests

(OGC) Proxy

  • Intercepts requests that can be served from cached tiles
  • Forwards other requests to other upstream server(s)

Miscellaneous

  • Caching HTTP Headers
  • "Auto" Expiry of old tiles
  • Error Reporting
    • Error Message
    • Empty Image
    • Status Code
  • Meta-Tiling
  • Watermarking
  • Max-Cached-Zoom: Upsample lower level tiles

Image Operations

  • Image data can be recompressed/optimized before being stored into a tile
    • Avoid double JPEG compression, i.e. request PNG from source WMS
    • Be agressive on compression when storing a tile (e.g. quantized PNG with max compression)
    • CPU time for dynamic recompressions can be lowered, at the cost of more bandwidth

Image Formats

  • PNG
    • Compression level
    • Quantization (color reduction to 8bit palette)
  • JPEG
    • Compression level
    • RGB/YCbCr
  • Mixed (JPEG + PNG)
    • Use for raster coverages on limited areas
    • PNG for tiles with transparency
    • JPEG otherwise

Handling Empty/Uniform Tiles

  • Strategy dependent on the cache backend
    • Symbolic linking
    • Reconstruct Paletted 1bit PNG from uniform color
    • For fully seeded caches, treat absent tile as fully transparent

Grids

  • Multiple Grids per Tileset (e.g. WGS84 + Mercator)
  • Handles Grid Aliasing
    • EPSG:3857/EPSG:900913
    • Non Standard Codes
  • Grid Subsets
    • min/max resolution
    • Restricted extent (with tolerance!)

Caches

  • Backend able to store tile data for given x,y,z
  • Extensible: Exists(), Get(), Put(), Delete()
  • Some backend specific hacks: symlinking, blank detection

Disk Cache

  • Mainly for dev, testing, or small tilesets
  • Pros
    • Simple, just provide a base directory
    • Fast
    • Supports blank tile detection
    • (Re-)Use custom layouts: TileCache, ArcGis, Worldwind, Template
  • Cons
    • Difficult to manage large number of files
    • Hits File System Limits
    • Wasted space due to File Sytem block size

Sqlite Cache

  • Stores tile data as blobs in an sqlite file (/path/to/{grid}/{tileset}/{z}.db)
  • Can be split into multiple db files (per dimension, grid, tileset. Configurable max number of tiles per db)
  • Pros
    • Single File
    • Support any schema (MBtiles, custom...)
    • Efficient Space (blank tile detection possible)
  • Cons
    • Some tweaking needed for very large caches (more than 1 Tb)
    • Concurrent Insertions are slow

MemCache

  • Ideal for Temporary Data: Forecasts, Sensors, ...
  • Pros
    • Easily distributed
    • Automatic pruning
  • Cons
    • Limited Storage

REST Caches

  • REST Back-end
    • Support GET/PUT/DELETE for storing tiles
    • Authentication/authorization plug-ins for S3, Azure, and Google Cloud Storage
    • Costs highly depend on the use case (number of applications, bandwidth)

(Geo)TIFF Cache

  • Specialized storage for satellite imagery basemaps
  • JPEG tiles are read directly from a constrained (Geo)TIFF file hierarchy
  • Configurable number of tiles per TIFF file
  • Templated access to file name
  • Pros
    • Efficient on disk usage (TIFF container)
    • TIFF files can be shared with other applications
  • Cons
    • Limited to JPEG data (no transparency)
    • No Concurrent writing to a single TIFF file
    • Updates/deletion of individual tiles does not reclaim storage space
  • Use-Case

Misc Caches

  • Riak (distributed, extensible, fault tolerant)
  • Berkeley DB
  • Couchbase (soon)
  • Redis (soon)

Composing Caches

  • Composite Caches Select different caches based on rules (zoom level, grid)
  • Fallback Caches
  • Multitier Caches mix caches with different performance/storage characteristics. If a tile is not found in the fast/small/expensive/transient cache, it will be looked for in the slow/large/cheap/permanent cache and moved into the first one for subsequent accesses.

Seeder: Pre-generating tiles

  • Multi thread/process seeding
  • "Drill-Down" mode to take advantage of caching on source WMS
  • Seed particular subset
    • Given Dimension
    • Regenerate old tiles
    • Restricted min/max zoom and/or extent
    • Restrict to geometry (e.g. seed to level 18 all areas where population density is over a given value)
  • Pruning mode

Dimensions

  • Multiple Dimensions per Tileset
  • E.g. Forecast Time, Elevation, Client ID
  • Multiple Types
    • List of Values
    • Intervals
    • Regular Expression
  • Use-Case: BoM MetEye
  • Coming soon: Tile assembling when multiple dimension values passed

Time Dimension

  • Each timestamp corresponds to a limited extent
  • Dynamic Mosaic given a requested time interval
  • Create Animated GIF from selected tiles (soon)

Documentation translation

  • Started to use transifex for simple collaboration
  • Attract further translations

How to Contribute?

  • Users
    • Beta and release candidate testing
    • Bug reports on Github, including test cases
    • Documentation (wiki, transifex, Github pull requests)
    • Answers on the mapserver-users list and IRC
  • Developers
    • Bug fixes/patches
    • New features
    • Binary builds

Project Status Report

FOSS4G-Europe 2015 - Como - 2015-07-15

schpidi.github.io/foss4g-e_2015_mapserver_status

Stephan Meißl stephan.meissl@eox.at @Schpidi

Thomas Bonfort tbonfort@terriscope.fr @tbonfort

 This work is licensed under a Creative Commons Attribution 4.0 International License.

Let's keep up the good work.

MapServer and the OSGeo community have revolutionized the field of GIS...

and you are all part of that revolution.