WebGL in Production

Content, Rendering and Integration in the BioDigital Human

Tarek Sherif

BioDigital

Goals

• Interactive 3D to make anatomical content engaging
• Move beyond the anatomy atlas
  • Tell stories
  • Bring content to life
  • Map data to and from 3D models
• Make it widely accessible through the web (including mobile!)

The BioDigital Human

• 3D content library
  • 5000+ anatomy objects
  • 500+ health conditions
• Rendering engine built on SceneJS
• User-created annotations and custom views
• API for embedding Human content into external websites

The BioDigital Human

• Key clients
  • Medical device manufacturers
  • Pharmaceutical
  • Educational
  • Medical students

Why WebGL?

• Pros:
  • It's everywhere!
    • (Isn't it?)
  • Use HTML/CSS/JavaScript for easy UI
  • Integrate into client web sites
  • Leverage existing web services

Why WebGL?

• Cons:
  • It's everywhere!
    • As long as you make it work!
  • JavaScript!!!
    • Common JS patterns are bad for performance

          
            object.transform({
              rotate: {
                axis: vec1.cross(vec2).normalize(),
                angle: Math.PI / 4
              } 
            })
          
        
      

WebGL in Production

• The pipeline

WebGL in Production

• Artists create models and animate them

WebGL in Production

• Export tool converts modeling format to runtime format that the WebGL engine can use efficiently
  • Meshes as binary arrays
  • Shader graphs as materials with known properties
  • Animations as transforms and morph targets

WebGL in Production

• WebGL!!!

WebGL in Production

• Deploy engine and content to our servers making them available to everyone

WebGL in Production

• Embed an iFrame with Human content in your page

WebGL in Production

• Manipulate scene in the iFrame through a JavaScript API

WebGL in Production

• Key challenges:
  • Content must be downloaded so its size should be limited
  • Running ubiquitously means dealing with limitations of low-end devices:
    • Memory and GPU limitations
    • Amplifies importance of optimization
    • But don't want to sacrifice quality on capable devices!
  • Making 3D navigation and concepts approachable to a general audience

Content

• Meshes built using Maya and ZBrush
• Textures: color, normal, specular, alpha

Content

• Animation:
  • Tweens on transforms, textures, opacity, etc.
  • Morph targets for more complex animations
  • Linear and Bezier interpolation

Content

• Accuracy:
  • Consult anatomical atlases and texts
  • Collaborate with medical professionals

Types of Content

• Static content
• Tween animations
• Morph targets
• Basic tours
• Complex tours

Content Challenges

• SIZE!
  • Artists want it beautiful: big textures, detailed geometry
  • Engineers want it to not crash on an iPhone
  • Tension between anatomical detail and application stability

Content Challenges

• Pipeline issues
  • Artists (mostly) don't write code
  • Have to map Maya properties or structures to GL variables and GLSL code

Example: Maya Ramp Node for Fresnel Effects

• Maya creates Fresnel effects using the general-purpose ramp node

Example: Maya Ramp Node for Fresnel Effects

• Maya ramp node:
  • Arbitrary gradient with unlimited stops
  • Color inputs can be textures
  • Interpolation factor can be anything

Example: Maya Ramp Node for Fresnel Effects

• Standard Fresnel equation

          
            float fresnel(eyeVec, worldNormal, color1, color2, bias, scale, power) {
              float facingRatio = dot(eyeVec, worldNormal);
              float f = bias + scale * pow(1.0 - facingRatio, power);

              return mix(color1, color2, f);
            }
          
        
      

Our Solution

• Limit Maya ramp node to:
  • Two color stops
  • No texture input (might be able to add this later)
  • Interpolation factor limited to facing ratio
• Extend fresnel function to accept two biases:
  • Center and edge biases (map to ramp node color stops)

Our Solution

• Maya ramp node

Our Solution

• SceneJS Fresnel equation

          
  float fresnel(vec3 viewDirection, vec3 worldNormal, float edgeBias, float centerBias, float power) {
      float fr = abs(dot(viewDirection, worldNormal));
      float finalFr = clamp((fr - edgeBias) / (centerBias - edgeBias), 0.0, 1.0);
      return pow(finalFr, power);
  }
          
        
      

Rendering

• Navigating anatomical content is difficult
  • Deeply nested
  • Hierarchical (did the user click on arm or hand or finger?)

Rendering

• Focus on giving users ways to view, emphasize, interact with information of interest:
  • Highlight object
  • Highlight region (defined by texture)
  • Dissect (remove an object)
  • Isolate (remove all other objects)
  • Transparency
  • Annotations
  • (+ text, audio, video, etc.)

Example

Rendering

• Some things are simple:
  • Highlight object: pick object, blend with highlight color in the fragment shader
• Some things are less so:
  • Annotations: map 3D point to 2D canvas position, manipulate DOM elements, check occlusion, follow morphing geometry

Rendering Challenges

• Mobile!
  • Rendering performance and memory can be severely limited on mobile devices
  • Possible to query some properties:
    • Variable limits (uniforms, varyings, texture units)
    • Shader precision
  • Not others:
    • GPU memory

Rendering Challenges

• ARGGGH... APPLE!!!

Working with mobile

• Shader Precision
  • On desktop mediump and highp tend to be the same
  • On mobile, not so

Shader Precision

• SceneJS used to set float precision to mediump in both the vertex and fragment shaders

          
              precision mediump float;
          
        

Shader Precision

• Result

Working with mobile

• Variable limits
  • Hardest limit to deal with so far is the limit on varyings (8!)
  • Limit on texture units (8 again!) uncovered a bug in SceneJS bookkeeping

Variable limits: Varyings

• SceneJS used to calculate the light vector and distance for each light in the vertex shader, pass to fragment shader in a varying
• Not hard to hit limit of 8 when using multiple lights

          
        SCENEJS_vViewLightVecAndDist0 = vec4(tmpVec3, length(SCENEJS_uLightPos0 - worldVertex.xyz));
        
        // ...
        
        SCENEJS_vViewLightVecAndDist1 = vec4(tmpVec3, length(SCENEJS_uLightPos1 - worldVertex.xyz));
          
        

Variable limits: Texture units

• SceneJS used to cycle through texture units for binding, assuming there would always be at least 10

          
        if (frameCtx.textureUnit > 10) {
            frameCtx.textureUnit = 0;    
        }
          
        

Solution

• Know the client!

          
        SceneJS.WEBGL_INFO.MAX_VARYING_VECTORS = gl.getParameter(gl.MAX_VARYING_VECTORS);
        SceneJS.WEBGL_INFO.MAX_TEXTURE_UNITS   = gl.getParameter(gl.MAX_COMBINED_TEXTURE_IMAGE_UNITS);

        if (gl.getShaderPrecisionFormat(gl.FRAGMENT_SHADER, gl.HIGH_FLOAT).precision > 0) {
            SceneJS.WEBGL_INFO.FS_MAX_FLOAT_PRECISION = "highp";
        } else if (gl.getShaderPrecisionFormat(gl.FRAGMENT_SHADER, gl.MEDIUM_FLOAT).precision > 0) {
            SceneJS.WEBGL_INFO.FS_MAX_FLOAT_PRECISION = "mediump";
        } else {
            SceneJS.WEBGL_INFO.FS_MAX_FLOAT_PRECISION = "lowp";
        }
          
        

Solution

• Use that knowledge!

          
        frameCtx.textureUnit = (frameCtx.textureUnit + 1) % SceneJS.WEBGL_INFO.MAX_TEXTURE_UNITS;
          
        

          
        var src = [
          "precision " + SceneJS.WEBGL_INFO.FS_MAX_FLOAT_PRECISION + " float;"
        ];
          
        

Solution

• For the varyings, we moved light vector calculations to the fragment shader
• Arguably less performant, but our tests show the effect to be negligible in practice

Human API

• Embed Human content in an external web page through an iFrame
• Basic interactions built-in
  • Mouse movement
  • Dissection, highlight, annotate, etc.
• UI can be customized through URL parameters

Human API

• For more customized interactions, use the JavaScript API
  • Communicates with iFrame using the window messaging API

          
              var human = new HumanAPI.Human("iFrameID"); 

              human.camera.flyTo({
                  eye: { z: -25 },
                  velocity: 20
              });

              human.pick.on("picked", function(e) {
                console.log(e.worldPos);
              });
          
        

Example

• About.com Diabetes Page

API Challenges

• Communication!
  • Documentation
  • Tutorials
  • Support
• Want to make everything we've discussed approachable to the average web developer
  • 3D navigation concepts
  • Anatomical concepts
  • Architecture of the Human

Get involved!

• Try out the API! https://developer.biodigital.com
• BioDigital summer internships!

Thanks!

• tarek@biodigital.com
• @thsherif
• www.biodigital.com