WebGL in Production – Content, Rendering and Integration in the BioDigital Human



WebGL in Production – Content, Rendering and Integration in the BioDigital Human

0 0


upenn-biodigital

BioDigital Human Presentation for UPenn, November 2015

On Github tsherif / upenn-biodigital

WebGL in Production

Content, Rendering and Integration in the BioDigital Human

Tarek Sherif

BioDigital

BioDigital's Goals

  • Use interactive 3D to make anatomical content engaging
  • Move beyond the anatomy atlas
    • Tell stories
    • Bring content to life
    • Map data to and from 3D models
  • Make it all widely accessible through the web (including mobile!)

The BioDigital Human

  • 3D content library
    • 5000+ anatomy objects
    • 500+ health conditions
  • Rendering engine built on SceneJS
  • User-created annotations and custom views
  • API for embedding Human content into external websites

The BioDigital Human

Key Clients
  • Medical device manufacturers
  • Pharmaceutical
  • Educational
  • Medical students

Why WebGL?

Pros
  • It's everywhere!
  • Use HTML/CSS/JavaScript for easy UI
  • Integrate into client web sites
  • Leverage existing web services
  • (Relatively) easy distribution
  • (Relatively) easy deployment of updates

In a nutshell...

  • Leverage the entire infrastructure of the Web as a platform

Why WebGL?

Cons
  • Web UX expectations
    • It should be fast
    • It should be easy
    • It should just work

Why WebGL?

Cons
  • JavaScript!!!
    • Memory management is hard
    • Common JS patterns are bad for performance
          
            object.transform({
              rotate: {
                axis: vec1.cross(vec2).normalize(),
                angle: Math.PI / 4
              } 
            })
          
        
      

WebGL in Production

The BioDigital Production Pipeline

The BioDigital Production Pipeline

  • Artists create models and animate them

The BioDigital Production Pipeline

  • Export tool converts modeling format to runtime format that the WebGL engine can use efficiently
    • Meshes as binary arrays
    • Shader graphs as materials with known properties
    • Animations as transforms and morph targets

The BioDigital Production Pipeline

  • WebGL!!!

The BioDigital Production Pipeline

  • Deploy engine and content to our servers making them available to everyone

The BioDigital Production Pipeline

  • Client can embed an iFrame with Human content in their page

The BioDigital Production Pipeline

  • Client manipulates scene in the iFrame through a JavaScript API

WebGL in Production

Key Challenges
  • Content must be downloaded so its size should be minimized
  • Running ubiquitously means dealing with limitations of low-end devices:
    • Memory and GPU limitations
    • Amplifies importance of optimization
    • But don't want to sacrifice quality on capable devices!
  • Making 3D navigation and concepts approachable to a general audience

Content

  • Meshes built using Maya and ZBrush
  • Textures: color, normal, specular, alpha

Content

  • Animation:
    • Tweens on transforms, textures, opacity, etc.
    • Morph targets for more complex animations
    • Linear and Bezier interpolation

Content

  • Accuracy:
    • Consult anatomical atlases and texts
    • Collaborate with medical professionals

Types of Content

Content Challenges

SIZE!
  • Artists want it beautiful: big textures, detailed geometry
  • Engineers want it to not crash on an iPhone
  • Tension between anatomical detail and application stability

Content Challenges

Pipeline Issues
  • Artists (mostly) don't write code
  • Have to map Maya properties or structures to GL variables and GLSL code
    • Authoring versus runtime representations of data

Example: Maya Ramp Node for Fresnel Effects

Example: Maya Ramp Node for Fresnel Effects

  • Maya creates Fresnel effects using the general-purpose ramp node

Example: Maya Ramp Node for Fresnel Effects

Maya Ramp Node
  • Arbitrary gradient with unlimited stops
  • Color inputs can be textures
  • Interpolation factor can be anything

Example: Maya Ramp Node for Fresnel Effects

Standard Fresnel Equation
          
            float fresnel(eyeVec, worldNormal, color1, color2, bias, scale, power) {
              float facingRatio = dot(eyeVec, worldNormal);
              float f = bias + scale * pow(1.0 - facingRatio, power);

              return mix(color1, color2, f);
            }
          
        

Our Solution

  • Limit Maya ramp node to:
    • Two color stops
    • No texture input (might be able to add this later)
    • Interpolation factor limited to facing ratio
  • Extend fresnel function to accept two biases:
    • Center and edge biases (map to ramp node color stops)

Our Solution

BioDigital Fresnel Node for Maya

Our Solution

SceneJS Fresnel Equation
          
  float fresnel(vec3 viewDirection, vec3 worldNormal, float edgeBias, float centerBias, float power) {
      float fr = abs(dot(viewDirection, worldNormal));
      float finalFr = clamp((fr - edgeBias) / (centerBias - edgeBias), 0.0, 1.0);
      return pow(finalFr, power);
  }
          
        
      

Rendering

  • Navigating anatomical content is difficult
    • Deeply nested
    • Hierarchical

Rendering

  • Focus on giving users ways to view, emphasize, interact with information of interest:
    • Highlight object
    • Highlight region (defined by texture)
    • Dissect (remove an object)
    • Isolate (remove all other objects)
    • Transparency
    • Annotations
    • (+ text, audio, video, etc.)

Example

The Human Heart

Rendering

  • Some things are simple:
    • Remove object: Disable its branch in the scene graph
  • Some things are less so:
    • Annotations: Map 3D point to 2D canvas position, manipulate DOM elements, check occlusion, follow morphing geometry

Rendering Challenges

MOBILE!
  • Rendering performance and memory can be severely limited on mobile devices
  • Possible to query some properties:
    • GL Variable limits (uniforms, varyings, texture units)
    • Shader precision
  • Not others:
    • GPU memory

Rendering Challenges

ARGGGH... APPLE!!!

Working with mobile

Shader Precision
  • On desktop mediump and highp tend to be the same
  • On mobile, not so

Shader Precision

  • SceneJS used to set float precision to mediump in both the vertex and fragment shaders
          
              precision mediump float;
          
        

Shader Precision

Result

Working with mobile

GL Variable Limits
  • Hardest limit to deal with so far is the limit on varyings (8!)
  • Limit on texture units (8 again!) uncovered a bug in SceneJS bookkeeping

GL Variable limits: Varyings

  • SceneJS used to calculate the light vector and distance for each light in the vertex shader, pass to fragment shader in a varying
  • Not hard to hit limit of 8 when using multiple lights
          
        SCENEJS_vViewLightVecAndDist0 = vec4(tmpVec3, length(SCENEJS_uLightPos0 - worldVertex.xyz));
        
        // ...
        
        SCENEJS_vViewLightVecAndDist1 = vec4(tmpVec3, length(SCENEJS_uLightPos1 - worldVertex.xyz));
          
        

GL Variable limits: Texture units

  • SceneJS used to cycle through texture units for binding assuming there would always be at least 10
          
        if (frameCtx.textureUnit > 10) {
            frameCtx.textureUnit = 0;    
        }
          
        

Solution

Know the Client!
          
        SceneJS.WEBGL_INFO.MAX_VARYING_VECTORS = gl.getParameter(gl.MAX_VARYING_VECTORS);
        SceneJS.WEBGL_INFO.MAX_TEXTURE_UNITS   = gl.getParameter(gl.MAX_COMBINED_TEXTURE_IMAGE_UNITS);

        if (gl.getShaderPrecisionFormat(gl.FRAGMENT_SHADER, gl.HIGH_FLOAT).precision > 0) {
            SceneJS.WEBGL_INFO.FS_MAX_FLOAT_PRECISION = "highp";
        } else if (gl.getShaderPrecisionFormat(gl.FRAGMENT_SHADER, gl.MEDIUM_FLOAT).precision > 0) {
            SceneJS.WEBGL_INFO.FS_MAX_FLOAT_PRECISION = "mediump";
        } else {
            SceneJS.WEBGL_INFO.FS_MAX_FLOAT_PRECISION = "lowp";
        }
          
        

Solution

Use That Knowledge!
          
        frameCtx.textureUnit = (frameCtx.textureUnit + 1) % SceneJS.WEBGL_INFO.MAX_TEXTURE_UNITS;
          
        
          
        var src = [
          "precision " + SceneJS.WEBGL_INFO.FS_MAX_FLOAT_PRECISION + " float;"
        ];
          
        

Solution

  • For the varyings, we moved light vector calculations to the fragment shader
  • Arguably less performant, but our tests show the effect to be negligible in practice

Human API

  • Embed Human content in an external web page through an iFrame
  • Basic interactions built-in
    • Mouse movement
    • Dissection, highlight, annotate, etc.
  • UI can be customized through URL parameters

Human API

  • For more customized interactions, use the JavaScript API
    • Communicates with iFrame using the window messaging API
          
              var human = new HumanAPI.Human("iFrameID"); 

              human.camera.flyTo({
                  eye: { z: -25 },
                  velocity: 20
              });

              human.pick.on("picked", function(e) {
                console.log(e.worldPos);
              });
          
        

Example

API Challenges

  • Want the average web developer to understand it all
    • 3D terminology and concepts
    • 3D navigation
    • Anatomical concepts
    • Architecture of the Human

Solution

  • Documentation
  • Tutorials
  • Support

Get involved!

Thanks!

0