Topics Scenarios and Motivation Image Formats Processing Pipelines - PowerPoint PPT Presentation

Web3D 2013 Tutorial : Medical and Volume Rendering with X3D Nicholas F. Polys, Ph.D. Andy Wood Virginia Tech Peter Leskovsky, John Congote, Esther Novo, Luis Kabongo, Ph.D. VicomTech

Topics • Scenarios and Motivation • Image Formats • Processing Pipelines • Presenting and Publishing (X3D, X3DOM)

Volume Rendering more than medical imaging • Geology • Meteorology • FlowVisualization • Microscopy • Paleontology • Non-invasive sensing: – Transportation security – Manufacturing QA ….

Web3D Medical Working Group • Reproducible rendering for stakeholders throughout the healthcare enterprise • An n-D Presentation must include: – Structured and interactive virtual environment display (2D & 3D objects and time series) – Platform-independent, royalty-free technology to enable vendor innovation – Can be rendered with or without stereoscopy – Openly-published

X3D Volume Rendering • Necessary and Sufficient node set for industry’s Greatest Common Denominator: – Volume Component : render styles – X3D version 3.3 • Two independent implementations: – www.h3d.org – www.instantreality.org

Volume Processing and Presentation Tools • Data – Sample xxxxx.dcm – X3D Content Examples http://www.web3d.org/x3d/content/examples/Basic/VolumeRendering/index.html – Volvis.org – http://www.osirix-viewer.com/datasets/ • Warning: some are compressed w/ jpg2000 ! • Tools – ImageJ : http://rsbweb.nih.gov/ij/ • Plugins: DICOM reader, DICOM header inspector – Seg3D.org – Slicer.org; ITK-Snap – X3D-Edit 3.3 • Viewers – H3D.org – InstantReality.org – MedX3DOM

X3D Volume Rendering • Composable Render Styles covering the state of the art – Formalizes parameters and transfer functions for 3D rendering & blending • BoundaryEnhancementVolumeStyle • CartoonVolumeStyle • ComposedVolumeStyle • EdgeEnhancementVolumeStyle • OpacityMapVolumeStyle • ProjectionVolumeStyle • ShadedVolumeStyle • SilhouetteEnhancementVolumeStyle • ToneMappedVolumeStyle Silhouette – Opacity Map Greatest Common Denominator Cartoon • Assign different RenderStyles to different segments, blend two volumes – BlendedVolumeStyle – SegmentedVolumeData – IsoSurfaceVolumeData • Clipping Planes are already specified in X3D 3.2 Rendering Component

Medical Interchange Profile • X3D 3.3 • Minimal X3D node set (footprint) to meet DICOM requirements: – Core – Time – Networking – Grouping – Rendering – Shape – Geometry3D – Geometry2D – Text – Lighting – Texturing – Interpolation – Navigation – Environmental effects – Event utilities – Texturing3D – Volume rendering Includes polygon, line and point rendering; metadata on any node

ISO Process • [ X3D 3.3 ] ISO/IEC DIS 1 19775-1:2008 – Clause 33 : Texturing3D Component – Clause 41 : Volume Rendering Component – Annex L: Medical Interchange Profile • Final Draft International Standard (FDIS) • Change document and unified spec drafts available to Web3D & DICOM members (* now published)

Online Videos & Examples • Web Video summary: – Extensible 3D (X3D) Volume Rendering • https://snoid.sv.vt.edu/medical/X3DVolumes/videos/Volume Vis-X3D-collected.mp4 (65 mb) • X3D Examples – http://www.web3d.org/x3d/content/examples/Basic/ VolumeRendering/index.html • For other other Videos, Images and Scenes using the VolumeData and VolumeRenderStyles of X3D 3.3 Clause 41, please visit: – https://snoid.sv.vt.edu/medical/X3DVolumes/

Volume Presentation Many techniques: • Volume rendering – 3DSplatting, ray tracing, pixelshaders – Established CPU and GPU algorithms • Surfaces – actual meshes • Segments – identifying voxels as groups • ISOSurfaces – rendered at a threshold I: Scenarios and Motivation

X3D Presentation Demo Screenshots • Nicholas Polys, Andy Wood Virginia Tech

Example Volume Rendering Styles (Head MRI, XML encoding) <Transform DEF='backdrop' > <VolumeData dimensions='.75 1 1' > <ImageTexture3D containerField="voxels" url='"./Segments/masked-vispart.nrrd"'/> <OpacityMapVolumeStyle /> </VolumeData> </Transform>

Example Volume Rendering Styles (Head MRI, optic segment) <ISOSurfaceVolumeData surfaceValues='.15' dimensions='.75 1 1' > <ImageTexture3D containerField="voxels" url='"./Segments/masked-optic.nrrd"'/> <CartoonVolumeStyle /> </ISOSurfaceVolumeData> (Head MRI, cerebrum segment) <VolumeData dimensions='.75 1 1' > <ImageTexture3D containerField="voxels" url='"./Segments/masked-cerebrum.nrrd"'/> <ComposedVolumeStyle> <CartoonVolumeStyle /> <EdgeEnhancementVolumeStyle gradientThreshold='.8' edgeColor='0 0 .5' /> </ComposedVolumeStyle> </VolumeData> I: Scenarios and Motivation

Image Formats • 2D image slices with order and spacing define the voxels • Typical image stacks as: – Tiff – Png – dcm • Sizing, smoothing, resampling can happen in 2.5D space (image stack)

Image J • Many useful (and programmable) functions to deal with image stacks: • Invert, Blur, resample, resize reduce bit depth • ImageJ : http://rsbweb.nih.gov/ij/ • Lots of Plugins: eg DICOM reader, DICOM header inspector III: Processing Pipelines

DICOM.dcm • Medical.nema.org • Ongoing work Item for n-Dimensional Presentation State • Working Group 11 (Display) • Extensive use cases documented • MPR and CPR parameters of members surveyed • Supplement draft in preparation

NRRD • Nearly Raw Raster Data • http://teem.sourceforge.net/nrrd/ • Nrrd and the teem library facilitates the pre- processing tasks (cropping, quantizing, resampling), as well as the basic analysis tasks which help to understand the structure of a dataset (histograms, maximum intensity projections, connected components) II: Image Formats

Seg3D Segment3D – a Gui tool With multiple methods for segmentation : - Neighborhood Growth - Thresholding Export .nrrd - all-in-one - separate files Seg3d.org III: Processing Pipelines

Slicer • Slicer.org

ITKSnap • http://www.itksnap.org

X3D Edit 3.3 • X3D-Edit is an open-source Extensible 3D (X3D) Graphics authoring tool for simple error- free editing, authoring and validation of X3D scenes. • Lots of translation and QA features • Stand-alone or Netbeans plug-in • https://savage.nps.edu/X3D-Edit/

Processing Scheme for DICOM Volume Use Cases acquisition Surfacing DICOM data Mesh: Other 3D Supp 132 data Segmentation (e.g. CAD, X3D) Segmentation: Supp 111 Compose WCS Model (Scene) for Rendering; assign appearances, views, etc. nDPS X3D Viewer workstation

Volume Processing Pipelines Source Data X3D Image <ImageTexture2D> Stack <VolumeData> Image DICOM Raw Processor <ImageTexture3D> -voxels Plugin Volume (ImageJ) <IsoSurfaceVolumeData> Volume Data NRRD <ImageTexture3D> -voxels Processing Volume (TEEM) <SegmentedVolumeData> Individual Segmentation <ImageTexture3D> -voxels Segments • Seg3D DICOM <ImageTexture3D> • Slicer Stack -segment identifiers • ITKSnap Segment IDs

Volume Image Data Source Data X3D Image <ImageTexture2D> Stack <VolumeData> Image DICOM Raw Processor <ImageTexture3D> -voxels Plugin Volume (ImageJ) <IsoSurfaceVolumeData> Volume Data NRRD <ImageTexture3D> -voxels Processing Volume (TEEM) <SegmentedVolumeData> Individual Segmentation <ImageTexture3D> -voxels Segments • Seg3D DICOM <ImageTexture3D> • Slicer Stack -segment identifiers • ITKSnap Segment IDs

Source Data • Stacks of 2D images with order and spacing data generated by imaging machinery • Some typical file formats for image stacks: – Tiff – Png – Dcm • Image processing techniques can accomplish much of the work for rendering in 2.5D – Smoothing, (simple) segmentation, resampling, contrast, etc.

Working with Images • Resolution – number of pixels in x and y dimensions • Channels – color images are 3 or 4 channel (RGB or RGBA), typical volume stack is single channel (greyscale) • Bit Depth – number of bits to encode value of a pixel, per channel • Volume data becomes large quickly: – 512 512x512 8-bit greyscale images: 134 million voxels and 128MB (raw) – 1024^3 volume, 16-bit RGB: 1.07 billion voxels and 48GB (raw) • ImageJ – Image processor designed to work with stacks – Many useful (and scriptable) functions: invert, blur, resample, resize, mask, etc. – Website: http://rsbweb.nih.gov/ij/

Volume Data Source Data X3D Image <ImageTexture2D> Stack <VolumeData> Image DICOM Raw Processor <ImageTexture3D> -voxels Plugin Volume (ImageJ) <IsoSurfaceVolumeData> Volume Data NRRD <ImageTexture3D> -voxels Processing Volume (TEEM) <SegmentedVolumeData> Individual Segmentation <ImageTexture3D> -voxels Segments • Seg3D DICOM <ImageTexture3D> • Slicer Stack -segment identifiers • ITKSnap Segment IDs