Presentation at the STI Cell BE Workshop "Geoscience and - - PowerPoint PPT Presentation

Prof.’s M. Halem and Ye Yesha "Geoscience and Aerospace Applications

• n a Potential Heterogenous Cell BE Cluster

at UMBC"

Presentation at the STI Cell BE Workshop

At Georgia Tech Univ. June 18-19, 2007 halem@umbc.edu

6/18/07 Page 2

Overview

• UMBC Background
• iCLASS Configuration
• Cell Research Application areas:
• Aerospace
• Geoscience
• Other
• Summary

6/18/07 Page 3

The University of Maryland Baltimore County (UMBC)

• One of the three research campuses

in the University of Maryland System

• UMBC is a tier 1 doctoral research

extensive university -- Carnegie Foundation

• Has 500 full time and 335 part time faculty
• 9.5K undergraduate and 2.5K grad. students
• Located in suburban Baltimore County,

between Baltimore and Washington DC.

• Focus on science, engineering, information technology

and public policy with ~$85M in external research funding

• NY Times, May 25,2006; Science Jnl March 25 “UMBC Science Program

Remaking Science Education”

• UMBC’s Meyerhoff Scholarship Program produces approx. 50

undergraduates/year; 90% matriculate to advanced degrees in science.

6/18/07 Page 4

CSEE @ UMBC

Computer Science & Electrical Engineering

• UMBC’s largest department with 45 faculty,

~1000 undergrad, ~200 grad students

Degree programs (graduate and undergraduate)

• Computer Science, Computer Engineering, Electrical Engineering
• UMBC is #1 in BS CS production and #18 in PhD production (see 2004

NSF data) among research universities

Many institutes, centers and labs

• Institute for Language and Information Technology, Center for Information

Security and Assurance, Center for Photonics, Lab For Advanced Information Technology, interdisciplinary Computational Lab for Autonomous Systems and Services (iCLASS), VLSI Lab, …

Breadth and focus in research areas

• ~ $5M/year in sponsored research from Government and industry
• Pervasive computing, AI, security, information retrieval, Computational

Science, graphics, databases, VLSI, …

6/18/07 Page 5

UMBC/CSEE

iCLASS Computing Configuration

• The CSEE department provides an ‘interdisciplinary Computational

Laboratory for Autonomic Systems and Services’ (iCLASS) within the UMBC School of Engineering that supports high productivity computational research across campus

• Mission of iCLASS is to prepare UMBC for large scale team science

research in partnership with universities, industry and government by enabling an optical based network cyberinfrastructure for prototyping of large compute and data intensive computational systems in the areas of aerospace, geosciences, defense and medical imaging

• Focus of research applications has been drawn from environmental and life

science modeling, web-based service oriented science and engineering, pervasive computing, communicating devices, visualization and data preservation

• The Computational Lab currently consists of an IBM JS20/JS21 cluster

“ bluegrit” and an Intel based IBM cluster of PCs “matisse” driving a Hyperwall for visualization. In addition, ‘bluegrit’ is part of the University System of Maryland Grid and a member of SURAgrid.

6/18/07 Page 6

iCLASS Compute Servers

Bluegrit

Head Node

• IBM X Series 346
• Red Hat Linux Enterprise 4.0
• Intel 2.8 GHz Intel Xenon with Hyper Threading
• 256 MB DDR SDRAM

Compute Nodes

• IBM Blade Center with 34 JS20 Blades and 12 JS21 Blades
• Red Hat Linux Enterprise 4.0
• 2 x 2.20 GHz PowerPC 970 Processors per JS20 Blade and

4x2.5 GHz Power PC 970 Processors

• Total number of processors is 116
• 512 MB DDR SDRAM per JS20 Blade and 2GB per JS21
• 40 GB hard disk per JS20 Blade and 0 hard disk per JS21
• Proposing to acquire a few Cell blades to upgrade ‘bluegrit’ to a

heterogeneous multicore processing system for exploratory research.

6/18/07 Page 7

Bluegrit

Storage

• IBM Storage System DS4300
• Red Hat Linux Enterprise 4.0
• 21 x 250 GB hard disks
• 5.25 TB total disk storage

iCLASS Compute Servers

Internal connectivity

• 4 Gbps connection between Blades

External Connectivity

• 1 Gbps burst fiber optic through Quest
• 10 Gbps fiber optic with Dynamic Dragon

Circuit Service from UMBC to UMCP/UMIACS, NASA/GSFC, NSA/LTS, NGIT/ McLean Va., NLR

6/18/07 Page 8

iCLASS Visualization

Tiled Display

• 6 x ThinkVision L200P
• Each with 1600 x 1200 resolution
• Driven by ATI All In Wonder X600
• A total resolution of ~ 11.5 Megapixels

Matisse

• 7 x IBM Think Centre Desktop Computers
• Red Hat Linux Enterprise 4.0
• Intel 3.3 GHz Intel Xenon with Hyper Threading
• 1 GB MB DDR SDRAM
• 80 GB hard disk
• Optiputer S/W
• Planning to acquire a small cluster of PS3s ( ~ 4-6) to

drive an upgraded Tile display and explore other uses.

6/18/07 Page 9

Exploratory Cell Research Applications

• Aerospace: Temperature & Ozone Radiance Gridding
• Geosciences: Global Circulation & Hurricane Predictions
• Ray Tracing Animations
• Hierarchical Image Segmentation for Medical Scanning
• --------- Other Academic Research Areas ---------------
• Digital Media
• Chemical Modeling
• Bioinformatics
• Sensor Web Simulations for Dynamic Congestion Pricing

6/18/07 Page 10

Aerospace: Cell Sounding Data Processing

• M. Halem & Ye.Yesha
• NASA, NOAA and DOD have very large satellite data holdings (PBs) from

many temperature and moisture sounding instruments flown over several

• decades. Current instruments have thousands of spectral channels.
• Atmospheric radiance data products are archived at different spatial and

spectral resolutions and temporal frequencies, and in different formats  Data transformations such as gridding, statistical sampling, subsetting, convolving, etc. are needed to produce long climate data records to study global warming. These transformations lend themselves well to the data parallel model which are ideal for Cell BE Processor.

Figure shows of one month of gridded AIRS data for a single channel

6/18/07 Page 11

Field of Views for AIRS(2382 spectral ch.) /AMSU(15 Spectral ch.)

2520 km (6 min. )/ granule 135 scan lines /granule 240 granules/ day

6/18/07 Page 12

Aerospace: Cell Sounding Data Processing (cont.)

• Web service tools developed at UMBC provide complex gridding services on-

demand for atmospheric radiance data sets utilizing the IBM JS20/21 cluster. However, reprocessing will consume the entire current system.

• Service algorithms are compute and data intensive to implement on serial

processors for gridding different instruments:

• On demand work flows for spatial/temporal/spectral subsetting of gridded fields
• Concatenation of radiances from high resolution spectral instruments using

FFT convolutions to match spectral radiances of lower resolution instruments

• Generating gridded AIRS and MODIS radiance data sets on demand from

satellite observations (L1) for user specified spatial-temporal regions with selected statistical aggregations

• Overlaying multi instrument gridded animations of over long time periods.

Motivation for Cell Processing:

• Services can be implemented as embarrassingly parallel operations.

E.g., Each SPUs can project a separate granule in parallel with all other granules. Convolutions can be performed with 1-D FFT’s in single precision to give performance factors of 10X to 15X over leading superscalar and vector CUs, (Williams et. al., CF’06,May 2006, Ischia,Italy).

Page 13

 AIRS ch. 2333(blues/purple), MODIS ch. 32,  MODIS cld free June-Sept. composite true color Earth

6/18/07 Page 14

Ozone Cell Processing Application

• The Ozone Monitoring Instrument (OMI) launched on the NASA Aura

spacecraft on July 15, 2004.

• OMI is a nadir-viewing UV-VIS spectrometer, with a 2600 km wide swath,

and a 13x24 km2 footprint, guaranteeing daily global coverage.

• OMI measures the back-scattered solar radiance in the wavelength range
• f 270 to 500 nm.
• The OMI Ozone Profile Algorithm is based on the optimal estimation

technique [C.D. Rodgers, 2000)] that has become standard in the field.

• It takes advantage of the hyperspectral capabilities of the OMI instrument

to improve the vertical resolution of the ozone profile below 20 km compared to those from the SBUV instruments that have flown on NASA and NOAA satellites since 1970.

6/18/07 Page 15

OMI Cell Processing (Cont.)

• OMI processing uses new approaches to calculate the required radiances

and Jacobians efficiently and to correct for rotational Raman scattering

• Planning to produce global daily ozone profiles at 36 x 48 km, an

unprecedented spatial resolution.

• The current implementation of the algorithm takes approximately 10

minutes per pixel on off the shelf commodity Intel processors.There are 1.3 million pixels per day which would take about 8000 processors to keep up with real time processing of every pixel.

• Profiles are computed largely independently, so 100 Cell blades should

be well suited to parallelize the code using independent SPUs. It should be possible to use the SPU SIMD instructions to compute multiple profiles simultaneously within each SPU at 5X over Intel processors.

• Averaging over 5X5 pixel regions would enable feasibility testing on

the proposed modest sized Cell blade cluster.

6/18/07 Page 16

Atmospheric Modeling on the IBM Cell Processor by M.Halem et. al.

Goddard Earth Observing System 5 Model (GEOS-5)

• Uses satellite data to improve hurricane prediction
• Different from regional methods in that it is based
• n physical parameterizations of climate processes
• Hurricane resolving global forecasts ( 25km grid)

Computations limit prediction:

• Extremely large data sets ( several terrabytes)

frequently being accessed from disk

• Need to simulate at 10 km resolution globally;

scales well for cluster use of 128 to 256 nodes Five day forecast results required within 6 hours for transmission to National Hurricane Forecast Center for inclusion in ensemble forecasts.

6/18/07 Page 17

Map 08 Hurricane Modeling

NASA Ames

• Mt. View, California

National Lambda Rail (10 - 40 Gb/s)

Northrop Grumman McLean, Virginia NASA Goddard Greenbelt, Maryland

DISTRIBUTED COMPUTING NODES NEXT-GENERATION NETWORKS UMBC SOC MODELING SERVICES SCIENCE ANALYSIS GEOS5/OGCM/ MOM4 GEOS5/OGCM/ MOM4 GEOS5/OGCM/ MOM4 GEOS5/OGCM/ POSEIDON GEOS5/OGCM/ POSEIDON GEOS5/OGCM/ POSEIDON UMBC and GSFC DATA PORTALS NOAA/NESDIS DATA and MODEL PORTALS EXTERNAL COLLABORATORS

Model to model & model to data validation / comparisons

6/18/07 Page 18

Cell Processor Dynamical Core Evaluation

• GEOS 5 uses the Arakawa finite difference stencil

3-D hyperbolic toy PDEs simulated with Mambo obtained

speedups of 20X and 10X in SP and 10X and 5X in DP

• Arakawa stencil needs 4 planes and is limited by local

store memory

• Propose the use of Lamda-Ram research project approach to

transfer large data sets from a remote TB memory cache over 10Ge optical network instead of from local disk

• Propose evaluation of real dynamical core employing :
• 8 Cell Synergistic Processing Units (SPU)
• SIMD pipeline
• 256K on chip memory (NOT cache)
• Minimize slower message passing branching for the dynamics stencil
• No prediction
• Difficult to keep pipeline filled

6/18/07 Page 19

Cell Processor Dynamical Core Evaluation

• Potential to improve Atmospheric Modeling predictions
• Computation suited for SPUs
• Convolutions defined by differential equations requires each point to be

updated by neighbors

• Well defined highly localized memory access
• Minimal branching
• Instruction level parallelizable
• Utilize large SIMD pipeline
• Thread level parallelizable
• Utilize of 8 SPUs
• Next Step: Work on 3D Matrix problems
• Williams et. al. 2006 have investigated
• Cactus WaveToy 3D PDE stencil computation
• Cell has 5x GFLOPS of IA64 at Double Precision
• Cell has 7x GFLOPS of IA74 at Single Precision
• Investigate larger Matrix problems

6/18/07 Page 20

High Definition Interactive Ray Tracing on CELL Processor using Coherent Grid Traversal

By D. Chapman & C. Lohr

6/18/07 Page 21

OVERVIEW

Ray tracing:

Used in movies and video games High image quality- shadows, reflections Very Slow- Test every ray/object

Perform Ray tracing quickly: Use Cell Processor

Use Coherent Grid Traversal Algorithm

Assumptions: All objects on processor

Ignore secondary rays and complex shading All objects are spheres

6/18/07 Page 22

Uniform Grid Algorithm

• Overlay Uniform Grid

6/18/07 Page 23

Cell PS3 Performance

• Benthin et al. 2006:

Ray Tracing Algorithm

• Coherent Bounding Value Hierarchy
• Large Scenes:

9x performance increase

• UMBC Results:

PS3: 12 FPS @ 1280 X 720 p with 16 spheres 20 FPS @ 720 X 480 p with 16 spheres Intel 1.73 GHz: 0.3 FPS @1280X720 p w 16 spheres (No assembly optimization, SPU code)

6/18/07 Page 24

Cell Applications to Medical Imaging

• J. Tilton /GSFC and M. Halem /UMBC
• Image segmentation consists of segmenting a pixel image by combining pixels

at different levels of detail, and producing coarser regions from the merging of regions of finer segmentations, while maintaining region boundaries at the full image spatial resolution. Image Segmentation transforms pixel-based analysis into region based analysis.

• A hierarchy of segmentations allows dynamic selection of the appropriate level
• f segmentation detail for each object of interest.
• HSEG is a hybrid of Hierarchical Step-Wise Optimization* region growing

together with spectral clustering – controlled by a spclust_wght parameter. (J.

• M. Beaulieu and M. Goldberg, “Hierarchy in picture segmentation: A stepwise
• ptimal approach,”IEEE Transactions on Pattern Analysis and Machine

Intelligence, vol. 11, no. 2, pp. 150-163, 1989.)

• RHSEG recursively subdivides the image data and then recombines the results

such that the number of regions handled at any point in the program is restrained.

Recursive HSEG (RHSEG) facilitates a highly efficient

parallel implementation

6/18/07 Page 25

Cell applications of RHSEG to medical imaging

• HSEG provides hierarchical segmentation of image (e.g., Landsat TM) or image-

like data (e.g., IMAGE spacecraft Radio Plasma Imager data)

• Recursive formulation (RHSEG) provides computational efficiency, and has an

effective parallel implementation. E.g. A full Landsat TM scene (6500x6500 by 6 bands) can be processed in two to eight minutes with 256 2.1 GHz CPUs (Thunderhead Beowulf Cluster).

• HSEGViewer provides a facility for visualizing and interacting with the HSEG

results, and allows a user to extract useful segmentation results from the HSEG segmentation hierarchy

• RHSEG and HSEGViewer have been extended as MED SEG and a CRADA

project between NASA and Bartron Medical Imaging has been initiated for the development of an extension to three-dimensional medical image analysis. Completed basic extension of HSEGViewer to enable viewing of arbitary 2-D data slices along a selected row, column or depth plane.

6/18/07 Page 26

Cell applications of RHSEG to medical imaging

View of segmentation of skull for depth plane 86 (out of 172): Grey Scale Image Four Region Segmentation

6/18/07 Page 27

Cell applications of RHSEG to medical imaging

View of segmentation of skull for row plane 128 (out of 256): Grey Scale Image Four Region Segmentation

6/18/07 Page 28

Cell applications of RHSEG to medical imaging

View of segmentation of skull for column plane 128 (out of 256): Grey Scale Image Four Region Segmentation

6/18/07 Page 29

Cell applications of RHSEG to medical imaging

CAT Scan Segmented Scan Bartron’s Med-Seg applied to body CAT scan

6/18/07 Page 30

Summary

• Aerospace and Geosciences have unique computational

requirements that Cell blades have potential to meet cost effectively.

• Medical Imaging and animated visualizations are ideal

applications for Cell processing.

• Substantial challenges remain in software development to

provide a wide range of service oriented science applications

• f Cell processing.
• UMBC proposing to explore multicore processing for selected

science service applications in collaboration with government agencies, industry and other universities.

6/18/07 Page 31

Other Academic Cell Research Applications:

6/18/07 Page 32

Cell applications of RHSEG to medical imaging

Advantages and Potential of Med-Seg

• Med-Seg provides enhancement of diagnosis power for a wide range of medical
• images. Built on HSEG, it could process 16 bit medical image data to reveal

information not normally seen with the human eye, which can normally differentiate only 8 to 10 bits

• Bartron’s 64 CPU parallel computer cluster provides HSEG results moderately

quickly, but is longer for large images. Proposing to test the Med-Seg algorithm

• n the cell blades.
• HSEGViewer provides the medical analyst ultimate control over selection of

segmentation results.

• Bartron currently seeking FDA approval for Med-Seg. Application to MRIs and

CAT scans currently under study.

• Bartron is exploring other possible uses with the Dept. of Defense, Dept. of

Agriculture and the Indian Health Service

6/18/07 Page 33

Open Surgical Simulation M.Olano

• Many facets to surgical training
• Cadavers, live animals, patients, simulation
• Simulation repeatable, allows more practice
• Today, primarily used for laproscopic
• Camera + constrained interaction
• UMBC working to simulate open surgery
• Cooperation with National Capitol Area

Medical Simulation Center, Uniformed Services University

• XFEM deformable organ simulation with cuts
• Need parallel FEM to achieve interaction
• Cell-cluster based XFEM

6/18/07 Page 34

Biochemistry of Life- S. Freeland

• Develop simulations of molecular evolution to better understand the roles of

chance and necessity in the early evolution of life on earth. We expect to derive a better understanding of the potential for expanding biochemistry through synthetic biology/biochemical engineering.

• Simulate the process by which mutations enter genetic information, producing

variation in the structure and function of proteins, which is then ‘filtered’ by natural

• selection. This theoretical framework directly explores the extent to which modern

computing can realistically reflect complex biochemical processes that are expensive and difficult to explore in a laboratory.

• We augment, shrink or merely change the chemical identities of elements such as

amino acids, to ask whether the very building blocks of life are non-randomly selected from the chemistry of this universe.

• These simulations require enormous computing power: in particular, it has

become apparent over recent decades that the ‘algorithm’ by which a linear, linked sequence of amino acids folds into a 3-dimensional structure imbued with biochemical function is one of enormous complexity.

• The proposed improvements to the super-computing cell infrastructure of UMBC

would provide an enormous increase in the scale and flexibility of the research that we are currently undertaking.

6/18/07 Page 35

Digital Media: Business Intelligence

• T. Finin, A. Joshi, Y. Yesha
• Social Media is a dynamic and growing area within the Digital

Media domain that includes blogs, wikis, forums, photo and video sharing sites, etc.

• “Social media describes the online tools and platforms that

people use to share opinions, insights, experiences, and perspectives with each other.” – Wikipedia, Feb 07

• UMBC is developing scalable solutions for deriving business

intelligence from digital media sources such as blogs

• Collaborating with a BI project on telecom graphs at

IRL/IBM SO

6/18/07 Page 36

Digital Media Analytics

• Temporal Analysis: Searching for

trends, bursts and events

• Opinion Extraction: Finding positive

and negative reviews using statistical NLP and ML

• Influence Detection: Using topic,

social structure, opinions, biases and temporal information

• Trust Propagation: Estimate trust

between two unknown blogs using sentiment or link polarity

• Community Extraction: Extracting

the social network of the blogs and finding the communities corresponding to topics of interest

Architectural Issues

• Timely aggregation of feeds
• Distributed indexing for faster performance
• Linguistic processing and spam detection
• Scalable, high volume, parallel query

processing

6/18/07 Page 37

Example: Simulating Pervasive Computing

Applications of SPEs in Pervasive Computing:

• Simultaneous monitoring and correlation of events in

near-realtime for

• Context awareness
• Reputation management
• Recourse and reciprocity
• Data quality
• Parallelized cross-layer analysis for improved

detection

6/18/07 Page 38

Cross-layer Analysis

Application Transport Link MAC/PHY

Commendations Commendations Accusations Accusations (to other devices) (to other devices) Routing attacks, Routing attacks, disruptions disruptions Unfair contention, Unfair contention, Jamming Jamming

Intrusion Detection Response

Packet dropping, Packet dropping, Mangling, injection Mangling, injection

Trust evolution, reputation management, Trust evolution, reputation management, recourse recourse

6/18/07 Page 39

Security,Trust, and Governance in Open Distributed Environments

• Many new computing and information systems are open,

heterogeneous and dynamic

• Examples: Grid computing, the web, SOC, P2P systems,

pervasive computing, MANETs, etc.

• Providing security and privacy in such systems is challenging
• We can not rely on traditional authentication-based schemes
• Recognizing “bad actors” in such systems is hard
• We are exploring new approaches using computational policies,

trust and reputation.

• These require significant computational capabilities

6/18/07 Page 40

Cell Technology for Digital Media Analytics

• IBM’s synergetic cell processing capability would allow large

scale text and graph analytics

• SIMD architecture makes it ideal for manipulating and

processing large (~50M nodes) graph matrices for social network analytics and community detection

• Optimized floating point computations provide an opportunity

to run influence detection algorithms on large, complex graphs with high performance

• Asymmetric thread runtime model would enable faster

creation and querying of large indices of text

6/18/07 Page 41

Graphics Projects: M. Olano

• Large scale simulation visualization
• Larger simulations bring larger models
• Develop new 64-bit parallel rendering algorithms
• CS Game Development Track
• Motivate students with games
• Teach general Cell and parallel programming

6/18/07 Page 42

Large Scale Simulation Visualization

• Compute advances bring huge simulations
• Number of primitives soon to exceed 32-bits
• Fine grids, small time steps, many frames
• Need new algorithms for rendering
• Large memory + large compute
• Parallel 64-bit Cell-based REYES algorithm
• On-the-fly pre-processing
• Fast first image, later exploration even faster
• Randomized algorithms for rendering

6/18/07 Page 43

Game Development Track

• UMBC developing a game development track
• Existing CS BS degree
• Restricted electives: Graphics, AI, Networking, etc.
• Emphasis on core CS experiences useful for games (and
• ther industries)
• Any Cell programming experience
• Any parallel programming experience
• Motivate with games, but teach with science

Backup

6/18/07 Page 45

Related Work

• Wald et al. 2006
• Coherent Grid Traversal
• Frustum Based
• UMBC Results with a PS3

(Lohr & Chapman)

12 FPS @ 1280 X 720 p with 16 spheres No acceleration or frustrum checking 20 FPS @ 720 X 480 p with 16 spheres Theoretical 26 Gflops

6/18/07 Page 46

Coherent Grid Traversal