THE MICROSCOPE FOR 21ST CENTURY DISCOVERY: MASSIVE & CAVE2 - - PowerPoint PPT Presentation
THE MICROSCOPE FOR 21ST CENTURY DISCOVERY: MASSIVE & CAVE2 PROFESSOR PAUL BONNINGTON, DIRECTOR OF E-RESEARCH CENTRE, MONASH UNIVERSITY AND DR WOJTEK GOSCINSKI, MANAGER, HIGH PERFORMANCE COMPUTING OF E-RESEARCH CENTRE, MONASH UNIVERSITY DR
PROFESSOR PAUL BONNINGTON, DIRECTOR OF E-RESEARCH CENTRE, MONASH UNIVERSITY AND DR WOJTEK GOSCINSKI, MANAGER, HIGH PERFORMANCE COMPUTING OF E-RESEARCH CENTRE, MONASH UNIVERSITY DR PAUL MCINTOSH, TECHNICAL LEAD, MASSIVE
THE MICROSCOPE FOR 21ST CENTURY DISCOVERY: MASSIVE & CAVE2
2
INSIGHT Lens
CAVE2 IMMERSIVE VISUALISATION DIGITAL SCIENTIFIC DESKTOPS
AUSTRALIAN SYNCH
MONASH BIOMEDICAL IMAGING RAMACCIOTTI CRYOEM
CAPTURE Light Source, Samples ANALYSIS Filters
MONASH RESEARCH CLOUD
MASSIVE
3
SHARE
IMAGING LOCUS
D A T A M A N A G E M E N T
Ranked in top 0.5% of universities worldwide
Clayton: imaging precinct
Globally unique imaging infrastructure - from an atom to a whole animal
▪
Imaging and Medical Beamline (Synchrotron)
▪
X-ray diffraction beamline (Synchrotron)
▪
Monash Biomedical Imaging
▪
Monash Centre for Electron Microscopy
▪
Monash Micro Imaging
▪
Cryo-Electron Microscope
Monash-NVIDIA
An Australian node of the NVIDIA Singapore Technology Centre With a focus on medical imaging, robotic vision and smart cities
INSIGHT Lens
CAVE2 IMMERSIVE VISUALISATION DIGITAL SCIENTIFIC DESKTOPS
AUSTRALIAN SYNCH
MONASH BIOMEDICAL IMAGING RAMACCIOTTI CRYOEM
CAPTURE Light Source, Samples ANALYSIS Filters
MONASH RESEARCH CLOUD
MASSIVE
7
SHARE
IMAGING LOCUS
D A T A M A N A G E M E N T
Partners ¡
Monash University Australian Synchrotron CSIRO
Affiliate Partners ¡
ARC Centre of Excellence in Integrative Brain Function ARC Centre of Excellence in Advanced Molecular Imaging
HPC ¡
150 active projects 500+ user accounts 100+ institutions across Australia
Interactive Vis ¡
200+ users
HPC for Characterisation Specialised Facility for Imaging and Visualisation
Instrument Integration ¡
Integrating with key Australian Instrument Facilities. Early numbers from 5 Synchrotron
166 merit allocated projects 725 users
Large cohort of researchers new to HPC ¡
M1 at Australian Synchrotron ¡
42 nodes (504 CPU-cores total) in
2.66GHz
RAM total)
GDDR5 per node (84 GPUs total) 153 TB of fast access parallel file system 4x QDR Infiniband Interconnect
M2 at Monash University
118 nodes (1,720 CPU-cores total) in four configurations:
running at 2.66GHz
RAM total)
(visualisation / high memory configuration)
RAM total)
running at 2.66GHz
RAM total)
running at 2.66GHz
GB RAM total) 264 GPUs & co-processors
coprocessors
coprocessors
345 TB of fast access parallel file system 4x QDR Infiniband Interconnect
M3 at Monash University
A Computer for Next-Generation Data Science 1,700 Intel Haswell CPU-cores 48 NVIDIA Tesla K80 GPU coprocessors for data processing and high end visualisation 8 NVIDIA Grid K1 GPUs for medium and low end visualisation that will support up to 16 users concurrently A 1.15 petabyte Lustre parallel file system 100 Gb/s Ethernet Mellanox Spectrum Supplied by Dell, Mellanox and NVIDIA
Steve Oberlin, Chief Technology Officer Accelerated Computing, NVIDIA Alan Finkel Australia’s Chief Scientist
Australian*Characterisa/on** Council*
* Australian*Synchrotron* * * Monash*University* *
MASSIVE'
* Characterisa1on' Virtual'Laboratory' *
Desktop' HPC' Cloud' Atom' Probe' CT'|'PET'|' MRI''' MicroCT' MX' IMBL' Infrared' PD'
SAXS' WAXS' XGray'
absorp1on'
spectrosc
Lab'for' Dynamic' Imaging' Monash' Biomedical' Imaging'
CryoEM' ' Monash'Micro' Imaging'
Key Australian Instruments Integrated with MASSIVE
4D rabbit pup lungs imaged at Australian Synchrotron, reconstructed and visualized on MASSIVE Fouras et al
4D rabbit pup lungs imaged at Australian Synchrotron, reconstructed and visualized on MASSIVE Fouras et al
4D rabbit pup lungs imaged at Australian Synchrotron, reconstructed and visualized on MASSIVE Fouras et al
BREATHING LIFE INTO PRETERM INFANT RESEARCH
NVIDIA K20
2 4 6 8 10 12 12 24 36 48 60 72 84 96 108 120 132 144 Time (minutes) Cores
2K - CT Wizard
MASSIVE 5 10 15 20 25 30 35 24 36 48 60 72 84 96 108 120 132 144 156 168 180 192 204 216 228 240 252 Time (minutes) Cores
4K - CT Wizard
MASSIVE
CSIRO XTRACT / XLI on MASSIVE
2 mins 15 mins
Hardware Layer Integration Systems View User view
Remote Desktop with Australian Synch credentials
Research undertaken using massive through beamline access at Australian Synchrotron
Other 6% Engineering 4% Physical Sciences 1% Environmental Sciences 6% Chemical Sciences 17% Earth Sciences 18% Agricultural and Veterinary Sciences 2% Medical and Health Sciences 11% Biological Sciences 35%Life Sciences Physical Sciences and Eng.
2012
74
2013
160
2014
218
160 180 200 220 240 260 140 120 100 80 60 40 20 Number of publications
User publications accepted or published, as reported in Project Leader Reports
MASSIVE underpins a wide range of research across the partners and
shows the aggregate publications in 2012, 2013 and 2014 across MASSIVE projects, as reported in annual Project Leader Reports.
ANALYSIS Filters INSIGHT Lens
AUSTRALIAN SYNCH
MONASH BIOMEDICAL IMAGING RAMACCIOTTI CRYOEM CAVE2 IMMERSIVE VISUALISATION DIGITAL SCIENTIFIC DESKTOPS MONASH RESEARCH CLOUD
CAPTURE Light Source, Samples
MASSIVE
19
SHARE
IMAGING LOCUS
D A T A M A N A G E M E N T
Data Sources
These data were merged and processed using XDS31, POINTLESS and SCALA32. Five per cent of the data sets were flagged as a validation set for calculation of the Rfree with neither a σ nor a low-resolution cut-off applied to the data. Experimental phases (Supplementary Table 1) were obtained by the MIRAS method; a native (Native1) data set and three heavy atom derivatives (ethylmercury phosphate, ammonium hexachloroiridate(iii) and iodine) were used for phasing. Experimental phasing was carried out using autoSHARP33; heavy atom positions were located using SHELXC/SHELXD34 and refined using SHARP35 with resulting isomorphous (acentric) and anomalous phasing powers of 0.982 and 0.950, respectively. The initial phases were improved by solvent flipping using SOLOMON36 and density modification using DM37, which dramatically increased the figure of merit (FOM) from 0.34 to 0.86. Such a large increase in FOM is probably due to the very high solvent content of the crystal (70.2%). One molecule was found per asymmetric unit and an initial model was generated using BUCCANEER38. Model building was performed using COOT39 while refinement was performed using PHENIX40, REFMAC41 and … The NAG model was made using the PRODRG server (http://davapc1.bioch.dundee.ac.uk/prodrg/). The final model also contains three glycerols, two chloride ions and four iodide ions. Crystallographic and structural analysis was performed using CCP4 suite43, WHATIF44 and MUSTANG45 unless otherwise specified. Figs 1, 2, 3, 4 and Supplementary Figs 2, 4, 5, 6 and 11 were generated in part using PYMOL46. Structural validation was performed using MolProbity47. In the final structure, two residues (L307 and Y486) are in disallowed regions in the Ramachandran plot. The MolProbity score is 1.56, which is in the 100th percentile of structures reported at this resolution. A summary of diffraction and refinement statistics can be found in Supplementary Table 1. The coordinates of perforin, together with the structure factors are deposited in the Protein Data Bank. All diffraction images are deposited in TARDIS (http://tardis.edu.au/) and are freely available.
NeCTAR Characterisation Virtual Laboratory Domain-specific Environments for Science
Atom Probe Atom Probe! CryoEM CryoEM ! Light Micros. Light Micros.! CT | PET | MRI CT | PET | MRI ! ANU ANU MicroCT MicroCT!
3D Slicer, AFNI / SUMA, Diffusion TK, FSL, MINC Tools, MRICRON, Trackvis Mango CCP, Chimera, EMAN, ImageJ, IMAGIC, IMOD, Phenix, Pymol, Relion, ShelXle, Slider, TiltPicker, VMD, XMIPP Daris MyTardis MyTardis, AMMRF DF
CAW, Crystallography, EFFOFF, Fourier Transform 3D & 4D, MRF, NN, POSminus, Pos to Obj Export, Rdf- kd, SDM-1D-calculate, SDM-2D-calculate, XRNG Editor
Tailored “workspaces” - tools for scientific data processing available on the Australian research cloud to all Australian researchers Underpinning fast & large parallel FS Accelerated rendering (NVIDIA K20, K80 and K1 GPUs)
UQueensland ANU MonashU USydney
Uptake:
NeCTAR Characterisation Virtual Laboratory Domain-specific Environments for Science
169$ 190$ 54$ 0$ 50$ 100$ 150$ 200$ <10$ 10*100$ >100$
Total&desktop&sessions&launched&used& per&user&
Investigate functional brain changes in presymptomatic HD (pre-HD) and early symptomatic HD (symp- HD), compared with controls, via fMRI. 35 pre-HD, 23 symp-HD (UHDRS ≥5), 32 controls.
Finding biomarkers for clinical trials in Huntingtons Disease – Prof Nellie Georgiou-Karistianis
Structural ¡
LongitudinalchangeinFA
AnteriorThalamicRadiation
% Change in FA % Change in FA % Change in FA Control Sym-HD Pre-HD Sym-HD Control Pre-HD p = 0.032 xyz : (10, -10, 0) p = 0.006 xyz : (-11, -4, 0) p = 0.023 xyz : (5, -28, -6) % Change in FA % Change in FA % Change in FA Symp-HD Symp-HD Pre-HDN=28 controls N=29 pre-HD N=29 symp-HD
Pre-HD Control ControlConnectivity ¡
Activation in working memory (N-back)
Functional ¡
Prof Nellie Georgiou-Karistianis - IMAGEHD
FSL$/$SPM$$ Interac/ve$Use$
Image$Capture$$
Batch$$ Processing$
MASSIVE' Image$Capture$ (DaRIS)$
Automatic capture Check-out data Check-in processed results (future) Batch submission
Neuroimage processing in the cloud from capture, processing, analysis and visualisation Interactive use, with batch submission (FSL & SPM)
One click access for the long tail
Secure VNC to access HPC was not easy for inexperienced users ¡ Native - Win, Linux, Mac ¡ Web using Guacamole ¡
0" 10" 20" 30" 40" 50" 60" 70" 80" 90" 100" 20/07/15" 21/07/15" 22/07/15" 23/07/15" 24/07/15" 25/07/15" 26/07/15" 27/07/15" 28/07/15" 29/07/15" 30/07/15" 01/08/15" 02/08/15" 03/08/15" 04/08/15" 05/08/15" 06/08/15" 07/08/15" 08/08/15" 09/08/15" 10/08/15" 11/08/15" 12/08/15" 13/08/15" 14/08/15" 15/08/15" 16/08/15" 17/08/15" 18/08/15" 19/08/15" 20/08/15" 21/08/15" 22/08/15" 23/08/15" 24/08/15" 25/08/15" 0" 10" 20" 30" 40" 50" 60" 20/07/15" 21/07/15" 22/07/15" 23/07/15" 24/07/15" 25/07/15" 26/07/15" 27/07/15" 28/07/15" 29/07/15" 30/07/15" 01/08/15" 02/08/15" 03/08/15" 04/08/15" 05/08/15" 06/08/15" 07/08/15" 08/08/15" 09/08/15" 10/08/15" 11/08/15" 12/08/15" 13/08/15" 14/08/15" 15/08/15" 16/08/15" 17/08/15" 18/08/15" 19/08/15" 20/08/15" 21/08/15" 22/08/15" 23/08/15" 24/08/15" 25/08/15"
GPGPU Desktop
ANALYSIS Filters INSIGHT Lens
AUSTRALIAN SYNCH
MONASH BIOMEDICAL IMAGING RAMACCIOTTI CRYOEM CAVE2 IMMERSIVE VISUALISATION DIGITAL SCIENTIFIC DESKTOPS MONASH RESEARCH CLOUD
CAPTURE Light Source, Samples
MASSIVE
29
SHARE
IMAGING LOCUS
D A T A M A N A G E M E N T
The Monash CAVE2
facility
tracking
Brains of ~100 study subjects (controls, pre-symptomatic and HD sufferers) visualised at once
Data: N. Georgiou-Karistianis ⬥ Visualisation: Y. Benovitski
Increasingly large cohort of long tail researchers with big data problems opportunities
Automated or semi-automated processing from the instrument Capturing new HPC and long tail researchers (10s of new users per beamline per week) Desktop interactive access e.g Psychology, wet lab biomed Desktop users progressing to batch access for big processing
Institutions Researchers from the following institutions have used MASSIVE since the start of the project:
Linux and Instrument Linux Instrument Bold indictates new additions in 2014
Australian National University Curtin University Deakin University James Cook University La Trobe University Massey University Monash University Queensland UniversityIndustry
IBM 4DX Australian Red Cross Blood Service AgResearch Ltd Carrick Gold Ltd GNS ScienceGovernment
Ames Laboratory IVEC Department of Primary Industries Museum Victoria State Library New South Wales Western Australian Museum DSTO NGV (National Gallery Vic) Synchrotron SOLEIL SA Museum Natural History Museum, UKUniversity Research Institute Medical
Key