Scalability of InfiniBand-Connected LNET Routers Team Light Coral - - PowerPoint PPT Presentation

▶
scalability of infiniband connected lnet routers
SMART_READER_LITE
LIVE PREVIEW

Scalability of InfiniBand-Connected LNET Routers Team Light Coral - - PowerPoint PPT Presentation

Oct 27, 2023 208 likes •385 views

Scalability of InfiniBand-Connected LNET Routers Team Light Coral Computer System, Cluster, and Networking Summer Institute Emily Baldwin Wheaton College Matthew Schauer Georgia Institute of Technology Jarrett Crews New Mexico Institute of

slide-1
SLIDE 1

Scalability of InfiniBand-Connected LNET Routers

Wheaton College Georgia Institute of Technology New Mexico Institute of Mining and Technology

Susan Coulter David Bonnie Christopher Hoffman Dane Gardner

Emily Baldwin Matthew Schauer Jarrett Crews

HPC-3 HPC-3 HPC-3 Instructor

Team Light Coral Computer System, Cluster, and Networking Summer Institute

LA-UR-14-26018

slide-2
SLIDE 2

Overview

Background Objective Cluster Set-Up Benchmark Methods Results Obstacles Future Work

slide-3
SLIDE 3

Background

— Lustre File System

— Servers — Network (LNET) router — Clients

— InfiniBand

— FDR – 56 Gb/s — IP over IB

— IOR Performance Benchmarks

slide-4
SLIDE 4

Objective

Is it possible to link multiple Lustre File Systems to a single LNET router? If so, what is the read/write performance of multiple file systems from many client nodes?

slide-5
SLIDE 5

Motivation

— LANL converting to Lustre from Panasas — Never more than one file system per LNET router

— Not cost-effective — Wasted router potential

— Multiple file systems per LNET router arrangement

— No loss in performance? — No significant change in router utilization?

— Potential for easier transition from legacy machines

slide-6
SLIDE 6

Cluster Set-Up

— IB

InfiniBand

— LNET

Lustre Network

— MDS/MDT Metadata Server/Target — MGS/MGT Management Server/Target — OSS/OST Object Storage Server/Target

slide-7
SLIDE 7

Cluster Set-Up

slide-8
SLIDE 8

Benchmark Methods

— IOR benchmarking tool

— Writes/reads variable amounts of data — Parameters for file size, block size, files per node, etc. — Reports bandwidth statistics

— eeyore

— Automates testing with IOR — Sequence a write, read, then simultaneous write/read — Script parameters include: file size, block size, nodes,

and processes per node

Op Mean Max Min Stdev w 535.73 544.19 528.31 5.64 r 410.05 416.34 405.79 3.88

slide-9
SLIDE 9

Benchmark Methods

— Run each test n times, collect mean and standard

deviation

— Test parameter combinations:

Number of Nodes File Size/ Process Block Size Processes/Node Total Transfer Size 6 32 GB 1 GB 1 192 GB 6 32 GB 512 MB 1 192 GB 6 32 GB 2 KB 1 192 GB 6 1 GB 1 GB 24 144 GB 6 1 GB 512 MB 24 144 GB 6 1 GB 2 KB 24 144 GB

slide-10
SLIDE 10

Results

32 GB files, 512 MB block size

Write, then read

  • 500 MB/s file system 1
  • Across 5 disks
  • 400 MB/s file system 2
  • Bad DIMM
  • Two file system bandwidth

is sum of individual file system bandwidths

  • Small standard deviation
  • Consistent results over

many test runs

0 ¡ 200 ¡ 400 ¡ 600 ¡ 800 ¡ 1000 ¡ 1200 ¡

FS ¡1 ¡ FS ¡1/FS ¡2 ¡

Bandwidth ¡(MB/s) ¡

Write ¡

0 ¡ 200 ¡ 400 ¡ 600 ¡ 800 ¡ 1000 ¡ 1200 ¡

FS ¡1 ¡ FS ¡1/FS ¡2 ¡

Bandwidth ¡(MB/s) ¡

Read ¡

slide-11
SLIDE 11

Results

32 GB files, 512 MB block size

Simultaneous write and read

  • Similar results to

sequential write and read

  • Large standard

deviation on two file system writes

0 ¡ 200 ¡ 400 ¡ 600 ¡ 800 ¡ 1000 ¡ 1200 ¡

FS ¡1 ¡ FS ¡1/FS ¡2 ¡

Bandwidth ¡(MB/s) ¡

Write ¡

0 ¡ 200 ¡ 400 ¡ 600 ¡ 800 ¡ 1000 ¡ 1200 ¡

FS ¡1 ¡ FS ¡1/FS ¡2 ¡

Bandwidth ¡(MB/s) ¡

Read ¡

slide-12
SLIDE 12

Results

32 GB files, 512 MB block size

200 400 600 800 1000 1200 200 400 600 800 1000 1200 1400

Router Throughput (MB/s) Test time (s)

LNET Router Throughput over Time

1 File System 2 File Systems Read Write Write/Read

slide-13
SLIDE 13

Discussion

— LNET routers scale beyond a single file system — Expected bottleneck does not exist in router

— Negligible router CPU load

— Two file systems performed at expected capacity — Scalability plausible

— Bandwidth trend may not continue

slide-14
SLIDE 14

Obstacles

— Lustre incompatibility with stock kernel

— Server and client utilities

— 10% bandwidth loss

— Removed LNET router — One file system performed slower than other

— Discovered bad DIMM

— Consistent results despite hardware issue

slide-15
SLIDE 15

Future Work

— Scalability of LNET routers to more file systems — More complex setups

— Lustre file system components on different servers — Heterogeneous networks connected partially with

InfiniBand and partially with Ethernet

— Multiple Lustre networks with varying number of

servers

— Multiple routers connecting many Lustre networks

slide-16
SLIDE 16

Thank You!

Wheaton College

Susan Coulter David Bonnie Christopher Hoffman Dane Gardner

Emily Baldwin Matthew Schauer Jarrett Crews

HPC-3 HPC-3 HPC-3 Instructor

Georgia Institute of Technology New Mexico Institute of Mining and Technology baldwinemb@gmail.com matthew.schauer.x@gmail.com jarrett.crews@gmail.com

slide-17
SLIDE 17

Questions?

Background Lustre, IOR, InfiniBand Objective >1 Lustre file systems, 1 LNET router Cluster Set-Up Lustre file system, LNET router Benchmark Methods bandwidth stats, parameters Results nearly double bandwidth, scalability plausible Obstacles Lustre kernel, 10% loss Future Work more file systems, more complex setups