Summary of Events on June 23, 2018 Matthew Berkheiser, DrPH, CIH, - - PowerPoint PPT Presentation

Matthew Berkheiser, DrPH, CIH, CSP

Associate Vice President and Chief Safety Officer Environmental Health & Safety, Sustainability and Emergency Management

Building Automation Systems (BAS) Network Access Procedure and Summary of Events on June 23, 2018

Contents

1 MD ANDERSON CANCER CENTER 2 WHAT IS THE BUILDING AUTOMATION SYSTEM (BAS)? 3 EVENT OF JUNE 23, 2018

Response Cause Process Changes

Our History

Monroe Dunaway Anderson created a charitable foundation in 1936. After his death, the MD Anderson Foundation provided funding and land to build a cancer hospital to serve the citizens of Texas.

Nurse helping a patient into one of the converted barracks

• n the Baker estate, 1951

Army Barracks were moved to the Baker family estate to act as inpatient wards and an operating room Laboratory at the Baker estate, 1944

Our Beginnings

Waiting room at the Baker estate, 1944 Baker estate,1942

Charles A. LeMaistre, M.D. President, 1978-1996 Peter Pisters, M.D. President, 2017 John Mendelsohn, M.D. President, 1996-2011 Ronald DePinho, M.D. President, 2011-2017 Randolph Lee Clark, M.D. President, 1946-1978 Ernst W. Bertner, M.D. Acting Director, 1942-1946

Our Leadership

To eliminate cancer in Texas, the nation and the world through outstanding programs that integrate patient care, research and prevention, and through education for undergraduate and graduate students, trainees, professionals, employees and the public.

Our Mission

We shall be the premier cancer center in the world, based on the excellence of our people, our research-driven patient care and our science.

Our Vision

• •

Our Core Values

Caring Integrity Discovery

• More than 137,000 people sought the superior care that

has made MD Anderson so widely respected—over 41,000 of whom were new patients.

• MD Anderson’s cancer clinical trial program is one
• f the largest of its kind. Almost 9,400 participants were

enrolled in 1,250+ clinical trials exploring innovative treatments.

Patient Care • FY17

Patient Care • FY17

Hospital admissions 28,793 Average number of inpatient beds 681 Outpatient clinic visits, treatments and procedures 1,441,403 Pathology/laboratory medicine procedures 12,700,333 Diagnostic imaging procedures 574,018 Surgery hours 70,460 Active clinical research protocols 1,255

Research

Research • FY17

• MD Anderson invested more than $844 million in

research.

• Every day, in our Texas Medical Center labs and our

research facilities in Bastrop County, scientists are making discoveries that are rapidly being translated into clinical care.

Research • FY17

• Our Moon Shots Program™ has 13 transdisciplinary

teams working to dramatically accelerate the conversion

• f scientific discoveries into clinical advances and

significantly reduce deaths caused by the disease.

• 150 clinical studies are being accelerated as a result of

the Moon Shots Program.

• MD Anderson ranks first in the number of research grants

awarded by the National Cancer Institute.

Our People

• More than 19,530 employees, including 1,743 faculty.
• A volunteer workforce composed of 1,141 on-site, trained

volunteers and 2,091 off-site, trained myCancerConnection survivor volunteers contributed 122,367 hours of service in FY17.

Our People

Locations

Houston area

Building Automation System

619,000 points and 20,000 controllers

• Operating room temperature

and air pressure

• Patient satisfaction and their

room comfort

• Pharmacy USP 800 compliance
• Integration with patient

scheduling (EPIC)

• Chilled water control systems

supporting equipment cooling (MRIs)

• Building water systems
• Building life safety and critical

systems monitoring and alarms

• Data centers and IT room

temperature, air flow controls, power monitoring

• Client critical equipment

monitoring

Systems Integration Support

Replication Event

On June 23, 2018, the MDA BAS team was contacted by the Patient Care Facilities O&M team concerning the abnormal operation of some

• f their facility chilled water and air handling equipment.
• Equipment was cycling on and off in the patient care buildings
• Graphics were displaying points randomly going in and out of

failure These events caused the O&M teams to place our air handling system in manual mode. We identified that the system was experiencing symptoms of data corruption, which is also commonly referred to as replication.

Replication Event -Impacts

• Exterior temperatures ranged from 77 to 90 degrees F and 84%

humidity

• Approximately 500 patients in beds
• On Saturdays we see approximately 400 out-patients for treatment
• Pressure issues in some pharmacy areas caused us to stop/relocate

pharmacy processes

• Elevated temperature and humidity inside patient care buildings
• 70 % humidity in operating suite with no cases scheduled

Response- Activate Incident Command

Response- Activate Incident Command

Response- Downtime Procedures

Cause

During construction of the League City project, a BAS vendor was using their company’s proprietary application on their

• laptop. The laptop had been used in the past to work on both our

research building databases and our patient care building databases. We believe data from our Research application was inadvertently copied to an image on his laptop and eventually copied to the Patient Care system causing a data mismatch and replication issue.

Database 1 Database 2 Corrupt Database

BAS Vendor Network Access Procedure

• Vendor owned computer equipment (laptops, iPads,

etc.) will no longer be allowed to access MD Anderson’s building controllers via direct connection or by utilizing the internal network via Ethernet or internal Wi-Fi.

• Vetted users utilizing their vendor provided computer
• r laptop can connect to MD Anderson assets by

utilizing MD Anderson’s Virtual Private Network (VPN) solution when connected on an external network.

Changes

BAS Vendor Network Access Procedure

• BAS vendors shall not be allowed to utilize any

proprietary software tools on MD Anderson network.

• Our internal BAS team has developed a script that

runs every 12 hours verifying we do not have replication occurring on the system.

Changes

BAS Vendor Network Access Procedure

New Building Construction:

• Vendor shall be responsible for providing a temporary

local network to meet the construction schedule or utilize the internal network when available.

• Vendor shall submit a detailed database merging plan

thirty days prior to substantial completion for owner review and approval.

• Once the submitted database merging plan is

approved by MDA BAS, we will merge the local construction database with the institution database.

Changes

BAS Vendor Network Access Procedure

Existing Building Addition and Renovation:

• Vendor will submit an outage request via the

Facilities Outage Notification (FON) 5 business days before the system modification.

• MDA BAS must approve the request prior to the

new building controller being configured and added to the network.

• Vendor shall not connect the building controller
• nline without MDA BAS supervision being

physically present at the site location.

Changes

Next Steps/Action Items

• IT Engineering to review and approve plans. Completed
• FM Sr. Staff to review and approve BAS Network Access
• Procedure. Completed
• Incorporate recommendations from MDA InfoSecurity on

additional recommendations to harden the BAS system.

• Plan to turn replication back on in the patient care system.
• Due to the events of June 23 caused by the replication

event we plan to implement a financial penalty through the construction contract process.

Questions