Platforms for Data Sharing, Data Analytics, and Data Visualization - - PowerPoint PPT Presentation

Platforms for Data Sharing, Data Analytics, and Data Visualization

Richard Glassco Noblis Consultant to FHWA Office of Operations: Research & Development

2

U.S. Department of Transportation ITS Joint Program Office

Agenda

• Overview of the Safety Pilot Model Deployment
• Size-Related Considerations
• Demonstration of the RDE
• Other CDS Program Tool development activities

3

U.S. Department of Transportation ITS Joint Program Office

RESEARCH DATA EXCHANGE

4

U.S. Department of Transportation ITS Joint Program Office

Goals of the Safety Pilot Model Deployment

• 1. Test connected vehicle operations in real-world

conditions

• 2. Understand how regular drivers use connected vehicle

technologies

• 3. Determine the safety benefits of a connected vehicle

5

U.S. Department of Transportation ITS Joint Program Office

Ann Arbor, Michigan, site of SPMD

6

U.S. Department of Transportation ITS Joint Program Office

SPMD TYPES OF MESSAGES

• Basic Safety Messages (BSMs)
• Signal Phase and Timing (SPaT) Messages
• MAP messages
• Traveler Information Messages (TIMs)

Plus contextual data:

• Weather
• Traffic counts

Plus documentation

7

U.S. Department of Transportation ITS Joint Program Office

SPMD Statistics

• 2836 vehicles driving on 73

miles of roadways (freeways and arterials)

• Each vehicle generates Basic

Safety Messages (position, speed, acceleration, etc.) every 10th of a second

• 2 Months of data (4 billion

BSMs) are on the RDE

• The SPMD data environment is

structured in 6 data sets, with a total sanitized volume of approximately 24 GB

• The original un-sanitized data

set was approximately 50GB

8

U.S. Department of Transportation ITS Joint Program Office

SPMD Data in the RDE

9

U.S. Department of Transportation ITS Joint Program Office

SPMD Contained Data that Could be Shared

• Some Data may not be shared, for example:
• It contains PII that cannot be removed
• It contains commercially sensitive information
• It contains intellectual property rights
• There are legal reasons the data cannot be released to the

public

• Such data may be stored in the FHWA’s Saxton Transportation

Operations Laboratory. Access to data in the Saxton Lab is restricted to researchers with appropriate qualifications and training.

10

U.S. Department of Transportation ITS Joint Program Office

Requirements for Data Submitted to the RDE

• PII is data that by itself or in combination with other data could be used to

determine the identity of individual private vehicles or persons from who driving data is obtained.

• Examples of PII include driver age, gender, social security number, or

address, and vehicle VIN number, license plate number, exact length and width, or weight, or other identifying characteristics.

• PII may also be derived from vehicle trajectories that reveal the origin or

destination of personal trips, or intermediate stops of personal nature such as a school, workplace, or frequent errand.

• Public transit vehicles and public vehicles such as maintenance trucks are not

subject to this PII restriction

11

U.S. Department of Transportation ITS Joint Program Office

PII had to be removed from the SPMD data while maintaining meaningfulness of the data PII had to be removed from the SPMD data while maintaining meaningfulness of the data

• To protect participants’ identity the RDE

team rid all data files of data elements that contain PII

• Data elements that could be paired with
• ther publicly available data were also

deleted

• Vehicle trajectories, with points collected

at 10Hz, revealed the identity of participants, therefore

□ Sanitization algorithms were

developed to truncate trajectories to mask trip origins and destinations

□ The algorithms were also applied to

dependent / related data elements

PII had to be removed from the SPMD data while maintaining meaningfulness of the data

• To protect participants’ identity the RDE

team rid all data files of data elements that contain PII

• Data elements that could be paired with
• ther publicly available data were also

deleted

• Vehicle trajectories, with points collected

at 10Hz, revealed the identity of participants, therefore

□ Sanitization algorithms were

developed to truncate trajectories to mask trip origins and destinations

□ The algorithms were also applied to

dependent / related data elements

12

U.S. Department of Transportation ITS Joint Program Office

Because of the Size of the Data Environment …

• Moved to storage on the Cloud rather than contractor's

server

• Developed subsetting tool

13

U.S. Department of Transportation ITS Joint Program Office

Full Year of SPMD Data

• USDOT has reserved the entire year of data for future

study

• Unsanitized, so it is not posted on the RDE – it will be

stored in the Saxton Lab

• Restricted access
• Special restrictions for video or audio data

14

U.S. Department of Transportation ITS Joint Program Office

LIVE DEMONSTRATION OF RDE

15

U.S. Department of Transportation ITS Joint Program Office

OTHER CDS TOOL DEVELOPMENT ACTIVITIES

16

U.S. Department of Transportation ITS Joint Program Office

Data Visualization Project

• Data visualization is #1 request of RDE stakeholders
• Uses data on the RDE to develop data visualization

tools

• Map and non-map based, interactive
• Applicable to multiple data environments

17

U.S. Department of Transportation ITS Joint Program Office

Samples from Data Visualization Project

18

U.S. Department of Transportation ITS Joint Program Office

Dynamic Interrogative Data Capture (DIDC)

19

U.S. Department of Transportation ITS Joint Program Office

BSM Emulator

20

U.S. Department of Transportation ITS Joint Program Office

OSADP - Companion Website for Software

• www.itsforge.net

21

U.S. Department of Transportation ITS Joint Program Office

References and Contact Info

• Research Data Exchange (RDE) website

https://www.its-rde.net/

• Open Source Application Development

Portal (OSADP): www.itsforge.net

Jon Obenberger, PhD, P.E. Senior Transportation Research Engineer, FHWA Jon.Obenberger@dot.gov