Pyodide: scientic Python compiled in WebAssembly Roman Yurchak - - PowerPoint PPT Presentation

▶

Sep 28, 2022 306 likes •573 views

Pyodide: scientic Python compiled in WebAssembly Roman Yurchak FOSDEM 2019 1 / 26 Iodide An interactive programming environment for scientists in the browser iodide.io Examples Lorenz Attractor World Happiness Eviction Notices in SF

SLIDE 1

FOSDEM 2019

Pyodide: scientic Python compiled in WebAssembly

Roman Yurchak

1 / 26

SLIDE 2

An interactive programming environment for scientists in the browser iodide.io Lorenz Attractor World Happiness Report Eviction Notices in SF

Iodide

Examples

2 / 26

SLIDE 3

Iodide overview

iodide.io

3 / 26

SLIDE 4

Architecture

Jupyter-like model

Browser UI User Server Kernel Data

Adapted from: jupyter.readthedocs.io/en/latest/architecture/how_jupyter_ipython_work.html#notebooks

4 / 26

SLIDE 5

Architecture

Jupyter-like model Iodide

Browser UI User Server Kernel Data Browser UI User Kernel Data

Adapted from: jupyter.readthedocs.io/en/latest/architecture/how_jupyter_ipython_work.html#notebooks

5 / 26

SLIDE 6

Jupyter like model

Local Install conda, Jupyter, then project-specic dependencies Remote Deploy in a container (binder etc.)

Sharing of notebooks

6 / 26

SLIDE 7

Jupyter like model

Local Install conda, Jupyter, then project-specic dependencies Remote Deploy in a container (binder etc.)

Iodide model

Local Deploy to a static webserver Just open it in your browser Remote Share a single le containing data, report, code and dependencies Just open it in your browser

Sharing of notebooks

7 / 26

SLIDE 8

Python scientic stack, compiled to WebAssembly

created by Michael Droettboom language plugin for Iodide

CPython interpreter

numpy, pandas, matplotlib

WebAssembly

A fast way to run compiled code in the browser

Related projects

PyPy.js, brython, RustPython github.com/iodide-project/pyodide

Pyodide

8 / 26

SLIDE 9

Pyodide example

9 / 26

SLIDE 10

Supported packages

CPython numpy pandas matplotlib supported experimental planned

10 / 26

SLIDE 11

Supported packages

CPython numpy pandas matplotlib Fortran scipy scikit-learn

BLAS / LAPACK

scikit-image supported experimental planned

11 / 26

SLIDE 12

Supported packages

CPython numpy pandas matplotlib Fortran scipy scikit-learn

BLAS / LAPACK

scikit-image supported experimental planned Pure python wheels on PyPi

12 / 26

SLIDE 13

Performance

Firefox: 4-8 slower for pure Python, 1-2 times slower for C-ext. Ideal scaling with the number of users.

github.com/iodide-project/pyodide/tree/master/benchmark

13 / 26

SLIDE 14

emscripten.org

Build process

Browser In memory filesystem, lz4 compressed Browser engine Emscripten compiler / toolchain CPython interpreter .wasm C extensions .wasm Python files .py 14 / 26

SLIDE 15

System calls

For example, ↓ os.open in Python ↓ CPython: call os_open_impl C function

15 / 26

SLIDE 16

Linux ↓ open system call to glibc ↓ Linux kernel

System calls

For example, ↓ os.open in Python ↓ CPython: call os_open_impl C function

16 / 26

SLIDE 17

Linux ↓ open system call to glibc ↓ Linux kernel Emscripten / WebAssembly ↓ Emscripten ↓ system call to musl libc ↓ WebAssembly engine

System calls

For example, ↓ os.open in Python ↓ CPython: call os_open_impl C function

17 / 26

SLIDE 18

Linux ↓ statvfs system call to glibc ↓ Linux kernel Emscripten / WebAssembly ↓ Emscripten : not implemented; return "safe and sane values" ✗ system call to musl libc ✗ WebAssembly engine

System calls (sometimes)

For example, ↓ os.statvfs in Python (disk space usage) ↓ CPython: call os_statvfs_impl C function Most system calls work, but there are some edge cases.

18 / 26

SLIDE 19

What doesn't work

Dicult network sockets multiprocessing host lesystem access Should work someday threads async

19 / 26

SLIDE 20

Pytest is supported: test collection and execution in the browser

Testing

20 / 26

SLIDE 21

Pytest is supported: test collection and execution in the browser

Testing

Test suites CPython: 380 test les / 497 pass (increasing, but some will never pass due to WebAssembly environment) numpy: 3145 passed, 42 failed (+ some collection failures), 47 skipped scikit-learn: WIP, looks promising. Some remaining issues with Fortran / LAPACK calls in scipy.

21 / 26

SLIDE 22

Planned language interoperability

22 / 26

SLIDE 23

Future work

increase the percentage of passing tests dynamic linking of BLAS/LAPACK in scipy possible in Emscripten 1.38.22 thanks to Kirill Smelkov

ptimize download sizes

threading and async support more packages Contributors welcome!

23 / 26

SLIDE 24

Application: in-browser data analytics

challenges of multi-user notebooks deployment running notebooks on the edge with uncertain/limited connectivity Iodide and Pyodide integrated into the OceJS apps store

nline / oine usage, synchronization in Dropbox etc

24 / 26

SLIDE 25

Brendan Colloran Hamilton Ulmer William Lachance Michael Droettboom Teon Brooks ...

Development team

25 / 26

SLIDE 26

github.com/iodide-project/pyodide @RomanYurchak

Thank you! Questions?

26 / 26