real-time with Linux real-time with Linux 06.02.2019 Lukas Pirl, - - PowerPoint PPT Presentation
real-time with Linux real-time with Linux 06.02.2019 Lukas Pirl, Daniel Richter, and Andreas Polze lecture on embedded operating systems Operating Systems & Middleware Group Hasso Plattner Institute at the University of Potsdam, Germany
06.02.2019 Lukas Pirl, Daniel Richter, and Andreas Polze lecture on embedded operating systems Operating Systems & Middleware Group Hasso Plattner Institute at the University of Potsdam, Germany based on Prof. Andreas Polze’s slides on real-time Linux (https://www.dcl.hpi.uni-potsdam.de/teaching/EOS17/Slides/08.3_rtLinux.pdf)
1 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
terminology motivation RTLinux history
architecture concepts
interrupt emulation, threads, scheduling, IO, synchronization, …
PREEMPT_RT
2 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
task when referring to abstract concepts in Linux, a synonym for “process” process when referring to the operating system concept (with virtual memory, handles, etc.) in Linux, a synonym for “task” thread when referring to the operating system concept (belonging to a process)
3 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
use standard operating system for real-time applications maturity level rich existing ecosystem
e.g., services, development tools, community
use existing knowledge/experiences use Linux for real-time applications POSIX compatibility
kernel modication possible nancial aspects
4 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
reduce costs for high-frequency tasks e.g., software-implemented AD/DA conversion software routing robotics e.g., to re-use frameworks for graphical interfaces and high-level functionality audio and video streaming use a standard Web server as control interface reduced development costs
inspired by http://www.yodaiken.com/papers/lwtalk.pdf
5 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
scheduling latency for real-time processing
# let’s ignore malloc,
device latencies, etc. for now
but in a general purpose OS, kernel threads might be non-preemptable
https://blog.emtrion.de/en/details/real-time-linux-part1-introduction-and-problematic-73.html
6 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
7 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
developed by Michael Barabanov as master’s thesis under the direction of Prof. Victor Yodaiken at New Mexico Institute of Mining and Technology development, ownership and rights moved to Finite State Machine Labs (FSMLabs) timeline 1997: Version 1: master’s thesis / research project 1999, October: Version 2: rst production version 2001, February: Version 3: rst industry-grade version
8 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
adds hard real-time capabilities to Linux interrupt emulation real-time scheduler periodic tasks high resolution timer non-RT tasks run as idle tasks of RT core real-time IPC between real-time and non-real-time tasks multi-platform X86, PowerPC, Alpha, MIPS, AMD Elan NetSC520
9 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
some other OS with “retrotted” real-time capabilities use similar concepts i.e., real-time operating system hosts a “virtualized” standard operating system a mix of GPL and commercial (FSMLabs) licenses partly patented (U.S. Patent 5,995,745) interrupt interception, hardware abstraction, etc.
10 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
Aarno, D. (2004). Autonomous path planning and real-time control-a solution to the narrow passage problem for path planners and evaluation of real- time linux derivatives for use in robotic control. Unpublished master’s thesis, Department of Numerical Analysis and Computer Science (NADA), KTH,
11 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
Aarno, D. (2004). Autonomous path planning and real-time control-a solution to the narrow passage problem for path planners and evaluation of real- time linux derivatives for use in robotic control. Unpublished master’s thesis, Department of Numerical Analysis and Computer Science (NADA), KTH,
12 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
layer of emulation software between “real” Linux and interrupt controller hardware patched into Linux source all hardware interrupts are caught by emulator Linux has no direct control over the interrupt controller so it does not inuence processing of real-time interrupts real-time interrupts handled in emulator don’t pass through to Linux
13 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
U.S. Patent 5,995,745
14 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
disabling a hardware interrupt unsets a variable within the emulator when hardware interrupt occurs, the variable is checked if set: Linux interrupt handler routine will be invoked if unset: Linux interrupt handler routine won’t be invoked
but but: a bit is set, which holds information about pending interrupts
re-enabling interrupts causes pending Linux interrupt handlers to be invoked
15 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
U.S. Patent 5,995,745
16 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
instructions for interrupt handling replaced with emulating macros
cli (clear interrupt ag) → S_CLI sti (set interrupt ag) → S_STI iret (interrupt return) → S_IRET
17 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
code modied for presentation purposes
18 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
code modied for presentation purposes
19 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
U.S. Patent 5,995,745
20 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
rtl_core.o: main module rtl_time.o: controls processor clocks rtl_sched.o: implements a real-time scheduler rtl_posixio.o: provides a POSIX-like interface to device drivers rtl_ipc.o provides POSIX-style blocking mutexes and semaphores mbuff.o: provides a shared memory between real-time and Linux tasks rtl_fifo.o: provides a real-time non-blocking FIFO rtl_debug.o adds support for source-level debugging rtl_com.o interface with serial ports
21 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
POSIX thread API for real-time threads all real-time tasks are threads in one real-time process per processor
22 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
23 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
24 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
1.000.000 priorities does not perform well with > 20 tasks EDF and RMS scheduler available
i.e., reprogramming of timer chip at each scheduling decision
periodic modes i.e., timer chip programmed once better performance
but not all periods available
25 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
real-time FIFOs implemented using soft interrupts non-blocking real-time interface enables communication between real-time and Linux tasks character device for Linux tasks shared Memory enables communication between real-time and Linux tasks support of mmap() in posixio.o
26 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
no surprises:
27 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
mutex options: lock counts, error checks, priority ceiling
28 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
29 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
30 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
soft interrupts are normal Linux kernel interrupts some Linux kernel functions can be called from them safely no hard real-time properties
31 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
very low latency usage of very limited function set
32 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
CLOCK_MONOTONIC: POSIX clock, runs at a steady rate, never adjusted or reset CLOCK_REALTIME: standard POSIX real-time clock CLOCK_RTL_SCHED: clock which is used by scheduler for task scheduling CLOCK_8254: used on non-SMP x86 machines for scheduling CLOCK_APIC: local APIC clock of the processor that executes clock_gettime
33 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
do as much as possible in (non-real-time) Linux processes e.g., GUI, le system I/O, networking, DB access be careful while implementing real-time tasks executed as kernel modules can easily crash whole system use debugger there is no memory protection in kernel space
34 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
buers are read/written each period for benchmarks: sampling must be done according to sampling theorem i.e., highest measurable frequency is half the sample frequency e.g. from above: 13μs
35 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
straightforward:
new kernel source kernel witch RTLinux patch kernel patched kernel kernel
36 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
improved scheduler performance LNet: hard real-time networking via Ethernet, Firewire 1394, USB 2.0 improved documentation test and validation tools for real-time modules removed kernel module semantic of real-time modules i.e., standard C programs can be real-time via RTLinux module loader
links functions of task modules to entry points in real-time modules
37 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
http://www.yodaiken.com/2005/01/01/rtlinux-is-easy-but-rt-is-dangerous
38 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
extended FIFO implementation integration into the Linux le system support of security attributes
39 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
40 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
now under umbrella term of Kernel and User Systems Programming (KUSP) work theme: “precise computation description, control, and measurement
that signicantly exceeds the capabilities of current practice in most systems”
now a patch to Linux on top of the PREEMPT_RT patch developed in 1998 at the University of Kansas at Information and Telecommunication Technology Center as a patch to Linux on top of UTIME extension to Linux
UTIME: run hardware timer as aperiodic device to increase temporal resolution with little overhead
https://www.ittc.ku.edu/kurt/ Srinivasan, B., Pather, S., Hill, R., Ansari, F., & Niehaus, D. (1998, June). A rm real-time system implementation using commercial o-the-shelf hardware and free software. In Real-Time Technology and Applications Symposium, 1998. Proceedings. Fourth IEEE (pp. 112-119). IEEE.
41 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
no layer between hardware and Linux is taking control of interrupts real-time tasks implemented as kernel modules or user-space programs which are then run by special, KURT-provided kernel module
Aarno, D. (2004). Autonomous path planning and real-time control-a solution to the narrow passage problem for path planners and evaluation of real- time linux derivatives for use in robotic control. Unpublished master’s thesis, Department of Numerical Analysis and Computer Science (NADA), KTH,
42 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
provides rm real-time ¹ synchronization of high-resolution timer using NTP microsecond scheduling resolution ability to dene a real-time schedule so-called e.g., multimedia somewhere between hard real-time … … and soft real-time
i.e., regarding timing requirements e.g., complexity, diversity of supporting services, cost model, not on dedicated or specialized hardware
to minimize jitter, processor busy-waits 50 μs before scheduled arrival time
¹ term coined by the authors
43 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
“normal mode” normal Linux plus high-resolution timers (UTIME) “mixed mode” “normal mode” plus real-time tasks can be run
non-real-time tasks are likely to cause high jitter for real-time tasks i.e.: soft real-time
e.g., for desktop applications “focused mode”
i.e., normal Linux is disabled/frozen
e.g., for embedded and special-purpose applications
44 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
developed in 2000 at Dipartimento di Ingeneria Aerospaziale, Politecnico di Milano by Professor Paolo Mantegazza approach similar to RTLinux supports original RTLinux API extended features
e.g., oating-point operations in interrupt handlers
multi-platform x86, x86_64, PowerPC, some ARM, m68k LGPL 2 license
45 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
Aarno, D. (2004). Autonomous path planning and real-time control-a solution to the narrow passage problem for path planners and evaluation of real- time linux derivatives for use in robotic control. Unpublished master’s thesis, Department of Numerical Analysis and Computer Science (NADA), KTH,
46 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
RTAI on a Pentium II, 233 MHz simultaneously servicing a heavily loaded Linux maximum periodic task iteration rate: 125 kHz 0-13 μs jitter UP 0-30 μs jitter SMP typical sampling task rate: 10 kHz (Pentium 100)
30 kHz Pentium-class CPU 10 kHz (486-class CPU) context switching time: ~ 4 μs
47 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
developed by Concurrent Computer Corporation implemented in RedHawk Linux, Suse Enterprise Real-time Linux applicable for symmetric multiprocessor systems high-priority tasks and interrupts are bound to a more shielded CPU i.e., protected/shielded from unpredictable processing activities conguration via processor anity
easily achievable in today’s standard Linux
with standard tools
48 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
ART-Linux ART-Linux – http://art-linux.sourceforge.net/ hard real-time, allows co-existence of non-real-time and real-time tasks focus on memory protection and avoidance of priority inversion among other things, interrupt handler re-implemented as periodic real-time task Linux/Resource Kernel Linux/Resource Kernel – http://www.cs.cmu.edu/~rajkumar/linux-rk.html timely, guaranteed, and enforced access to physical resources via so-called reserves guaranties via reserve admission control, resource scheduling, limits, accounting RED-Linux RED-Linux – formally at http://linux.ece.uci.edu/RED-Linux/SDK/ adds preemption points in Linux kernel for lower preemption delay adds hierarchical real-time scheduling and a framework to dene policies for it
49 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
QLinux QLinux – http://lass.cs.umass.edu/software/qlinux/ allocates CPU and disk bandwidth fairly and guaranteed for soft real-time fair and guaranteed network packet scheduling for individual ows and aggregates Vieira et al.: “SMART Scheduling for Linux SMART Scheduling for Linux” (Workshop of Real Time Linux, Vienna, 1999) Scheduling Multimedia Applications Real-Time: app. adapts to resources available scheduling based on deadlines for (soft) real-time applications and priorities Childs et al.: “The Linux-SRT integrated multimedia operating system: Bringing QoS to The Linux-SRT integrated multimedia operating system: Bringing QoS to the desktop the desktop” (Real-Time Technology and Applications Symposium, Taiwan, 2001) predictable QoS scheduling for standard Linux (binary compatible) scheduling policies for CPU and memory bandwidth
50 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
51 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
<= 2004: no serious eorts to merge real-time patches into mainline # maybe due to partly commercial character/aspects 2004 – 2009: a few patches posted on the mailing list with heaps of discussion scheduler maintainer, Ingo Molnar, (re-)wrote a combined set of patches developers from KURT, RTLinux and others joined in for development & maintenance also, a lot of work by IBM and RedHat due to, again, military contracts 2009 – 2013: maintenance with part-time funding from RedHat ~ 2014: hobbyist project (maintenance) due to cut funding 2015: initiative by Linux Foundation: found, connected and funded ve developers >= 2016: mainly refactoring for easier integration into mainline
https://wiki.linuxfoundation.org/realtime/rtl/blog#preempt-rt-history
52 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
does not introduce a nano- or micro kernel under “normal” Linux as, e.g., RTLinux or RTAI but modies the kernel to add “native” real-time support as, e.g., KURT or Linux/Resource Kernel
CONFIG_PREEMPT_RT makes kernel preemptable at virtually any point
except in kernel critical sections replaces all kernel spinlocks with mutexes makes all interrupts threaded
i.e., attached interrupt handler just wakes thread with actual interrupt handler code
latencies: higher average, but lower in worst-case and standard deviation
53 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
CONFIG_PREEMPT, CONFIG_PREEMPT_RT_BASE, CONFIG_PREEMPT_RT_FULL
historically collectively referred to as PREEMPT_RT see previous slide
CONFIG_PREEMPT_VOLUNTARY (default)
via explicit preemption/rescheduling points in kernel code typical for desktop systems
CONFIG_PREEMPT_NONE
kernel code is never preempted (historic default)
lowest overhead due to reduced context switching, highest latency
typical for server systems
54 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
ABI-compatible: user space can remain untouched
improved POSIX real-time API support priority inheritance protocol for mutexes high-resolution timers avoids priority inversion in interrupt handlers heavily prots from / leverages Linux ecosystem (its time complexity == its name) 100 Hz, 250 Hz, 300 Hz, … depends on hardware fully merged: CONFIG_HIGH_RES_TIMERS (through threading) e.g., system conguration (e.g., CPU shielding via cpusets), benchmarks, community
55 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
O(1) replaced with Completely Fair Scheduler (CFS) in 2.6.23
multiple scheduling policies available:
SCHED_FIFO: runnable higher-priority threads immediately interrupt lower-priority threads
until done or blocking POSIX-compliant
SCHED_RR: like SCHED_FIFO, but threads get re-queued when exceeding a time quantum
POSIX-compliant
SCHED_DEADLINE: deadline scheduling using EDF
added in 3.14 three additional parameters: runtime, period, deadline Constant Bandwidth Server (CBS) algorithm computes deadlines
so that each task runs for at most its runtime in its period, avoids interference between dierent tasks
Kernel docs are excellent! ;)
56 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019
motivation RTLinux
PREEMPT_RT
: Why a standard OS for real-time? Why Linux? history
architecture concepts : master thesis of 1997 : hard real-time, real-time scheduler, high resolution timer : catch interrupts before they reach Linux, re-play when appropriate : real-time modules, threads, scheduling, IO, IPC, synchronization, … : KURT, RTAI, and more : make the standard Linux kernel preemptable threaded interrupt handlers, mutexes → spinlocks, preemption modes, scheduling, …
57 real-time with Linux lecture on embedded operating systems lukas.pirl@hpi.de 06.02.2019