File System Project Seminar File System Interfaces Prof. Andreas - - PowerPoint PPT Presentation

File System Project Seminar File System Interfaces

• Prof. Andreas Polze

Andreas Grapentin, Sven Köhler Max Plauth, Jossekin Beilharz, Felix Eberhardt Hasso Plattner Institute

1

Polze, Grapentin, Köhler Plauth, Beilharz, Eberhardt 18.10.2017 File System Seminar Interfaces Chart 2

Background

Interfaces Multiple File Systems May be Present

Polze, Grapentin, Köhler Plauth, Beilharz, Eberhardt 18.10.2017 File System Seminar Interfaces Chart 3

/ proc usr home share bin

btrfs procfs ext4 nfs The operating system needs to provide a common interface to different file system drivers, as well as the user space applications.

Block Buffer

Interfaces A Common Abstraction

Polze, Grapentin, Köhler Plauth, Beilharz, Eberhardt 18.10.2017 File System Seminar Interfaces Chart 4

• pen

program

readdir

Virtual File System ext4 proc fs btrfs nfs socket disk

kernel

Interfaces Virtual File Systems

Polze, Grapentin, Köhler Plauth, Beilharz, Eberhardt 18.10.2017 File System Seminar Interfaces Chart 5

■ First introduced with SunOS 2.0 in 1985 (for NFS) ■ Nowadays most major operating systems provide a VFS or comparable means (Linux, *BSD, macOS, Windows, …) ■ File System In User Space (FUSE) is a cross-platform VFS, fed from User Space processes ■ Desktop Enviroments often provide their own VFS (KDE – KIO, Gnome – GIO)

program VFS FUSE FUSE driver block buffer disk GIO

Virtual File System Tasks

Polze, Grapentin, Köhler Plauth, Beilharz, Eberhardt 18.10.2017 File System Seminar Interfaces Chart 6

■ Provides an abstraction layer between applications and file systems ■ Presents a common file model ■ Caches inodes, directory entries, block buffers (e.g. LRU) ■ May include syncronization/locking mechanisms ■ Ease implementation for new file system drivers by auto-completing interfaces (e.g. use mmap to implement read/write)

2

Polze, Grapentin, Köhler Plauth, Beilharz, Eberhardt 18.10.2017 File System Seminar Interfaces Chart 7

Linux VFS

• cfs

• utlines the Linux storage stack as of Kernel version 4.0

• pen(2)

• sector on disk
• bio_vec cnt
• bio_vec index
• bio_vec list
• sector cnt

• ver Ethernet

• verlayfs

Virtual File System in Linux The Basic Building Blocks (4+2)

Polze, Grapentin, Köhler Plauth, Beilharz, Eberhardt 18.10.2017 File System Seminar Interfaces Chart 8

■ struct super_block – represents a mounted system, links the root directory. ■ struct inode – represents an existing file ■ struct file – represents an open file, with pointer ■ struct dentry – represents a directory entry, links a name to an inode ■ struct file_system_type – used to mount a system, builds a superblock ■ struct vfsmount – a device+mountpoint pair

Virtual File System in Linux Relation Of The Four Basic Objects

Polze, Grapentin, Köhler Plauth, Beilharz, Eberhardt 18.10.2017 File System Seminar Interfaces Chart 9

Storage Device proc1 proc2 file file dentry dentry inode superblock fd fd f_dentry f_dentry d_inode i_sb

Dentry cache Hardlink

Virtual File System in Linux OOP-ish?

Polze, Grapentin, Köhler Plauth, Beilharz, Eberhardt 18.10.2017 File System Seminar Interfaces Chart 10

AbstractInode ________________ + size + permission … ________________ + create = 0 + link = 0 + unlink = 0 … InodeExt4 InodeNFS InodeProcFS

Object oriented programming looks like the ideal paradigm here. But Linux, as most kernels, uses C. Virtual methods, polymorphism and inheritance require additional effort in C.

Virtual File System in Linux How To Do Virtual Methods and Objects in C

Polze, Grapentin, Köhler Plauth, Beilharz, Eberhardt 18.10.2017 File System Seminar Interfaces Chart 11

( )

GObject

/

PT2 <= 2013

Virtual File System in Linux How To Do Virtual Methods The Linux Way (I)

Polze, Grapentin, Köhler Plauth, Beilharz, Eberhardt 18.10.2017 File System Seminar Interfaces Chart 12

struct inode { unsigned long i_ino; umode_t i_mode; uid_t i_uid; gid_t i_gid; kdev_t i_rdev; loff_t i_size; struct timespec i_atime; struct timespec i_ctime; struct timespec i_mtime; struct super_block *i_sb; struct inode_operations *i_op; const struct file_operations *i_fop; void *i_private; }

include/linux/fs.h#L566

Virtual File System in Linux How To Do Virtual Methods The Linux Way (II)

Polze, Grapentin, Köhler Plauth, Beilharz, Eberhardt 18.10.2017 File System Seminar Interfaces Chart 13

struct inode_operations { int (*create) (struct inode *, struct dentry *, int); struct dentry * (*lookup) (struct inode *, struct dentry *); int (*link) (struct dentry *, struct inode *, struct dentry *); int (*unlink) (struct inode *, struct dentry *); int (*symlink) (struct inode *, struct dentry *, const char *); int (*mkdir) (struct inode *, struct dentry *, int); int (*rmdir) (struct inode *, struct dentry *); int (*mknod) (struct inode *, struct dentry *, int, dev_t); int (*rename) (struct inode *, struct dentry *, struct inode *, struct dentry *); int (*readlink) (struct dentry *, char *,int); int (*follow_link) (struct dentry *, struct nameidata *); void (*truncate) (struct inode *); int (*permission) (struct inode *, int); int (*setattr) (struct dentry *, struct iattr *); int (*getattr) (struct vfsmount *mnt, struct dentry *, struct kstat *); ... };

inode->i_op->create(...)

Virtual File System in Linux How To Do Virtual Methods The Linux Way (III)

Polze, Grapentin, Köhler Plauth, Beilharz, Eberhardt 18.10.2017 File System Seminar Interfaces Chart 14

struct file_operations { struct module *owner; loff_t (*llseek) (struct file *, loff_t, int); ssize_t (*read) (struct file *, char *, size_t, loff_t *); ssize_t (*write) (struct file *, const char *, size_t, loff_t *); int (*readdir) (struct file *, void *, filldir_t); int (*ioctl) (struct inode *, struct file *, unsigned int, unsigned long); int (*mmap) (struct file *, struct vm_area_struct *); int (*open) (struct inode *, struct file *); int (*flush) (struct file *); unsigned long (*get_unmapped_area)(struct file *, ...); ... };

Your new FS driver needs to fill in those function pointers and provide

{super,inode,file,dentry}_operations to the corresponding data struct.

“this”-pointer

Virtual File System in Linux Adding A New File System (Overview)

Polze, Grapentin, Köhler Plauth, Beilharz, Eberhardt 18.10.2017 File System Seminar Interfaces Chart 15

• 1. Create a new loadable kernel module
• 2. Fill a struct file_system_type
• 3. Register your file system
• 4. Implement get_sb to load and create a superblock
• 5. Implement file, inode, superblock and dentry operations
• 6. Implement iget to load an arbitrary inode and iput to save them
• 7. Load super block and create inode for your root directory

Virtual File System in Linux Adding A New File System (Register)

Polze, Grapentin, Köhler Plauth, Beilharz, Eberhardt 18.10.2017 File System Seminar Interfaces Chart 16

static struct file_system_type demofs_fs_type = { .owner = THIS_MODULE, .name = ”demofs", .get_sb = demofs_get_sb, .kill_sb = kill_block_super, .fs_flags = FS_REQUIRES_DEV, }; static int __init init_tue_fs(void) { return register_filesystem(&demofs_fs_type ); } static void __exit exit_tue_fs(void){ unregister_filesystem(&demofs_fs_type); }

Start a new kernel module: Register filesystem at startup default cleanup, can be specialized I need a device (no procfs, …) required for mounting

Virtual File System in Linux Adding A New File System (Mounting)

Polze, Grapentin, Köhler Plauth, Beilharz, Eberhardt 18.10.2017 File System Seminar Interfaces Chart 17

static struct dentry *demofs_mount(struct file_system_type *fs_type, int flags, const char *dev_name, void *data) { return mount_bdev(fs_type, flags, dev_name, data, demofs_fill_super); } static int demofs_fill_super(struct super_block *sb, void *data, int silent) { struct demofs_sb *info; struct inode *root; int ret; sb->s_blocksize = DEMOBSIZE; sb->s_blocksize_bits = blksize_bits(DEMOBSIZE); if (!(info = kzalloc(DEMOFS_SB_SIZE, GFP_KERNEL))) return -ENOMEM; if ((ret = demofs_blk_read(sb, 0, info, DEMOFS_SB_SIZE))) goto fail; sb->s_fs_info = info; root = demofs_iget(sb, ROOT_DIR_INODE); ... }

In most cases let the kernel handle block device access

Virtual File System in Linux Adding A New File System (Using The Block Buffer)

Polze, Grapentin, Köhler Plauth, Beilharz, Eberhardt 18.10.2017 File System Seminar Interfaces Chart 18

int demofs_blk_read(struct super_block *sb, unsigned long pos, void *buf, size_t buflen) { struct buffer_head *bh; unsigned long offset; size_t segment; while (buflen > 0) {

• ffset = pos & (DEMOBSIZE - 1);

segment = min_t(size_t, buflen, DEMOBSIZE - offset); bh = sb_bread(sb, pos >> DEMOBSBITS); if (!bh) return -EIO; memcpy(buf, bh->b_data + offset, segment); brelse(bh); buf += segment; buflen -= segment; pos += segment; } return 0; }

decrement ref counter

Virtual File System in Linux Adding A New File System (iget, iput)

Polze, Grapentin, Köhler Plauth, Beilharz, Eberhardt 18.10.2017 File System Seminar Interfaces Chart 19

iget – Load inode based on index number What to do if the file system has no index numbers (e.g. FAT)? FAT creates a fake number based on the first data block.

Virtual File System in Linux Starting Point: Examples In The Kernel

Polze, Grapentin, Köhler Plauth, Beilharz, Eberhardt 18.10.2017 File System Seminar Interfaces Chart 20

linux/fs/ramfs – Very simple implementation using libfs linux/fs/romfs – Example using a block device

Virtual File System in Linux Other stuff – Page Cache

Polze, Grapentin, Köhler Plauth, Beilharz, Eberhardt 18.10.2017 File System Seminar Interfaces Chart 21

3

Polze, Grapentin, Köhler Plauth, Beilharz, Eberhardt 18.10.2017 File System Seminar Interfaces Chart 22

File System In User Space

File System In User Space Operations

Polze, Grapentin, Köhler Plauth, Beilharz, Eberhardt 18.10.2017 File System Seminar Interfaces Chart 23

Design very much resembles the Linux VFS. FUSE-C API follows the a similar function-pointer pattern, but combines file and inode operations into one struct.