Finish Proj 3A NOW! No deadline extension for the rest of quarter - PowerPoint PPT Presentation

Finish Proj 3A NOW! No deadline extension for the rest of quarter • Project 0 resubmission for autograding : June 1  Project 0 score =max(old score, old score *0.10 + new score *0.90).  Donot print “shell>” prompt. • Project 3A (May 29).  Harness code is released. • Optional Project 3B (June 4). -- You can use Project 3B to replace midterm OR one of project scores: Project 1, 2, 3A. • Exercise Set 2 (June 4 Thursday 12:30pm) 1 5/21/2015

File Systems CS170 Fall 2015. T. Yang

What to Learn? • File interface review • File-System Structure  File-System Implementation  Directory Implementation • Allocation Methods of Disk Space  Free-Space Management  Contiguous allocation  Block-oriented indexing – Unix inode structure

Files • File concept:  Contiguous logical address space in a persistent storage (e.g. disk). • File structure  None - sequence of words, bytes  Simple record structure – Lines – Fixed length – Variable length  Complex Structures: Formatted document • Who decides the structure:  Operating system  Program

File Attributes • Name – only information kept in human-readable form • Identifier – unique tag (number) identifies file within file system • Type – needed for systems that support different types • Location – pointer to file location on device • Size – current file size • Protection – controls who can do reading, writing, executing • Time, date, and user identification – data for protection, security, and usage monitoring • Information about files are kept in the directory structure, which is maintained on the disk

File Operations • Create • Open(F i )  search the directory structure on disk for entry F i  move the content of entry to memory • Close (F i ) –  move the content of entry F i in memory to directory structure on disk • Write • Read • Reposition within file (e.g. seek) • Delete • Truncate

Access Methods • Sequential Access read next write next reset • Direct Access read n write n position to n read next write next rewrite n n = relative block number

File System Abstraction • Directory  Group of named files or subdirectories  Mapping from file name to file metadata location • Path  String that uniquely identifies file or directory  Ex: /cse/www/education/courses/cse451/12au • Links  Hard link: link from name to metadata location  Soft link: link from name to alternate name • Mount  Mapping from name in one file system to root of another

UNIX File System API • create, link, unlink, createdir, rmdir  Create file, link to file, remove link  Create directory, remove directory • open, close, read, write, seek  Open/close a file for reading/writing  Seek resets current position • fsync  File modifications can be cached  fsync forces modifications to disk (like a memory barrier)

File System Interface • UNIX file open is a Swiss Army knife:  Open the file, return file descriptor  Options: – if file doesn’t exist, return an error – If file doesn’t exist, create file and open it – If file does exist, return an error – If file does exist, open file – If file exists but isn’t empty, nix it then open – If file exists but isn’t empty, return an error – …

Example of Linux read, write, and lseek int main() { int file=0; char buffer[15]; if((file=open("testfile.txt",O_RDONLY)) < -1) return 1; if(read(file,buffer,14) != 14) return 1; printf("%s\n",buffer); if(lseek(file,5,SEEK_SET) < 0) $ cat testfile.txt This is a test file return 1; if(read(file,buffer,19) != 14) $ ./testing return 1; This is a test printf("%s\n",buffer); is a test file return 0; }

Protection • File owner/creator should be able to control:  what can be done  by whom • Types of access Example in Linux  Read  Write  Execute  Append  Delete  List

Access Lists and Groups in Linux • Mode of access: read, write, execute • Three classes of users RWX  a) owner access 7 1 1 1 RWX  b) group access 6 1 1 0 RWX  c) public access 1 0 0 1 • Ask manager to create a group (unique name), say G, and add some users to the group. • For a particular file (say game ) or subdirectory, define an appropriate access. owner group public chmod 761 game Attach a group to a file chgrp G game

Windows Access-Control List Management

Directory Structure • A collection of nodes containing information about all files Directory Files F 1 F 2 F 3 F 4 F n Both the directory structure and the files reside on disk Backups of these two structures are kept on tapes

A Typical File-system Organization on a Disk Partition

Operations Performed on Directory • Search for a file • Create a file • Delete a file • List a directory • Rename a file • Traverse the file system

Directory with single-Level or two-level • A single directory for all users • Two -level

Tree-Structured Directories

Directory with acyclic graph structure • Name Resolution: The process of converting a logical name into a physical resource (like a file)  Traverse succession of directories until reach target file  Global file system: May be spread across the network

Building a File System • File System: Layer of OS that transforms block interface of disks (or other block devices) into Files, Directories, etc. • File System Components  Disk Management: collecting disk blocks into files  Naming: Interface to find files by name, not by blocks  Protection: Layers to keep data secure  Reliability/Durability: Keeping of files durable despite crashes, media failures, attacks, etc • User vs. System View of a File  User’s view: Durable Data Structures  System call interface: – Collection of Bytes (UNIX)  System’s view (inside OS): – Collection of blocks (a block is a logical transfer unit, while a sector is the physical transfer unit on disk) – Block size  sector size; in UNIX, block size is 4KB Kubiatowicz’s cs162 UCB

Translating from User to System View File System • What happens if user says: give me bytes 2 — 12?  Fetch block corresponding to those bytes  Return just the correct portion of the block • What about: write bytes 2 — 12?  Fetch block  Modify portion  Write out Block • Everything inside File System is in whole size blocks  For example, getc() , putc()  buffers something like 4096 bytes, even if interface is one byte at a time • From now on, file is a collection of blocks Kubiatowicz’s cs162 UCB

File System Design • Data structures  Directories: file name -> file metadata – Store directories as files  File metadata: how to find file data blocks  Free map: list of free disk blocks • How do we organize these data structures?  Device has non-uniform performance

Design Challenges • Index structure  How do we locate the blocks of a file? • Index granularity  What block size do we use? • Free space  How do we find unused blocks on disk? • Locality  How do we preserve spatial locality? • Reliability  What if machine crashes in middle of a file system op?

File System Workload • Studying application workload and characteristics can help feature prioritization or optimization of design • What should be considered?  File sizes – Are most files small or large? – Which accounts for more total storage: small or large files?  File access pattern – Small file, large file? – Random access vs sequential access?

File System Workload • File sizes  Are most files small or large? – SMALL  Which accounts for more total storage: small or large files? – LARGE

File System Workload • File access  Are most accesses to small or large files?  Which accounts for more total I/O bytes: small or large files?

File System Workload • File access  Are most accesses to small or large files? – SMALL  Which accounts for more total I/O bytes: small or large files? – LARGE

File System Workload • How are files used?  Most files are read/written sequentially  Some files are read/written randomly – Ex: database files, swap files  Some files have a pre-defined size at creation  Some files start small and grow over time – Ex: program stdout, system logs

Designing the File System: Access Patterns • Sequential Access: bytes read in order ( “ give me the next X bytes, then give me next, etc. ” )  Most of file accesses are of this flavor • Random Access: read/write element out of middle of array ( “ give me bytes i — j ” )  Less frequent, but still important, e.g., mem. page from swap file  Want this to be fast – don ’ t want to have to read all bytes to get to the middle of the file • Content-based Access: ( “ find me 100 bytes starting with JOSEPH ” )  Example: employee records – once you find the bytes, increase my salary by a factor of 2  Many systems don ’ t provide this; instead, build DBs on top of disk access to index content (requires efficient random access) A. Joseph UCB CS162. Spr 2014