Operating System Structure and System Calls

An operating system (OS) is the program that controls the hardware and gives every other program a cleaner, safer machine to run on. It plays two roles at once. As a resource manager it shares the CPU, memory, disks and devices among competing programs and keeps them from damaging each other. As an extended machine it hides awkward hardware behind abstractions: processes instead of raw CPU registers, address spaces instead of physical RAM, files instead of disk sectors.

User programs cannot be trusted with the hardware directly, so the hardware itself provides at least two privilege levels. Ordinary code runs in user mode; the OS kernel runs in kernel mode. A system call is the controlled doorway between them: the only way a user program can ask the kernel to do something privileged, such as reading a file, creating a process or allocating memory.

The protection mechanism (dual mode, privileged instructions, traps and interrupts), the system-call path from a C library function to a kernel handler and back, the main categories of system calls, and the classic ways of structuring a kernel are the base that every later OS topic relies on.

The CPU keeps a mode bit in a status register. In the convention used by Silberschatz, Galvin and Gagne, mode bit 0 means kernel mode and 1 means user mode; other architectures use privilege rings (x86 has rings 0 to 3, with the kernel in ring 0 and applications in ring 3). The numeric encoding is an architecture convention (0 or 1, or a ring number); the rule that matters is:

Privilege rule. An instruction that could let one program harm another program or the OS is privileged. Executing a privileged instruction in user mode does not run it: the hardware raises an exception and transfers control to the kernel, which normally terminates the program.

Reading the clock versus setting the timer. Reading the clock is harmless; setting the timer is privileged. The two are easy to confuse.

Traps, exceptions and interrupts transfer control to designated handlers, often through vectors/tables. An event from user mode enters kernel mode; an event already in kernel mode need not change privilege. System-call dispatch-table lookup is distinct from hardware handler entry. They differ in cause.

A page fault is a fault that the kernel usually repairs; the faulting instruction is then restarted. A division by zero is usually not repairable and ends the process with a signal. A system-call trap returns to the instruction after the trap.

The kernel sets a hardware timer before giving the CPU to a user program. When it expires it interrupts, and the kernel runs again even if the program is in an infinite loop. A timer guarantees periodic regain of control and enables quantum-based round robin even for a compute-bound infinite loop. Device interrupts can also cause event-driven preemption without a timer; periodic time slicing in Round Robin and Multilevel Queue Scheduling needs the periodic timer mechanism.

Programmers rarely issue the trap themselves. They call an API function, such as POSIX read() in the C library, and the library wrapper does the machine-level work.

Mode switch versus context switch. An ordinary returning system call by a user-mode thread costs two privilege transitions (user to kernel, kernel to user). A nonreturning exit call has no return-to-user transition for that caller; count fork parent/child continuations only when the question includes them. It causes a context switch (a different process gets the CPU) only if the call blocks, for example a read() that must wait for the disk, or if the kernel decides to schedule someone else on the way out. "Every system call causes a context switch" is false.

Linux on x86-64 passes up to six arguments in registers. Whatever the method, the kernel must check every user pointer before use, because a malicious program can pass an address belonging to the kernel.

strlen, sqrt and qsort run entirely in user mode. printf formats in user mode and calls write() only when the stdio buffer must be flushed. malloc usually carves blocks out of memory it already owns and calls the kernel (brk or mmap on Linux) only when it needs more.

A program writes 10,000 bytes to a file.

This simplified comparison counts only write calls, their two user/kernel transitions and their overhead; it excludes open, close and any other system calls in both versions.

If each counted write call costs an assumed overhead of 1 microsecond beyond the copying work, version A spends 10 ms in overhead and version B spends 3 microseconds. The difference is the reason stdio buffering exists.

A user-mode program executes, in order: an add, a read of the time-of-day clock, a load of the timer register, and a trap instruction. The first two run normally. The timer load is privileged and raises an exception, so control goes to the kernel; with the usual policy the process is terminated and the trap instruction is never reached. If instead the program had skipped the timer load, the trap would enter the kernel legitimately.

Linux is monolithic with loadable modules. MINIX 3 and QNX are microkernels. Windows and macOS are usually described as hybrids.

Microkernel cost. In a microkernel, a file read is a message from the application to the file server, possibly another message to the disk driver, and replies back, each crossing the kernel. A monolithic kernel serves the same read with one trap and in-kernel function calls. More mode switches, not more privilege, is why microkernels are slower.

Related material. A virtual machine monitor (hypervisor) runs whole operating systems as guests. A guest kernel believes it is in kernel mode, but its privileged instructions trap to the hypervisor, which emulates them. This is the trap mechanism applied one level down.

Common misconceptions.