# Running NixOS

This chapter describes various aspects of managing a running NixOS system, such as how to use the `systemd` service manager.

## Service management

In NixOS, all system services are started and monitored using the systemd program. Systemd is the “init” process of the system (i.e. PID 1), the parent of all other processes. It manages a set of so-called “units”, which can be things like system services (programs), but also mount points, swap files, devices, targets (groups of units) and more. Units can have complex dependencies; for instance, one unit can require that another unit must be successfully started before the first unit can be started. When the system boots, it starts a unit named `default.target`; the dependencies of this unit cause all system services to be started, file systems to be mounted, swap files to be activated, and so on.

The command `systemctl` is the main way to interact with `systemd`. Without any arguments, it shows the status of active units:

    $ systemctl
    -.mount          loaded active mounted   /
    swapfile.swap    loaded active active    /swapfile
    sshd.service     loaded active running   SSH Daemon
    graphical.target loaded active active    Graphical Interface
    ...

You can ask for detailed status information about a unit, for instance, the PostgreSQL database service:

    $ systemctl status postgresql.service
    postgresql.service - PostgreSQL Server
              Loaded: loaded (/nix/store/pn3q73mvh75gsrl8w7fdlfk3fq5qm5mw-unit/postgresql.service)
              Active: active (running) since Mon, 2013-01-07 15:55:57 CET; 9h ago
            Main PID: 2390 (postgres)
              CGroup: name=systemd:/system/postgresql.service
                      ├─2390 postgres
                      ├─2418 postgres: writer process
                      ├─2419 postgres: wal writer process
                      ├─2420 postgres: autovacuum launcher process
                      ├─2421 postgres: stats collector process
                      └─2498 postgres: zabbix zabbix [local] idle

    Jan 07 15:55:55 hagbard postgres[2394]: [1-1] LOG:  database system was shut down at 2013-01-07 15:55:05 CET
    Jan 07 15:55:57 hagbard postgres[2390]: [1-1] LOG:  database system is ready to accept connections
    Jan 07 15:55:57 hagbard postgres[2420]: [1-1] LOG:  autovacuum launcher started
    Jan 07 15:55:57 hagbard systemd[1]: Started PostgreSQL Server.

Note that this shows the status of the unit (active and running), all the processes belonging to the service, as well as the most recent log messages from the service.

Units can be stopped, started or restarted:

    $ systemctl stop postgresql.service
    $ systemctl start postgresql.service
    $ systemctl restart postgresql.service

These operations are synchronous: they wait until the service has finished starting or stopping (or has failed). Starting a unit will cause the dependencies of that unit to be started as well (if necessary).

## Rebooting and shutting down

The system can be shut down (and automatically powered off) by doing:

    $ shutdown

This is equivalent to running `systemctl poweroff`.

To reboot the system, run

    $ reboot

which is equivalent to `systemctl reboot`. Alternatively, you can quickly reboot the system using `kexec`, which bypasses the BIOS by directly loading the new kernel into memory:

    $ systemctl kexec

The machine can be suspended to RAM (if supported) using `systemctl suspend`, and suspended to disk using `systemctl hibernate`.

These commands can be run by any user who is logged in locally, i.e. on a virtual console or in X11; otherwise, the user is asked for authentication.

## User sessions

Systemd keeps track of all users who are logged into the system (e.g. on a virtual console or remotely via SSH). The command `loginctl` allows querying and manipulating user sessions. For instance, to list all user sessions:

    $ loginctl
       SESSION        UID USER             SEAT
            c1        500 eelco            seat0
            c3          0 root             seat0
            c4        500 alice

This shows that two users are logged in locally, while another is logged in remotely. (“Seats” are essentially the combinations of displays and input devices attached to the system; usually, there is only one seat.) To get information about a session:

    $ loginctl session-status c3
    c3 - root (0)
               Since: Tue, 2013-01-08 01:17:56 CET; 4min 42s ago
              Leader: 2536 (login)
                Seat: seat0; vc3
                 TTY: /dev/tty3
             Service: login; type tty; class user
               State: online
              CGroup: name=systemd:/user/root/c3
                      ├─ 2536 /nix/store/10mn4xip9n7y9bxqwnsx7xwx2v2g34xn-shadow-4.1.5.1/bin/login --
                      ├─10339 -bash
                      └─10355 w3m nixos.org

This shows that the user is logged in on virtual console 3. It also lists the processes belonging to this session. Since systemd keeps track of this, you can terminate a session in a way that ensures that all the session’s processes are gone:

    $ loginctl terminate-session c3

## Control groups

To keep track of the processes in a running system, systemd uses control groups (cgroups). A control group is a set of processes used to allocate resources such as CPU, memory or I/O bandwidth. There can be multiple control group hierarchies, allowing each kind of resource to be managed independently.

The command `systemd-cgls` lists all control groups in the `systemd` hierarchy, which is what systemd uses to keep track of the processes belonging to each service or user session:

    $ systemd-cgls
    ├─user
    │ └─eelco
    │   └─c1
    │     ├─ 2567 -:0
    │     ├─ 2682 kdeinit4: kdeinit4 Running...
    │     ├─ ...
    │     └─10851 sh -c less -R
    └─system
      ├─httpd.service
      │ ├─2444 httpd -f /nix/store/3pyacby5cpr55a03qwbnndizpciwq161-httpd.conf -DNO_DETACH
      │ └─...
      ├─dhcpcd.service
      │ └─2376 dhcpcd --config /nix/store/f8dif8dsi2yaa70n03xir8r653776ka6-dhcpcd.conf
      └─ ...

Similarly, `systemd-cgls cpu` shows the cgroups in the CPU hierarchy, which allows per-cgroup CPU scheduling priorities. By default, every systemd service gets its own CPU cgroup, while all user sessions are in the top-level CPU cgroup. This ensures, for instance, that a thousand run-away processes in the `httpd.service` cgroup cannot starve the CPU for one process in the `postgresql.service` cgroup. (By contrast, it they were in the same cgroup, then the PostgreSQL process would get 1/1001 of the cgroup’s CPU time.) You can limit a service’s CPU share in `configuration.nix`:

    systemd.services.httpd.serviceConfig.CPUShares = 512;

By default, every cgroup has 1024 CPU shares, so this will halve the CPU allocation of the `httpd.service` cgroup.

There also is a `memory` hierarchy that controls memory allocation limits; by default, all processes are in the top-level cgroup, so any service or session can exhaust all available memory. Per-cgroup memory limits can be specified in `configuration.nix`; for instance, to limit `httpd.service` to 512 MiB of RAM (excluding swap) and 640 MiB of RAM (including swap):

    systemd.services.httpd.serviceConfig.MemoryLimit = "512M";
    systemd.services.httpd.serviceConfig.ControlGroupAttribute = [ "memory.memsw.limit_in_bytes 640M" ];

The command `systemd-cgtop` shows a continuously updated list of all cgroups with their CPU and memory usage.

## Logging

System-wide logging is provided by systemd’s journal, which subsumes traditional logging daemons such as syslogd and klogd. Log entries are kept in binary files in `/var/log/journal/`. The command `journalctl` allows you to see the contents of the journal. For example,

    $ journalctl -b

shows all journal entries since the last reboot. (The output of `journalctl` is piped into `less` by default.) You can use various options and match operators to restrict output to messages of interest. For instance, to get all messages from PostgreSQL:

    $ journalctl -u postgresql.service
    -- Logs begin at Mon, 2013-01-07 13:28:01 CET, end at Tue, 2013-01-08 01:09:57 CET. --
    ...
    Jan 07 15:44:14 hagbard postgres[2681]: [2-1] LOG:  database system is shut down
    -- Reboot --
    Jan 07 15:45:10 hagbard postgres[2532]: [1-1] LOG:  database system was shut down at 2013-01-07 15:44:14 CET
    Jan 07 15:45:13 hagbard postgres[2500]: [1-1] LOG:  database system is ready to accept connections

Or to get all messages since the last reboot that have at least a “critical” severity level:

    $ journalctl -b -p crit
    Dec 17 21:08:06 mandark sudo[3673]: pam_unix(sudo:auth): auth could not identify password for [alice]
    Dec 29 01:30:22 mandark kernel[6131]: [1053513.909444] CPU6: Core temperature above threshold, cpu clock throttled (total events = 1)

The system journal is readable by root and by users in the `wheel` and `systemd-journal` groups. All users have a private journal that can be read using `journalctl`.

## Cleaning up the Nix store

Nix has a purely functional model, meaning that packages are never upgraded in place. Instead new versions of packages end up in a different location in the Nix store (`/nix/store`). You should periodically run Nix’s garbage collector to remove old, unreferenced packages. This is easy:

    $ nix-collect-garbage

Alternatively, you can use a systemd unit that does the same in the background:

    $ systemctl start nix-gc.service

You can tell NixOS in `configuration.nix` to run this unit automatically at certain points in time, for instance, every night at 03:15:

    nix.gc.automatic = true;
    nix.gc.dates = "03:15";

The commands above do not remove garbage collector roots, such as old system configurations. Thus they do not remove the ability to roll back to previous configurations. The following command deletes old roots, removing the ability to roll back to them:

    $ nix-collect-garbage -d

You can also do this for specific profiles, e.g.

    $ nix-env -p /nix/var/nix/profiles/per-user/eelco/profile --delete-generations old

Note that NixOS system configurations are stored in the profile `/nix/var/nix/profiles/system`.

Another way to reclaim disk space (often as much as 40% of the size of the Nix store) is to run Nix’s store optimiser, which seeks out identical files in the store and replaces them with hard links to a single copy.

    $ nix-store --optimise

Since this command needs to read the entire Nix store, it can take quite a while to finish.