equivariant

Catproofing my keyboard shortcuts

2026-04-22T00:00:00+00:00

Problem

I have a number of global keyboard shortcuts configured in my desktop environment (currently Xfce). They are:

Shortcut	Action
Alt+P	Open launcher
Alt+F	Open home folder
Alt+X	Open web browser
Alt+O	Open text editor
Shift+Alt+Return	Open terminal
Shift+Print	Screenshot full screen
Ctrl+Print	Screenshot selection
Shift+Ctrl+Print	Screenshot selection (to clipboard)
Alt+B	Music: next track
Alt+C	Music: play/pause
Alt+Z	Music: previous track
Alt+D	Music: show current track
Alt+F1	Open applications menu
Alt+Q	Lock screen

My cat likes to flop over on my keyboard, and this can wreak havoc due to the global shortcuts. The main culprits were:

Alt+F, Alt+X, Alt+O, Shift+Alt+Return: If held down, opens a huge number of windows.
Alt+B, Alt+C, Alt+Z, Alt+D: Starts making noise, which can be jarring if I'm trying to do something else.

Mitigation: rate limiting

The first mitigation I implemented was rate limiting: instead of opening hundreds of web browser windows, it's limited to a more manageable number. I hacked together a small Python script to do the rate limiting: ratelimit. It is invoked like so:

$ ratelimit --help
usage: ratelimit [-h] [-n N] [-i I] [-k KEY] [-v] cmd [args ...]

Only allow a program to run N times every I seconds

positional arguments:
  cmd               Command to run
  args              Arguments to command

options:
  -h, --help        show this help message and exit
  -n, --count N     Max executions per interval (default 3)
  -i, --interval I  Interval to limit execution count within (seconds, default 60)
  -k, --key KEY     Unique identifier for this command (defaults to command)
  -v, --verbose     Enable debug output

Mitigation: music player

To defuse the music player shortcuts, I modified them so they only act if the music player is currently running. Previously, they would launch the music player if it wasn't already running. Now, I must explicitly start the music player first.

I'm using Strawberry, so I changed the command from strawberry --play-pause to pgrep strawberry && strawberry --play-pause (and similar for the other shortcuts).

Final configuration

The configuration I ended up with is as follows:

Shortcut	Command	Note
Alt+P	`dmenu_run`	dmenu
Alt+F	`ratelimit -- /usr/bin/caja --no-desktop /home/me`
Alt+X	`ratelimit -- /usr/bin/firefox`
Alt+O	`ratelimit -- /usr/bin/gvim`
Shift+Alt+Return	`ratelimit -- /usr/bin/xfce4-terminal`
Shift+Print	`screenshot`	screenshot
Ctrl+Print	`screenshot --select`
Shift+Ctrl+Print	`screenshot --select --clip`
Alt+B	`sh -c 'pgrep strawberry && ratelimit -- /usr/bin/strawberry --next'`
Alt+C	`sh -c 'pgrep strawberry && ratelimit -- /usr/bin/strawberry --play-pause'`
Alt+Z	`sh -c 'pgrep strawberry && ratelimit -- /usr/bin/strawberry --previous'`
Alt+D	`sh -c 'pgrep strawberry && ratelimit -- /usr/bin/strawberry --show-osd'`
Alt+F1	`xfce4-popup-applicationsmenu`
Alt+Q	`xflock4`

This has been working reasonably well for me.

Printer troubleshooting (personal notes)

2026-01-26T00:00:00+00:00

Printers are notoriously unreliable. Here, I record printing issues I've encountered and how I fixed them. This saves me time if I encounter them again a year or two later and don't recall the solution. Perhaps it will also be useful for other people who encounter the same issues.

I'm currently using a Brother HL-L2300D with CUPS.

Ensure printer is on and auto power off is disabled

There have been times when my printer was off and I didn't notice. This happened because I had accidentally enabled auto power off.

Print a copy of the "printer settings" page: turn off the printer, turn it back on, then press the Go button three times. Check if AUTO POWER OFF is set to "OFF".

If it isn't, press Go four times. All the LEDs will blink twice when auto power off is enabled, or once when it is disabled. To confirm, print another copy of the "printer settings" page.

Reference: https://support.brother.com/g/b/faqend.aspx?c=eu_ot&lang=en&prod=hll2352dw_eu&ftype3=2029&faqid=faq00100216_510

Ensure printer is not paused

Open the CUPS administration page at http://localhost:631/printers/. If the status is "Paused", click the printer's name, then do Maintenance → Resume Printer.

Duplex/double-sided printing doesn't work from Gnome print settings

The Gnome print settings seem to be ignored. Instead, find the printer on http://localhost:631/printers/, then go to Administration → Set Default Options. Set Duplex to "DuplexNoTumble".

DIY drop bar pogies

2025-11-26T00:00:00+00:00

Pogies are a good way to keep your hands warm and dry when biking in cold or rainy weather. In the article Make Your Own Ultralight Pogies on Bikepacking.com, Greg Maino explains how to sew a pair of lightweight, waterproof, uninsulated pogies. The pattern in the article works best for flat bars, and requires some modification to work well with drop bars. In this article, I describe those modifications.

If you're looking for a ready-made product, consider Pogie Lites from BIKEIOWA. They work for both flat bars and drop bars.

Here is my modified pattern: template.jpg

The image is 123.5 pixels per inch, if you wish to print it. If you are hand drawing the pattern, you can use the annotated measurements as a guide. The shape of the curves isn't critial, so just eyeball it.

After tracing the pattern onto fabric, flip it horizontally and trace it again so that there are two copies of it back to back, as shown in step 2 of the original guide. This allows you to make each pogie out of a single piece of fabric and eliminates a seam at the top of the pogie.

Materials used:

HyperD 300 - 1 yard (58" wide)
4 mm elastic cord and toggles
Gutermann Mara 70 - Tex 40 polyester thread

I did not find the loop or snaps from the original guide helpful for drop bars, so I omitted them.

Here is what the pogie looks like after sewing and cutting off the excess fabric from the seam allowance:

Here it is installed on a bike:

And a close-up of where the elastic cords go:

See the Pogie Lites V3.0 Dropbar Installation video for how to install them. I use the method shown at 1:50, but also cinch the bottom cord as shown at 0:50.

Bike light duty cycle

2025-11-18T00:00:00+00:00

I noticed in a video that my bike's rear light (CatEye ViZ450) appeared to be flashing, despite it being in "constant on" mode. This indicated to me that it was using pulse width modulation (PWM)—repeatedly turning on and off at a high frequency in order to appear dimmer. This surprised me, as the light does not have different brightness modes available. One can cycle between constant mode and three different flashing modes, but there is no control over brightness.

I was curious what the PWM duty cycle was, so I hooked up a quick circuit using a phototransistor (similar to my janky tachometer):

On the oscilloscope, it looked like this:

The measurements indicate it's running at 100 Hz with a 10% duty cycle. No wonder it appeared to flash on the camera, which probably had a framerate of 30 or 60 fps.

After thinking about it a little more, I realized why they used such a low duty cycle. The most intense flashing mode, presumably intended for emergencies, has a very bright flash a couple times a second. It likely uses the full duty cycle during that flash.

Extech EX330 temperature calibration

2022-11-11T00:00:00+00:00

I noticed that the temperature readings on my Extech EX330 multimeter didn't match the readings on two different thermostats, so I decided to try calibrating it. Someone on the EEVblog forums contacted Extech support about calibration, who provided these directions:

4.1.0 mVDC Adjustment
4.1.1. Set the EX330 to mVDC.
4.1.2. Apply 100.0mVDC to the input sockets of the EX330.
NOTE: The display should indicate "100.0 mV."
4.1.3. Adjust VR3 until you have a reading of 100.0mVDC.

4.2.0 VAC Adjustment
4.2.1. Set the EX330 to VAC.
4.2.2. Apply 1.000VAC/ 60Hz into the input sockets of the EX330.
NOTE: The display should indicate "1.000 VAC".
4.2.3. Adjust VR2 until the UUT displays a reading of 1.000VAC.

4.3.0 Capacitance Adjustment
4.3.1. Set the EX330 to CAP.
4.3.2. Apply 100.0nF into the input sockets of the EX330.
NOTE: The display should indicate "100.0 nF".
4.3.3. Adjust VR1 until the UUT displays a reading of 100.0nF.

4.4.0 Temperature Adjustment
4.4.1. Set the EX330 to Deg C.
4.4.2. Apply 25°C into the input sockets of the EX330 using a type K thermocouple wire.
NOTE: The display should indicate "25°C".
4.4.3. Adjust VR5 until the UUT displays a reading of 25°C.

4.4.4. Apply 750°C into the input sockets of the EX330.
NOTE: The display should indicate "750°C".
4.4.5. Adjust VR6 until the UUT displays a reading of 750°C.

4.4.6. Set the EX330 to Deg F.
4.4.7. Apply 77°F into the input sockets of the EX330.
NOTE: The display should indicate "77°F".
4.4.8. Adjust VR7 until the UUT displays a reading of 77°F.

I don't have a stable 25°C or 750°C reference, so I adapted the calibration procedure to use an ice bath and boiling water.

Calculate boiling point

Due to lower air pressure, water boils at a lower temperature at higher altitudes. The barometric formula can be used to calculate the pressure at a given altitude:

$\begin{aligned} P &= P_0 \exp\left(\frac{-gMh}{RT}\right),\\ \small{\text{where }} & \small{P_0 = \text{air pressure at sea level} \approx 760 \,\mathrm{mmHg}}\\ & \small{g = \text{acceleration due to gravity} \approx 9.807 \,\mathrm{m}/\mathrm{s}^2}\\ & \small{M = \text{molar mass of air} \approx 0.02896 \,\mathrm{kg}/\mathrm{mol}}\\ & \small{h = \text{altitude above sea level}}\\ & \small{R = \text{ideal gas constant} \approx 8.315 \,\mathrm{J}/(\mathrm{mol}\cdot\mathrm{K})}\\ & \small{T = \text{outdoor air temperature}}. \end{aligned}$

The final calculation is not very sensitive to air temperature, so you can fix it at 15°C (288.15 K).

The vapor pressure of a liquid at a given temperature is the pressure at which it evaporates. At lower pressure, a liquid evaporates at a lower temperature. The relation between vapor pressure and temperature can be approximated with the Antoine equation, shown here with the empirical constants for water:

$T_\mathrm{°C} \approx \frac{1730.63}{8.07131 - \log_{10} P_\mathrm{mmHg}} - 233.426.$

All together, the boiling point in Celsius at a given altitude $h$ in meters is:

$T_\mathrm{boil,°C} \approx \frac{1730.63}{5.1905 + 0.00005146 \cdot h} - 233.426.$

Here's a table, with altitudes rounded to the nearest 100 m:

Altitude above sea level	Boiling point of water
0 m	100°C (212°F)
300 m	99°C (210°F)
600 m	98°C (208°F)
900 m	97°C (207°F)
1200 m	96°C (205°F)
1500 m	95°C (203°F)
1800 m	94°C (201°F)
2200 m	93°C (199°F)
2500 m	92°C (198°F)

Calibrate

Open the multimeter

Remove the silicone case, battery cover, and the 3 screws for the back cover. Find the potentiometers labeled VR5, VR6, and VR7 located on the back (battery side) of the circuit board.

Ice bath calibration, °C

Put a couple dozen ice cubes in a large bowl and add just enough cold water to cover the ice. After a few minutes, the ice bath will be 0°C. Attach the temperature probe to the multimeter, set it to °C, and submerse the probe in the water. Take care that it is not touching the sides of the bowl. Now adjust VR5 with a screwdriver until the display reads 0°C.

Boiling water calibration, °C

Put a pot of water on a burner and heat it until it begins boiling. Set the multimeter to °C and submerse the probe in the water, taking care not to let it touch the sides of the pot. Adjust VR6 until the display reads the boiling point you calculated above (e.g., 98°C at 600 m altitude).

Boiling water calibration, °F

Switch the multimeter to °F. Adjust VR7 until the display reads the proper boiling point (e.g., 208°F for 600 m altitude).

Onsite backups with zrepl

2022-03-14T00:00:00+00:00

For some time I've been using a homegrown solution for onsite backups. It works fine, but required me to manually prune old snapshots periodically. Rather than extend my script to support pruning, I decided to use a prebuilt solution.

zrepl is a tool that handles periodic snapshotting, sending snapshots to other zpools (possibly remote), and pruning old snapshots. It does exactly what I needed, and more.

Installation

For Arch Linux users, a package for zrepl is available in the AUR. For other distributions, see the documentation.

cd aur
auracle download -r zrepl
cd zrepl
less PKGBUILD  # Always inspect AUR packages!
makepkg -sic

Configuration

On my system, I have my root filesystem in the dataset zroot/ROOT/default, a HDD-backed zpool named slow, and a dataset containing lectures and other large media files at slow/bulk. I configured zrepl to back up the root dataset to slow/zrepl and take a daily snapshot of slow/bulk.

# /etc/zrepl/zrepl.yml

global:
  logging:
    - type: syslog
      format: human
      level: warn

jobs:
  # Periodically send a snapshot of the root filesystem to slow/zrepl.
  - name: zroot_to_slow
    type: push
    connect:
      type: local
      # Must match listener_name from sink below.
      listener_name: slow_zrepl_sink
      # A name for the system being backed up. I called it `local` because
      # there's only one system involved. The hostname would also be fine.
      client_identity: local
    filesystems:
      "zroot/ROOT/default": true
    snapshotting:
      type: periodic
      prefix: zrepl_
      interval: 1h
    pruning:
      keep_sender:
        - type: not_replicated
        - type: regex
          # Keep snapshots not created by zrepl
          negate: true
          regex: "^zrepl_.*"
      keep_receiver:
        - type: grid
          # Hourly for a day
          # Daily for 6 months
          # Weekly for 2 years
          # Monthly for 10 years
          grid: 1x1d(keep=all) | 180x1d | 104x1w | 120x30d
          regex: "^zrepl_.*"

  # This is the dataset where the snapshots are sent to.
  - name: slow_zrepl_sink
    type: sink
    serve:
      type: local
      # Must match listener_name from push above.
      listener_name: slow_zrepl_sink
    root_fs: slow/zrepl
    recv:
      placeholder:
        encryption: inherit

  # Take a daily snapshot of slow/bulk.
  - name: snapshot_bulk
    type: snap
    filesystems:
      "slow/bulk": true
    snapshotting:
      type: periodic
      prefix: zrepl_
      interval: 24h
    pruning:
      keep:
        - type: grid
          # Daily for 3 months
          # Weekly for 6 months
          # Monthly for 1 year
          grid: 90x1d | 26x1w | 12x30d
          regex: "^zrepl_.*"
        - type: regex
          # Keep snapshots not created by zrepl
          negate: true
          regex: "^zrepl_.*"

Starting the service

First, create the destination dataset:

sudo zfs create -o readonly=on -o mountpoint=none slow/zrepl

Start zrepl:

sudo systemctl enable --now zrepl.service

Wait until the first snapshot is sent:

sudo zrepl status

Give the destination dataset a mountpoint so you can easily access its snapshots:

sudo zfs set mountpoint=/slow/zrepl slow/zrepl/local/zroot/ROOT/default

You should now be able to access the snapshots at /slow/zrepl/.zfs/snapshot/zrepl_YYYYMMDD_HHMMSS_000. Note that the .zfs directory is hidden by default, so you must navigate to it directly.

Monitoring

Set a calendar reminder to manually test backups monthly. You should verify that the backups are current and you are actually able to restore from them.

Add a cron job that checks for errors from zrepl in the last day:

sudo crontab -e

#min hour dom mon dow cmd
0    0    *   *   *   journalctl -S -1d -p err -t zrepl -q

If any errors occurred in the last day, they will be mailed to root.

This has the potential to miss errors if, for example, the computer is powered off when the cron job is scheduled to run. That could be fixed, but I find this to be sufficient for my needs.

Onsite backup script for ZFS

2022-03-12T00:00:00+00:00

On my home computer, I employ a 3-2-1 backup strategy¹:

Primary copy of data on mirrored SSDs.
Daily offsite backups with Borg.
Daily onsite backups to a HDD-backed zpool.

In this article, I describe my onsite backups.

I have since switched to using zrepl instead of a homegrown backup script. My configuration is documented here.

The script described in this article is still useful, as it is relatively simple and does not require a specialized daemon.

How it works

The backup script sends incremental backups of a dataset to another zpool using zfs send. To facilitate incremental backups without requiring the source dataset to retain a full snapshot, it uses ZFS bookmarks.

Here is what the backups look like:

NAME                                     USED  AVAIL     REFER  MOUNTPOINT
zroot/ROOT/default                       199G   169G      199G  /
zroot/ROOT/default#2022-03-12.133001        -      -      199G  -
slow/backupsend                         1.58T   772G      218G  /slow/backupsend
slow/backupsend@2022-03-08.133001       1.02G      -      216G  -
slow/backupsend@2022-03-09.133001       2.70G      -      216G  -
slow/backupsend@2022-03-10.133001       1.32G      -      216G  -
slow/backupsend@2022-03-11.133001       1.18G      -      217G  -
slow/backupsend@2022-03-12.133001          0B      -      218G  -

Here, zroot/ROOT/default is the dataset being backed up and slow/backupsend is the dataset where the daily backups are being stored. While the script is running, there will be an additional temporary snapshot and bookmark on the source dataset.

The script

Here is the script, which I saved as /usr/local/bin/zfs-rootbackupsend:

#!/bin/bash
set -euxo pipefail

if [ "$EUID" -ne 0 ]; then
    echo 'Run as root'
    exit 1
fi

src='zroot/ROOT/default'
dst='slow/backupsend'

now="$(date -u +%Y-%m-%d.%H%M%S)"

# Determine bookmark corresponding to latest snapshot.
lastbook="$(zfs list -t bookmark -H -o name -s name "$src" | tail -n 1)"

# Snapshot the source dataset. Take a bookmark so that we can send the next
# incremental snapshot from this point even after deleting the snapshot.
zfs snapshot "$src@$now"
zfs bookmark "$src@$now" "$src#$now"

if ! zfs list "$dst" >/dev/null 2>&1; then
    # Initial full send.
    zfs send "$src@$now" | zfs receive -o readonly=on "$dst@$now"
else
    # Determine the latest snapshot of the destination dataset.
    last="$(zfs list -t snapshot -H -o name -s name "$dst" | tail -n 1 | cut -d@ -f2)"

    # Send the changes since the last snapshot on the destination dataset.
    zfs send -i "$src#$last" "$src@$now" | zfs receive "$dst@$now"
fi

# Remove the snapshot of the source dataset to free up space. The bookmark
# still remains and can be used as the incremental source for the next backup.
zfs destroy "$src@$now"

# Remove bookmark corresponding to previous snapshot. Only the latest bookmark
# is needed to make an incremental snapshot, so there's no point keeping older
# bookmarks around.
if [[ -n "$lastbook" ]]; then
    zfs destroy "$lastbook"
fi

# Take a snapshot of the data that's unique to the non-root zpool.
# NOTE: If you're using this script, you probably don't need this and can
#       comment out this line.
zfs snapshot "slow/bulk@$now"

Set it as executable:

sudo chmod 755 /usr/local/bin/zfs-rootbackupsend

Add a cron entry to /etc/cron.d/zfs-rootbackupsend:

SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
30 5 * * * root nice chronic /usr/local/bin/zfs-rootbackupsend

Pruning backups

To prune old backups down to monthly:

prune() {
    zfs list -H -o name -t snapshot "$1" > snaps
    # DATASETNAME@YYYY-MM-DD
    #            \---9---/
    prefix="$(($(echo -n "$1" | wc -c)+9))"
    uniq -w "$prefix" snaps > keep
    comm -23 snaps keep > destroy
    while read s; do echo "$s"; echo sudo zfs destroy "$s"; done < destroy
}
prune slow/backupsend
prune slow/bulk
rm snaps keep destroy

3-2-1 rule: 3 copies of the data, 2 different media types, 1 offsite copy. ↩

Reconstructing audio from its spectrogram

2022-01-18T00:00:00+00:00

Derived from spectrogram by Wikipedia user Aquegg, public domain.

A spectrogram is a visualization of the frequencies present in audio. Time is shown on the x-axis and frequency is shown on the y-axis. The above spectrogram is from the Wikipedia article Spectrogram and was generated from audio of someone speaking the words "nineteenth century".

The discrete Fourier transform

Let $x$ be a signal whose value is sampled at a rate $R$ (e.g., 44100 samples/second). Let $x[n]$ denote the $n^\text{th}$ sample, i.e., the value of the signal at time $n/R$ .

The discrete Fourier transform (DFT) takes a signal and calculates which frequencies are present in it. For example, if $x$ were the sum of a sine wave with frequency 500 Hz and another with frequency 600 Hz, then the DFT would have peaks at 500 Hz and 600 Hz. More precisely, if $X = \operatorname{DFT}(x)$ , then $X[k]$ is the coefficient of the sinusoid with frequency $kR/N$ in the original signal, where $N$ is the total number of samples. Note that $X[k]$ is a complex number. The DFT is typically calculated with an algorithm called the fast Fourier transform (FFT).

Window functions and the short-time Fourier transform

A window function is a function that is multiplied against a signal elementwise to "focus" on a small part of the signal and ignore the rest:

The window function is the one in the middle. This one is the Hann window.

The short-time Fourier transform (STFT) is the DFT calculated over many small windows of the signal. More precisely, $\operatorname{STFT}(x)[m,k] = \operatorname{DFT}(x_m * w)[k],$ where $*$ denotes elementwise multiplication and $x_m$ is the signal shifted over by $m$ places ( $x_m[n] = x[m+n]$ ).

Let's break that down: The elementwise multiplication $x_m * w$ zeros out all but a small part of the signal starting at position $m$ . Then we apply the DFT and extract the $k^\text{th}$ element, i.e., the coefficient associated with frequency $kR/N$ . Putting that back together, $\operatorname{DFT}(x_m * w)[k]$ is the magnitude of frequency $kR/N$ starting at time $m/R$ in the original signal.

A spectrogram is the square of the STFT: $\operatorname{spec}(x)[m,k] = |\operatorname{STFT}(x)[m,k]|^2.$ Typically the spectrogram isn't computed at every value of $m$ , but instead on wider-spaced, overlapping windows.

Inverting the spectrogram

The DFT comes with a counterpart to undo the transformation, the inverse discrete Fourier transform (IDFT). It has the property that $\operatorname{IDFT}(\operatorname{DFT}(x)) = x$ .

Suppose we have computed the STFT to be $X = \operatorname{STFT}(x)$ . We can then compute the inverse short-time Fourier transform (ISTFT) by $\operatorname{ISTFT}(X) = \sum_m \operatorname{IDFT(X[m,\cdot])},$ where " $\cdot$ " denotes an entire row of the array. This assumes that overlapping window functions always sum to 1, which can be achieved through normalization. The ISTFT has the property that $\operatorname{ISTFT}(\operatorname{STFT}(x)) = x$ .

Now suppose we are given a spectrogram $S = \operatorname{spec}(x)$ . The spectrogram throws away some information relative to the STFT: by squaring each element, we lose the phase, the amount by which each sine wave is shifted horizontally. If we were to invert the spectrogram by simply taking its ISTFT, it would be unintelligible because the phases would be incorrect.

Thankfully, there is an iterative algorithm due to Griffin and Lim (1984) to estimate the phase: $\begin{aligned} x_0 &= \operatorname{ISTFT(S)};\\ X_{i+1} &= \operatorname{ISTFT}\left( S \cdot \frac{\operatorname{STFT}(x_i)}{|\operatorname{STFT}(x_i)|} \right). \end{aligned}$ After a few iterations, this yields a decent approximation of the original signal ( $x_{10} \approx x$ ).

Implementation

The following implementation is written in Python 3.10.1 and uses NumPy 1.21.5, SciPy 1.7.3, and Pillow 9.0.0.

# To install dependencies:
#   python3 -m venv venv
#   source venv/bin/activate
#   pip install numpy==1.21.5 scipy==1.7.3 Pillow==9.0.0

from PIL import Image
import numpy as np
from scipy.signal.windows import hann
from scipy.io import wavfile

The STFT and ISTFT are defined as before. Since audio signal are real-valued, we can use NumPy's rfft and irfft functions, which compute the DFT and IDFT for real-valued signals.

def rstft(x, win, hop):
    """
    Compute the short-time Fourier transform (STFT) of a real-valued signal.

    Parameters
    ----------
    x : array_like
        Signal to compute the STFT of.
    win : int
        Width of the Hann window.
    hop : int
        Number of samples to skip between windows. If `hop < win`, the windows
        will overlap. Normalization to compensate for window overlap is
        performed automatically.

    Returns
    -------
    y : array_like
        STFT of `x`. Axis 0 is frequency and axis 1 is time.
    """
    # The Hann window has the property that at 50% overlap, overlapping windows
    # sum to 1. For denser overlaps, we compute a scaling factor.
    w = hann(win)
    scale = 2 * hop/win

    # Round length up to the nearest multiple of `hop`.
    length = ((x.shape[0] - win - 1) // hop + 1) * hop + win
    x_pad = np.array(x)
    x_pad.resize(length)

    # Compute the STFT.
    y = []
    for t in range(0, length - win + hop, hop):
        y.append(np.fft.rfft(x_pad[t:t+win] * w))
    return scale * np.array(y).T

def irstft(y, win, hop):
    """
    Compute the inverse short-time Fourier transform (ISTFT) of a real-valued
    signal.

    Parameters
    ----------
    y : array_like
        STFT to calculate the ISTFT of. Axis 0 is frequency and axis 1 is time.
    win : int
        Width of the Hann window.
    hop : int
        Number of samples to skip between windows. If `hop < win`, the windows
        will overlap.

    Returns
    -------
    x : array_like
        ISTFT of `y`.
    """
    length = win + hop * (y.shape[1] - 1)
    x = np.zeros((length,))
    for t in range(y.shape[1]):
        x[t*hop:t*hop+win] += np.fft.irfft(y[:,t])
    return x

Now Griffin and Lim's algorithm:

def reconstruct(y, iterations, win, hop):
    """
    Reconstruct a signal from its spectrogram.

    Parameters
    ----------
    y : array_like
        Spectrogram. Linear scale, not log scale. Axis 0 is frequency and
        axis 1 is time.
    iterations : int
        Number of iterations to run the algorithm. More iterations produces a
        better reconstruction.
    win : int
        Width of the Hann window.
    hop : int
        Number of samples to skip between windows. If `hop < win`, the windows
        will overlap.

    Returns
    -------
    x : array_like
        Reconstructed signal.

    References
    ----------
    (1) Griffin, D.; Lim, J. Signal Estimation from Modified Short-Time Fourier
    Transform. IEEE Transactions on Acoustics, Speech, and Signal Processing
    1984, 32 (2), 236–243. https://doi.org/10.1109/TASSP.1984.1164317
    """
    x = irstft(y, win, hop)
    for _ in range(iterations):
        yi = rstft(x, win, hop)
        x = irstft(y * yi/np.abs(yi), win, hop)
    return x

Demo

Let's try it out! First, read the image, normalize it, and flip it upsize-down so the low frequencies are at the start of the array:

im = Image.open("spec.png").convert("L")  # Convert to grayscale.
im = np.flipud(np.asarray(im, dtype=np.float64) / 255)

Since spectrograms are typically drawn with a logarithmic color map, we'll need to undo that:

min_db = -70  # Tunable paramter to adjust volume.
spec = 10**(min_db * im / 10)

In order to get the frequency of the reconstructed audio correct, we need to know the frequency range depicted in the spectrogram. In this example, the spectrogram ranges from 0-10 kHz. From this, we can calculate the correct amount of overlap between windows.

By the Nyquist theorem, we need a sample rate of $R \ge 2 f_\text{max}$ in order to represent a signal with max frequency $f_\text{max}$ . From this, we can compute the correct hop between adjacent windows: $\begin{aligned} T &= \frac{N}{R} = \frac{\text{win} + \text{hop} \cdot (W - 1)}{2 f_\text{max}} \\ \implies \text{hop} &= \frac{2 f_\text{max} T - \text{win}}{W - 1}, \end{aligned}$ where $T$ is the duration of the audio and $W$ is the width of the spectrogram in pixels.

When computing the DFT of a real-valued signal, half the coefficients are redundant (they're the complex conjugate of the other half), so they are omitted from the spectrogram. Hence the window length is twice the height of the spectrogram, excluding the first row which is the DC (nonperiodic) component.

max_freq = 10_000
duration = 1
win = 2 * (spec.shape[0] - 1)
hop = round((2 * max_freq * duration - win) / (spec.shape[1] - 1))

Now we can reconstruct the audio signal:

x = reconstruct(spec, 5, win, hop)

Finally, we normalize the signal so that it doesn't clip, convert to 16-bit PCM, and save to a WAV file:

x /= np.max(np.abs(x))
x_s16 = (x * (2**15-1)).astype(np.int16)
rate = round(x.shape[0] / duration)
wavfile.write('audio.wav', rate, x_s16)

Here's the result of running that on the spectrogram at the beginning of the post:

Using T201-V1.0 ESP8285 module with Arduino IDE

2020-08-05T00:00:00+00:00

In my previous post I went over the hardware and programming process for a T201-V1.0 ESP8285 module that was taken from a Gosund WP3 smart plug. This module can also be used with Arduino as a "Generic ESP8285 Module".

Wire up the module on a breadboard according to the diagram below. To program the module, I'm using a FTDI Basic Breakout board from SparkFun with the jumper on the rear switched to the 3.3V setting. If you're using a different USB-to-serial adapter, the pin order will be different on J2.

(Download KiCad schematic)

More information about how to connect the module can be found in the ESP8285 Arduino Core documentation. It's necessary to use an external 3.3V power supply, as the USB-to-serial adapter is unable to supply enough power to the ESP8285 module. I used a benchtop power supply.

Next, install the ESP8266 Arduino Core as described in the documentation. Once installed, change the following settings under the "Tools" menu:

Setting	Value
Board	Generic ESP8285 Module
Builtin Led	14
Reset Method	no dtr (aka ck)

The "no dtr (aka ck)" setting will pull the DTR pin low while programming, which is required to place the ESP8285 in programming mode. After it finishes, it will issue a soft reset and let DTR float high.

The rest can be left at their defaults. Open the "Blink" example sketch under File→Examples→ESP8266→Blink. Plug in the USB-to-serial adapter and upload the sketch with Sketch→Upload. When you see Connecting... in the console, press the reset switch (SW1 in the schematic). A successful upload will look something like this:

Executable segment sizes:
IROM   : 228656          - code in flash         (default or ICACHE_FLASH_ATTR) 
IRAM   : 26752   / 32768 - code in IRAM          (ICACHE_RAM_ATTR, ISRs...) 
DATA   : 1248  )         - initialized variables (global, static) in RAM/HEAP 
RODATA : 688   ) / 81920 - constants             (global, static) in RAM/HEAP 
BSS    : 24880 )         - zeroed variables      (global, static) in RAM/HEAP 
Sketch uses 257344 bytes (26%) of program storage space. Maximum is 958448 bytes.
Global variables use 26816 bytes (32%) of dynamic memory, leaving 55104 bytes for local variables. Maximum is 81920 bytes.
esptool.py v2.8
Serial port /dev/ttyUSB0
Connecting........_____...
Chip is ESP8285
Features: WiFi, Embedded Flash
Crystal is 26MHz
MAC: c4:4f:33:xx:xx:xx
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 1MB
Compressed 261504 bytes to 193165...
Wrote 261504 bytes (193165 compressed) at 0x00000000 in 17.1 seconds (effective 122.1 kbit/s)...
Hash of data verified.

Leaving...
Soft resetting...

Once it's done, the LED should start blinking.

ESP8285 module from a Gosund WP3 smart plug

2020-08-03T00:00:00+00:00

The Gosund WP3 smart plug is a Wifi-controllable relay:

Hardware

Inside, it contains an ESP8285 module labeled "T201-V1.0". The ESP8285 is a variant of ESP8266 with 1 MiB on-chip flash.

The following pins are exposed:

Front rectangular pads		Rear rectangular pads		Rear circular pads
Silkscreen	Function	Silkscreen	Function	Silkscreen	Function
RST	EXT_RSTB	3V3	VDD	RST	EXT_RSTB
AD	ADC	GND	GND	IO0	GPIO0
13	GPIO13	RX	UART Rx	IO2	GPIO2
04	GPIO4	TX	UART Tx	IO4	GPIO4
05	GPIO5	12	GPIO12	IO5	GPIO5
		14	GPIO14	IO13	GPIO13

Refer to the ESP8285 datasheet (rehosted copy) for pin descriptions.

I soldered headers to the front and rear rectangular pads as well as the two circular pads that didn't have redundant pads, IO0 and IO2.

Programming

To program, you'll need a 3.3V USB-to-serial board. I used a FTDI Basic Breakout board from SparkFun, with the jumper on the rear switched to the 3.3V setting.

Make the following connections between your ESP8285 module and the USB-to-serial board:

Serial	ESP8285 module
GND	GND and IO0*
3V3**	3V3
TXO	RX
RXI	TX
DTR	RST

* Needed to put the module ESP8285 in programming mode.
** Labeled 5V on my board because that is its default voltage.

Plug the USB-to-serial board into your computer. Run esptool to confirm that you can make a connection:

$ esptool.py -p /dev/ttyUSB0 read_mac
esptool.py v2.8
Serial port /dev/ttyUSB0
Connecting....
Detecting chip type... ESP8266
Chip is ESP8285
Features: WiFi, Embedded Flash
Crystal is 26MHz
MAC: c4:4f:33:xx:xx:xx
Uploading stub...
Running stub...
Stub running...
MAC: c4:4f:33:xx:xx:xx
Hard resetting via RTS pin...

To back up the firmware:

$ esptool.py -p /dev/ttyUSB0 read_flash 0 0x100000 fwbackup.bin
esptool.py v2.8
Serial port /dev/ttyUSB0
Connecting.....
Detecting chip type... ESP8266
Chip is ESP8285
Features: WiFi, Embedded Flash
Crystal is 26MHz
MAC: c4:4f:33:xx:xx:xx
Uploading stub...
Running stub...
Stub running...
1048576 (100 %)
1048576 (100 %)
Read 1048576 bytes at 0x0 in 99.0 seconds (84.7 kbit/s)...
Hard resetting via RTS pin...

Unfortunately, I did not back up my firmware before flashing it, so I don't have a copy of the Gosund WP3 firmware.

To flash a new program to the chip, such as Tasmota, do the following:

$ wget https://github.com/arendst/Tasmota/releases/download/v8.4.0/tasmota.bin
$ esptool.py -p /dev/ttyUSB0 write_flash -fs 1MB -fm dout 0 tasmota.bin
esptool.py v2.8
Serial port /dev/ttyUSB0
Connecting.....
Detecting chip type... ESP8266
Chip is ESP8285
Features: WiFi, Embedded Flash
Crystal is 26MHz
MAC: c4:4f:33:xx:xx:xx
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Compressed 606368 bytes to 427780...
Wrote 606368 bytes (427780 compressed) at 0x00000000 in 38.0 seconds (effective 127.7 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

The -fs 1MB is there because the ESP8285 has 1 MiB of flash. The -fm argument is described on the esptool documentation.

To boot into the program you flashed:

Disconnect the USB cable
Unplug IO0 to take the chip out of programming mode
Connect RST to 3.3V
Reconnect the USB cable

If you flashed Tasmota, you should be able to connect with picocom or any other serial terminal. Issue the status command to see the current status:

$ picocom --baud 115200 --echo --omap crcrlf /dev/ttyUSB0 
picocom v3.1

port is        : /dev/ttyUSB0
flowcontrol    : none
baudrate is    : 115200
parity is      : none
databits are   : 8
stopbits are   : 1
escape is      : C-a
local echo is  : yes
noinit is      : no
noreset is     : no
hangup is      : no
nolock is      : no
send_cmd is    : sz -vv
receive_cmd is : rz -vv -E
imap is        : 
omap is        : crcrlf,
emap is        : crcrlf,delbs,
logfile is     : none
initstring     : none
exit_after is  : not set
exit is        : no

Type [C-a] [C-h] to see available commands
Terminal ready
status
08:40:52 CMD: status
08:40:52 RSL: stat/tasmota_XXXXXX/STATUS = {"Status":{"Module":1,"DeviceName":"Tasmota","FriendlyName":["Tasmota"],"Topic":"tasmota_XXXXXX","ButtonTopic":"0","Power":0,"PowerOnState":3,"LedState":1,"LedMask":"FFFF","SaveData":1,"SaveState":1,"SwitchTopic":"0","SwitchMode":[0,0,0,0,0,0,0,0],"ButtonRetain":0,"SwitchRetain":0,"SensorRetain":0,"PowerRetain":0}}

You can also access Tasmota's web UI via the tasmota_XXXXXX-XXXX Wifi network. More instructions specific to Tasmota can be found in their documentation.

References

DIY tachometer

2020-07-11T00:00:00+00:00

If you have a photoresistor and an oscilloscope, making a simple tachometer is straightforward.

Build this circuit:
Make a high-contrast mark on the thing you're measuring the speed of:
Power it on and hold the photoresistor a couple millimeters away:
Check the oscilloscope:

The smooth signal is from the photoresistor and the square wave is from the fan's internal tachometer. As you can see, they have the same frequency. In my case, the mark passes the photoresistor twice per revolution, so the speed is:

$44.0\,\mathrm{Hz} {\huge /} \left(2\,\frac{\mathrm{passes}}{\mathrm{revolution}}\right) \cdot 60\,\frac{\mathrm{s}}{\mathrm{min}} = 1320\,\text{RPM}.$

12V 4-pin PC fan controller (AVR)

2020-07-07T00:00:00+00:00

Over the weekend I built a PC fan controller using an Arduino Diecimila (containing a ATmega168 microcontroller). As a learning exercise, I decided to write the firmware with avr-libc and forgo use of the Arduino environment and libraries.

The device has a potentiometer to control the fan speed and outputs the current RPM over serial. The hardware is straightforward: the fan is connected as described in Intel and Noctua's documentation and the supporting circuitry for the ATmega168 is contained in the Arduino itself. After building the circuit on a breadboard, I captured the schematic in KiCad.

The firmware uses several peripherals:

ADC - read potentiometer value
PWM (TIMER1) - send control signal to fan at 25 kHz
External interrupt INT0 - count pulses from fan's tachometer
Real time clock (TIMER0) - keep track of time to calculate RPM
UART - send current RPM over serial console

I suspect the firmware would have been significantly shorter if written using the Arduino libraries. I may revisit this in the future and make a comparison.

The project files are available on GitLab: https://gitlab.com/equivariant/pc_fan_control

References

External monitor brightness control with Redshift and ddcutil

2020-07-07T00:00:00+00:00

Redshift is a program for Linux that adjusts the color temperature of the screen based on the time of day. I find my screen more comfortable to look at when the color temperature is reduced, especially at night. Redshift also has an option to adjust the brightness, but it does the adjustment in software, so the backlight on the display remains the same. This results in a washed out image at lower brightness levels.

Most monitors support an interface called DDC/CI to adjust the parameters found in the monitor's OSD. ddcutil is a program to send these commands to the monitor. In particular, it can adjust the brightness of the monitor's backlight.

To set it up, first install Redshift and ddcutil:

sudo pacman -S redshift ddcutil

Configure Redshift:

# ~/.config/redshift.conf

[redshift]
temp-day=5500
temp-night=3500
location-provider=manual
fade=0

[manual]
lat=66.058
lon=-74.135

ddcutil requires access to I²C devices. Enable the i2c-dev module:

echo i2c-dev | sudo tee /etc/modules-load.d/i2c-dev.conf
sudo modprobe i2c-dev

Confirm that ddcutil can see your monitor:

sudo ddcutil detect

Set NOPASSWD for ddcutil in /etc/sudoers so that Redshift doesn't need to run as root:

sudo visudo

Add near the end:

youruser ALL = (root) NOPASSWD: /usr/bin/ddcutil setvcp 10 *, /usr/bin/ddcutil setvcp 12 *

Features 10 and 12 and brightness and contrast, respectively. You can get a list of all the features supported by your monitor with sudo ddcutil capabilities. Confirm that you can now adjust the brightness and contrast without entering a password:

ddcutil setvcp 10 50
ddcutil setvcp 12 25

Now add a hook to Redshift to adjust the brightness and contrast:

# ~/.config/redshift/hooks/ddcutil

#!/bin/bash
case $1 in
period-changed)
    case $3 in
    night)
        # 10=brightness, 12=contrast
        sudo ddcutil setvcp 10 50
        sudo ddcutil setvcp 12 35
        ;;
    transition)
        sudo ddcutil setvcp 10 63
        sudo ddcutil setvcp 12 43
        ;;
    daytime)
        sudo ddcutil setvcp 10 76
        sudo ddcutil setvcp 12 61
        ;;
    esac
    ;;
esac

Mark it executable:

chmod +x ~/.config/redshift/hooks/ddcutil

Run Redshift and confirm that it adjusts the brightness/contrast on startup:

redshift

Weather.gov API

2020-06-26T00:00:00+00:00

The United States National Weather Service (NWS) offers an easy to use weather API. To fetch data for a given latitude and longitude, use the following query:

curl -H 'accept: application/geo+json' 'https://api.weather.gov/points/33.3815,-118.4332'

{
  ...
  "properties": {
    ...
    "gridId": "LOX",
    "gridX": 142,
    "gridY": 16,
    "forecast": "https://api.weather.gov/gridpoints/LOX/142,16/forecast",
    "forecastHourly": "https://api.weather.gov/gridpoints/LOX/142,16/forecast/hourly",
    "forecastGridData": "https://api.weather.gov/gridpoints/LOX/142,16",
    "observationStations": "https://api.weather.gov/gridpoints/LOX/142,16/stations",
    ...
  }
}

The response is in GeoJSON format, but there's no need to know the details of that format to use the data.

The NWS operates a number of Weather Forecast Offices (WFO), each responsible for forecasts in a portion of the US. The WFO responsible for that latitude/longitude is .properties.gridId. Each WFO splits the region it's responsible for into a grid. The grid cell corresponding to the requested latitude/longitude is [.properties.gridX, .properties.gridY].

To get the forecast in a machine-parsable, numeric format, use the URL under .properties.forecastGridData. It is also available in human-readable format in 12-hour intervals under .properties.forecast and 1-hour intervals under .properties.forecastHourly. For now, we'll focus on the machine-readable forecast data:

curl -H 'accept: application/geo+json' 'https://api.weather.gov/gridpoints/LOX/142,16'

{
  ...
  "properties": {
    ...
    "temperature": {
      "uom": "wmoUnit:degC",
      "values": [
        {
          "validTime": "2020-06-26T04:00:00+00:00/PT2H",
          "value": 14.444444444444445
        },
        {
          "validTime": "2020-06-26T06:00:00+00:00/PT2H",
          "value": 15
        },
        ...
      ]
    },
    "windSpeed": {...},
    "probabilityOfPrecipitation": {...},
    ...
  }
}

The first thing you'll notice if you try this out is that there's a ton of data returned. Let's focus on one property, .properties.temperature. The unit of the measurement is given under .properties.temperature.uom (°C, in this case). The forecast for each time interval is under .properties.temperature.values. The interval is given in ISO 8601 interval format.

In addition to forecasts, you can request raw observations. Recall that our first query gave a list of observation stations under .properties.observationStations. Let's take a look:

curl -H 'accept: application/geo+json' 'https://api.weather.gov/gridpoints/LOX/142,16/stations'

{
  ...
  "features": [
    {
      "id": "https://api.weather.gov/stations/KAVX",
      "geometry": {
        "type": "Point",
        "coordinates": [
          -118.41456,
          33.40421
        ]
      },
      ...
    },
    {
      "id": "https://api.weather.gov/stations/AIDC1",
      "geometry": {
        "type": "Point",
        "coordinates": [
          -118.3305,
          33.338
        ]
      },
      ...
    },
    ...
  ],
  ...
}

This list includes all the stations within the 2.5 km grid area. We can fetch the most recent observations made by a particular weather station like so:

curl -H 'accept: application/geo+json' 'https://api.weather.gov/stations/KAVX/observations?limit=1'

{
  ...
  "features": [
    {
      ...
      "properties": {
        ...
        "timestamp": "2020-06-26T10:08:00+00:00",
        "temperature": {
          "value": 13.9,
          "unitCode": "unit:degC",
          "qualityControl": "qc:V"
        },
        "dewpoint": {
          "value": 12.8,
          "unitCode": "unit:degC",
          "qualityControl": "qc:V"
        },
        "windDirection": {
          "value": 110,
          "unitCode": "unit:degree_(angle)",
          "qualityControl": "qc:V"
        },
        ...
      }
    }
  ]
}

There's plenty of other data available, so be sure to take a look at the API documentation (click the "Specification" tab). Note that this API barely scratches the surface of what data the National Weather Service has to offer. If you take a look around their website, you'll find many different APIs run by various departments in the NWS, such as the GIS web services and the National Digital Forecast Database.

Syncing files between Android and a Linux desktop

2020-06-18T00:00:00+00:00

When I want to copy photos or other files from my Android (LineageOS) phone to my desktop computer, I typically plug in a USB cable and transfer with MTP. Today I was reminded of Syncthing by this Lobsters post, which allows you to sync folders between systems, including Android. Syncthing is available in the Arch Linux and F-Droid package repositories, so installation was simple. On my desktop, I enabled the systemd user service, as suggested on the Arch Wiki. On both ends, I disabled NAT traversal, global discovery, and relaying (leaving only local discovery), as I only intend to sync on my LAN. Device discovery and setting up the synced folders were straightforward.

It's a nice piece of software. Everything just works without any fuss.

Python packaging, dependency management, and development tools

2020-05-30T00:00:00+00:00

For most of the time I've been using Python, I've managed dependencies either with my system package manager or with a simple pip requirements.txt file and venv (or virtualenv in Python 2). This has served me well: it's simple and covers most use cases.

In the past few years, there has been a surge of interest in improving Python dependency management. Many dependency management tools are now available, some old, some new, including Pipenv, pyenv, Poetry, and conda. The PyPA working group recommends Pipenv (as of 2020-05-30), but I found it to be prohibitively slow last time I tried it. I decided to give Poetry a try on a small project after reading positive reviews of it online. So far I'm happy with it. It's fast, easy enough to use, and hasn't gotten in the way too much. I can see myself using it on future projects.

I also gave tox a try, having previously only used unittest or pytest directly. tox is not an alternative to unittest or pytest, but rather a tool for running your tests in a number of different environments. In my case, I only have one environment I wish to test in (Python 3.8), but I can see it being useful when authoring programs that need to run under many different Python versions.

Black is a tool to standardize Python code formatting. Similar tools for other languages include gofmt and rustfmt. I would recommend automatic code formatters to everyone. Not having to think about code style reduces unnecessary mental load when programming, allowing you to focus on the functionality of your software rather than its appearance.

Here is how I set up the environment for my latest project:

poetry new myproject
cd myproject
git init
curl -O .gitignore https://raw.githubusercontent.com/github/gitignore/master/Python.gitignore
poetry add -D black tox
poetry add dependency1 dependency2 ....

If you need to run a command inside the project's environment, you can use poetry run mycommand or poetry shell. This is similar to sourcing the activate script in a venv.

Here is how I configured tox (see here):

# tox.ini
[tox]
isolated_build = true
envlist = py38

[testenv]
whitelist_externals = poetry
commands =
    poetry install
    poetry run black --check .
    poetry run pytest tests/

Before committing any changes, I run Black to reformat my code and tox to run tests:

poetry run black .
poetry run tox

I'm satisfied with this new workflow. It's simple enough to set up and use that I can see myself using it instead of pip+venv in the future. Poetry has the added benefit of handling all the packaging work you would otherwise have to do manually, though I haven't needed that for this project.

Backblaze hard drive stats and time series databases

2020-05-12T00:00:00+00:00

Backblaze publishes quarterly hard drive statistics. The data files are in CSV format and contain one row per day per drive. The model and capacity are given, as well as a large array of SMART attributes. The SMART data is relatively sparse — most drive models only report a few SMART attributes. A couple potential uses for this dataset:

Calculate the reliability of different drives. The failure rate can then be taken into account when calculating the lifetime cost of a storage array.
Use SMART data to predict future drive failures.

My goal today, however, was to try out some time series databases. I began importing the data into InfluxDB using a simple script built on influxdb-python, but stopped after I realized it was going to take about 10 hours to run (5k records/sec ingestion rate). Running the import script without actually connecting to InfluxDB was projected at about 2 hours, so it's entirely plausible that the majority of the overhead was on the Python side of things. It would be interesting to rewrite the import script in a lower level language like Rust or Go and see how it compares. I decided not to play around with InfluxDB more today, because I already use it fairly often at work.

Next, I tried out TimescaleDB, a Postgres extension for time series data. I've had good experiences with Postgres in the past, so it seemed promising. To import the Backblaze CSV data, I used the timescaledb-parallel-copy program, which chunks the CSV data and runs several COPY commands in parallel. It took about 45 minutes to import the full dataset (70k records/sec ingestion rate).

TimescaleDB has a neat compression feature that takes advantage of the fact that time series tend to be fairly predictable. They claim to achieve 91-96% compression on real-world datasets. When I tried this out on the Backblaze dataset, the Postgres data directory ended up 53% larger. A VACUUM FULL did not resolve it. I'm likely doing something wrong. More experimentation needed!

File sizes with ZFS compression

2020-05-12T00:00:00+00:00

For OpenZFS:

ls -lh shows the logical size of the file, the number of bytes you would see if you were to read the file
du -h --apparent-size shows the logical size of the file
du -h shows the space used on disk by the file

Typically du -h is less than du -h --apparent-size if you have compression on.

Demo:

$ # Create file compressible down to ~33%.
$ head -c $((33*1024**2)) /dev/urandom > data
$ yes meow | head -c $((67*1024**2)) >> data

$ ls -lh data
-rw-r--r-- 1 nobody nobody 100M Jan 24 14:38 data
$ du -h --apparent-size data
100M    data
$ du -h data
34M data

$ zfs get used,referenced,logicalused,logicalreferenced,compressratio zroot/foo
NAME       PROPERTY           VALUE     SOURCE
zroot/foo  used               33.6M     -
zroot/foo  referenced         33.6M     -
zroot/foo  logicalused        100M      -
zroot/foo  logicalreferenced  100M      -
zroot/foo  compressratio      2.98x     -

Zpool size doesn't match ZFS root dataset size

2020-05-11T00:00:00+00:00

I noticed that the total size of a zpool doesn't match the used+avail for the root dataset:

$ zpool status zroot
...
NAME                                    STATE     READ WRITE CKSUM
zroot                                   ONLINE       0     0     0
  mirror-0                              ONLINE       0     0     0
    /dev/disk/by-id/dm-name-cryptroot1  ONLINE       0     0     0
    /dev/disk/by-id/dm-name-cryptroot2  ONLINE       0     0     0
...

$ zpool list zroot
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zroot   460G   192G   268G        -         -    24%    41%  1.00x    ONLINE  -

$ zfs list zroot
NAME    USED  AVAIL     REFER  MOUNTPOINT
zroot   192G   254G       24K  none

Notice that the zpool reports a size of 460 GiB. This size is consistent with the alloc and free properties for the zpool: $\text{alloc} + \text{free} = 192\text{ GiB} + 268\text{ GiB} = 460\text{ GiB} = \text{size}.$

However, the sum of the used and avail values for the root dataset is lower: $\text{used} + \text{avail} = 192\text{ GiB} + 254\text{ GiB} = 446\text{ GiB} < 460\text{ GiB}.$

It's about 3% lower: $\frac{460\text{ GiB} - 446\text{ GiB}}{460\text{ GiB}} = 3.04\%.$

This value comes from the spa_slop_shift ZFS module parameter (see zfs(4)). By default, it is set to 5, meaning 1/2^5=3.125% of the space is set aside.

Raidz

For raidz, the zpool size is reported differently. Rather than showing the size of each disk (which is equal to the size of the zpool for mirrored disks), zpool list shows the sum of the disks. For example:

$ zpool status slow
...
NAME                                              STATE     READ WRITE CKSUM
slow                                              ONLINE       0     0     0
  raidz1-0                                        ONLINE       0     0     0
    /dev/disk/by-id/wwn-0x5000000000000001-part1  ONLINE       0     0     0
    /dev/disk/by-id/wwn-0x5000000000000002-part1  ONLINE       0     0     0
    /dev/disk/by-id/wwn-0x5000000000000003-part1  ONLINE       0     0     0
...

$ zpool list  slow
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
slow   10.9T  4.49T  6.42T        -         -     1%    41%  1.00x    ONLINE  -

Each of the three disks is 4 TB (not TiB), so the zpool size is reported as $3 \cdot 4\text{ TB} \cdot 1000^4\,\frac{\text{byte}}{\text{TB}} {\Large /} 1024^4\,\frac{\text{byte}}{\text{TiB}} = 10.9\text{ TiB}.$

For raidz, you must take into account the space reserved for parity data. For a raidzP (P=1,2,3) group, about (N-P)*X of the space is available, where X is the size of each disk: $(N-P) \cdot X = (3 - 1) \cdot 4\text{ TB} \cdot 1000^4\,\frac{\text{byte}}{\text{TB}} {\Large /} 1024^4\,\frac{\text{byte}}{\text{TiB}} = 7.27\text{ TiB}.$

So with 3.125% reserved for spa_slop_shift, we should expect the root dataset to report 7.27 TiB * (1-0.03125) = 7.04 TiB for used+avail. Indeed, 2.99 TiB + 4.05 TiB = 7.04 TiB:

$ zfs list slow
NAME    USED  AVAIL     REFER  MOUNTPOINT
slow   2.99T  4.05T      330K  /slow

References

https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSReservedSpaceVaries

Distributed systems: causal broadcast

2020-05-10T00:00:00+00:00

These are my notes for Lindsey Kupuer's spring 2020 distributed systems course.

Lecture 3/4 - partial orders and Lamport clocks

Lecture 5 - vector clocks

Lecture 6 - delivery guarantees

Lecture 7 - causal broadcast

Causal broadcast is an algorithm for implementing causal delivery. Recall that if $m_1$ 's send happens before $m_2$ 's send, then under causal delivery $m_1$ must be delivered before $m_2$ (with respect to the happens before relation). Here is the causal broadcast algorithm:

Initialize a local clock to $[0,0,...,0]$ for each process.
If a message from $P_1$ is delivered at $P_2$ (recall that this is different from it being received), then increment $P_2$ 's local clock in the $P_1$ position.
If a message is to be sent from $P_1$ , increment its own position in its local clock, then include the local clock when sending the message.
A message from $P_1$ is only delivered at $P_2$ if: a. The $P_1$ position in the message's timestamp must be one greater than the $P_1$ position in $P_2$ 's local clock. That is is, this must be the next expected messages from $P_1$ . b. All other positions of the message's timestamp must be less than or equal to the corresponding positions in $P_2$ 's local clock. That is, there must be no missing messages from other processes.

Causal ordering can also be used to find a consistent global snapshot of a system. One algorithm to do this is the Chandy-Lamport algorithm. This will be discussed in the next lecture.

References

UCSC CSE138 spring 2020 lecture 7: causal broadcast [video]

Distributed systems: delivery guarantees

2020-05-09T00:00:00+00:00

These are my notes for Lindsey Kupuer's spring 2020 distributed systems course.

Lecture 3/4 - partial orders and Lamport clocks

Lecture 5 - vector clocks

Lecture 6 - delivery guarantees

Lecture 7 - causal broadcast

A channel is the connection between two processes. A message can be queued when it is received on a channel and later delivered to a process.

An example is FIFO delivery, where a message is queued until all messages that were sent before it (with respect to the sender's local clock) have been received. A FIFO anomaly is when message $A$ is sent before $B$ , but $B$ is received first. This is also a causal anomaly. FIFO delivery can be implemented by tagging messages with a sender ID and a sequence number that is locally incremented on each send. If we receive a message with sender ID and sequence number $(p,s)$ , we queue it until $(p,s-1)$ is delivered. This algorithm assumes reliable delivery: it fails if messages can be lost. FIFO delivery can also be implemented using acknowledgements. If $A$ sends a message to $B$ , it waits for an acknowledgement from $B$ before sending its next message. In practice, many protocols use TCP to implement FIFO delivery.

Another example of a delivery guarantee is causal delivery: if $m_1$ 's send happens before (see lectures 3/4 for definition) $m_2$ 's send, then $m_1$ 's delivery happens before $m_2$ 's delivery. It can be implemented by attaching vector clocks to each message sent; this will be discussed in the next lecture.

In totally-ordered delivery, if a process delivers $m_1$ before $m_2$ , then all processes deliver $m_1$ before $m_2$ . This ensures messages appear in the same order on all processes. A FIFO anomaly isn't a total order violation. This gives us a hierarchy of delivery guarantees:

$\text{causal} \implies \text{FIFO} \implies \emptyset$

$\text{totally-ordered} \implies \emptyset$

References

UCSC CSE138 spring 2020 lecture 6: delivery guarantees [video]

Disable Anki 2.1 update check

2020-05-08T00:00:00+00:00

Anki 2.1.23 displays a notification when a new version is released. There's an option to skip a particular version by clicking "Ignore this update", but no option to disable all future update checks.

Here's a small Python script to do that. Save it as disable_update_check.py:

from pprint import pprint
import aqt
pm = aqt.profiles.ProfileManager()
pm.setupMeta()
pm.load(pm.profiles()[0])
print('Current settings:')
pprint(pm.meta)
print('Will set "updates" to False.')
if input('Continue? [y/n] ') == 'y':
    pm.meta['updates'] = False
    pm.save()
    print('Done.')
else:
    print('No changes made.')

Back up your Anki settings (~/.local/share/{Anki,Anki2} on Linux), ensure Anki is not running, then execute the script:

$ python disable_update_check.py
Current settings:
{'created': 999999999,
    'defaultLang': 'en',
    'disabledAddons': [],
    'firstRun': False,
    'id': 99999999999999999999,
    'lastMsg': -1,
    'last_addon_update_check': 999999999,
    'suppressUpdate': False,
    'updates': True,
    'ver': 0}
Will set "updates" to False.
Continue? [y/n] y
Done.

Anki will no longer display a notification when a new version is released.

DNA Chisel: library for DNA constraint optimization

2020-04-22T00:00:00+00:00

DNA Chisel is a library for modifying DNA sequences to conform to a set of constraints, optionally maximizing an objective function. For example, it can be used to optimize the codon usage of a gene for a particular organism subject to the constraint that the gene must not contain a particular restriction enzyme cut site.

KLEE: symbolic execution for software testing

2020-04-09T00:00:00+00:00

KLEE is a symbolic execution engine for LLVM: it executes programs symbolically, starting with the set of all possible inputs. At each branch point in the program, constraints are added according to the branch condition and the program is forked. For example, if (x < 10) forks into a branch with an x < 10 constraint and a branch with a x >= 10 constraint. When KLEE reaches an assertion, pointer dereference, or other unsafe operation, it uses a SMT solver to check if there are inputs that would cause the operation to fail given the current constraints.

The authors used a number of heuristics to reduce the number of SMT queries by a factor of 10. To reduce memory usage, KLEE uses copy-on-write data structures to store program state.

To allow KLEE to work on real world programs, system calls are mocked. KLEE was run against GNU Coreutils, BusyBox, the MINIX core utilities, and an operating system kernel, and found bugs in all of them. It achieved greater line coverage than all of the manually constructed test suites. KLEE can also be used to check for equivalence between two programs by calling both and asserting their output is the same. This was run on GNU Coreutils and BusyBox and to uncover correctness errors.

References

C. Cadar, D. Dunbar, and D. Engler, “KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs,” USENIX Symposium on Operating Systems Design and Implementation, vol. 8, pp. 209–224, 2008, Accessed: Apr. 10, 2020. [Online]. Available: https://llvm.org/pubs/2008-12-OSDI-KLEE.pdf
https://klee.github.io/

Distributed systems: vector clocks

2020-04-08T00:00:00+00:00

These are my notes for Lindsey Kupuer's spring 2020 distributed systems course.

Lecture 3/4 - partial orders and Lamport clocks

Lecture 5 - vector clocks

Lecture 6 - delivery guarantees

Lecture 7 - causal broadcast

Vector clocks are similar to Lamport clocks, but make a stronger guarantee: for events $A$ and $B$ , $A \to B$ if and only if $\operatorname{VC}(A) < \operatorname{VC}(B)$ . Vector clocks are assigned as follows:

For each process, initialize a (local) length- $n$ vector to all zero, where $n$ is the number of processes.
If process $p$ has an event, increment position $p$ of process $p$ 's counter by 1.
If process $p$ sends a message, increment position $p$ of its counter by 1 and send the counter with the message.
If process $q$ receives a message, increment position $q$ of process $q$ 's counter by 1 and take the elementwise maximum with the received counter.

If $\operatorname{VC}(A)$ and $\operatorname{VC}(B)$ are vector clocks, then we define $\operatorname{VC}(A) < \operatorname{VC}(B)$ if and only if $\operatorname{VC}(A)_i \le \operatorname{VC}(B)_i$ for all $i$ and $\operatorname{VC}(A) \ne \operatorname{VC}(B)$ .

References

UCSC CSE138 spring 2020 lecture 5: vector clocks [video]

Neural network model extraction

2020-04-08T00:00:00+00:00

Natural language models can be "stolen" from a black box API. The attacker provides random sentences, then uses the output of the API to train its own model. The authors found the following:

If both models were pretrained with the same dataset, the model can be extracted much more effectively.
The method did not require confidence scores from the black box API, only hard labels.
If the random sentences are more similar to the dataset that the black box API was trained on, extraction is more effective.
Current known defenses are:
- Detecting and blocking queries likely to be from an attack;
- Watermarking outputs of the black box API.

Such an attack could be used in the following scenarios:

Extract the model, then use the extracted model to construct adversarial examples against the original model.
Extract the model, then extract training data from that extracted model. This could be used to extract private data that the black box API was trained on.
Extract the model, then use it directly to avoid calling out to the black box API.

References

https://www.cleverhans.io/2020/04/06/stealing-bert.html
K. Krishna, G. S. Tomar, A. P. Parikh, N. Papernot, and M. Iyyer, “Thieves on Sesame Street! Model Extraction of BERT-based APIs,” arXiv:1910.12366 [cs], Jan. 2020, Accessed: Apr. 08, 2020. [Online]. Available: https://arxiv.org/abs/1910.12366
G. Hinton, O. Vinyals, and J. Dean, “Distilling the Knowledge in a Neural Network,” arXiv:1503.02531 [cs, stat], Mar. 2015, Accessed: Apr. 08, 2020. [Online]. Available: https://arxiv.org/abs/1503.02531
F. Tramèr, F. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart, “Stealing Machine Learning Models via Prediction APIs,” arXiv:1609.02943 [cs, stat], Oct. 2016, Accessed: Apr. 08, 2020. [Online]. Available: https://arxiv.org/abs/1609.02943

Distributed systems: partial orders and Lamport clocks

2020-04-07T00:00:00+00:00

These are my notes for Lindsey Kupuer's spring 2020 distributed systems course.

Lecture 3/4 - partial orders and Lamport clocks

Lecture 5 - vector clocks

Lecture 6 - delivery guarantees

Lecture 7 - causal broadcast

A synchronous network is one where message delivery time is bounded. An asynchronous network is one where it is not.
A "happens before" relation describes the order of events in a distributed system:
- If $A$ appears before $B$ in the same process, then $A \to B$ ;
- If $A$ is a send and $B$ is the corresponding receive, then $A \to B$ ;
- If $A \to B$ and $B \to C$ , then $A \to C$ .
A partial order is a set $S$ $S$ with a binary relation $\le$ $\leq$ with the following properties:
- Reflexive: $a \le a$ ;
- Antisymmetric: if $a \le b$ and $b \le a$ , then $a = b$ ;
- Transitive: if $a \le b$ and $b \le c$ , then $a \le c$ .
The " $\to$ " relation is a partial order without reflexivity.
Lamport clocks:
- To assign Lamport clocks to events:
  - Initialize counter to 0 for all processes.
  - Increment the counter on each event.
  - Send the counter with every message.
  - When receiving a message, set counter to $1+\max(\text{own counter},\text{received counter})$ .
- If $A \to B$ , then $\operatorname{LC}(A) \le \operatorname{LC}(B)$ .

References

Collecting local system stats with Prometheus on Arch Linux

2019-08-25T00:00:00+00:00

Table of Contents

Install Prometheus
Install Grafana
- Add Prometheus data source
Install node exporter
- Add to Grafana
Install blackbox exporter
- Configure blackbox exporter
- Add to Grafana

Install Prometheus

sudo pacman -S prometheus

Increase metrics retention time and listen only on loopback interface (the first ExecStart entry clears the existing ExecStart value):

sudo systemctl edit prometheus

[Service]
ExecStart=
ExecStart=/usr/bin/prometheus \
    --config.file=/etc/prometheus/prometheus.yml \
    --storage.tsdb.path=/var/lib/prometheus/data \
    --storage.tsdb.retention.time=1y \
    --web.listen-address=127.0.0.1:9090

Start Prometheus and confirm it's listening on the loopback interface only:

sudo systemctl start prometheus
sudo ss -lptun
sudo systemctl enable prometheus

Confirm the prometheus target is healthy: http://localhost:9090/targets

Install Grafana

sudo pacman -S grafana

Configure Grafana:

sudo vim /etc/grafana.ini

[server]
http_addr = 127.0.0.1
[auth.anonymous]
enabled = true
org_role = Editor

Start Grafana and confirm it's listening only on the loopback interface:

sudo systemctl start grafana
sudo ss -lptun
sudo systemctl enable grafana

Add Prometheus data source

Go to http://localhost:3000/login and log in with username admin and password admin. Change the password to keep it from nagging you every time you log in.

Go to Configuration -> Data Sources and click "Add data source". Choose "Prometheus" and use the default settings.

Go to Create -> Import and import Grafana.com dashboard ID "2" (Prometheus Stats). Select the Prometheus data source.

Confirm that stats appear under the new dashboard.

Install node exporter

The node exporter collects system metrics.

sudo pacman -S prometheus-node-exporter

Configure it to only listen on the loopback interface:

sudo vim /etc/conf.d/prometheus-node-exporter

NODE_EXPORTER_ARGS="--web.listen-address=127.0.0.1:9100"

Start the service and confirm it's only listening on the loopback interface:

sudo systemctl start prometheus-node-exporter
sudo ss -lptun
sudo systemctl enable prometheus-node-exporter

Go to http://localhost:9100/metrics and confirm that metrics appear.

Configure Prometheus to pull from the node exporter:

sudo vim /etc/prometheus/prometheus.yml

scrape_configs:
  - job_name: 'node'
    static_configs:
    - targets: ['localhost:9100']

Reload Prometheus to make it aware of the new target:

sudo systemctl reload prometheus

Go to http://localhost:9090/targets and confirm that the localhost target appears.

Add to Grafana

Go to Grafana and import a new dashboard (http://localhost:3000/dashboard/import). Use Grafana.com dashboard ID "1860" (Node Exporter Full). Select the "Prometheus" data source. Go to the new dashboard and confirm that the last 5 minutes of metrics look sane.

Install blackbox exporter

The blackbox exporter probes arbitrary network endpoints.

sudo pacman -S prometheus-blackbox-exporter

Edit the systemd unit to set a config file and only listen on the loopback interface:

sudo systemctl edit prometheus-blackbox-exporter

[Service]
ExecStart=
ExecStart=/usr/bin/prometheus-blackbox-exporter \
    --config.file="/etc/prometheus/blackbox.yml" \
    --web.listen-address=127.0.0.1:9115

Configure a generic IPv4 ICMP (ping) probe:

sudo vim /etc/prometheus/blackbox.yml

modules:
  icmp:
    prober: icmp
    timeout: 5s
    icmp:
      preferred_ip_protocol: "ip4"

sudo chown root:prometheus /etc/prometheus/blackbox.yml

Start the service and confirm it's only listening on the loopback interface:

sudo systemctl start prometheus-blackbox-exporter
sudo ss -lptun
sudo systemctl enable prometheus-blackbox-exporter

Confirm that it's working: * Go to http://localhost:9115/metrics and confirm that metrics about the blackbox exporter service are present. * Go to http://localhost:9115/probe?target=prometheus.io&module=icmp and confirm that the ICMP probe works. Afterward, it should appear in the "Recent Probes" section on http://localhost:9115/.

Configure blackbox exporter

Configure Prometheus to pull from the blackbox exporter. Replace the values in the targets list with appropriate values for your network setup. The gateway and next hop can be determined with mtr -b 8.8.8.8 (8.8.8.8 is Google's public DNS server, which is generally reliable).

sudo vim /etc/prometheus/prometheus.yml

scrape_configs:
  - job_name: 'blackbox_meta'
    static_configs:
      - targets: ['localhost:9115']
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets: ['8.8.8.8', '<<NEXT_HOP_IP>>', '<<YOUR_GATEWAY_IP>>', '127.0.0.1']
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 'localhost:9115'

The blackbox_meta job collects metrics about the exporter itself. The blackbox job collects results from the ICMP probes.

The /probe endpoint requires two URL parameters:

The module parameter is set to a fixed value of icmp in the params section. This matches the module name configured in /etc/prometheus/blackbox.yml.
The target parameter is set in the relabel_configs section. Read the relabel_configs docs for full details on how it works:
- When Prometheus collects metrics, it iterates over the targets listed in the static_configs section. __address__ is the address of the target currently being polled. The first relabel sets the target URL parameter to be this value (the label __param_NAME is passed as the URL parameter NAME).
- The second relabel copies this value to the instance label so that we don't end up with all the metrics being attributed to localhost:9115.
- The third relabel sets the __address__ to localhost:9115, since that's where the /probe endpoint is located. Without this, Prometheus would try to request metrics from the target (http://8.8.8.8/probe?target=8.8.8.8&module=icmp) rather than the blackbox exporter (http://localhost:9115/probe?target=8.8.8.8&module=icmp).

Reload Prometheus to start collecting metrics:

sudo systemctl reload prometheus

Go to http://localhost:9090/targets and confirm that the blackbox and blackbox_meta targets appear with all of the endpoints you configured above (in my configuration, there are a total of 5 endpoints across both).

Add to Grafana

Go to Grafana and import a new dashboard (http://localhost:3000/dashboard/import). Use Grafana.com dashboard ID "7587" (Prometheus Blackbox Exporter). Select the "Prometheus" data source. Go to the new dashboard and confirm that the last 5 minutes of metrics look sane.

Migrating a nonredundant zpool to raidz1 (without a spare disk)

2019-07-14T00:00:00+00:00

I had a zpool pool1 backed by a single 3TB disk. I ordered two 4TB disks and wanted to migrate the data from pool1 to a new raidz1 zpool pool2 backed by the 3TB disk and the two 4TB disks. Here's how.

Create a zpool with the two 4TB disks and a placeholder file in place of the 3TB disk:

$ sudo fdisk -l /dev/dev/disk/by-id/wwn-0x5000000000000001
Disk /dev/disk/by-id/wwn-0x5000000000000001: 2.75 TiB, 3000592942592 bytes, 5860533091 sectors
$ sudo truncate -s $((3000592942592-1024**3)) /fakedisk
$ sudo zpool create \
>   -O compression=on \
>   -O encryption=on \
>   -O keyformat=passphrase \
>   -O keylocation=file:///etc/cryptfs.key \
>   -f \
>   pool2 raidz1 /fakedisk /dev/disk/by-id/wwn-0x500000000000000[23]
$ sudo zpool offline pool2 /fakedisk
$ sudo rm /fakedisk

Some notes:

When ZFS is using a whole disk rather than a file or partition, it applies its own partitioning scheme that uses some of the space on the disk. I made the placeholder file 1GiB smaller than the actual disk as padding for the space used by ZFS. The vdev will be expanded to use the full space later.
I'm using ZFS 0.8.x encryption on the new zpool. This is optional.
The WWN is a stable identifier that uniquely identifies the disk. It typically encodes the manufacturer and serial number. wwn-0x..1 is the old 3TB disk and wwn-0x..[23] are the two new 4TB disks.
The -f is required to allow disks and files to be mixed in the same raidz group.

Here's what the zpool looks like after creation:

$ zpool list pool2
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
pool2  8.17T  1.20M  8.17T        -     2.72T     0%     0%  1.00x  DEGRADED  -
$ zfs list pool2
NAME    USED  AVAIL     REFER  MOUNTPOINT
pool2   559K  5.27T      128K  /pool2

The total zpool size is about 9TB (8.17TiB): one 3TB disk and 3TB from each 4TB disk. The expandsize is 1TB+1TB+1TB=3TB (2.72TiB) because that is how much the total zpool size would increase if the 3TB disk were replaced with a 4TB disk. The root ZFS dataset shows 3TB+3TB=6TB (5.27TiB) available because every 2 blocks written to the zpool are encoded into 2 data blocks and 1 parity block, i.e., 1/3 of the space is dedicated to redundancy.

Normally zfs send -R would be sufficient to move the data over to the new zpool, but because I'm migrating from a zpool encrypted with dm-crypt (i.e., no encryption at the ZFS layer) to a ZFS-encrypted zpool, it is not an option. Instead, I created two helper scripts to send all snapshots and datasets:

#!/bin/bash
# send-snapshots
set -eux
src="$1"
dst="$2"
first="$(zfs list -Hr -o name -t snapshot "$src" | head -1)"
last="$(zfs list -Hr -o name -t snapshot "$src" | tail -1)"
zfs send "$first" | zfs recv -u "$dst"
zfs send -I "$first" "$last" | zfs recv -u "$dst"
zfs mount "$dst"

#!/bin/bash
# send-datasets
set -eux
src="$1"
dst="$2"
zfs list -Hr -t filesystem,volume -o name "$src" \
  | tail +2 \
  | cut -d/ -f2 \
  | while read -r ds; do ./send-snapshots "$src/$ds" "$dst/$ds"; done

Send the data:

$ chmod +x send-snapshots send-datasets
$ sudo ./send-datasets pool1 pool2

Verify that the contents of pool2 are as expected.

Destroy pool1 and replace the placeholder file in pool2 with the 3TB disk from pool1. The -f is required because ZFS will see the label on the old disk and halt the replacement as a safety measure.

$ sudo zpool destroy pool1
$ sudo zpool replace pool2 /fakedisk /dev/disk/by-id/wwn-0x5000000000000001

The zpool will begin resilvering:

$ zpool status pool2
  pool: pool2
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress [...]
config:

    NAME                                                STATE     READ WRITE CKSUM
    pool2                                               DEGRADED     0     0     0
      raidz1-0                                          DEGRADED     0     0     0
        replacing-0                                     DEGRADED     0     0     0
          /fakedisk                                     OFFLINE      0     0     0
          /dev/disk/by-id/wwn-0x5000000000000001-part1  ONLINE       0     0     0
        /dev/disk/by-id/wwn-0x5000000000000002-part1    ONLINE       0     0     0
        /dev/disk/by-id/wwn-0x5000000000000003-part1    ONLINE       0     0     0

errors: No known data errors

Eventually the zpool will reach a healthy state:

$ zpool status pool2
  pool: pool2
 state: ONLINE
  scan: resilvered [...]
config:

    NAME                                              STATE     READ WRITE CKSUM
    pool2                                             ONLINE       0     0     0
      raidz1-0                                        ONLINE       0     0     0
        /dev/disk/by-id/wwn-0x5000000000000001-part1  ONLINE       0     0     0
        /dev/disk/by-id/wwn-0x5000000000000002-part1  ONLINE       0     0     0
        /dev/disk/by-id/wwn-0x5000000000000003-part1  ONLINE       0     0     0

errors: No known data errors

Now that the actual 3TB disk is present in pool2, we can expand the vdevs to use the entire disk:

$ zpool list pool2
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
pool2  8.17T  2.21T  5.96T        -     2.72T     0%    27%  1.00x    ONLINE  -
$ zpool online -e pool2 /dev/disk/by-id/wwn-0x500000000000000[123]
$ zpool list pool2
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
pool2  8.17T  2.21T  5.96T        -         -     0%    27%  1.00x    ONLINE  -

Personal Arch Linux setup

2019-04-24T00:00:00+00:00

This is how I have my computer set up. I don't expect this document to be useful for other people.

Table of Contents

Installation
Non-root user
Desktop environment
Screen locker
Keyboard shortcuts
NetworkManager
Sound
- Bluetooth headset
"Apple" keyboard
Console utilities
Development tools
Docker
Desktop programs
Firefox
HDD ZFS pool
Backups
- Snapshots to HDD
- Offsite
Prometheus

Installation

Follow Arch Linux with mirrored, encrypted ZFS root.

Connect to network:

dhcpcd

Non-root user

useradd -m -G wheel,network,sys,uucp me
chmod 700 /home/me
passwd me
useradd -m -G network guest
passwd guest
pacman -S sudo
visudo

Uncomment the line %wheel ALL=(ALL) ALL and save. Log out and log back in as regular user.

Desktop environment

Install video drivers:

pacman -S nvidia nvidia-utils cuda cudnn
systemctl reboot

Install basic desktop environment:

pacman -S xfce4 xf86-input-{synaptics,wacom} firefox vim 
cp /etc/X11/xinit/xinitrc .xinitrc

Replace the last few lines with

exec startxfce4

Remap caps lock to escape:

# ~/.Xmodmap
# Double check that ~/.xinitrc loads this file.
# If not, add `xmodmap "$HOME/.Xmodmap"` before the `exec` line.
clear Lock
!keycode 66 = Escape Caps_Lock  ! <Shift>Escape for caps lock
keycode 66 = Escape

Run exec startx to start DE.

Screen locker

Configure shortcuts in xfce4-keyboard-settings:

Alt+Q       xflock4

Install xautolock to lock the screen after 20 minute timeout:

pacman -S xautolock

In xfce4-session-settings under "Application Autostart", add an entry for xautolock:

/bin/bash -c 'xset dpms 0 0 0; xautolock -time 20 -locker /usr/bin/xflock4 > ~/.xautolock.log 2>&1'

Confirm that screen lock works. Confirm that you can't bypass the lock with Ctrl-Alt-F7.

Keyboard shortcuts

Follow Catproofing my keyboard shortcuts.

NetworkManager

pacman -S networkmanager network-manager-applet
systemctl enable NetworkManager systemd-resolved
systemctl start NetworkManager systemd-resolved
ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

The router provided by my ISP has a bug that causes DNS requests to hang for 5 seconds. To work around it:

nmcli con
nmcli con modify MyNetworkName ipv4.dns-options single-request

To revert, set ipv4.dns-options to '' instead of single-request.

Sound

pacman -S pipewire pipewire-media-session pipewire-pulse xfce4-pulseaudio-plugin xfce4-notifyd pavucontrol clementine gst-libav gst-plugins-{base,good,bad,ugly}

Add PulseAudio applet to panel. Log out and log back in, then confirm sound works.

Bluetooth headset

Reference

$ pacman -S bluez-utils blueman-applet

Configure BlueZ to automatically power on Bluetooth adapter:

# /etc/bluetooth/main.conf
[Policy]
AutoEnable=true

Power on the headset. Start BlueZ and connect to the headset:

$ systemctl enable bluetooth
$ systemctl start bluetooth
$ bluetoothctl
[bluetooth]# power on
[bluetooth]# agent on
[bluetooth]# default-agent
[bluetooth]# scan on
[bluetooth]# devices
[bluetooth]# pair XX:XX:XX:XX:XX:XX
[bluetooth]# connect XX:XX:XX:XX:XX:XX

Confirm audio is working in headset. You may need to switch the output device in xfce4-pulseaudio-plugin/pavucontrol. Mark the headset trusted so that it can initiate the connection:

[bluetooth]# trust XX:XX:XX:XX:XX:XX

Create the following script to automatically switch to the headset when powered on:

# ~/.local/bin/btautoswitch
#!/bin/bash
while true; do
    if pactl list sinks | fgrep bluez_output.XX_XX_XX_XX_XX_XX.a2dp-sink >/dev/null; then
        if pactl info | fgrep 'Default Sink: alsa_output.pci-XXXX_XX_XX.X.analog-stereo' >/dev/null; then
            pactl set-default-sink bluez_output.XX_XX_XX_XX_XX_XX.a2dp-sink
        fi
    fi
    sleep 0.5
done

Add it to autostart in xfce4-session-settings. You may also need to add blueman-applet. Start btautoswitch manually now:

$ btautoswtich

Power off the headset. Audio should go back to the previous audio sink. Power on the headset. Audio should move to the headset.

If something goes wrong, the following commands may be useful:

[bluetooth]# show  # Show Bluetooth controller status
[bluetooth]# info  # Show Bluetooth device status
$ journalctl -fu bluetooth  # Watch BlueZ logs
$ journalctl -fk  # Watch kernel messages for new devices
$ pactl list cards  # See if headset is known by PulseAudio
$ pactl list sinks  # See if headset is recognized as audio sink by PulseAudio
$ blueman-manager  # Blueman GUI

"Apple" keyboard

The keyboard I'm currently using presents itself as an Apple keyboard. This requires a couple workarounds.

Move keyboard before autodetect in /etc/mkinitcpio.conf, then run sudo mkinitcpio -P
Set the function keys to be function keys instead of multimedia keys: echo 2 | sudo tee /sys/module/hid_apple/parameters/fnmode
Make it permanent by adding options hid_apple fnmode=2 to /etc/modprobe.d/hid_apple.conf

Console utilities

pacman -S mlocate ncdu openssh parallel virtualbox virtualbox-host-modules-arch wget whois ripgrep the_silver_searcher moreutils bind-tools android-{tools,udev} cronie
gpasswd -a me vboxusers
systemctl enable man-db.timer systemd-timesyncd cronie
systemctl start man-db.timer systemd-timesyncd cronie

Ignore backups and directories with many files in /etc/updatedb.conf:

PRUNEPATHS = "[...] /slow/backup /slow/bulk/datasets"

Development tools

pacman -S python base-devel python-{numpy,scipy,matplotlib,seaborn,scikit-learn,pandas,pip} python2-{pip,dbus} git pandoc ipython ipython2 jupyterlab tensorflow-opt-cuda tensorboard rustup texlive-most wireshark
gpasswd -a me wireshark

Docker

zfs create -o mountpoint=/var/lib/docker zroot/docker
pacman -S docker
gpasswd -a me docker

Desktop programs

pacman -S firefox xorg-xset redshift caja dmenu gvim maim dzen2 pidgin pidgin-libnotify thunderbird keepassx2 aspell-en eom gimp gnumeric inkscape picard liferea mpv soundconverter transmission-gtk mcomix engrampa youtube-dl ttf-dejavu ttf-liberation noto-fonts{,-cjk,-emoji,-extra}

Set units in Caja to IEC.

Install auracle-git. Install the following:

auracle download anki freecad-weekly-appimage insect zotero-bin
cd $pkg; makepkg -sic  # for each

Firefox

about:preferences settings:

General:
- Tabs:
  - Disable "Ctrl+Tab cycles through tabs in recently used order"
- Browsing:
  - Enable "Use autoscrolling"
  - Enable "Always show scrollbars"
  - Disable "Recommend extensions as you browse"
  - Disable "Recommend feaures as you browse"
Home:
- New Windows and Tabs: Set all to "Blank Page"
- Firefox Home Content: Disable all
Search:
- Search Suggestions: Disable all
Privacy & Security:
- Enhanced Tracking Protection: Strict
- Logins and Passwords:
  - Disable "Autofill logins and passwords"
- Address Bar - Firefox Suggest: Disable all except "Bookmarks"
- Firefox Data Collection and Use:
  - Disable "Allow Firefox to send technical and interaction data to Mozilla"
  - Disable "Allow Firefox to make personalized extension recommendations"
  - Disable "Allow Firefox to install and run studies"

about:config (or user.js) settings:

// Don't mangle URLs
user_pref("keyword.enabled", false);
user_pref("browser.fixup.alternate.enabled", false);

// Don't close the window when the last tab is closed.
user_pref("browser.tabs.closeWindowWithLastTab", false);

// Send origin but not full URL.
// Not sending the referrer at all breaks too many sites.
user_pref("network.http.referer.XOriginTrimmingPolicy", 2);

// Disable clipboard API to avoid exposing clipboard contents on middle click.
// This has been fixed for links in <https://bugzilla.mozilla.org/show_bug.cgi?id=1521396>,
// but middle clicking on other elements still exposes your clipboard contents.
user_pref("middlemouse.paste", false);
user_pref("dom.event.clipboardevents.enabled", false);
user_pref("dom.events.asyncClipboard", false);

// Service Workers can make requests even after a page is closed. This is a
// privacy risk.
user_pref("dom.serviceWorkers.enabled", false);

Theme: "Light". The "System theme" doesn't highlight the selected tab.

Extensions:

Vimium config:

unmapAll
map j scrollDown
map k scrollUp
map gg scrollToTop
map G scrollToBottom
map d scrollPageDown
map u scrollPageUp
map i enterInsertMode
map f LinkHints.activateMode
map F LinkHints.activateModeToOpenInNewTab

HDD ZFS pool

Create /etc/systemd/system/zfs-load-key.service (see ZFS#Unlock at boot time: systemd):

[Unit]
Description=Load encryption keys
DefaultDependencies=no
Before=zfs-mount.service
After=zfs-import.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/bash -c '/usr/bin/zfs load-key -a'

[Install]
WantedBy=zfs-mount.service

Place key in /etc/cryptfs.key. Import the pool and enable services:

zpool import
zpool import slow
systemctl enable zfs-scrub@slow.timer
systemctl start zfs-scrub@slow.timer
systemctl enable zfs-load-key.service
systemctl start zfs-load-key.service

Backups

Snapshots to HDD

See Onsite backups with zrepl.

Offsite

Using Borg to back up to rsync.net.

In /usr/local/bin/rsyncnet-backup:

#!/bin/bash
set -eu

if [ "$EUID" -ne 0 ]; then
    echo 'Run as root'
    exit 1
fi

# # /root/.ssh/config
# Host whatever.rsync.net
#    User 12345
#    IdentityFile ~/.ssh/rsyncnet
#    ServerAliveInterval 10
#    ServerAliveCountMax 30
export SSH_HOST='whatever.rsync.net'
export BORG_REPO="$SSH_HOST:borg/myhostname"
export BORG_PASSPHRASE="$(cat /path/to/borg.key)"

# Based on https://borgbackup.readthedocs.io/en/stable/quickstart.html#automating-backups

info() { printf "\n%s %s\n\n" "$( date )" "$*" >&2; }
trap 'echo $( date ) Backup interrupted >&2; exit 2' INT TERM

info "Checking quota"

ssh "$SSH_HOST" quota

info "Starting backup"
borg create \
    --remote-path=borg1 \
    --verbose \
    --filter AME \
    --list \
    --stats \
    --show-rc \
    --compression lz4 \
    --exclude-caches \
    --one-file-system \
    --exclude '/home/*/.cache' \
    --exclude '/home/*/.cargo' \
    --exclude '/home/*/.conda' \
    --exclude '/home/*/.go' \
    --exclude '/home/*/.julia' \
    --exclude '/home/*/.npm' \
    --exclude '/home/*/.platformio' \
    --exclude '/home/*/.pyenv' \
    --exclude '/home/*/.rustup' \
    --exclude '/home/*/.vagrant.d/boxes' \
    --exclude '/home/*/Code/3p' \
    --exclude '/home/*/Downloads' \
    --exclude '/home/*/Music' \
    --exclude '/home/*/aur' \
    --exclude '/var/cache/*' \
    --exclude '/var/lib/docker' \
    --exclude '/var/lib/prometheus' \
    --exclude '/var/log/journal' \
    --exclude '/var/tmp/*' \
    ::'{utcnow}' \
    /etc \
    /home \
    /root \
    /var

info "Pruning repository"
borg prune \
    --remote-path=borg1 \
    --list \
    --show-rc \
    --keep-daily 7 \
    --keep-weekly 4 \
    --keep-monthly 4

Set appropriate permissions:

sudo chown root:root /usr/local/bin/rsyncnet-backup /path/to/borg.key
sudo chmod 755 /usr/local/bin/rsyncnet-backup
sudo chmod 600 /path/to/borg.key

Add to /etc/anacrontab:

1 120 rsyncnet-backup nice chronic /usr/local/bin/rsyncnet-backup

Prometheus

See Collecting local system stats with Prometheus on Arch Linux.

Arch Linux with mirrored, encrypted ZFS root

2019-04-23T00:00:00+00:00

Goals:

The root filesystem is ZFS.
The root filesystem is encrypted. The boot partition does not need to be.
The root filesystem supports data checksums.
If a drive fails, the system should boot with no manual intervention.

ZFS on Linux 0.8 supports native encryption. This page was written before 0.8 was released, so native encryption was not evaluated as an option. The ArchWiki now has instructions for using native ZFS encryption on the root filesystem. Consider following those instructions instead.

On the other hand, it may still be worth avoiding ZFS native encryption due to issues with zfs send/recv on natively encrypted datasets.

Installation

Prepare Arch Linux ISO with ZFS

For installation and recovery, it's useful to have a USB flash drive with ZFS already installed. From an Arch Linux system:

sudo -i
pacman -S archiso
cp -r /usr/share/archiso/configs/releng archlive
cd archlive

Add the archzfs repository. The key is the one listed here and here.

echo $'[archzfs]\nServer = http://archzfs.com/$repo/x86_64' >> pacman.conf
cp pacman.conf airootfs/etc/
echo 'systemctl enable pacman-archzfs-init.service' >> airootfs/root/customize_airootfs.sh
mkdir airootfs/etc/systemd/scripts
curl -L -o airootfs/etc/systemd/scripts/archzfs.asc https://archzfs.com/archzfs.gpg

Create airootfs/etc/systemd/system/pacman-archzfs-init.service with the following contents:

[Unit]
Description=Adds archzfs to pacman keyring
Requires=pacman-init.service
After=pacman-init.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/pacman-key --add /etc/systemd/scripts/archzfs.asc
ExecStart=/usr/bin/pacman-key --lsign-key DDF7DB817396A49B2A2723F7403BD972F75D9D76

[Install]
WantedBy=multi-user.target

Add packages:

echo $'linux-headers\narchzfs-linux' >> packages.x86_64

Build and copy to a USB flash drive:

mkarchiso -v -C pacman.conf .
dd bs=4M if=out/archlinux-*.iso of=/dev/sdX status=progress oflag=sync

Replace /dev/sdX with the path to your USB flash drive.

Begin installation

Follow the Installation Guide up to (but not including) the partitioning step.

Partitioning & filesystems

In the following, I will assume you are installing on /dev/sda and /dev/sdb.

Partition drives

I use BIOS because my motherboard's UEFI firmware is finicky. If you don't, adapt the partition scheme for UEFI as necessary. GPT supports drives ≥2 TiB, while MBR does not. Even though it was not strictly necessary for these two drives, I chose GPT over MBR for uniformity, as I have another drive >2 TiB.

Partition /dev/sda. Repeat for /dev/sdb. The sizes of partitions 2 and 3 must match between the two drives.

gdisk /dev/sda
o
n <enter> <enter> +1M ef02
n <enter> <enter> +1G fd00
n <enter> <enter> +464G 8309
w
sfdisk -A /dev/sda 1

The sfdisk command sets the bootable flag on the protective MBR (PMBR). My motherboard will not boot to a drive that doesn't have this bit set.

This results in the following partition scheme:

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048            4095   1024.0 KiB  EF02  BIOS boot partition
   2            4096         2101247   1024.0 MiB  FD00  Linux RAID
   3         2101248       975179775   464.0 GiB   8309  Linux LUKS

Run lsblk --discard. If the DISC-GRAN and DISC-MAX columns are nonzero, then TRIM (a.k.a. discard) is supported. Take note of whether or not TRIM is supported on /dev/sda and /dev/sdb.

Set up dm-crypt partition

cryptsetup luksFormat --type luks2 /dev/sda3
cryptsetup luksFormat --type luks2 /dev/sdb3
cryptsetup open /dev/sda3 cryptroot1 --allow-discards
cryptsetup open /dev/sdb3 cryptroot2 --allow-discards

If your drives do not support TRIM or you are not comfortable with the tradeoffs of enabling TRIM on an encrypted volume, omit the --allow-discards option.

Create the ZFS pool

Create a pool and a dataset for the root filesystem. The dataset can be named anything, but you cannot use the pool's root dataset (zroot) as the root filesystem.¹ I chose zroot/ROOT/default to match the naming scheme commonly used for boot environments.

zpool create zroot mirror /dev/mapper/cryptroot[12]
zfs set mountpoint=none acltype=posixacl xattr=sa zroot
zfs create -o mountpoint=none zroot/ROOT
zfs create -o mountpoint=/ zroot/ROOT/default
zpool export zroot
zpool import -d /dev/disk/by-id -R /mnt zroot

When creating the datasets, it will complain that it cannot mount /. This is okay.

OpenZFS uses a "cachefile" to record the devices in the pool. We must copy this file into our new Arch Linux system so that it knows where to look for the pool's devices:²

zpool set cachefile=/etc/zfs/zpool.cache zroot
mkdir -p /mnt/etc/zfs
cp /etc/zfs/zpool.cache /mnt/etc/zfs/

Set up RAID1 boot partition

We will be using GRUB 2 as our bootloader. While GRUB 2 does support booting from LUKS1-encrypted ZFS partitions, the ZFS support is not well maintained.³ Instead, we will use an unencrypted ext4 boot partition mirrored with MD/RAID.

mdadm --create --level=1 --metadata=1.0 --raid-devices=2 /dev/md/boot /dev/sd[ab]2
mkfs.ext4 /dev/md/boot
mkdir /mnt/boot
mount /dev/md/boot /mnt/boot

The 1.0 metadata format places the metadata at the end of the partition to avoid interfering with the bootloader.

Install base packages and chroot

Select mirrors as described in the Installation Guide, then install the base packages, bootloader, and the zfs-linux-lts package:

pacstrap -K /mnt base linux-lts linux-firmware vim grub mdadm zfs-linux-lts

I recommend running a LTS kernel with ZFS. Because ZFS depends on internal kernel functions and structures, changes in the kernel can break ZFS. Using a LTS release gives the OpenZFS developers some time to update ZFS after a new kernel release.

Generate a fstab and chroot into the system:

genfstab -U /mnt >> /mnt/etc/fstab
cp /etc/systemd/scripts/archzfs.asc /mnt/root/
arch-chroot /mnt

Add the archzfs mirror in the new system:

echo $'[archzfs]\nServer = http://archzfs.com/$repo/x86_64' >> /etc/pacman.conf
pacman-key --add /root/archzfs.asc
pacman-key --lsign-key DDF7DB817396A49B2A2723F7403BD972F75D9D76
pacman -Sy

Continue with the Installation Guide until you reach the initramfs (mkinitcpio) step.

Initramfs

Set the following hooks in /etc/mkinitcpio.conf:

HOOKS=(base udev keyboard autodetect microcode modconf kms keymap consolefont block mdadm_udev encrypt encrypt2 zfs filesystems fsck)

Compared to the default, we have added mdadm_udev encrypt encrypt2 zfs between block and filesystems. Importantly, encrypt must come after keyboard keymap consolefont so that we can input the password.

The udev-based encrypt hook only supports unlocking a single encrypted volume. The systemd-based sd-encrypt hook supports unlocking multiple volumes, but unfortunately it is not compatible with the zfs hook.⁴ We will instead use a workaround and make a copy of the encrypt hook to unlock the second drive:

cp /usr/lib/initcpio/install/encrypt /etc/initcpio/install/encrypt2
cp /usr/lib/initcpio/hooks/encrypt /etc/initcpio/hooks/encrypt2
sed -i s/cryptdevice/cryptdevice2/ /etc/initcpio/hooks/encrypt2
sed -i s/cryptkey/cryptkey2/ /etc/initcpio/hooks/encrypt2

Now generate a new initramfs:

mkinitcpio -P

Bootloader

Install GRUB to both drives:

grub-install --target=i386-pc /dev/sda
grub-install --target=i386-pc /dev/sdb

Find the UUID of the encrypted partitions:

blkid /dev/sd[ab]3

Configure GRUB to pass those in as kernel parameters in /etc/default/grub:

GRUB_CMDLINE_LINUX="cryptdevice=UUID=11111111-1111-1111-1111-111111111111:cryptroot1:allow-discards cryptdevice2=UUID=22222222-2222-2222-2222-222222222222:cryptroot2:allow-discards"

As before, omit the :allow-discards if your drives do not support TRIM. Now generate your GRUB config. The ZPOOL_VDEV_NAME_PATH tells ZFS to use full paths (/dev/disk/by-uuid/dm-uuid-...) rather than short names (dm-uuid-...).

ZPOOL_VDEV_NAME_PATH=1 grub-mkconfig -o /boot/grub/grub.cfg

Save that variable to /etc/environment so that it doesn't need to be provided when running grub-mkconfig in the future:

echo ZPOOL_VDEV_NAME_PATH=1 >> /etc/environment

Confirm root zpool detected

Open /boot/grub/grub.cfg in a text editor and locate the menuentry 'Arch Linux' section. A few lines down there will be a line like linux ... root=ZFS=zroot/ROOT/default. If yours is root=ZFS=/ROOT/default (no zroot), then GRUB failed to detect your root zpool. A workaround is given on the Arch Wiki (bug report):

Open /etc/grub.d/10_linux in a text editor and replace

rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`

with

rpool=`zdb -l ${GRUB_DEVICE} | grep " name:" | cut -d\' -f2`

Run ZPOOL_VDEV_NAME_PATH=1 grub-mkconfig -o /boot/grub/grub.cfg again and confirm that the root was properly detected as root=ZFS=zroot/ROOT/default.

If you run grub-mkconfig again in the future (e.g., after a grub package update), be sure to look over /boot/grub/grub.cfg afterward. You may need to repeat the steps above.

Set root password, unmount, and reboot

passwd
exit  # leave chroot
umount /mnt/boot
zpool export zroot
systemctl reboot

First boot

We're not done! When the new system is booted up, regenerate the ZFS cachefile and enable ZFS-related services:

zpool set cachefile=/etc/zfs/zpool.cache zroot
systemctl enable zfs.target zfs-import-cache.service zfs-mount.service zfs-import.target zfs-zed.service

ZFS uses the system's hostid to determine if a pool is already open by a different OS.⁵ The hostid isn't available in the initramfs, so we'll need to save a copy of it to a file:

zgenhostid "$(hostid)"

Now regenerate your initramfs and reboot:

mkinitcpio -P
systemctl reboot

If the initramfs is unable to open the pool, it may kernel panic. You can always rescan devices and force the pool open by temporarily adding these kernel parameters:

zfs_import_dir=/dev/disk/by-id zfs_force=1

If this happens, you may need to save the ZFS cachefile and regenerate the initramfs:

zpool set cachefile=/etc/zfs/zpool.cache zroot
mkinitcpio -P

Quota (optional)

ZFS performance degrades when the pool is too full. It's helpful to set the quota to about 80% of the pool size to prevent this from happening:

zpool get size zroot
# NAME   PROPERTY  VALUE  SOURCE
# zroot  size      460G   -

zfs set quota=368G zroot

Monitoring

Set up local mail server

pacman -S opensmtpd s-nail

Configure OpenSMTPD for local delivery. Replace the default configuration at /etc/smtpd/smtpd.conf with the following:

table aliases file:/etc/smtpd/aliases
listen on localhost
action "local" mbox alias <aliases>
match for local action "local"

Start mail server:

systemctl start smtpd.service
systemctl enable smtpd.service

Add alias to forward mail from root to your user, then regenerate aliases:

echo 'root: youruser' >> /etc/smtpd/aliases
smtpctl update table aliases

Test mail:

mailx -s 'hello to root' root <<<'example message'
mailx -s 'hello to youruser' youruser <<<'example message'
cat /var/mail/youruser

You should see both of the test messages in /var/mail/youruser. Now install a local mail client (or configure your existing client to receive local mail). Mutt is a reasonable option; it can be installed with pacman -S mutt.

Configure mdadm to mail on events

In /etc/mdadm.conf, add the following option:

MAILADDR youruser

Start monitoring service:

systemctl start mdmonitor.service
systemctl enable mdmonitor.service  # May already be enabled.

Test it:

mdadm --monitor --scan --oneshot --test

Configure smartd to mail on errors

Install smartmontools:

pacman -S smartmontools

Replace the existing DEVICESCAN line in /etc/smartd.conf (replace youruser with your username):

DEVICESCAN -a -n standby,6,q -m youruser -M test

This line tells smartd to email youruser if any errors are encountered. It scans once every 30 minutes (default for smartd, see smartd(8)), skipping drives in standby mode. If a device is skipped 6 times, it will by scanned regardless of whether or not the drive is in standby.

Start the service:

systemctl start smartd.service

You should receive a test email for each drive. You can now remove the -M test from /etc/smartd.conf, then reload and enable the service.

systemctl reload smartd.service
systemctl enable smartd.service

Configure ZFS to mail on events

In /etc/zfs/zed.d/zed.rc, add an email address:

ZED_EMAIL_ADDR="youruser"

Temporarily enable notification for all events, then restart ZED:

cp /usr/lib/zfs/zed.d/generic-notify.sh /etc/zfs/zed.d/all-notify.sh
echo 'ZED_NOTIFY_INTERVAL_SECS=1' >> /etc/zfs/zed.d/zed.rc
systemctl restart zfs-zed.service

Generate an event by taking a snapshot:

zfs snap zroot/ROOT/default@meow
zfs destroy zroot/ROOT/default@meow

Confirm you received an email about the events. Remove the notifications for all events, then restart ZED again. You will still receive notifications for important events (see /etc/zfs/zed.d/*-notify.sh).

rm /etc/zfs/zed.d/all-notify.sh
sed -i '/^ZED_NOTIFY_INTERVAL_SECS=1$/d' /etc/zfs/zed.d/zed.rc
systemctl restart zfs-zed.service

Set up periodic ZFS scrub

Create /etc/systemd/system/zfs-scrub@.timer:

[Unit]
Description=Monthly zpool scrub on %i

[Timer]
OnCalendar=*-*-01 03:00:00
AccuracySec=1h
Persistent=true

[Install]
WantedBy=multi-user.target

Create /etc/systemd/system/zfs-scrub@.service:

[Unit]
Description=zpool scrub on %i

[Service]
Nice=19
IOSchedulingClass=idle
KillSignal=SIGINT
ExecStart=/usr/bin/zpool scrub %i

Enable the timer for your root pool:

systemctl enable zfs-scrub@zroot.timer
systemctl start zfs-scrub@zroot.timer

Set up periodic ZFS trim

Skip this section if you chose not to use TRIM.

Create /etc/systemd/system/zfs-trim@.timer:

[Unit]
Description=Weekly zpool trim on %i

[Timer]
OnCalendar=weekly
AccuracySec=1h
Persistent=true

[Install]
WantedBy=multi-user.target

Create /etc/systemd/system/zfs-trim@.service:

[Unit]
Description=zpool trim on %i

[Service]
Nice=19
IOSchedulingClass=idle
KillSignal=SIGINT
ExecStart=/usr/bin/zpool trim %i

Enable the timer for your root pool:

systemctl enable zfs-trim@zroot.timer
systemctl start zfs-trim@zroot.timer

Maintenance

Upgrading your system

The archzfs repo lags behind the Arch core repository by a few days when there are kernel updates. You will sometimes see an error like this:

# pacman -Syu
:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...
error: failed to prepare transaction (could not satisfy dependencies)
:: installing linux (5.9.2.arch1-1) breaks dependency 'linux=5.9.1.arch1-1' required by zfs-linux

You have a couple options:

Don't upgrade right now. Wait for archzfs to catch up with the current Linux version.
Press on and do a partial upgrade. This is not supported by the Arch Linux team!

If you choose to press on with a partial upgrade, you'll need to skip packages containing kernel modules, as Arch Linux does not guarantee backward compatibility of these modules with older kernels.⁶

mods=$(pacman -Qo /usr/lib/modules | awk '{print $(NF-1)}')
hold=$({ echo "$mods"; expac -l'\n' '%D' $mods | grep = | cut -d= -f1; } | sort | uniq | paste -sd,)
pacman -Su --ignore $hold

Before confirming the upgrade, carefully review the list of packages to be upgraded.

Viewing the status

Viewing mdadm status:

mdadm --detail /dev/md/boot

# /dev/md/boot:
#         Version : 1.0
#   Creation Time : x
#      Raid Level : raid1
#      Array Size : x
#   Used Dev Size : x
#    Raid Devices : 2
#   Total Devices : 2
#     Persistence : Superblock is persistent
# 
#     Update Time : Sun Apr 14 17:06:03 2019
#           State : clean 
#  Active Devices : 2
# Working Devices : 2
#  Failed Devices : 0
#   Spare Devices : 0
# 
#            Name : archiso:boot
#            UUID : x
#          Events : 122
# 
#     Number   Major   Minor   RaidDevice State
#        3       8        x        0      active sync   /dev/sda2
#        2       8        x        1      active sync   /dev/sdb2

Viewing ZFS status:

zpool status

#  pool: zroot
# state: ONLINE
#  scan: scrub repaired 0B in 0h10m with 0 errors on Sun Apr 14 12:42:19 2019
#config:
#
#   NAME                        STATE     READ WRITE CKSUM
#   zroot                       ONLINE       0     0     0
#     mirror-0                  ONLINE       0     0     0
#       /dev/mapper/cryptroot1  ONLINE       0     0     0
#       /dev/mapper/cryptroot2  ONLINE       0     0     0
#
#errors: No known data errors

Silent errors

If a drive has silent data corruption, ZFS will detect it and increment its checksum error counter in zpool status. We should kick the drive out of the MD/RAID configuration before it causes problems. RAID1 does not perform checksums, so if a drive has corruption, MD/RAID will blindly copy it to the other drive.

mdadm --fail /dev/md/boot $DRIVE

Run fsck on the boot partition:

umount /boot
fsck.ext4 /dev/md/boot
mount -a

If you're concerned that there was silent corruption in the boot partition, reinstall GRUB and regenerate the initramfs in case they got corrupted by MD/RAID copying bad data:

grub-install --target=i386-pc $GOOD_DRIVE
grub-mkconfig -o /boot/grub/grub.cfg
mkinitcpio -P

Now get the list of packages that have files in /boot and reinstall them:

find /boot -type f | xargs pacman -Qo 2>/dev/null
pacman -S $PACKAGES_FROM_PREVIOUS_COMMAND

Hard failures

If a drive becomes unavailable, mdadm and zpool status will both show a "degraded" state.

Replacing a drive

If you haven't already done so, remove the drive from MD/RAID and ZFS:

zpool offline zroot $DRIVE
zpool detach zroot $DRIVE
mdadm --fail /dev/md/boot $DRIVE  # if drive still active
mdadm --remove /dev/md/boot $DRIVE

If the drive is not listed in mdadm --detail /dev/md/boot, you may need to replace $DRIVE with failed or detached in the mdadm commands.

Power down your computer, remove the old drive, and add a new one. Power the computer back on. When the system comes back up, mdadm and ZFS will show a degraded state.

Follow the partitioning and dm-crypt steps above on the new drive. I recommend reusing the same name for the dm-crypt partition as the failed drive (cryptroot1 or cryptroot2). Install GRUB:

grub-install --target=i386-pc /dev/sdx

Add the drive to mdadm and ZFS. In this example, I will assume /dev/sdx is the new drive and its dm-crypt partition is mapped to /dev/mapper/cryptrootY.

mdadm --add /dev/md/boot /dev/sdx2
zpool attach zroot $GOOD_DRIVE /dev/mapper/cryptrootY

Finally, update the UUID in /etc/default/grub to the one shown by blkid /dev/sdx2 and generate a new GRUB config:

grub-mkconfig -o /boot/grub/grub.cfg

Interrupted kernel upgrade

If a kernel upgrade is interrupted, you may end up with an unbootable system with the following error:

Loading Linux linux-lts ...
error: file `/vmlinuz-linux-lts' not found.
Loading initial ramdisk ...
error: you need to load the kernel first.

This isn't unique to this setup, but the recovery is a little more complicated. Boot into the USB flash drive created earlier. Chroot into your system and reinstall the relevant packages:

cryptsetup open /dev/sda3 cryptroot1 --allow-discards
cryptsetup open /dev/sdb3 cryptroot2 --allow-discards
zpool import -d /dev/disk/by-id -R /mnt zroot
mount /dev/md/boot /mnt/boot
arch-chroot /mnt
pacman -S mkinitcpio systemd linux-lts
exit
umount /mnt/boot
zpool export zroot
systemctl reboot

After rebooting, reinstall all packages for good measure, then reboot again:

pacman -Qqn | sudo pacman -S -
systemctl reboot

Pacman preserves the installation reason when reinstalling packages, so doing this won't mess up pacman -Qe/pacman -Qe.

GRUB will try to set the root disk as root=ZFS=zroot/, which is not a valid dataset due to the trailing slash. ↩
This step isn't required if you choose to let ZFS scan all devices in the initramfs, but this is potentially slow. If you choose to do this, you'll need to add the kernel parameter zfs_import_dir=/dev/disk/by-id. ↩
One OpenZFS developer commented that "nobody at upstream GRUB cares much about ZFS support" (dead link). However, the issue was later deleted and OpenZFS 2.1.0 ships with a grub2 compatability feature set, so perhaps the situation has changed. ↩
There is an alternative systemd-based sd-zfs hook that is compatible with the sd-encrypt hook, but I have not tested it. ↩
For example, if two systems were accessing the pool over iSCSI. ↩
Most packages that install to /usr/lib/modules/*/extramodules do not specify a version constraint in their dependencies. This is because Arch Linux does not support partial upgrades, which is exactly what we're doing here. ↩

Queuing theory basics part 2: multiple servers

2019-01-29T00:00:01+00:00

These are my notes for the following three lectures on queueing theory:

Part 1 - notes for Lec-30 Queueing Models

Part 1 - notes for Lec-31 Single Server Queueing Models

Part 2 - notes for Lec-32 Multiple Server Queueing Models

Table of Contents

Multiple servers, infinite queue (M/M/c)
- Distribution of queue lengths
- Expected queue length and wait time
Multiple servers, finite queue (M/M/c/N)
- Distribution of queue lengths
- Expected queue length and wait times
Simulation

Multiple servers, infinite queue (M/M/c)

Consider the case with $c$ servers and infinite queue capacity. Assume people form a common line and are served by the first available server. Additionally, assume all servers have identical service rate $\mu$ .

The effective service rate depends on the number of people in the system. If there are $n \le c$ people are in the system, then the service rate is $n \mu$ . If there are $n \ge c$ people in the system, then all servers are busy, so the service rate is $c \mu$ .

Distribution of queue lengths

We can now compute a recurrence similar to the single server case:

$\begin{aligned} p_n(t+h) &=\; P(\text{1 arrival, 0 service}) \cdot p_{n-1}(t)\\ &\;+ P(\text{0 arrival, 1 service}) \cdot p_{n+1}(t)\\ &\;+ P(\text{0 arrival, 0 service}) \cdot p_n(t)\\ &=\; \lambda h \cdot (1 - \mu_{n-1} h) \cdot p_{n-1}(t)\\ &\;+ (1 - \lambda h) \cdot \mu_{n+1} h \cdot p_{n+1}(t)\\ &\;+ (1 - \lambda h) \cdot (1 - \mu_n h) \cdot p_n(t)\\ &\approx \lambda h p_{n-1}(t) + \mu_{n+1} h p_{n+1}(t) + (1 - \lambda h - \mu_n h) p_n(t)\\ \implies \frac{p_n(t+h) - p_n(t)}{h} &= \lambda p_{n-1}(t) + \mu_{n+1} p_{n+1}(t) - (\lambda + \mu_n) p_n(t). \end{aligned}$

At steady-state, the distribution is time-independent, so $\frac{p_n(t+h) - p_n(t)}{h} = 0$ . Thus

$\lambda p_n{n+1} + \mu_{n+1} p_{n+1} = (\lambda + \mu_n) p_n.$

Similarly, we find $\mu_1 p_1 = \lambda p_0$ (or $p_1 = \frac{\lambda}{\mu_1} p_0$ ). Now

$\begin{aligned} \lambda p_0 + \mu_2 p_2 = (\lambda + \mu_1) p_1 = \lambda p_1 + \mu_1 p_1 = \lambda p_1 + \lambda p_0\\ \implies \mu_2 p_2 = \lambda p_1 \implies p_2 = \frac{\lambda}{\mu_2} p_1. \end{aligned}$

The recurrence continues similarly:

$p_n = \frac{\lambda^n}{\prod_{i=1}^n \mu_i} p_0 = \begin{cases} \frac{\rho^n}{n!} p_0 &\text{ if } n \le c\\ \frac{\rho^n}{c! c^{n-c}} p_0 &\text{ if } n \ge c \end{cases}.$

Now we can compute $p_0$ :

$\begin{aligned} 1 = \sum_{n \ge 0} p_n = \sum_{n=0}^{c-1} \frac{\rho^n}{n!} p_0 + \sum_{n \ge c} \frac{\rho^n}{c!c^{n-c}} p_0 = p_0 \left( \sum_{n=0}^{c-1} \frac{\rho^n}{n!} + \frac{\rho^c}{c!} \frac{1}{1-\rho/c} \right)\\ \implies p_0 = \left( \sum_{n=0}^{c-1} \frac{\rho^n}{n!} + \frac{\rho^c}{c!} \frac{1}{1-\rho/c} \right)^{-1}. \end{aligned}$

Expected queue length and wait time

There are only people waiting if there are $n \ge c$ people in the system, so the expected number of people waiting is

$\begin{aligned} L_q &= \sum_{n \ge c} (n-c) p_n = \sum_{n \ge 0} n p_{n+c} = \sum_{n \ge 1} n \frac{\rho^{n+c}}{c! c^n} p_0\\ &= \frac{p_0 \rho^{c+1}}{c! c} \sum_{n \ge 1} n \frac{\rho^{n-1}}{c^{n-1}}\\ &= \frac{p_0 \rho^{c+1}}{c! c} \frac{\mathrm{d}}{\mathrm{d} (\rho/c)} \sum_{n \ge 1} \frac{\rho^n}{c^n}\\ &= \frac{p_0 \rho^{c+1}}{c! c} \frac{\mathrm{d}}{\mathrm{d} (\rho/c)} \left(\frac{1}{1-\rho/c} - 1\right)\\ &= \frac{p_0 \rho^{c+1}}{c! c} \left( \frac{1}{(1-\rho/c)^2} \right)\\ &= \frac{p_0 \rho^{c+1}}{(c-1)!(c-\rho)^2}. \end{aligned}$

A similar calculation can be carried out for the expected number of people in the system, $L_s = L_q + \rho$ . With these, can can again calculate the expected wait time and service time using Little's law:

$L_q = \lambda W_q, \qquad L_s = \lambda W_s.$

In the case of multiple servers, the calculation of utilization is more complicated. It can be thought of as the average probability that any one server is busy, assuming people are uniformly distributed across servers. That is,

$U = \sum_{n=0}^{c-1} \frac{n}{c} p_n + \sum_{n \ge c} p_n = \frac{\rho}{c}.$

As before, we can plot the latency with respect to utilization:

import matplotlib.pyplot as plt
import numpy as np
from scipy.special import factorial

def plot_latency(c, **kwargs):
   rho = np.linspace(0, c, 1000)
   p0 = 1/(sum(rho**n/factorial(n) for n in range(0, c)) +
           rho**c/factorial(c)/(1-rho/c))
   utilization = rho/c
   latency = p0*rho**c/(factorial(c-1)*(c-rho)**2) + 1
   plt.plot(utilization, latency, label=f'{c} servers', **kwargs)

plt.figure(figsize=(9, 6))
plt.ylim(0, 10)
plt.xlabel('utilization')
plt.ylabel('latency')
for c, alpha in [(2, 0.25), (8, 0.5), (32, 0.75), (128, 1.0)]:
   plot_latency(c, alpha=alpha, color='blue')
plt.legend()
plt.savefig('mmc-latency.png')
plt.show()

We can also plot the latency with respect to the arrival rate, holding the service rate fixed. So long as $\lambda \le c \mu$ , the arrival rate is equal to the throughput of the system (if $\lambda > c \mu$ , the queue will grow indefinitely). This demonstrates that adding more servers reduces the latency:

import matplotlib.pyplot as plt
import numpy as np
from scipy.special import factorial

rho = np.linspace(0, 5, 100)
def plot_latency(c, **kwargs):
   p0 = 1/(sum(rho**n/factorial(n) for n in range(0, c)) +
           rho**c/factorial(c)/(1-rho/c))
   latency = p0*rho**c/(factorial(c-1)*(c-rho)**2) + 1
   plt.plot(rho, latency, label=f'{c} servers', **kwargs)

plt.figure(figsize=(9, 6))
plt.ylim(0, 10)
plt.xlabel('throughput')
plt.ylabel('latency')
plot_latency(5)
plot_latency(6)
plt.legend()
plt.savefig('mmc-throughput.png')
plt.show()

Multiple servers, finite queue (M/M/c/N)

Consider a finite queue of capacity $N$ feeding into $c$ servers, where people arrive at rate $\lambda$ and each server has rate $\mu$ . In the following, assume $N > c$ .

Distribution of queue lengths

The recurrence for the distribution over the number of people in the system is the same as the M/M/1/N queue:

$p_n = \begin{cases} \frac{\rho^n}{n!} p_0 &\text{ if } n \le c\\ \frac{\rho^n}{c!c^{n-c}} p_0 &\text{ if } n \ge c \end{cases}.$

We can now compute the probability that there are 0 people in the system:

$\begin{aligned} 1 = \sum_{n=0}^N p_n &= \sum_{n=0}^{c-1} \frac{\rho^n}{n!} + \sum_{n=c}^N \frac{\rho^n}{c!c^{n-c}} p_0\\ &= p_0 \left( \sum_{n=0}^{c-1} \frac{\rho^n}{n!} + \frac{\rho^c}{c!} \cdot \frac{1 - (\rho/c)^{N-c+1}}{1 - \rho/c} \right)\\ \implies p_0 &= \left( \sum_{n=0}^{c-1} \frac{\rho^n}{n!} + \frac{\rho^c}{c!} \cdot \frac{1 - (\rho/c)^{N-c+1}}{1 - \rho/c} \right)^{-1}. \end{aligned}$

Expected queue length and wait times

We can compute the expected queue length similar to the M/M/c case:

$\begin{aligned} L_q &= \sum_{n=c}^N (n-c) p_n = \sum_{n=0}^{N-c} n p_{n+c} = \sum_{n=0}^{N-c} n \frac{\rho^{n+c}}{c!c^n} p_0\\ &= \frac{\rho^{c+1} p_0}{(c-1)!} \cdot \frac{1 - (\rho/c)^{N-c+1} - (N-c+1)(1-\rho/c)(\rho/c)^{N-c}}{(c-\rho)^2}. \end{aligned}$

And the expected number of people in the system:

$\begin{aligned} L_s &= \sum_{n=0}^N n p_n\\ &= p_0 \rho \sum_{n=0}^{c-2} \frac{\rho^n}{n!} + \frac{p_0 \rho^c}{c!} \bigg( c \frac{1 - (\rho/c)^{N-c+1}}{1-\rho/c}\\ &\qquad+ \frac{\rho}{c} \frac{1 - (\rho/c)^{N-c+1} - (N-c+1)(1-\rho/c)(\rho/c)^{N-c}}{(1-\rho/c)^2} \bigg). \end{aligned}$

The probability that the queue is full and thus someone is rejected is $p_N = \frac{\rho^N}{c!c^{N-c}} p_0$ . As with the M/M/1/N queue, there is an effective arrival rate $\lambda_\text{eff} = \lambda (1-p_N)$ , from which we can derive the service and wait times $W_s = L_s/\lambda_\text{eff}$ and $W_q = L_q/\lambda_\text{eff}$ .

Simulation

Here is a Jupyter notebook simulating a single- and multiple-server infinite queue:

Queuing theory basics part 1: single server

2019-01-29T00:00:00+00:00

These are my notes for the following three lectures on queueing theory:

Part 1 - notes for Lec-30 Queueing Models

Part 1 - notes for Lec-31 Single Server Queueing Models

Part 2 - notes for Lec-32 Multiple Server Queueing Models

Table of Contents

Notation
Single server, infinite queue (M/M/1)
- Distribution of queue lengths
- Expected queue length and wait time
Single server, finite queue (M/M/1/N)

In a queueing model, people arrive at the queue, wait in line, and then are serviced by a server. There may be a single server or multiple servers (sharing a single line). The queue capacity may be finite or infinite. In a finite queue, people may be unable to join the queue if it is full.

Arrivals and servers are each modeled by a distribution over their rates.

Arrivals are typically modeled by a Poisson distribution with rate $\lambda$ (e.g., 4 per hour).
Service times are typically modeled by an exponential distribution with rate $\mu$ (e.g., 5 per hour). The mean service time is then $\frac{1}{\mu}$ .

In the following, we will assume that arrivals and servers follow a Poisson and exponential distribution, respectively. These two distributions are memoryless, meaning that arrivals are independent of one another, as are service times. Note that if we have an infinite queue capacity and $\lambda > \mu$ , then the queue will grow indefinitely.

The queueing discipline determines which person in line gets serviced next. An example queueing discipline is first-in, first-out (FIFO), where people are serviced in the order in which they arrive. Steady-state behavior is often the same regardless of queueing discipline. In the following, we will assume a FIFO queueing discipline.

Notation

Queues can be specified in Kendall's notation. The notation is A/S/c/K/N/D, where

A = arrival process, with "M" denoting a memoryless process.
S = service time distribution, with "M" denoting a memoryless distribution of service time.
c = number of servers
K = queue capacity
N = population size (total population from which arrivals are drawn)
D = queueing discipline, e.g., FIFO

For example, a M/M/1/∞/∞/FIFO queue is a memoryless queue with 1 server, infinite queue capacity, infinite population, and FIFO queueing discipline. If unspecified, we assume K/N/D is ∞/∞/FIFO. For example, M/M/1 = M/M/1/∞/∞/FIFO.

Single server, infinite queue (M/M/1)

Let's consider the case of a queue with infinite capacity and a single server. Assume that arrivals are Poisson distributed with rate $\lambda$ and service times are exponentially distributed with rate $\mu$ .

Distribution of queue lengths

In a Poisson distribution, no two events occur at the same instant. Let $h$ be an infinitesimal interval in which exactly 0 or 1 event occurs.

Let $p_n(t)$ to be the probability that $n$ people are in the system (waiting in line or being serviced) at time $t$ . Then

$\begin{aligned} p_n(t+h) =\; &P(\text{1 arrival, 0 service}) \cdot p_{n-1}(t)\\ + &P(\text{0 arrival, 1 service}) \cdot p_{n+1}(t)\\ + &P(\text{0 arrival, 0 service}) \cdot p_n(t). \end{aligned}$

As $h$ is defined to be the interval in which at most 1 event occurs, there is no $P(\text{1 arrival, 1 service}) \cdot p_n(t)$ term.

In a Poisson distribution, the probability of $k$ events occurring in an interval $\tau$ is

$P(k \text{ events in interval } \tau) = f(k; \tau) = e^{\lambda \tau} \frac{(\lambda \tau)^k}{k!}.$

This distribution has mean $E[k] = \sum_{k \ge 0} k f(k; \tau) = \lambda \tau$ . As we have defined $h$ to be an interval with at most 1 event,

$P(1 \text{ event}) = 0 \cdot P(0 \text{ events}) + 1 \cdot P(1 \text{ event}) = E_{f(k; h)}[k] = \lambda h$

Therefore

$\begin{aligned} p_n(t+h) =\; &\lambda h \cdot (1 - \mu h) \cdot p_{n-1}(t)\\ + &(1 - \lambda h) \cdot \mu h \cdot p_{n+1}(t)\\ + &(1 - \lambda h) \cdot (1 - \mu h) \cdot p_n(t). \end{aligned}$

To first order in h,

$p_n(t+h) \approx \lambda h p_{n-1}(t) + \mu h p_{n+1}(t) + (1 - \lambda h - \mu h) p_n(t),$

thus

$\frac{p_n(t+h) - p_n(t)}{h} = \lambda p_{n-1}(t) + \mu p_{n+1}(t) - (\lambda + \mu) p_n(t).$

At steady state, we expect a unchanging distribution of queue lengths, so

$0 = \frac{p_n(t+h) - p_n(t)}{h} \implies \lambda p_{n-1} + \mu p_{n+1} = (\lambda + \mu) p_n. \qquad{(1)}$

When the queue is empty, no service occurs ( $P(\text{0 service}) = 1$ ), so

$\begin{aligned} p_0(t+h) &= P(\text{0 arrivals, 1 service}) \cdot p_1(t) + P(\text{0 arrivals, 0 service}) \cdot p_0(t)\\ &= (1 - \lambda h) \cdot \mu h \cdot p_1(t) + (1 - \lambda h) \cdot 1 \cdot p_0(t)\\ &\approx \mu h \cdot p_1(t) + (1 - \lambda h) p_0(t). \end{aligned}$

Hence, at steady state,

$0 = \frac{p_0(t+h) - p_0(t)}{h} = \mu p_1(t) - \lambda p_0(t) \implies \mu p_1 = \lambda p_0. \qquad{(2)}$

By (1) ( $n=1$ ) and (2),

$\begin{aligned} &\lambda p_0 + \mu p_2 = (\lambda + \mu) p_1 = \lambda p_1 + \mu p_1 = \lambda p_1 + \lambda p_0\\ &\implies p_2 = \frac{\lambda}{\mu} p_1 = \left(\frac{\lambda}{\mu}\right)^2 p_0. \end{aligned}$

Let $\rho = \frac{\lambda}{\mu}$ . By a similar argument, $p_n = \rho^n p_0$ for all $n \ge 1$ . As this is a probability distribution,

$1 = \sum_{n \ge 0} p_n = p_0 \sum_{n \ge 0} \rho^n = p_0 \cdot \frac{1}{1-\rho} \implies p_0 = 1 - \rho.$

Therefore

$p_n = \rho^n p_0 = \rho^n (1 - \rho). \qquad{(3)}$

Expected queue length and wait time

Using the distribution of the number of people in the system (3), we can derive the expected number of people in the system (recall that $\rho = \frac{\lambda}{\mu}$ ):

$\begin{aligned} L_s = E_p[n] &= \sum_{n \ge 0} n \rho^n (1-\rho)\\ &= (1-\rho) \rho \sum_{n \ge 1} n \rho^{n-1}\\ &= (1-\rho) \rho \frac{\mathrm{d}}{\mathrm{d}\rho} \sum_{n \ge 1} \rho^n = (1-\rho) \rho \frac{\mathrm{d}}{\mathrm{d}\rho} \frac{1}{1-\rho}\\ &= (1-\rho) \rho \frac{1}{(1-\rho)^2} = \frac{\rho}{1-\rho}. \end{aligned}$

The expected number of people being serviced is the expected proportion of time the system is nonempty, i.e., $1 - p_0 = 1 - (1-\rho) = \rho$ . Thus the expected number of people waiting to be serviced is $L_q = L_s - \rho$ .

The expected time spent in the system (including time being serviced) is related to the queue length by Little's law: $L_s = \lambda W_s$ . A similar relation applies to the expected time spent waiting to be serviced: $L_q = \lambda W_q$ .

The utilization is the probability that the server is busy:

$U = 1 - p_0 = 1 - (1 - \rho) = \rho.$

If we plot the service time (i.e., latency) with respect to utilization, we see that it begins increasing rapidly around $U=0.8$ :

import matplotlib.pyplot as plt
import numpy as np

rho = np.linspace(0, 1, 100)
latency = 1/(1-rho)

plt.figure(figsize=(9, 6))
plt.ylim(0, 10)
plt.xlabel('utilization')
plt.ylabel('latency')
plt.plot(rho, latency)
plt.savefig('mm1-latency.png')
plt.show()

Single server, finite queue (M/M/1/N)

The recurrence for $p_n = \rho^n p_0, n \le N$ remains the same, and we can use it to compute $p_0$ :

$\begin{aligned} 1 = \sum_{n=0}^N p_n = \sum_{n=0}^N \rho^n p_0 = p_0 \cdot \frac{1-\rho^{N+1}}{1-\rho}\\ \implies p_0 = \frac{1-\rho}{1-\rho^{N+1}}. \end{aligned}$

With this, we can compute the expected number of people in the system:

$\begin{aligned} L_s &= \sum_{n=0}^N n p_n = \sum_{n=0}^N n \rho^n p_0 = p_0 \rho \sum_{n=0}^N n \rho^{n-1} = p_0 \rho \frac{\mathrm{d}}{\mathrm{d}\rho} \sum_{n=0}^N \rho^n\\ &= p_0 \rho \frac{\mathrm{d}}{\mathrm{d}\rho} \frac{1-\rho^{N+1}}{1-\rho} = p_0 \rho \cdot \frac{1 + N \rho^{N+1} - (N+1) \rho}{(1-\rho)^2}. \end{aligned}$

A similar calculation may be performed for $L_q$ , or we can reason through it as follows: As people are unable to join the queue when it is full, there is an effective arrival rate $\lambda_\text{eff} < \lambda$ . A person may only join the queue if there are fewer than $N$ others in the system, which happens with probability $1-p_N$ . Thus $\lambda_\text{eff} = \lambda (1 - p_N)$ . From this, we can compute the expected queue length, service time, and wait time:

$L_q = L_s - \frac{\lambda_\text{eff}}{\mu}, \qquad W_s = \frac{L_s}{\lambda_\text{eff}}, \qquad W_q = \frac{L_q}{\lambda_\text{eff}}.$

Continued in part 2...