Blog on blog.gnoack.org

Landlock Path Walk Inversion Experiment

Fri, 08 May 2026 21:51:04 +0000

Objective

Landlock supports multiple nested sandboxes, so when an operation with filesystem path is attempted, it has to check it against up to 16 nested policies.

The way it currently does that is by constructing a matrix of the requested access rights per layer and checking off these matrix entries during the path walk. This is efficient, but it also increases code complexity and I have long had a nagging doubt about whether it is worth the tradeoff.

After all, the mental model for using Landlock is that each layer gets checked independently from the next. A more “natural” way to implement this would be to:

loop over the layers first, and
then do the path walk inside (multiple times).

To be clear, my confidence that this would be acceptable performance-wise was always low, but then again, performance can sometimes be counterintuitive, and so it seemed like it might be worthwhile exploring.

The refactoring would be a bigger change, but luckily we have a very comprehensive suite of kernel selftests in Landlock – so at least for the sake of doing that experiment, it was feasible to vibe code the refactoring. (The code is absolutely not meant to be submitted like that, but it is good enough to get performance numbers of the general idea.)

The experiment failed – it worsened performance noticeably. I’m publishing it anyway, so that others don’t have to attempt the same.

I also found it amazing that I can play with the performance aspects of such a large refactoring without implementing it myself. ✨ LLMs are underused for such experiments.

Hypothesis

The (highly speculative) hypothesis of this experiment is:

By refactoring the code to loop over the layers first, and doing the path walk inside (multiple times),

the code becomes simpler to reason about,

and keeps reasonable performance.

Implementation

I gave the refactoring task to an LLM agent. This code is not production ready and entirely unreviewed, but it does pass the very comprehensive Landlock test suite, which remained unmodified, so I have good confidence that the relevant aspects are retained.

Measurements

Effects on Landlock code complexity

The measurement was done using:

wc -l security/landlock/*.[ch]

The lines of code in Landlock went down from 7245 (2006 in fs.c) to 6815 (1726 in fs.c) lines.

The usual caveats apply:

Lines of code are a poor measurement of actual complexity.
It might have deleted some KUnit tests along the way. Whether that is legitimate due to simplification is a subjective question.

But it’s at least an indication that it does indeed simplify the code.

The full refactoring and benchmarking code are kept on a Git branch at https://github.com/gnoack/linux/tree/landlock-invert.

Effects on Performance

This is measured using an improved version of fs_bench.

fs_bench benchmarks how long it takes to check Landlock policies during open(), and it does so in an intentionally very-bad-case scenario, namely one where the opened file is in a deeply nested filesystem location from the root, and where the operation is in the end not permitted, so that Landlock has to walk the path all the way up to the root.

This is chosen to amplify the effect and to make it more measurable. In more practical scenarios, the number of nested Landlock domains is likely to be low (e.g., 1), and the number of nested directories is also low (e.g. <10).

The resulting performance graphs show that independent of the number of nested directories, adding more nested Landlock sandboxes does not scale well after the refactoring attempt.

Noteworthy results:

The refactoring attempt does not scale well when adding more Landlock sandboxes.

Even with a small and more realistic directory nesting depth of only 10, the performance degradation of doing the path walk twice when adding a second Landlock layer is clearly visible.

In the case of only one layer, the performance improved slightly, presumably because the code is just simpler now, but this only becomes properly measurable at a high number of nested directories. In the more realistic scenario with only 10 directories, the effect is not measurable with the clock resolution.

Directory nesting depth	before refactoring (clocks)	after refactoring (clocks)
10	6	6
100	32	29 (90%)
1000	469	384 (82%)
10000	6087	4676 (77%)

Conclusion

❌ The refactoring was a bad idea and it reduces performance.

The current path walk implementation with the layer matrix performs better, even in scenarios with small numbers.

Full results

Kernel A: before (benchmarking/bzImage-before)
Kernel B: after (benchmarking/bzImage-after)

Values are “System” clock ticks from times(2) (smaller = faster).

fs_bench

Scenario	before	after	Δ (B − A)	B/A
fs_bench / depth=10000, baseline	6	6	0	1.000
fs_bench / depth=10000, 1 domains	6087	4676	-1411	0.768
fs_bench / depth=10000, 2 domains	6072	9502	3430	1.565
fs_bench / depth=10000, 4 domains	6281	19933	13652	3.174
fs_bench / depth=10000, 8 domains	6452	44484	38032	6.895
fs_bench / depth=10000, 16 domains	7007	99500	92493	14.200
fs_bench / depth=1000, baseline	4	4	0	1.000
fs_bench / depth=1000, 1 domains	469	384	-85	0.819
fs_bench / depth=1000, 2 domains	453	731	278	1.614
fs_bench / depth=1000, 4 domains	506	1867	1361	3.690
fs_bench / depth=1000, 8 domains	530	3826	3296	7.219
fs_bench / depth=1000, 16 domains	595	8684	8089	14.595
fs_bench / depth=100, baseline	3	4	1	1.333
fs_bench / depth=100, 1 domains	32	29	-3	0.906
fs_bench / depth=100, 2 domains	34	59	25	1.735
fs_bench / depth=100, 4 domains	37	123	86	3.324
fs_bench / depth=100, 8 domains	40	299	259	7.475
fs_bench / depth=100, 16 domains	47	678	631	14.426
fs_bench / depth=10, baseline	4	3	-1	0.750
fs_bench / depth=10, 1 domains	6	6	0	1.000
fs_bench / depth=10, 2 domains	6	9	3	1.500
fs_bench / depth=10, 4 domains	7	15	8	2.143
fs_bench / depth=10, 8 domains	7	30	23	4.286
fs_bench / depth=10, 16 domains	8	61	53	7.625

net_bench

Scenario	before	after	Δ (B − A)	B/A
net_bench / baseline	49	49	0	1.000
net_bench / 2 domains	9	9	0	1.000
net_bench / 4 domains	8	9	1	1.125
net_bench / 8 domains	9	8	-1	0.889
net_bench / 16 domains	10	9	-1	0.900

scoped_bench

The “scoped_bench” ran as well, but the results were all zero. (I have not investigated this further; net_bench and scoped_bench were not expected to take a big performance hit from the refactoring.)

Shell Tricks: Extract text between two regexes

Fri, 08 May 2026 19:15:34 +0000

To extract text between two regular expressions, use sed -n '/first_regex/second_regex/p'.

For instance, to ad-hoc extract the function definition of the test tsync_override_log_subdomains_off, use:

sed -n '/TEST_F(audit, tsync_override_log_subdomains_off)/,/^}/p' \
    tools/testing/selftests/landlock/audit_test.c

This assumes that the function ends with a single } on an individual line. But that tends to be a given.

awk offers similar features. This invocation finds the definitions of static functions with report_fixup in their names, and prefixes these findings with the filenames:

awk '/^static.*report_fixup/,/^}/ { print FILENAME, $0; }' *.c

But well, awk is awkward.

Landlock Multithreaded Policy Enforcement

Mon, 13 Apr 2026 18:44:37 +0000

With Linux 7.0, Landlock gains the new LANDLOCK_RESTRICT_SELF_TSYNC feature.

With this new flag to the landlock_restrict_self(2) system call, the Landlock policy enforcement is applied to the entire process rather than just the calling thread. (The naming is analogous to the similarly named SECCOMP_FILTER_FLAG_TSYNC flag for Seccomp-BPF.)

The old workaround

This works around Landlock’s need for libpsx in multithreaded environments. Libpsx is a user-space library which uses low-level trickery to make every thread call the Landlock system call directly. This worked to some extent, but had the drawback that with that, all the different threads were enforcing the same policy, but it still resulted in technically distinct Landlock sandboxes - and that makes a difference in some corner cases.

Programming language support

C

Simple C programs which enforce Landlock at the start of main() won’t need this feature, but it can be useful when you are using Landlock in scenarios that are already multithreaded, like multithreaded frameworks or implicitly multithreaded programming languages.

Go

The Go-Landlock library moves to version 0.8.0 and offers support for this new feature. To update, update your dependency on Go-Landlock to version 0.8.0.

No change in API usage is needed. As Go is inherently multithreaded, Landlock policy enforcements through Go-Landlock will automatically use it if the kernel supports it.

References

More details are in:

https://github.com/landlock-lsm/go-landlock/issues/53 - Go-Landlock bug
https://wiki.gnoack.org/LandlockMultithreadedEnforcement - kernel implementation notes

For the curious, I’ll give a talk about this feature at the upcoming Go meetup in Zurich on 2026-04-23.

Nutella is in space

Thu, 09 Apr 2026 20:35:42 +0000

The big question for science is: Does it stay on the ground in the jar, or does it form floating lumps?

Parameterized testing: You want Inputs and Outputs

Wed, 11 Feb 2026 23:58:31 +0000

Is your test in the business of calculating the expected output from the inputs?

Then your test is duplicating the logic from the code under test.

There is a reason why too much logic in tests is frowned upon: If you implement the same logic twice, you are prone to repeat the same mistakes in the test which you already made in the real implementation.

for c in [
  Case(socket_type=SOCK_STREAM, fruit="Apple"),
  Case(socket_type=SOCK_STREAM, fruit="Orange"),
  Case(socket_type=SOCK_DGRAM,  fruit="Apple"),
  Case(socket_type=SOCK_DGRAM,  fruit="Orange"),
]:
  # Oh no, surprise logic in the test!
  # This becomes hard to find when the test gets longer.
  expected = 1
  if c.socket_type == SOCK_STREAM and c.fruit == "Orange":
    expected = 0  # Expecting nothing in this case

  self.assertEqual(expected, operation(c.socket_type, c.fruit))

Better is to flatten out the expected results into the test table and remove that logic from the test:

for c in [
  Case(socket_type=SOCK_STREAM, fruit="Apple",  expected=1),
  Case(socket_type=SOCK_STREAM, fruit="Orange", expected=0),  # special case
  Case(socket_type=SOCK_DGRAM,  fruit="Apple",  expected=1),
  Case(socket_type=SOCK_DGRAM,  fruit="Orange", expected=1),
]:
  self.assertEqual(c.expected, operation(c.socket_type, c.fruit))

A non-trivial real-world example (from the Go-Landlock library).

This is one of these articles that I’m writing just so that I can point to it later. In a code review. :)

She sells Z-Shells at the Z-Shore

Wed, 11 Feb 2026 23:49:43 +0000

Mickaël’s FOSDEM talk about the Island sandboxing tool:

https://fosdem.org/2026/schedule/event/EW8M3R-island/ (with 📽️ video!)

The Island sandboxing tool comes with tight integration with the zsh, to make sandboxing a part of the everyday workflow.

When you use that integration, the directory that you are in determines the Landlock policy which is then applied to the commands that you run.

🐚

Martin Luther King Day

Mon, 19 Jan 2026 20:39:26 +0000

The time is overdue. Stevie Wonder is amazing. 🎉

FOSDEM 2026: It is happening!

Sun, 18 Jan 2026 23:20:25 +0000

I will be at FOSDEM 2026 this year, with a hunger for Belgian waffles and a big bag of freshly printed Landlock stickers.

I am looking forward to meeting you all again there!

If you are dropping by, and want to chat in person, send me a note!

I am collecting links over at https://wiki.gnoack.org/FosdemTwentySix. (You might want to double check regarding a potential railway strike on the 30th.)

Google Big Sleep: Linux Vulnerabilities

Tue, 06 Jan 2026 19:45:46 +0000

I am excited to share that my team at Google, Google Big Sleep, has published the reports for a few (now fixed) Linux kernel vulnerabilities:

The full public list of issues discovered by Google Big Sleep is at https://goo.gle/bigsleep.

Lore links in mutt

Wed, 24 Dec 2025 11:21:34 +0000

The following script takes an email on stdin and constructs the associated lore.kernel.org link based on the email’s Message-ID header. The script copies the link to the Wayland clipboard and posts a notification.

The escaping story is not perfect, but at least it is secure. (It only supports the most common characters in the message ID, but then it also can’t accidentally be misused for escaping shenanigans.)

The script email-to-lore:

#!/bin/sh
awk '
/^Message-ID: <[a-zA-Z0-9/+=.@_-]+>/ {
  match($2, /<([^>]+)/, a);
  system("wl-copy -- https://lore.kernel.org/all/" a[1] "/");
  system("notify-send --expire-time=5000 \"Copied to clipboard\" https://lore.kernel.org/" a[1] "/");
}
'

The script is registered in the mutt configuration in the following way:

macro pager l "| /path/to/email-to-lore<enter>" "Copy a Lore link to the clipboard"

When you now hit l in the mutt email view (pager), you get a desktop notification indicating that the link has been copied to the clipboard and you can use the usual shortcuts to paste it.

FYI: Existing alternatives: An alternative to this is the Python lorifier script, which adds the Lore link to the mail headers before display. lorifier is probably smarter about detecting the mailing list which the mail was posted to, whereas my awk script only generates the “all” link.

Listening on an ephemeral TCP port

Fri, 14 Nov 2025 19:26:44 +0000

The classic TCP client/server interaction

When a TCP client calls connect(2), it usually gets assigned a port number from the ephemeral port range which is not used yet.

The server, on the other hand, usually calls bind(2) to set the port that it will listen on.

Some less known fun facts

Servers that listen on random ports

While a server will usually bind(2) itself to a fixed port number, it can absolutely omit that, and then, when it calls listen(2), the server will get an ephemeral port assigned by the kernel which it will then listen on. (This is similar to setting the bound port number to 0.)

Clients that use a specific port number

If a client socket uses bind(2) before connect(2), the client can enforce using a specific port number for its own side.

Reference

The best description I found of this is in ip(7):

An ephemeral port is allocated to a socket in the following circumstances:

the port number in a socket address is specified as 0 when calling bind(2);

listen(2) is called on a stream socket that was not previously bound;

connect(2) was called on a socket that was not previously bound;

sendto(2) is called on a datagram socket that was not previously bound.

So the behaviour of bind(2) actually works the same on both client and server sockets. It’s just in daily use where it only gets used for server sockets.

Previous discussion on the kernel mailing lists

Integer overflow checking with C23

Sun, 02 Nov 2025 20:48:02 +0100

While I wasn’t looking, C23 has standardized checked integer arithmetic functions, replacing the old “__builtin_mul_overflow()” compiler intrinsics that shipped with GCC and Clang.

#include <stdckdint.h>
bool ckd_add(type1 *result, type2 a, type3 b);
bool ckd_sub(type1 *result, type2 a, type3 b);
bool ckd_mul(type1 *result, type2 a, type3 b);

The operations are performed equivalently to doing them in the mathematically reasonable way (as normal integers in ℤ, not modulo-something), and then truncating them to fit into the *result integer. The function returns true if the resulting value could not be represented in *result’s integer type.

To check for overflow of two integers a and b, define a variable for the calculation result and do:

int result;
if (ckd_mul(&result, a, b)) {
  errx(1, "a * b overflow or wraparound");
}

These semantics remove having to worry of the unintuitive difference between (undefined) integer overflow and (well-defined) unsigned integer wraparound. (The “arithmetic” ways for checking overflow/wraparound differ substantially for signed and unsigned integers, otherwise.)

The feature is available in: GCC 14+, Clang 18+, both released in Q2 2024.

Further references

The Integer section of the SEI C Coding Standard wiki has some excellent guidance for correct overflow and wraparound checks, also for older compilers. Specifically, these sections are:

David Svoboda, who is involved here, is the same person who proposed the extension to the C23 standard.

Jens Gustedt’s Blog has a longer post on the topic as well with more practical examples and a discussion of how to emulate the new interface based on the older macros that GCC and LLVM offered before C23.

The __builtin_mul_overflow() macro and friends were introduced in GCC 5 (2015) and Clang 3.4 (2014).

Landlock Your Vibe Coding

Sat, 02 Aug 2025 20:08:48 +0200

We’ve all heard the horror stories where an unsandboxed coding agent deletes the user’s entire home directory. As a mitigation, many vibe coding tools are now using Docker to contain the agent’s actions.

Unfortunately, Docker is not a good tool to keep the vibe coding agents at bay. To quote Docker’s own documentation (emphasis mine):

The docker group grants root-level privileges to the user. For details on how this impacts security in your system, see Docker Daemon Attack Surface.

So while it keeps the agent from doing harm outside of the Docker container, installing Docker also removes the security boundary between your normal user and root. Docker is not something that your normal user account should have access to.

So as a stop-gap measure, for systems where you don’t want to install Docker, here is a little ad-hoc sandboxing script which runs Google’s gemini-cli in a Landlock sandbox:

#!/bin/sh

RDIR=execute,read-file,read-dir

# A reasonable subset of write accesses
RWDIR=$RDIR,write-file,remove-dir,remove-file,make-dir,make-reg,make-sock,make-fifo,make-sym,refer,truncate

if [ "$PWD" = "$HOME" ]; then
    echo "Run this from your project directory only"
    exit 1
fi

mkdir -p $HOME/.gemini  # ensure this exists, no-op if it preexists

setpriv \
  --landlock-access fs \
  --landlock-rule path-beneath:$RWDIR:$HOME/.gemini \
  --landlock-rule path-beneath:$RDIR:/etc \
  --landlock-rule path-beneath:$RDIR:/bin \
  --landlock-rule path-beneath:$RDIR:/usr \
  --landlock-rule path-beneath:$RDIR:/lib \
  --landlock-rule path-beneath:$RWDIR:$PWD \
  /usr/bin/gemini

The tool used here it setpriv(1) from the util-linux package.

Obligatory warning: This Landlock invocation restricts access to the file system: It can keep an agent from reading your SSH keys and from seeing of modifying the content of your files outside ~/.gemini and the current directory. It does not restrict access to networking, Unix signals and a variety of other things. It keeps your files safe, but you still should not store your prod database credentials within the agent’s reach.

Argument passing adventures

Fri, 02 May 2025 13:54:24 +0200

When I posted about the Emacs mouse bug in February, I looked into the patch set again, and of course spotted more mistakes. This patch simplifies and corrects some inconsistent logic in where CAP_SYS_ADMIN was required. The change does not make a difference for existing terminal mouse driver programs like GPM or Consolation, but it is noteworthy for the coding aspect:

Fantastic new ways of passing arguments

In the kernel code, the decision whether to apply CAP_SYS_ADMIN is made by looking at an “enum” value (sel_mode) which defines the operation to be run, in a struct which also carries arguments:

struct {
    char  subcode;
    short xs, ys, xe, ye;
    short sel_mode;
};

And in most cases, sel_mode was also used like an enum.

However, at some point someone added the operation TIOCL_SELMOUSEREPORT and found that it required an additional argument. But the existing argument struct did not have space for it! What to do?

Instead of implementing the operation with a different argument struct, the implementers then presumably figured that it would be easier to implement if they re-used the existing struct. So they made TIOCL_SELMOUSEREPORT the number 16 (in binary: 10000) and then used the lower 4 bits of the same enum as an additional argument. (It’s the 90’s. Go for it!)

None of this got documented either (Chaos is the price of freedom), so I also ended up submitting documentation for it to the TIOCLINUX man page, in the section about TIOCLINUX’s TIOCL_SETSEL subcode (highlight mine):

TIOCL_SELMOUSEREPORT

Make the terminal report (xs, ys) as the current mouse location using the xterm(1) mouse tracking protocol (see console_codes(4)). The lower 4 bits of sel_mode (TIOCL_SELBUTTONMASK) indicate the desired button press and modifier key information for the mouse event.

[…]

All of these features are luckily highly obscure and seldomly looked at – I really hope that no one will actually need this documentation again, but if they do, it’s documented now.

What’s the takeaway here?

Admittedly, I should have read the kernel code better and should not have tripped over this. But also, in my defense, people are trained to recognize conventional programming patterns. If code looks conventional on the surface but deviates in surprising ways, it becomes easy to miss. Don’t do this.

Along the way I also wrote up my understanding of Linux’s terminal mouse support. (I should note that I am not a terminal driver expert, and it’s possible that I am misunderstanding parts of it.)

The patch is now in all stable kernels (6.12.26+, 6.14.5+). Versions before 6.7 are unaffected.

I broke Emacs mouse support on the console

Wed, 12 Feb 2025 17:11:09 +0100

I accidentally broke Emacs mouse support on the Linux console. o_O

The TIOCLINUX patch for disabling dangerous IOCTLs for the Linux console driver (background discussed on the Wiki) ended up accidentally making the mouse cursor invisible on the Linux console (the proper text mode one, not xterm).

I apologize for breaking this. We luckily found a patch for it and it is now rolling out to stable Linux kernels.

If your mouse does not work as expected in Emacs on the Linux console, please update your kernel to a newer release.

It boggles me that this managed to slip through the cracks in the initial review, after we had done such an exhaustive search for remaining users in Debian code search. Lesson learned.

References

Git without a hosted platform

Wed, 05 Feb 2025 21:16:03 +0100

People have apparently started believing that in order to use git, you’d have to also have to host the project on Github or Gitlab:

A few weeks ago, a friend told me that he keeps his pass password repository on Github.
Beej wrote a guide on git and the Github workflow is very prominent in it
I came home from FOSDEM with a Github and a Gitlab sticker, but not with one for git send-email or cgit.

I feel that is is underreported that you can easily host Git repos on a normal Unix account. I use this all the time myself for my personal projects – I maintain far more repositories locally over SSH than I expose on hosted platforms. This is not visible on the public internet, but maybe that makes it especially mention-worthy.

How it’s done

All you need is an account on a Unix/Linux machine, which you can reach over SSH.

On the remote side, set up the repo:

mkdir -p ~/repos/foobar.git
cd ~/repos/foobar.git
git init --bare

On the local side, set up the remote:

git remote add foobar user@hostname:repos/foobar.git
git push --set-upstream foobar main

There is not much more to it. Now you can push and pull.

A more detailed description that also covers SSH keys can be found in the Git Book.

What if I need more?

The approach scales and extends to bigger endeavors as well.

Web UI: You can set up cgit on a web server to render your repositories.
Code review: Some projects do code reviews over git send-email. The Linux kernel is known for this and scales up to thousands of developers.

If a project grows and you want to work with people who can’t wrap their heads around this, you can still fall back to a hosted platform after the fact and push your repository there later.

Dan Bernstein on sandboxing

Thu, 23 Jan 2025 22:32:27 +0100

Thoughts on sandboxing, found via Aaron Swartz’s weblog, who in turn found it in a writeup about Dan Bernstein’s 2005 research activities (at the end):

The software installed on my newest home computer contains 78 million lines of C and C++ source code. Presumably there are hundreds of thousands of bugs in this code. Removing all of those bugs would be quite expensive. Fortunately, there’s a less expensive way to eliminate most security problems.

Consider, for example, a compressed music track. The UNIX ogg123 program and underlying libraries contain 50000 lines of code whose sole purpose is to read a compressed music track in Ogg Vorbis format and write an uncompressed music track in wave format.

UNIX makes it possible (though unnecessarily difficult) to build a safeogg program that does the same conversion and that has no other power over the system. Bugs in the 50000 lines of code are then irrelevant to security: if the input is from an attacker who seizes control of safeogg, the most the attacker can do is write arbitrary output, which is what the input source was authorized to do anyway.

I’m sharing this here because much of the reasoning behind Landlock traces its roots back to the same train of thoughts (and also because Dan Bernstein lais down the reasoning very nicely).

Choice of sandboxing mechanisms

This is 2005, so the UNIX mechanisms that he suggests for sandboxing are:

chroot(2) to limit access to the filesystem (requires root, or CAP_SYS_CHROOT on Linux)
using a temporary unique UID to limit access to other processes (requires root)
setrlimit(2) with a hard RLIMIT_NOFILE of 0 to limit access to the network
setrlimit(2) with other resource limits to limit RAM resource use

(Unfortunately, some of these require higher privileges than what normal programs run as, a property that these mechanisms also share with classic Linux Security modules like AppArmor and SELinux.)

One thing where Landlock’s approach differs slightly is that we believe that the mechanisms for sandboxing should be available to unprivileged processes so that they become maximally useful – software authors for general purpose software generally can not expect that their software runs with high privileges, and neither should they – it is absurd if a process needs higher privileges in order to drop privileges. (Seccomp was an attempt at a Sandboxing mechanism that did not require higher privileges, but Seccomp had other practical issues).

Sandboxing should be a developer task

But the overlap to Landlock’s approach is very large – especially when it comes to the idea that the use of the sandboxing mechanism should be integrated into a program’s design: It becomes the programmer’s responsibility to design the sandbox, and this results in narrower sandboxes:

I should emphasize here that there’s a fundamental difference between this project and typical sandboxing projects described in the literature. The goal of a typical sandboxing project is to apply as many restrictions as possible to a program while receiving no help from the programmer; the problem is that this doesn’t stop all attacks.¹ In contrast, I insist on an extreme sandbox guaranteeing complete security, and I then ask how the programmer’s time can be minimized. As in Section 1, programs are not static objects; the programmer cooperates with the sandboxing tools.

Networking

Related to that, Dan Bernstein has also made the proposal in the past that it should be possible to disable the network with a dedicated disablenetwork(void) function.

In 2009-2010, Michael Stone sent a Linux patchset for that. That patchset unfortunately did not make it, but it will hopefully eventually be possible with Landlock! – Mikhail Ivanov’s “Support socket access-control” patch set for Landlock should make it possible to forbid new network connections in most cases. (There are tiny exceptions; for the details, see my presentation on it starting around minute 34.)

I continue to be very excited for this feature to land. It’ll let users define reasonably detailed policies for what kinds of networking you still want to use, but one of the largest use cases continues to be the case where a program absolutely does not want to do any networking.

Conclusion

I expect this strategy to produce invulnerable computer systems: restructure programs to put almost all code into extreme sandboxes; eliminate bugs in the small volume of remaining code. I won’t be satisfied until I’ve put the entire security industry out of work.

We’re getting there. 🚀

Again, the full writeup from 2005 is available at https://cr.yp.to/cv/activities-20050107.pdf and worth a read.

https://www.usenix.org/publications/library/proceedings/sec96/goldberg.html, for example, described a sandboxing tool that applied some restrictions to Netscape’s “DNS helper” program. The subsequently discovered libresolv bug was a security hole in that program despite the sandbox. Imposing heavier restrictions would have meant changing the program. ↩︎

Upcoming conferences 2025 Q1

Fri, 17 Jan 2025 07:44:45 +0100

I will be attending the following conferences:

Vintage Computer Festival Zurich, January 18-19, 2025
FOSDEM 2025, February 1-2, 2025
- I am collecting a few interesting talks over on the wiki
- Obligatory Landlock talk: Sandbox IDs with Landlock
- We have Landlock stickers this time :)

If you happen to be at one of these venues, if you are interested in sandboxing, or if you just want to chat, let me know via email. :)

Talk: Update on Landlock: IOCTL support

Sun, 13 Oct 2024 11:15:42 +0200

📢 I gave a talk about the recent changes in Landlock and its new support for restricting IOCTL usage at the Linux Security Summit Europe 2024 in Vienna:

🌍 Talk page | 🎥 Video on YouTube | 😎 We have stickers!

Talk Summary

The Landlock security module lets Linux processes restrict what they can do and puts developers in charge of defining appropriate sandboxing policies for their programs. We will give a brief overview over Landlock’s current features, recent developments, and talk about what is next. We will discuss in more detail Landlock’s new support for restricting the use of IOCTL and the design considerations and trade-offs that went into it.

In other news

I finally took the time to finish up the mathematical writeup of how Landlock’s file system access rights are composed on the wiki. All of this should be obvious from the documentation, but it can still be helpful to have a mathematical model to check against.

Git: Multirepo patch set

Tue, 10 Sep 2024 21:03:55 +0200

In the context of Alejandro Colomar stepping down as a man-pages maintainer :(, I learned from him that it is possible with git to create an email patch set that spans multiple target repositories. Specifically, he pointed to a review thread by Jiri Olsa where this was done, and whose outline takes a similar shape to this (simplified):

[PATCH proj v3 0/3] foobar: Add transmogrifier
- [PATCH proj v3 1/3] foobar: Prepare flux compensator
- [PATCH proj v3 2/3] foobar: Add transmogrifier feature
- [PATCH man v3 3/3] foobar.1: Document transmogrification

Other than in normal review threads, every mail subject is prefixed with an additional tag here, to indicate which project it belongs to.

It was not obvious to me how to best achieve it with the existing tools, so I attempted to reconstruct how he had done it, and ended up with the workflow that I will describe below. This is not the same workflow that Jiri Olsa has been using after all, but it’s close, and unusual enough that it’s maybe worth documenting here.

1. Pull both original repositories in the same local repository

We first make sure that all necessary commits are stored in the same local repository, by pulling in a second remote repository with git fetch.

This is unusual, but it is possible. As a result, your local repository contains two unrelated commit histories, for instance one for the linux kernel and one for the man pages:

You can simplify your life here a bit by using git worktrees – with these, your can keep doing your Linux work in a separate directory to your man page work, but still keep all of the changes in the same repository.

In the example above, we have prepared two commits in the linux-feature branch, based on the linux-base branch from upstream, as well as one commit in the man-feature branch, based on man-base.

Example

Starting from two repositories /tmp/git/linux-origin and /tmp/git/man-origin, let’s fetch both repositories into the same local repository and create a worktree for each:

gnoack:/tmp/git$ git clone /tmp/git/linux-origin linux
Cloning into 'linux'...
done.
gnoack:/tmp/git$ cd linux
gnoack:/tmp/git/linux(main)$ git tag linux-base
gnoack:/tmp/git/linux(main)$ git remote add man-origin /tmp/git/man-origin
gnoack:/tmp/git/linux(main)$ git fetch man-origin
remote: Enumerating objects: 12, done.
remote: Counting objects: 100% (12/12), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 12 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
Unpacking objects: 100% (12/12), 948 bytes | 948.00 KiB/s, done.
From /tmp/git/man-origin
 * [new branch]      main       -> man-origin/main
gnoack:/tmp/git/linux(main)$ git worktree add /tmp/git/man man-origin/main
Preparing worktree (detached HEAD efbfa45)
HEAD is now at efbfa45 4

We now have a worktree for the man pages at /tmp/git/man and one for Linux at /tmp/git/linux.

We now prepare the Linux and man page commits on top of these git repositories in the usual way, and make sure that we create branches for the pristine states called man-base and linux-base, and call our work branches man-feature and linux-feature.

2. Format a patch set that spans two otherwise unrelated branches

Create a giant patch set that spans two branches. Indicating the desired commit range by using dotted range notation twice.

This invocation is compatible with common flags like -v3 for the patch set version and --cover-letter.

We use --subject-prefix so that the mail subject is additionally qualified with an abbreviation for the main project.

git format-patch -v3 --cover-letter \
   --subject-prefix="PATCH proj"    \
   linux-base..linux-feature        \
   man-base..man-feature

Remember from the gitrevisions(7) man page that a revision range in Git parlance is a set of commits, which are not necessarily ordered in a single linear chain. (Somewhat surprisingly, they are also not all connected here, as you would think from reading the man page.)

git format-patch includes some additional features for controlling the ordering, but in practice it seems that commits are ordered by date if they are not connected.

3. Final hand-editing

Hand-edit the subject prefix in the man page commit’s email, so that the man page commit stands out among the others for the reviewers.

And sure enough, after git send-email, the mail thread has the expected form (in mutt):

 [PATCH proj v3 0/3] foobar: Add transmogrifier
 ├─>[PATCH proj v3 1/3] foobar: Prepare flux compensator
 ├─>[PATCH proj v3 2/3] foobar: Add transmogrifier feature
 └─>[PATCH man v3 3/3] foobar.1: Document transmogrification

Upcoming talk at LSS Europe 2024

Thu, 29 Aug 2024 22:33:24 +0200

📢 I will give a talk about the recent changes in Landlock and its new support for restricting IOCTL usage at the Linux Security Summit Europe 2024 in Vienna:

The Landlock security module lets Linux processes restrict what they can do and puts developers in charge of defining appropriate sandboxing policies for their programs. We will give a brief overview over Landlock’s current features, recent developments, and talk about what is next. We will discuss in more detail Landlock’s new support for restricting the use of IOCTL and the design considerations and trade-offs that went into it.

🌍 Link: https://sched.co/1ebVW

Landlock IOCTL control in Linux 6.10

Mon, 15 Jul 2024 21:39:21 +0200

Landlock’s IOCTL support for device files has landed in Linux 6.10 🚀.

I also merged the IOCTL support in the Go-Landlock library, so you can easily play around with it already.

In other news, I started informally documenting Landlock usage in C on my wiki on the LandlockUsage page.

LLM-generated docs are usually worthless

Fri, 01 Mar 2024 09:09:35 +0100

This has always been true, but it begs repeating:

Every docstring should provide additional information about the thing that it is documenting, which is not obvious from the thing’s name and its type information.

And the corollary:

If an LLM could generate the same docstring for you, based on the name and type information alone, then the docstring is not adding any value.

This applies independent of whether it was actually written by a LLM, or whether you yourself just mindlessly filled in the blanks without thinking.

Writing a docstring is supposed to make you think.

Opening /proc/self/fd/1 is not the same as dup(1)

Mon, 19 Feb 2024 20:45:12 +0100

Introduction

Unix processes have file descriptors which point to file descriptions (struct file in Linux). Multiple file descriptors can point to the same file description, for instance by duplicating them with dup(2), or by passing them across process boundaries using fork(2) or UNIX Domain Sockets (unix(7)).

For a long time, I was under the impression that that was also what happened behind the scenes when opening /dev/fd/${FD} (a.k.a. /proc/${PID}/fd/${FD}) on Linux. I thought I would get a new file descriptor which is also pointing to the same file description, similar to if you were calling dup(fd). This is wrong!

This feature is mis-documented

The misunderstanding is even documented in my earlier edition of “The Linux Programming Interface” (section 5.11) (but it has been fixed in newer editions, as Michael Kerrisk points out in the comments below):

Opening one of the files in the /dev/fd directory is equivalent to duplicating the corresponding file descriptor. Thus, the following statements are equivalent:
fd = open("/dev/fd/1", O_WRONLY);
fd = dup(1);

This is a reasonably simple explanation which is close enough to reality for many practical use cases, and which is true on other Unixes, but it is not fully accurate on Linux. (The book is very comprehensive and useful nevertheless.)

This RedHat bug from 2000 discusses how that behaviour was apparently changed in Linux 1.3.34. The aforementioned equivalence between the open(2) and dup(2) calls is called the “Plan9 semantics” there.

proc_pid_fd(5) gives usage examples, but does not go into a lot of detail on the exact semantics in the case of open(2).

`/dev/fd/*` behave different on other Unixes

On top of that, the behavior is implemented differently on other Unixes.

From a FreeBSD 14 box:

$ ./dup -dup > out; cat out; echo
1d
$ ./dup -proc > out; cat out; echo
1d

On FreeBSD, the result of open("/dev/fd/1", O_WRONLY); does share the same file description with the original file descriptor, as if we were calling dup(1).

Part 1: An experiment!

It turns out, opening /dev/fd/*, /proc/${PID}/fd/* or /proc/self/fd/* (proc_pid_fd(5)) results in a separate file description (struct file) being allocated for you, but it refers to the same underlying file on disk.

You can try it out with the following program:

$ cat dup.c
#include <err.h>
#include <fcntl.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>

int usage(const char *name) {
  printf("Usage: %s [-dup|-proc]\n", name);
  return 0;
}

int main(int argc, char *argv[]) {
  int fd;

  if (argc != 2) {
    return usage(argv[0]);
  }

  if (!strcmp(argv[1], "-dup")) {
    fd = dup(1);  // stdout
    if (fd < 0) {
      err(1, "dup");
    }
  } else if (!strcmp(argv[1], "-proc")) {
    fd = open("/dev/fd/1", O_WRONLY);
    if (fd < 0) {
      err(1, "open /dev/fd/1");
    }
  } else {
    return usage(argv[0]);
  }

  write(1, "1", 1);
  
  write(fd, "d", 1);
  close(fd);
}

When we build and run this program, we can see that the behavior of dup(2) and open(2) is actually different!

Duplicating the file descriptor using dup(2)

$ make dup
cc -Wall -static    dup.c   -o dup
$ ./dup -dup > out; cat out; echo
1d
$

In the dup(2) case, the struct file is actually shared – both file descriptors refer to the exact same file description. The first write(2) updates the file description’s file position (f_pos). The second write(2) uses the exact same file description, so it sees the updated file position, and the byte gets written after the one that was written before.

Duplicating the file descriptor through `/proc`

$ ./dup -proc > out; cat out; echo
d
$

In the proc_pid_fd(5) case, we see only one byte written to the output file. So there are two struct files created – and they use independent positions f_pos in the file, which are both set to 0 initially.

The first write(2) through stdout (fd 1) updates the file position from 0 to 1.
The second write(2) uses a separate file description and overwrites the byte that was previously written.

That’s why we can only see “d” in the output.

Other file types

So far, this was a bit confusing. It’s definitely inconsistent with the theory that opening /dev/fd/* does the same as dup(2). But what happens for other file types than regular files?

TCP Sockets: Can not be reopened through `/proc`

You can try this out by redirecting stdout to a socket, using the obscure /dev/tcp extension in bash¹:

$ nc -l 9999 &
[1] 4166
$ ./dup -proc >/dev/tcp/localhost/9999
dup: open /dev/fd/1: No such device or address
[1]+  Done                    nc -l 9999
$

The error here is ENXIO: No such device or address.

For sockets, the /proc/self/fd/* entry is a symlink to a name like socket:[16902].

lrwx------ 1 gnoack gnoack 64 Feb 17 23:12 1 -> 'socket:[16902]'

Pipes: Can be reopened through `/proc`

However, a pipe can be reopened through /dev/fd/1, for example like this:

$ ./dup -proc | cat ; echo
1d

…and this works even though the pipe’s symlink looks like this:

l-wx------ 1 gnoack gnoack 64 Feb 17 23:10 1 -> 'pipe:[15895]'

Part 2: What is really happening

First, let’s recall the in-kernel VFS structure:

The following things happen in a sequence:

A user space process calls open("/proc/self/fd/1")
System call handler:
- parses flags
- does the path walk, which eventually invokes proc_pid_get_link():
  - fs/proc/base.c:proc_pid_get_link():
    - invokes proc_fd_link() through a callback
      - fs/proc/fd.c:proc_fd_link(): looks up the original struct file* from the target task and returns the ->f_path that existed on that struct file (through an output pointer argument).
    - invokes nd_jump_link(), which sets the result of the path walk in nameidata to the previously set path!
- eventually calls path_openat().
  - namei.c:path_openat(): Always allocates a new struct file with alloc_empty_file()
  - namei.c:do_open(): calls vfs_open(), which in turn calls do_dentry_open()
  - open.c:do_dentry_open():
    - first initializes the file ops from the inode: f->f_op = fops_get(inode->i_fop)
    - the calls the “open” file operation: f->f_op->open

Where did the `no_open` pointer come from?

For the TCP socket above, f->f_op->open is set to the no_open function, which unconditionally returns ENXIO. So that socket can’t be reopened through /proc.

The decision which f_op->open is used for each file is done in inode.c:init_special_inode, for sockets and pipes.

Summary

Every call to open(2) results in a new struct file* being allocated.
The resulting struct file* refers to an existing inode, even for special files like pipes.
Not all of the special files support this kind of re-opening.

These /dev/tcp/... files do not actually exist: bash treats these paths specially and really just calls the BSD socket API itself… but we can use it here to write directly into a socket. ↩︎

Go-Landlock: Networking support

Wed, 10 Jan 2024 21:23:00 +0100

In Linux 6.7, Konstantin Meskhidze introduced TCP Networking support for Landlock. 🚀 I am happy to announce that Go-Landlock is one of the first libraries using it, and it has a demo tool.

Demo

If you are running Linux 6.7 or higher and have Landlock enabled, you can try it out like this:

Install the tool to ~/go/bin:

go install github.com/landlock-lsm/go-landlock/cmd/landlock-restrict-net@latest

Invoke the tool using:
```
landlock-restrict-net -tcp.bind 8080 /usr/bin/nc -l 127.0.0.1 8080
```
This invokes the command nc -l 127.0.0.1 8080 under a Landlock policy where 8080 is the only TCP port which can be bound with bind(2). The command listens on TCP port 8080, so this works.

You can see the effect of the sandbox by changing one of the two port numbers:

landlock-restrict-net -tcp.bind 8081 /usr/bin/nc -l 127.0.0.1 8080

In this case, the nc command will fail to listen on 8080, because this port can not be bound. The sandbox works. 🎉

Go API

Landlock is not only useful to sandbox other programs from the outside, but it shines when programs sandbox themselves.

The Go API for Go-Landlock’s networking support is in line with the API that Go-Landlock already provides for file system restrictions:

err := landlock.V4.BestEffort().RestrictNet(
    landlock.BindTCP(8080),
    landlock.ConnectTCP(25),
    landlock.ConnectTCP(587),
)

This will enable a Landlock policy on the calling process, where:

only TCP port 8080 can be bound with bind(2), and
only TCP ports 25 and 587 can be connected to with connect(2).

The RestrictNet() method is a new addition next to the previously existing RestrictPaths(). These methods restrict either networking or file system accesses.

The other new addition is the generic Restrict() method, which restricts both networking and file system access at the same time.

Non-TCP protocols are currently unaffected by Landlock. In particular, UDP can still be freely used independent of any enabled Landlock policies.

Public tweets are not public, haha!

Wed, 12 Jul 2023 20:38:02 +0200

Twitter

Posting: You can tweet, but the public can’t read it¹.
Reading: Most of my timeline is garbage.

Mastodon

Posting: Everything is quasi-public, and instance operators are all-knowing gods².
Reading: All of my timeline is garbage.

I’m sad for Mastodon. It would have been an opportunity to reverse the attention-grabbing engagement-through-outrage mechanisms, which have already annoyed me on other social media platforms. Somehow this did not work out.

The Blogosphere

Posting: Everything is public, and it does not try to pretend otherwise.
Reading: Readers control what they are reading through their RSS subscriptions.

And this is the winner.

Summary

I’m ditching Twitter.
I’m not joining Mastodon.
I keep operating this trusty weblog.

If you don’t have one yet, get yourself an RSS Reader app, and subscribe via RSS! (see link at the top)

Yes! Reading posts requires a Twitter account now! ↩︎
Guarantees are difficult to make in a federated environment. ↩︎

The Design of Mailprint

Thu, 22 Jun 2023 22:04:56 +0200

I wrote a tool, Mailprint, for printing out nice-looking emails from Mutt and other classic Mail user agents.

(You might remember from the last article that I have recently spent some time to polish my e-mail setup for kernel development. While this is not the hip thing to do any more, some of those mails are difficult to understand, and I occasionally print them out.)

This article describes Mailprint’s internals and design philosophy. I’m fond of this approach, because it fits nicely into the UNIX environment and is reusable.

If you are interested in using Mailprint, you can find its homepage at https://gnoack.github.io/mailprint/.

Design philosophy

Orthogonal software design, in the UNIX-style, building one tool for each task, is great. This applies in particular to side-projects, because it is a way to achieve a lot more with less work.

Mailprint is a tool which could have existed in similar form in the 80s already.

It makes use of a pipeline of other tools to do its job:

GNU troff (groff) to format pages
ImageMagick’s convert to convert profile pictures from various image formats

Interface to the outside

To the outside, Mailprint is designed to be used as part of a UNIX pipeline, reading a plain text email and outputting PDF:

cat ~/.Mail/Inbox/cur/foobar123 | mailprint > out.pdf

This makes it easy to hook up Mailprint to mutt and other classic MUAs.

Internal design

⚠️ This section is not up-to-date any more.

More recent versions of Mailprint generate PDF directly with a PDF generation library and do not start additional child processes.

Internally, Mailprint generates groff source code in the “mom” dialect from the input email and feeds it through a UNIX pipeline of processing tools:

The steps implemented in Go are:

Parsing emails into email headers and bodies. (code)
Looking up the sender’s profile picture: This is also a separate Go library, picon.
Converting the email to groff source code of the “mom” flavor, which is the heart of the program. (code)

The remaining steps are accomplished by invoking external UNIX programs:

ImageMagick convert converts the discovered profile pictures from various source formats into the PDF format.
preconv belongs to the groff suite and converts Unicode characters into input that GNU troff understands.
groff finally creates the PDF from the input source.

The pipeline is invoked from Mailprint’s top-level run() function. (code)

Turning it inside-out

If you want to build a more custom Mailprint pipeline, you can also make Mailprint expose its groff intermediary format using the -output.format=mom option, and chain it with groff yourself:

mailprint -face.picon=false -output.format=mom | preconv | groff -mom -Tpdf | lpr

Lei, the Local Email Interface

Sun, 11 Jun 2023 20:13:00 +0200

Intro

When I came back to the email account where I first subscribed to LKML, it was full.

But if you are anyway using a local email client like mutt, there is a better option than subscribing via email: The lei program lets you import Linux mailing list archives into local maildirs:

The server side needs to run the public-inbox software.

Set up

First, create a directory for your maildirs:

mkdir -p .Mail/lei

Arch Linux installs lei into an unusual location. In this case, add this to .bashrc:
alias lei=/usr/bin/vendor_perl/lei

Subscription/Search management

Subscribe to various mailing lists and to the mails that go to yourself (Query Language):

# Linux Security Module list
lei q --only=https://lore.kernel.org/linux-security-module \
      --output="${HOME}/.Mail/lei/lsm"                     \
      'rt:12.months.ago..'

# Linux Man Pages list
lei q --only=https://lore.kernel.org/linux-man \
      --output="${HOME}/.Mail/lei/man"         \
      'rt:12.months.ago..'

# Mails where I am in To: or Cc:, and the surrounding threads
lei q --only=https://lore.kernel.org/all \
      --output="${HOME}/.Mail/lei/tome"  \
      --threads                          \
      'a:myemail@example.com rt:36.months.ago..'

The lei q command is not issuing a single-shot query, but it is storing the query locally as a subscription.

These subscriptions are called “searches” in lei lingo, and their results are stored in the given maildir directories.

To see all the searches you have, use:

lei ls-search

To remove an existing search, use

lei forget-search "${HOME}/.Mail/lei/lsm"  # example

Regularly: Update subscriptions/searches

To update all lei subscriptions and populate new content into the maildirs:

lei up --all

Now you are all set, and you can read the mailing lists with the MUA of your choice:

mutt -f ~/.Mail/lei/lsm

References

Thanks to derkling, who pointed me to lei!

The empty Makefile is valid

Wed, 10 May 2023 19:00:00 +0200

This is less of a “Today I learned” article, but more like a Unix trick that took me too long to realize.

Create a new directory with a hello world C program:

$ ls
$ cat > hello.c
#include <stdio.h>

int main() {
  puts("Hello, world!");
  return 0;
}

💡 The empty or non-existent Makefile is a valid makefile for this C program:

$ make hello                # build "hello"
cc     hello.c   -o hello
$ ./hello
Hello, world!

Landlock: Best Effort mode

Sun, 05 Mar 2023 10:34:09 +0100

One of Landlock’s strengths is that you can deploy the same program on multiple kernel versions, and make it use the best available sandboxing on each.

This “best effort” approach is already implemented for you in the Go-Landlock library and in the Rust Landlock library. But what if you need to implement it yourself?

Note: This article assumes previous knowledge of Landlock - you can read more about the abstract model in the Landlock File System Access Model document which I’ve written previously, or in the official Landlock documentation.

ABI versions and matching access rights

Landlock exposes its feature set in the form of a numeric Landlock ABI version.

In order to implement the fallback correctly, you need to know which file system access rights are supported in which ABI version.

The (simplified) compatibility chart as of Linux 6.2 is:

ABI	Linux	File system access rights
1	5.13	almost all of the basic file operations (compare kernel docs)
2	5.19	+`LANDLOCK_ACCESS_FS_REFER` (reparenting files)
3	6.2	+`LANDLOCK_ACCESS_FS_TRUNCATE` (truncating files)

Further below, we will define this support matrix in C.

The problem: Refer is different

The backwards compatibility works differently for “refer” than for other access rights.

Skip this section if you are only interested in the C code.

Each Landlock ABI version is characterized by three disjoint sets of file system operations: The Always Forbidden operations, the Configurable operations and the Always Permitted operations.

The Configurable operations are the ones which already have names in the landlock.h header. With increasing ABI versions, all relevant operations should become “Configurable”, in particular the “Always permitted” operations.

In ABI v1, an set of useful file system operations is configurable in Landlock, but there is also a longer list of known operations which is always permitted (Landlock can not restrict them). Finally, some more complicated interactions are always forbidden because they would interfere with Landlock’s guarantees:

When introducing ABI v2, the “refer” operation, which was previously forbidden, became configurable:

With ABI v2, it is now possible to reparent (link(2) or rename(2)) files between different directories, as long as the involved files and directories meet certain constraints about their access rights.

The behavior of “refer” is different to “truncate” and other future access rights, which fall back to Always Permitted in earlier ABI versions, and so “refer” needs a different fallback logic.

If a program needs to do a “refer” (reparent files between directories), using ABI v1 is not an option.

The canonical documentation for this interaction is the kernel documentation.

Implementation of best-effort fallback in C

To implement the fallback logic, we will (a) define the compatibility table, and then, (b) depending on whether your program needs to do file reparenting, implement a slightly different fallback logic based on that compatibility table.

This question decides which approach to use.

The question to ask is:

Does your program need to do any file reparenting (“refer”)?

File reparenting means: Creating hard links or moving files or directories, between different(!) directories (compare the kernel documentation).

Common part: ABI version compatibility table

We define a small C array to hold the file system access rights which are supported by the different Landlock ABI versions. We store one __u64 bitmask for each ABI version.

__u64 landlock_fs_access_rights[] = {
  (1ULL << 13) - 1,  /* ABI v1                 */
  (1ULL << 14) - 1,  /* ABI v2: add "refer"    */
  (1ULL << 15) - 1,  /* ABI v3: add "truncate" */
};

I’m keeping it short here for brevity, but you can also use the constants from linux/landlock.h.

Case 1: the sandboxed program does not need to reparent files

Use this approach, if you don't use link(2) and rename(2) across directory boundaries.

If the sandboxed program does not need to reparent files, the best-effort logic is to simply remove the unsupported rights from ruleset_attr.handled_access_fs.

This code needs to be inserted after you have filled a struct landlock_ruleset_attr variable, but before you create the ruleset from it:

int abi = landlock_create_ruleset(NULL, 0,
                                  LANDLOCK_CREATE_RULESET_VERSION);
if (abi <= 0) {
  /*
   * This kernel does not let us use Landlock.
   * Best effort: Give up, and do not restrict the process.
   */
  return 0;
}
if (abi > ARRAY_SIZE(landlock_fs_access_rights)) {
  /* Kernel from the future: Treat as highest known ABI version. */
  abi = ARRAY_SIZE(landlock_fs_access_rights);
}
ruleset_attr.handled_access_fs &= landlock_fs_access_rights[abi-1];

Don't forget this!

Additionally, before you add a “path beneath” Landlock rule to the ruleset, make sure that the requested access rights fit into the previously restricted handled_access_fs rights:

path_beneath.allowed_access &= ruleset_attr.handled_access_fs;

Case 2: the sandboxed program needs to reparent files

If the sandboxed program does need to reparent files, the best-effort logic works the same as above, but it has a special case for ABI version 1. In this ABI version, file reparenting does not work under Landlock at all, and so we have to give up on it.

This code needs to be inserted after you have filled a struct landlock_ruleset_attr variable, but before you create the ruleset from it:

int abi = landlock_create_ruleset(NULL, 0,
                                  LANDLOCK_CREATE_RULESET_VERSION);
switch (abi) {
  case -1:
  case 0:  /* should not happen */
    /*
     * This kernel does not let us use Landlock.
     * Best effort: Give up, and do not restrict the process.
     */
    return 0;
  case 1:
    /*
     * This kernel only supports Landlock ABI v1.
     * We need the "refer" right, but it's not supported in ABI v1.
     * Best effort: Give up, and do not restrict the process.
     */
    return 0;
  default:
    if (abi > ARRAY_SIZE(landlock_fs_access_rights)) {
      /* 
       * A kernel with Landlock suppport from the future!
       * Treat it as the highest known ABI version.
       */
      abi = ARRAY_SIZE(landlock_fs_access_rights);
    }
    ruleset_attr.handled_access_fs &= landlock_fs_access_rights[abi-1];
}

Additionally, before you add a “path beneath” Landlock rule to the ruleset, make sure that the requested access rights fit into the previously restricted handled_access_fs rights:

path_beneath.allowed_access &= ruleset_attr.handled_access_fs;

Case 3: Sometimes this, sometimes that

This is the complicated case which we also implemented in the Go and Rust Landlock libraries. These libraries figure out at runtime which of the two cases we are in, and then use the adequate approach as above.

Conclusion

In the long run, when Landlock has stabilized more, and as older kernels become less relevant, this complication will hopefully go away. Until then, I’m hopeful that most users can simply use the simple approach in Case 1 above, because their programs do not need the “refer” right.

References

I’ve previously written about this more in the abstract at: Landlock File System Access Model.
https://landlock.io/ - Official Landlock page
Landlock kernel documentation

Thank you Mickaël, for pointing out some issues in this article!

Landlock truncation support in Linux 6.2

Mon, 20 Feb 2023 21:20:00 +0100

Linus Torvalds released Linux 6.2 yesterday, and this kernel release supports restricting the truncate(2) and ftruncate(2) operations with Landlock. (The kernel patch set has more information and discussion.)

You can try this out today with the go-landlock library, which already supports this feature. To forbid file truncation when using go-landlock, update your RestrictPaths() invocation to use Landlock version 3 as follows:

err := landlock.V3.BestEffort().RestrictPaths(
    landlock.RODirs("/usr", "/bin"),
    landlock.RWDirs("/tmp"),
)

Most existing users will only need to exchange V2 for V3. When using landlock.V3 this way, file truncation is forbidden by default.

The RWFiles() and RWDirs() helpers grant the truncation right when used on a file or directory. (It comes hand in hand with the right to open files for writing.)

Why use Go-Landlock for sandboxing?

Mon, 14 Nov 2022 20:00:00 +0100

This is the article version of a talk I presented a the Zurich Gophers Meetup on 2022-10-25, with some less relevant parts shortened.

The slides are also available at https://blog.gnoack.org/talks/go-landlock.

Motivation

I started to get interested in computer security about 20 years ago. In the late 90’s and early 2000’s, buffer overflow exploits were all the rage and started to be more widely understood and researched.

This image depicts a high level overview over a common pattern of a technical attack (“exploit”) on a computer program:

The attack channel can take many forms, such as over the network, by invoking a privileged executable file, or by distributing crafted input files.

The attacker talks to the attacked process through the same channels like any other user would do. But unlike another user's input, the attacker's input is maliciously crafted to trick the attacked process into misinterpreting or wrongly validating it, and thereby doing things on behalf of the attacker that it was not originally designed to do.

This can range from exposing process-local memory (e.g. Heartbleed), to exposing the contents of files that were not intended to be exposed (e.g. directory traversal attacks), to even giving the attacker full control over the attacked process (e.g. a classic buffer overflow exploit).

Now all of this would be less critical, if the attacked process only had access to the resources that it needs for execution, but as it turns out, in the normal UNIX model, access rights are usually determined by the user that a program runs as, and that often means that these programs have access to significantly more things than they need. (Ambient Authority).

For instance, programs that you run on your desktop are going to have access to your bank documents, your (hopefully encrypted) SSH and PGP keys, your cookies, your git repositories, etc.

– so: Let’s limit this ambient access!

Philosophy

I can’t speak for Mickaël Salaün’s vision for sandboxing, but this is mine, and it aligns very well with Landlock’s approach.

First of all, I hold the belief that:

Software authors are generally well-meaning. They want their software to work, and to be secure. If provided with the right tools for securing applications, they will use them.

However, if we take a look at the adoption of unprivileged sandboxing on Linux (namely, seccomp-bpf), it shows that there are only a handful of programs making use of it – but what is the reason for that?

Which leads me to this hypothesis:

It is too difficult for software authors to confine their software to have just the access that the software needs, even though these software authors are in the best position to reason about the required scope of access.

The Landlock approach

There are two main points that I’d like to push for in Go-Landlock, which I think make it more usable for sandboxing than other approaches:

1. Make it really easy to use

The existing confinement approaches on Linux tend to be too difficult to set up or maintain, which means that they don’t get used as much as they should.

2. Make sandboxing enablement part of program initialization

This is the other main idea – it should be up to each process to enable its own sandbox.

The rough approach is:

The program initializes itself – parses flags, opens the necessary files, named UNIX sockets, etc.
The program restricts its own access (using Landlock)
The program starts processing untrusted input from potentially malicious sources

Note: If programs sandbox themselves, they can drop more permissions, because they can also drop the permissions that were required for their initialization phase.

UNIX enforces permissions when opening files, so an already opened file can continue to get used by a process, even when it does not have the permissions to open the file again.

Other operating systems

These ideas are not new - OpenBSD has demonstrated the feasability of this approach with pledge() and unveil(), which is now used in many of OpenBSD’s userland programs. Various slide decks on the topic can be found at https://www.openbsd.org/events.html.

Capsicum on FreeBSD is another unprivileged sandboxing mechanism, which is used in Chromium and other programs (see its homepage). Capsicum is a flexible capability system, but it also has a larger API surface.

Other Linux sandboxing technologies

A deeper discussion of seccomp-bpf and the various Linux Security Modules would be beyond the scope of this article.

For a discussion of the other unprivileged sandboxing mechanism, see my previous article about it on this blog.

The other Linux sandboxing mechanisms are either only available to privileged users, or they are still very difficult to set up.

How to use Go-Landlock

This diagram shows the overall approach for program initialization when using Landlock (it’s a more detailed view of the program initialization overview diagram from above):

The important parts here are:

Primarily, Landlock is a Linux kernel feature – the access policies enforced through Landlock apply to both the enforcing program, as well as all newly forked subprocesses, independent of the libraries being used to do these accesses.
The Go-Landlock library is only required to configure and enforce the Landlock policy.

The steps required to enforce Landlock are:

Step 1: Make sure your kernel supports Landlock

For development, it’s recommended to double check that Landlock is already working, so that you can try out your policies. Landlock is already enabled on many major distributions today.

On most systems, you can check whether Landlock is working using cat /sys/kernel/security/lsm.

On systems that do not have it yet:

Make sure Landlock is compiled into the kernel.
Set the lsm=landlock boot parameter (or configure it in CONFIG_LSM at build time).

For deployment, you can pick whether your program should insist on running on a Landlock-enabled kernel, or whether it should fall back to using weaker (or no) Landlock policies on older systems:

Step 2: State what file accesses you are going to do!

This is the only function invocation your program needs in order to confine itself in a Landlock policy:

landlock.V2 determines the Landlock ABI version. A higher ABI version means that more operations can be restricted. (Alternatively, you can also spell out the exact operations that you want to restrict.)
.BestEffort() is an optional call – it configures the library to gracefully degrade to weaker Landlock policies on systems that do not have the requested Landlock features.
.RestrictPaths() is the final call which enforces the rules.
The arguments to .RestrictPaths() are file system hierarchies that the process intends to still access going forward. Access to all paths other than the listed ones will be forbidden, as much as the Landlock ABI permits. (Landlock does not currently restrict all possible file system operations, but that’s eventually the goal. A more detailed look is in the appendix below.)
landlock.RODirs and landlock.RWDirs are shortcuts to identify “general read-only” operations and “general read-write operations”. This can also be configured in more detail if needed.

…and that’s it. After this invocation, your program will be unable to work with files other than the ones specified or the ones that have already been opened before.

Link to the full documentation

Examples

This section will list a few practical examples.

Image converter

This is a basic image conversion tool in the spirit of ImageMagick’s “convert”. Multimedia processing libraries are often optimized for performance and can be particularly prone to programming bugs, and we want to protect against attackers providing malicious image files as input.

In this example, the Landlock invocation happens right at the start and is landlock.V2.BestEffort().RestrictPaths():

We simply restrict to no file system access at all.
But we fall back to enforcing weaker Landlock policies or none, if it’s not supported by the system where the program is running.

Link to the full example

Web Server

This simple wiki software only needs access to the directory where the wiki pages are stored.

The Go-Landlock invocation is:

err = landlock.V2.BestEffort().RestrictPaths(
    landlock.RWDirs(*storeDir),
)

This gives the process the right to serve and edit the wiki pages, but removes the rights for all other file paths (within the limits of what is possible on the current system).

(Small side note: The net.Listen() call happens before Landlock enablement – I am using this with a named UNIX domain socket.)

Link to the full example

The Go-Landlock example tool

The Go-landlock library comes with a simple example tool, which lets you play with Landlock from bash, similar to the one in the samples/landlock subdirectory in the kernel source, but written using the Go library:

The command to install the landlock-restrict tool is:

go install github.com/landlock-lsm/go-landlock/cmd/landlock-restrict@latest

The above shell transcript starts bash under a Landlock policy where it only has access to these files:

Read access to /usr and /lib is needed for shared libraries and the bash executable itself.
Read access to /etc is required for some common configuration files.
Write access to /dev is needed for some specific files like /dev/null, /dev/stdout and others. (Could be made more specific.)
Write access to $HOME: We rewire the $HOME environment variable to a temporary directory for our subprocess and set $TMPDIR to a directory within it, so we can just grant write access to $HOME. (This trick makes it possible to use a coarser policy, but it lets the sandboxes process execute in a slightly less usual environment.)

Link to the landlock-restrict example tool

Current Landlock limitations

Some things that Landlocked processes can never do are:

They can not manipulate the file system topology (e.g. mount, pivot_root)
Landlocked processes have the NO_NEW_PRIVS flag – you can not execute suid root binaries
Restricted use of ptrace() (debugging other processes)

Also,

Landlock is in development
Some file operations are not restrictable yet
But it’s already limiting the most common ones and it’s already usable

What file system operations are restrictable?

Landlock gains features and makes it possible to restrict more file system operations. To make this observable, Landlock’s ABI is versioned. As of November 2022, the current ABI is V2.

The list of restrictable file system operations is documented in the Landlock documentation.

Warning: Not all file system operations can be restricted yet. Currently, notable exceptions are file truncation (as a form of file modification), and various ways to observe presence and metadata of files (but not reading their contents).

In future Landlock ABI versions, new operations will start to be configurable. Currently ongoing patch sets (as of November 2022) are:

File truncation (currently scheduled for inclusion in kernel 6.2)
Chmod and Chown (patch set in review)
Networking support (patch set in review)

Summary

In short, please try it out! The invocation is as simple as:

err := landlock.V2.BestEffort().RestrictPaths(
    landlock.RODirs("/usr", "/bin"),
    landlock.RWDirs("/tmp"),
)

I would love to hear your feedback!

Further Links

For further reference, here are some additional links:

Go-Landlock:
- On Github: https://github.com/landlock-lsm/go-landlock
- Docs: https://pkg.go.dev/github.com/landlock-lsm/go-landlock/landlock
Landlock kernel project:
- Page: https://landlock.io/
- Kernel doc: https://docs.kernel.org/userspace-api/landlock.html
Mailing list: https://lore.kernel.org/landlock (To subscribe, mail landlock+subscribe@lists.linux.dev)
Some more documentation:
- Landlock File System Access Model (discussing the kernel API and configurable file system operations in a more mathematical way).

The feasibility of pledge() on Linux

Sat, 16 Jul 2022 10:36:20 +0200

So, there was a post by Justine Tunney about her port of OpenBSD’s pledge() to her own libc, the Cosmopolitan libc.

She is also calling out that previous attempts at this were flawed:

There’s been a few devs in the past who’ve tried this. I’m not going to name names, because most of these projects were never completed. […] The projects that got further along also had oversights like allowing the changing of setuid/setgid/sticky bits. So none of the current alternatives should be used.

My own seccomp-scopes project which I worked on from 2016 onwards is one of these attempts, so I feel I should explain the reasons why I stopped pursuing this approach of unprivileged sandboxing, and what I think needs to get done to do it right.

At the high level, the main problem is that seccomp-bpf does its filtering at the level of system calls and software libraries generally do not give guarantees about which system calls they are using under the hood. This does not even hold for libc implementations.

You can’t predict the syscalls a program will do

Here are some ways in which glibc makes it hard to predict which system calls it will do:

glibc replaces existing uses of system calls with newer variants. A call to the open() libc function used the openat(2) syscall under the hood, and that is just one of many examples. This changes between glibc versions.
glibc initializes parts of the library on demand when they are first used, and that may involve system calls that should better be forbidden. So this initialization needs to ideally be done before enforcing seccomp.
glibc makes use of shared libraries for commonly used functionality (nsswitch.conf). Administrators can flexibly install additional ways of doing name lookups, but any attempt at reasoning about this will need to involve these shared libraries as well.

For example: If a program calls gethostbyname() for the first time, the following things happen:

It looks up /etc/nsswitch.conf to find the shared libraries that implement hostname lookup (system calls: various file accesses)
It loads these shared libraries (system calls: various file accesses, various address space manipulation syscalls)
It calls these shared libraries to do name lookup (system calls: you can’t tell anymore)

There is additionally the problem that at the system call layer, DNS lookups are indistinguishable from other UDP socket operations, so allow-listing DNS will probably allow other UDP traffic as well.

So, to summarize: Attempting to implement a pledge() like call with seccomp-bpf and independent of a specific libc is an inherently brittle approach, which involves keeping up-to-date lists of system calls on different kernel versions and architectures and their use by different libcs. The complexity and feature-richness of glibc (particularly libnss) makes this particularly difficult. Any libc-independent pledge() library would need to get updated in sync with glibc updates, or it would run the risk that glibc starts using a syscall that it doesn’t allow-list, breaking the programs that use it.

Justine Tunney’s pledge() implementation works around these problems by (a) only supporting her own, simpler, libc implementation, and (b) only supporting the x86-64 architecture. I’m really happy to see that this works well together, but I’m afraid it’s a mistake to think this implementation can be “ported” to glibc which is used for the bulk of Linux distributions.

Restricting by file path

In OpenBSD, pledge() was always path-aware, until they moved that part into the separate unveil() call.

Seccomp-bpf can only filter syscalls by their direct arguments, so the filter can see the value of the pointer to the path name, but not the path name itself in the memory referenced by that pointer.

There are more advanced techniques to inspect pointer memory, but using these safely involves separate supervisor processes or more complicated constrained ways to control what processes do – you need to take security very seriously pull that off, and doesn’t map to a call to a single C function like pledge() anymore.

Landlock promises to fix this in the future

Unprivileged sandboxing continues to be difficult on Linux, for the moment, and it’s no surprise that the main users of seccomp-bpf are either dedicated sandboxing or containerization tools, or projects where security is a major focus, like web browsers, OpenSSH or Tor. But we should not give up yet. :)

The Landlock LSM offers a better approach for unprivileged sandboxing, although it can’t currently restrict the same number of operations yet as seccomp-bpf can. Landlock can solve the above problems, because:

Landlock filters security-sensitive operations at the point when these operations are done in the kernel, not at the system call layer. This makes it architecture independent and removes the need to keep up-to-date lists of system calls.
Landlock can easily filter on file paths and other relevant in-memory properties that can not be observed by seccomp-bpf at the system call interface.

If you want to try it out, Landlock is already enabled on some Linux distributions (i.e. Arch Linux). A simple call to Landlock (using the Go library) is:

err := landlock.V1.BestEffort().RestrictPaths(
    landlock.RODirs("/usr", "/bin"),
    landlock.RWDirs("/tmp"),
)

Some further links:

Summary

As shown above, seccomp-bpf makes it more difficult than necessary to sandbox processes. It is available on a wide range of Linux distributions, but it’s currently not practical to use for the bulk of software linked to glibc, and it’s not possible to restrict operations by file path in BPF.

Landlock is not rolled out to all Linux distributions yet, and it still has some known gaps in its current version, but it has a significantly simpler API and a much simpler implementation in the kernel than what would be required in userspace to work around the problems of seccomp-bpf.

And simplicity is a great property for security features to have. I can wholeheartedly recommend having a look at it.

XKB With Sway

Thu, 02 Dec 2021 18:24:17 +0100

This explains how to configure multiple keyboards in Sway, and how to use advanced configuration when regular xkb_options are not enough.

Sway configuration

Luckily, Sway has great support for multiple keyboards with different layouts, so it nicely adjusts to whatever keyboard you have plugged in or can use separate configurations for a physical keyboard and the laptop keyboard.

First, identify the identifiers for the keyboards you have plugged in, using the following command:

$ swaymsg -t get_inputs

Look for the lines starting with Identifier: for the device you’re interested in.

Next, add one section for each of your keyboards to ~/.config/sway/config:

input "1278:32:PFU_Limited_HHKB-Classic" {
	xkb_layout "us"
	xkb_model "hhk"
	xkb_options "compose:ralt"
	xkb_capslock "disabled"
}

input "1133:49948:Logitech_USB_Keyboard" {
	xkb_file ".xkb/keymap/logitech"
}

The example above lists a HHKB (“Happy Hacking”) keyboard where right Alt acts as the Compose key. More options are documented in the man pages sway-input(5) as well as xkeyboard-config(7).

In the case where the existing XKB options are not sufficient, you need to refer to a separate XKB configuration file, as described below.

XKB configuration

To configure my Logitech keyboard, the plan was to have the key mapping reasonably close to the HHKB layout.

In my case, I have two files under ~/.xkb (the directories need to first be created): This is the ~/.xkb/keymap/logitech file:

// Derived from `setxkbmap -print`,
// added the "+unixy" part to xkb_symbols.
xkb_keymap {
	xkb_keycodes  { include "evdev+aliases(qwerty)"	};
	xkb_types     { include "complete"	};
	xkb_compat    { include "complete"	};
	xkb_symbols   { include "pc+us+inet(evdev)+unixy"	};
	xkb_geometry  { include "pc(pc105)"	};
};

This file is the output from running setxkbmap -print, but adds the +unixy part to the xkb_symbols section. This refers to the ~/.xkb/symbols/unixy file with the following content:

partial alphanumeric_keys
xkb_symbols "unixy" {
	// exchange backspace and backslash
	key <BKSL> {	[ BackSpace, Backspace ] };
	key <BKSP> {	[ backslash, bar ] };

	// caps is ctrl
	key <CAPS> {	[ Control_L ] };
	modifier_map Control { <CAPS> };

	// right alt is compose
	key <RALT> {	[ Multi_key ] };
};

Error Handling Links

Fri, 10 Sep 2021 16:54:54 +0200

This is a collection of interesting literature on the subject of error handling which I had collected to research my own blog post on the subject a while ago.

“A Philosophy of Software Design” by John Ousterhout has a significant section on error handling that I found worth a read. It takes a slightly different angle and gives some good examples on how to design programs in a way so that errors cannot happen, or only happen in the right places.
The Smalltalk-80 Blue book had the weirdest error handling mechanism: You’d call the error: method on your own object, and the interactive system pops up a dialog. Some Smalltalk errors are recoverable in unusual ways through the interactivity of the system - you can define methods on the go if your program calls a method that doesn’t exist… :)
Common Lisp has a “Conditions” system which is similar to exceptions in that it propagates up the stack, but it inserts a layer in the middle between handling (“catching”) and signaling (“raising”) exceptions. The middle layer can define recovery strategies which the handling layer can then choose from.

I found the Common Lisp stuff in the “Practical Common Lisp” book. I cannot claim that I managed to wrap my head around that one. The chapter is online at https://gigamonkeys.com/book/beyond-exception-handling-conditions-and-restarts.html . Apparently someone wrote a full book on the Common Lisp condition system recently, but I haven’t read that. https://amazon.com/Common-Lisp-Condition-System-Mechanisms/dp/148426133X

Apart from that, the books I looked through were not very helpful. I feel that the Ousterhout approach of explaining this is a useful one, talking about ways to design programs so that the complications of error handling are reduced.

Other notable “error handling philosophies” I have only some links to…

https://python.org/dev/peps/pep-0020/ The Zen of Python (“errors should never pass silently”)
The Midori error model http://joeduffyblog.com/2016/02/07/the-error-model/ (Midori was an experimental OS developed at Microsoft)
My own research culminated in this article: https://blog.gnoack.org/post/error_handling/. In hindsight, I took a much too academic and prescriptive approach there, and so it doesn’t get read much. The article might be biased towards the “deployed and monitored” kind of software.

Another thing I heard people discuss is the rule to “act on an error in only one place”, but I failed to find a canonical source for it. (It’s the rule that is most commonly violated if someone is logging the same error at multiple layers in the stack, leading to log spam.)

Sandboxing with Landlock

Fri, 16 Apr 2021 14:20:09 +0200

Linux 5.13 is going to include the unprivileged sandboxing feature Landlock, which has been in the making for many years now.

Unprivileged means that you don’t need special system-level privileges like root or CAP_MAC_ADMIN to use it, as some other Linux sandboxing mechanisms do. Philosophically speaking, you should not need additional special privileges just to drop the privileges that you already have.

How it works

With Landlock, processes can restrict themselves to only use a specified set of file paths, and it lets them control which operations are available on these paths and their subdirectories (for example opening for reading, writing, and directory operations¹).

A similar sandboxing feature has been around in OpenBSD for a while now with unveil(): This slide deck and talk by Bob Beck talks about the lessons learned and has examples on how to sandbox software with it, which should translate well to Landlock.

Download

To make it easier to play with, I forked the Landlock example tool from the kernel sources and made it compile standalone: You can get it at https://github.com/gnoack/landlockjail, or you can just download a precompiled statically linked version.

At the moment, there are still some gaps, like stat(), which can still be done even in landlocked processes. But it’s moving in the right direction, and at least accesses to the files’ contents can already be restricted. ↩︎

Writing Tests in Go

Wed, 10 Feb 2021 22:00:06 +0100

Tests should be correct by inspection

Tests require a different approach than normal code. We don't have tests for tests, so tests need to be correct by inspection -- and the main technique to achieve this is to get rid of the generality of the production code, and exercise only very narrow and specific scenarios.

Decide what test to write

The goal of testing is to increase confidence

A test's purpose is to increase the confidence that you have in your program's correctness. The next test to write is the test that promises the highest ratio of additional confidence for the work required to write it.

Do test all your business decisions

Test all your code that is testable. Prioritize testing the code that (a) is important to work correctly (where confidence needs to be high) or (b) has become too complicated (where confidence has dropped too low).

Don’t test I/O and side effects

Controversial part ahead!

Sometimes, production code will do I/O or other side effects that are hard to simulate reproducably in a test. In these cases, it can sometimes be advisable to separate the I/O parts of the production code out and only test the rest - the bulk of your logic. It's key to only separate out only the smallest possible part that has the side effect, to keep as much as possible of the program testable.

“But Günther”, I hear you saying, “what if there is a bug in these I/O parts”? Well - that’s why you need to keep them tiny and stare at them very long in a code review, to make sure they work without a test. It sounds heretic to leave this untested, but the alternative is to keep the I/O and other program logic together. That’ll easily make your more regular program logic hard to test, and that’s often not worth the trade off. After all, our goal is to increase confidence, not to reach full line coverage.

Special case: There are also some types of I/O which don’t affect your program logic, such as logging and incrementing performance counters. It’s ok to leave those untested without separating them out - these are battle proof APIs designed for quickly adding and removing them from your code, and they do not play a role for your programs correctness. It would give you little additional confidence to test them.

Structure your tests into Given/When/Then

Stick to the three test phases!

Stick with the setup/exercise/verify style of tests. Some people call this also Given/When/Then or the “Four Phase test” (counting tear-down as separate phase).

It does not matter what you call this pattern, but “Given/When/Then” tends to read nice in comments. Delimiting the section explicitly with comments is optional, but serves another useful purpose: You cannot use helper functions that span multiple of these purposes. (Those helper functions would be bad style.)

func TestSpellOut(t *testing.T) {
  // Given
  s := speller.New("en")

  // When
  got := s.SpellOut(42)

  // Then
  want := "forty-two"
  if got != want {
    t.Errorf("SpellOut(%q); got %q, want %q", 42, got, want)
  }
}

Test helpers

Test helper functions should be the primary way to share functionality between tests.

Each helper serves only one of the test phases

Make sure that each test helper helps with only one of the test phases: setup, exercise or verify.

In particular, resist the temptation to extract the whole test into a helper method, so that you can call it with differing parameters. The way this is done in Go instead is with table-driven tests and t.Run() if needed.¹

Setup helpers

The checklist for setup helpers

A setup helper is called from a test's setup phase and should have the following properties:

It accepts a testing.TB² as first parameter.
It calls t.Helper() at the start.
It may not return an error, but will call t.Fatal() on error.
- It is ok to fail early in setup helpers.
- Tests don’t need to check errors manually, making them easier to read.
If it sets up state that needs to be undone after the test, it calls t.Cleanup().

This is a helper method for setting up a Speller object from the previous example:

func setUpSpeller(t testing.TB, path string) speller.Speller {
  t.Helper()

  s, err := speller.Load(path)
  if err != nil {
    t.Fatalf("Could not set up speller: speller.Load(%q): %v",
             path, err)
  }

  t.Cleanup(s.Close)
  return s
}

Exercise helpers

Don't.

Only introduce helpers for the test's execute phase if the invocation is unbearable to read.

It’s good practice to exercise the component under test directly without such a helper. Using the component directly is a good test for its API’s usability and will often uncover small possibilities for API improvements in the process.

Verification helpers

Alternatives to assertion libraries

Common Go style discourages the use of assertion helper libraries, which are common in most other languages.

While I don’t necessarily agree with that, the next best thing you can do within the bounds of this style is to extract helpers that do the necessary comparisons for you, if necessary. The ground rule I use is: The if should be part of the top-level test function, but the condition in that check can call the helper.

These helpers do not need to have testing.TB passed in - they are just regular utility functions.
They should probably return a boolean, or other object representing the result of the check (like cmp.Diff)

General advice on writing tests

A lot of general advice from other testing frameworks is applicable to Go too.

Make each test exercise a specific narrow scenario

Each test should exercise a specific and narrow scenario. There should be little overlap between tests. Any bug introduced in the code should ideally only break a single test, which has narrow focus and is related to the broken functionality.

Start with the assertion

Start writing tests with the assertion, and work from there upwards by filling out the missing variables that aren’t defined yet. This approach helps me to focus each test around one specific and narrow behaviour that I want to test.

One test for the happy path, plus one for each error

With simple functions with inputs, outputs and errors, I like to have one main test for the happy path, plus one for each error, whose test input is based on the one for the happy path, with a modification.

See my article on Shared base fixtures for examples.

Avoid `TestMain`, use setup helper functions

If you don’t know what TestMain is, good!

TestMain is a way to share common test setup and teardown between multiple test cases, without them calling a setup helper explicitly. In most cases, this is better done with a setup helper function as described above.

There is one case where TestMain is acceptable, which is the case where the test setup is so expensive to run that it will be significantly faster to only do it once.

t.Run() is a way of structuring your test output, but it’ll also make sure that subtests can cancel with t.Fatal(), without affecting the execution of other subtests. ↩︎
testing.TB is an interface for the common methods in testing.T and testing.B, so it can be used from tests and benchmarks alike. ↩︎

Don't check emptyness before loops

Tue, 06 Oct 2020 07:12:04 +0200

If your plan is to loop over a collection: You shouldn’t check for it being non-empty first.

Don’t do this:

if (!stuff.empty()) {
  for (Item item : stuff) {
    // ...
  }
}

Do this instead:

for (Item item : stuff) {
  // ...
}

Benefits:

Shorter
Less nesting
Same performance

I used to do this too at some point in the past, because I thought this would have performance benefits, but in reality, the performance benefit is hardly measurable and generally falls into the category “premature optimization”.

Even if there is a little extra code happening between the if and the for, the performance impact needs to be significant for this to make a difference in most cases, and it’s probably not worth the additional code convolution.

Go-style coroutines in the FN language

Mon, 14 Sep 2020 11:15:00 +0100

My toy Lisp fn is an inherently single-threaded language, but with its built-in stack-inspection capabilities, it’s easy to build coroutines on top.

The implementation only has about 20 lines of code, including comments. This article contains a simplified implementation in-line. In only rewrote some comments to make more sense in context, and removed confusing technicalities like fn’s custom method call syntax, the use of dynamic-wind and remarks about tail calls.

Prerequisites

In most programming languages, the call stack is a chain of stack frames which ends in the program’s entry point.

A conventional call stack

Stack frames are on the heap in fn.

In the case of C, this call stack is a contiguous memory area, but in fn, the stack frames are actually allocated objects on the heap which have pointers to each other. Each stack frame also stores the information required to resume it.

Functions can get a handle on their own stack frame

Additionally, fn has a natively implemented helper called $get-frame. When called, $get-frame returns the caller’s own stack frame as Lisp object which can be manipulated. You can access it from an interactive fn session as well:

fn> ($get-frame)
#<Frame () @1ff4437b5780>
fn> (frame-caller ($get-frame))
#<Frame () @1ff4437b97c4>

Implementation

A queue of suspended coroutines

For Go-style coroutine behaviour, we keep track of a queue of call stacks of currently suspended coroutines. A coroutine which yields execution queues itself last, and returns into the first call stack from that queue.

A queue of suspended call stacks

Starting new coroutines

A new coroutine is started with thread-new!. thread-new! suspends its caller and itself becomes the entry point for the new coroutine which immediately executes. The helper function thread-die! unschedules the current thread from execution by scheduling another coroutine without adding itself back to the queue.

(defn thread-new! (thunk)
  ;; Suspend current call stack.
  (queue-push! *thread-queue* (frame-caller ($get-frame)))
  ;; Make sure we don't accidentally
  (set-frame-caller! ($get-frame) '*should-never-be-used*)
  ;; Call thunk, then call thread-die!.
  (thunk)
  (thread-die!))

(defn thread-die! ()
  (set-frame-caller! ($get-frame)
                     (queue-pop! *thread-queue*))
  nil)

Switching between coroutines

In some languages, next-thread! would be called yield

To switch between coroutines, all you need to do is to have multiple call stacks and switch between them. In fn, this is done in the next-thread! function. next-thread!, when called, will swizzle out the calling stack frame stored in its own stack frame. That way, the function can return into a different call stack than the one it was invoked from.

next-thread! returns into a different call stack than it was called from

next-thread! is implemented as follows:

(defn next-thread! ()
  ;; Push our own frame's caller onto the queue.
  (queue-push! *thread-queue* (frame-caller ($get-frame)))
  ;; Remove a suspended stack frame from the queue,
  ;; and set it as our own caller, so we'll return to it.
  (set-frame-caller! ($get-frame)
                     (queue-pop! *thread-queue*))
  nil)

Caveats about tail calls

Manipulating your own stack frame leads to a simple and short implementation, but the devil is in the details.

In particular, in languages with implicit and automatic tail calls, an invocation such as (set-frame-caller! ($get-frame) ...) cannot be put at the end of a function, of you would manipulate the caller of a function that won’t ever return again. To work around this problem, functions like next-thread! and thread-die! return nil explicitly at the end, so that set-frame-caller! does not get invoked with a tail call.

The full coroutine implementation is in the fn repository in examples/threads.fn.

Rendering drawings with tuhirender

Mon, 06 Jul 2020 00:04:10 +0200

Recently, almost all my digital drawings were made with a Bamboo Slate tablet and Tuhi. It was good fun to take Tuhi’s drawing data and create my own renderer tuhirender, which can be used from the command line to produce some more advanced effects and tune more of the rendering knobs.

tuhirender replaces Tuhi’s own renderer by building on Tuhi’s JSON-based output format for line coordinates and pen pressure data.

Basic invocation

The basic invocation is:

tuhirender -width 200 -fit -o out.png < in/1593622352.json

All parameters are in principle optional, but this invocation shows the most commonly used options. -width specifies the desired output width in pixels.

Animate a drawing

To animate the drawing progress, use the gif output format by passing the flag --format gif and specifying a matching filename with -o out.gif:

An animated drawing

Simplify line segments.

With the --simplify flag, tuhirender simplifies all lines. See my previuos post on the topic: Go: Path simplification library.

Original and simplified version side-by-side

This picture is a bit of an extreme example to demonstrate the simplification. When tweaking the parameters a bit, the simplification is less visible. At some point, I might be able to use this for outputting vector graphics at reduced file size and acceptable quality loss.

Some sample pictures

A bigger picture: Winged Victory of Samothrace

This is the biggest picture I’ve drawn so far with the tablet; I’m happy with how it turned out. The hatching renders nicely and produces the desired shades.

A cartoon drawing: A man, a dog and a sausage

This cartoon drawing is simpler and has a bigger zoom, so it’s easier to see how the change in pen pressure is rendered.

A gopher

The gopher picture has a lot of nuanced Pen pressure, and arguably it looked slightly better on paper. At the moment, tuhirender renders pen pressure by modifying the line width on the go. I suspect that changing the line opacity might be better, but I’m still looking for an elegant algorithm to fix up the artifacts between line segments when doing that.

Go: Test Cleanup in Go 1.14

Thu, 14 May 2020 09:26:41 +0200

In Go 1.14, the new T.Cleanup() function schedules defer-like work to be done after the current test. This makes it possible to write very comfortable setup helper functions:

func SetUpFrobnicator(t *testing.T) *frob.Frobnicator {
    t.Helper()

    f := frob.New()    // Do setup.
    t.Cleanup(f.Close) // Schedule cleanup for later.
    return f
}

func TestFoobar(t *testing.T) {
    f := SetUpFrobnicator(t)

    // (Run the test itself, using f.)
}

Previously, presumably the most idiomatic thing would have been to return a “cancel” callback from the helper function and rely on the test to defer it.

I’m almost a bit surprised that the Go test framework offers this now - otherwise, compared to other languages, Go’s test code is very similar in style to production code (e.g. it doesn’t even have assert functions).

Why Software Design?

Wed, 13 May 2020 21:10:10 +0200

The main technique in software design is this: You look at the entirety of your system and you decompose it into pieces that are more manageable and have clear interfaces. This approach is usually referred to as modularization.

But what’s the point of investing such an effort if in the end only the externally visible properties of a software matter?

In his classic paper “On the Criteria To Be Used in Decomposing Systems into Modules”¹, David Parnas gives the following rationale:

David Parnas

The benefits expected of modular programming are (1) managerial – development time should be shortened because separate groups would work on each module with little need for communication; (2) product flexibility – it should be possible to make drastic changes to one module without a need to change others; (3) comprehensibility – it should be possible to study the system one module at a time. The whole system can therefore be better designed because it is better understood.

To paraphrase this in today’s terminology, the benefits of modularization are:

(1) To enable independent work on different modules through clear interfaces which reduce communication overhead.

Conway's Law

A more sarcastic take on cross-team communication overhead is Conway’s Law, which states that any system designed by an organization will reflect that organization’s communication structure.

(2) To simplify future changes, by aligning the code so that anticipated changes are easy.

Need to anticipate changing requirements!

Changes are easy when they are contained within a module or crossing few module boundaries. The difficulty is to identify likely future changes and lay out the system accordingly, so that they will be a fit with the design and won’t require changes to the bigger architecture.

As David Parnas says himself in the conclusion to his paper:

We propose instead that one begins with a list of difficult design decisions or design decisions which are likely to change. Each module is then designed to hide such a decision from the others.

Modules should not rely on each other beyond what their interfaces guarantee, so that if you change a module, the change is contained behind its interface and not observable to others.

(3) To increase confidence in functionality, by making reasoning about relevant aspects easier.

Easier reasoning for module users

The key here is that the interface to each module should give a more abstract guarantee which hides the module’s implementation details. If the right abstraction is picked, the module’s users will be able to reason about it in simpler terms and avoid dealing with details.

Deep classes

In a good modularization, the interface definition is small and the contained functionality is large (a property which John Ousterhout calls “Deep Classes”)². This minimizes the effort to use a module and maximizes its benefit.

Besides reasoning about behavior, a good system decomposition can also help reasoning about other aspects. For example, it’s easier to reason about crash resilience if different parts of the system are deployed as separate services.

Easier reasoning within modules

Finally, a suitable system decomposition also simplifies reasoning within a module, which now has a clear contract to fulfill, as defined by the module’s interface.

There is more to be said about modularization techniques and different approaches on how to deal with the uncertainties of changing requirements, but that has to go on a follow-up article. Subscribe to this blog with your RSS reader to get notified of future articles.

David Parnas, On the Criteria To Be Used in Decomposing Systems into Modules (1972) ↩︎
John Ousterhout, A Philosophy of Software Design (2018) ↩︎

Go: Sharing templates across multiple pages

Sun, 03 May 2020 11:10:01 +0100

Renaissance gopher Johannes Gopherberg after inventing html/template

I’m using html/template much too seldom to remember how to properly share common template blocks across multiple pages. Today I had to figure it out for the second time. It’s about time to document it!

The approach presented here both (1) makes it possible to share template blocks across multiple logical pages, and (2) makes it easy to use them using tmpl.Execute(writer, data) in your HTTP handlers.

In particular, with this approach, all pages share a common top-level definition of an HTML page, as opposed to maintaining separate header and footer templates whose opening and closing tags would need to be kept in sync.

From the text/template documentation:

Clone can be used to prepare common templates and use them with variant definitions for other templates by adding the variants after the clone is made.

With Clone(), you can have a separate template.Template instance for each of your pages, while making them share a common set of base templates at the same time.

The way you’d like to construct these goes something like this:

Constructing separate template collections from a shared base

None of this is really a secret. However, before .Clone() existed, it wasn’t as easy, and it’s still easy to find a lot of outdated advice.

Example

Let’s go step by step.

Directory structure for this example

The base templates live in templates/base/*.html. These include page.html for the top-level definition of an HTML file, a dummy definition for the main page content in main.html, as well as various helpers that may be reused across various pages.

The per-page templates live in templates/${PAGENAME}/*.html. A minimal page just redefines main.html to have the necessary content.

The templates in foo/ override main.html from base/.

Create the base templates

First, define the base templates with the page.html definition as root template for all template instantiations.

base := template.New("page.html")
base  = template.Must(
    base.ParseGlob("templates/base/*.html")))

We name the template collection page.html, so that page.html will be used as the default template to render for all of our pages, when it gets called through .Execute().

The templates/base/page.html template defines the top-level HTML structure with navigation bars and core site elements, and includes the main page content main.html with a block action:

<html>
<head><title>...</title></head>
<body>
  <!-- navigation, main site elements, etc -->
  {{- block "main.html" . -}}{{- end -}}
</body>
</html>

Derive the specialized templates for the `foo` handler

In the package or file for the foo handler, create a local var fooTmpl *template.Template where you derive the base template:

fooTmpl := template.Must(baseTmpl.Clone())
fooTmpl  = template.Must(
    fooTmpl.ParseGlob("templates/foo/*.html"))

We define page content in the templates/foo/main.html template:

<h1>Hello, world!</h1>
<p>This is an example template.</p>

Use the specialized templates in your handler

Finally, all that’s needed to use the specialized templates in a handler is to just call .Execute():

func handleFoo(w http.ResponseWriter, req *http.Request) {
        data := fooData{Key: "Value"}
        fooTmpl.Execute(w, data)
}

It will automatically instantiate the collection of templates starting from page.html.

Go: Path simplification library

Sun, 05 Apr 2020 14:13:52 +0200

A few weeks ago, I got myself a Bamboo Slate tablet, and had a lot of fun playing around with the output. The Linux software for talking to that device is Tuhi, which is GUI driven, but also dumps the raw point and pressure data in JSON format into a directory.

As a side effect of playing around with that, I published a 2d path simplification library which uses the Ramer-Douglas-Peucker algorithm.

Given a sequence of 2D points, the algorithm determines a subset of these points so that the path drawn by these is still reasonably close to the original. Here is an example output:

Original and simplified version side-by-side

I found the Ramer-Douglas-Peucker algorithm to be very clever, my attempts to come up with my own algorithm had terrible results in comparison.

The library is here:

Source: https://github.com/gnoack/path
Godoc: https://godoc.org/github.com/gnoack/path

I have a vague idea to use this for simplifying diagram drawings and straightening out imprecisely drawn shapes, replacing squiggly boxes and arrows with properly aligned ones. I’m not sure whether I’m going in the right direction with this approach, given that I’m not making use of pressure data and my knowledge of the shapes I want to recognize; we’ll see whether it’ll turn out to work.

Go: The lack of generics

Sun, 05 Apr 2020 11:53:00 +0200

Go’s lack of generics means that people can’t overengineer in the way they are used to, so they need to put more thought into their designs.

There are some use cases where abstractions with generics are better to the current ones (e.g. the sort package), but by and large, in other languages, people use them to build towers of abstractions which you’ll later need to wrap your head around, which introduce conceptual duplication (e.g. mocking and predicate frameworks) and which lead to a fragmentation of Go style (as would be the case if people start building abstractions to hide the go keyword, like threading and future libraries).

The Setup-Cleanup problem

Wed, 08 Jan 2020 21:37:28 +0100

A fully automated cleanup strategy.

This is a comparison of different programming idioms for performing cleanup work.

Common use cases where this is used include:

Locking and unlocking synchronization locks.
Opening and closing files.
Canceling asynchronous operations that aren’t needed any more.
Allocation and deallocation.

A naive solution: Cluttered cleanups

int frobnicate() {
        set_up();

        int err = foo();
        if (err) {
                clean_up();  // <-- 1.
                return err;
        }

        if (!bar()) {
                clean_up();  // <-- 2.
                return ERROR_ACCESS;
        }

        baz();

        clean_up();          // <-- 3.
        return SUCCESS;
}

There are now three places where clean_up() is being called (Code duplication), and they are all far away from the call to set_up(). When the code changes a bit, it’s easy to miss one and introduce cleanup issues. This is not a well-maintainable approach.

Linux kernel style (C)

The Linux kernel uses gotos to coordinate a function’s cleanup:

int frobnicate()
{
        int rc = 0;

        set_up();

        rc = foo();
        if (rc)
                goto out;

        if (!bar()) {
                rc = EACCES;
                goto out;
        }

        baz();

out:
        clean_up();  // <--
        return rc;
}

All function exits happening after set_up() need to jump to the label where the clean_up() is done.

Note that this pattern can be nested: With multiple independent setup and cleanup steps, the function ends with a sequence of multiple cleanup operations and multiple exit labels (example).

A longer explanation is at Eli Bendersky’s website.

Resource Acquisition is Initialization (C++)

In C++, setup and cleanup is frequently bound to object lifetimes, which are deterministic there. This is an example using Abseil’s absl::MutexLock class:

void MyClass::Frobnicate() {
  absl::MutexLock l(&mutex_);

  Foo();  // under lock
  Bar();  // under lock
  Baz();  // under lock
}

The lifetime of the l variable ends with the function scope, and absl::MutexLock’s constructor and destructor are locking and unlocking the mutex.

This pattern can also be nested. C++ guarantees that variable destruction happens in the inverse order of construction.

The RAII pattern (Wikipedia) is also used in other languages like Rust (Drop trait).

To a similar extent, similar functionality to RAII is also available in C, when using the GCC-specific __attribute__((__cleanup__(f))) language extension. This extension lets you attach a cleanup function to a variable scope, but the cleanup is defined in the place where the variable is defined, not in the type.

As readers pointed out, a newer variant of C#’s using statement implements a similar non-nested cleanup bound to variable lifetime as well. See the documentation comment by Morgens Heller Gabe below for examples.

“Defer” statement (Go)

Go has a language feature for cleanup:

func (m *MyClass) Frobnicate() {
  m.mu_.Lock()
  defer m.mu_.Unlock()

  Foo()  // under lock
  Bar()  // under lock
}

The defer statement defers a function call until the current function exits.

The Zig language has syntactically similar defer and errdefer statements. Other than in Go, Zig executes the cleanup when leaving the current scope rather than when leaving the function.

“With” blocks and friends (Python, C#, Java)

Python implements a syntax which makes it easy to acquire resources during the execution of a nested block of commands.

def readfullfile(filename):
  with open(filename) as f:
    print("opened", filename)
    return f.read()

The object returned by open() implements the context manager protocol, which means that it can be used in a with statement. When the with-scope exits, the cleanup functions will automatically get invoked (e.g. to close a file).

Cleanups get invoked in all cases when execution leaves the scope, no matter whether it happens through a regular exit, an early return or an exception.

Update: Multiple readers have pointed out that both C# and Java now support equivalent scoped statements as well:

In C# with the using statement (see documentation and Mogens Heller Grabe’s comment with examples below).

in Java with the try-with-resources syntax. (examples)

In all of these cases, the used object must implement a special interface with a cleanup method, such as IDisposable in C# and AutoCloseable in Java.

Macros (Lisp)

Lisp macros are used similarly as Python’s with statements, but offer more ways to shoot yourself in the foot (Example from the book Practical Common Lisp by Peter Seibel):

(with-open-file (stream "/some/file/name.txt" :direction :output)
  (format stream "Some text."))  ; while file is opened

Here, with-open-file is a macro caring about opening and closing (spec). The user of the macro provides a sequence of commands to be executed with the opened file stream, e.g. (format stream "Some text.").

These macros are often named to start with the word with, but there are exceptions, e.g. save-excursion in Emacs Lisp.

From the caller perspective, this behaves similar as Python’s with statements, but they are usually expanded into a use of unwind-protect or a similar primitive depending on the Lisp dialect (comparable to try…finally).

Try-Finally blocks (Java, C#, Python, …)

try blocks are primarily used for catching exceptions but in many languages they offer a finally clause as well, which is guaranteed to execute whenever the try scope exits, including on early returns and exceptions.

setUp();
try {
  doWork();  // between setup and cleanup
} finally {
  cleanUp();
}

The construct tends to lead to deeper nesting when multiple cleanups need to be done.

Some languages have finally-like constructs which are independent of exceptions, like Smalltalk with Block»ensure:, unwind-protect in Common Lisp (on C2 wiki) and dynamic-wind in Scheme. (Scheme has the additional complication that it needs to deal with continuations.)

Todo lists (languages with closures)

In languages with closures or higher order functions, one way to defer work for later is to save functions to be executed later in a list:

def frobnicate():
  todos = []

  setUp()
  todos.append(cleanUp)

  doWork()  # between setup and cleanup

  for f in todos:
    f()

One variant of this is addCleanup in Python’s unittest library for deferring work to be done after the test execution. Note that similarly to defer in Go, this lets us place the cleanup right together with the setup code, so that we can easily keep them in sync:

class FooTest(unittest.TestCase):
  # ...

  def test_service(self):
    srv = StartService()
    self.addCleanup(srv.Stop)

    self.assertEqual(42, srv.Invoke())  # while running

Update: Since version 1.14, Go supports the same with T.Cleanup() to simplify test cleanup.

Passing higher-order functions (Ruby, Lisp?)

In Ruby, it’s common to pass down functions to other functions, as a way of structuring control flow and also for cleanup purposes. This invocation of File.open executes the passed function (in between do…end) with the opened file and closes the file afterwards.

File.open("output.txt", "a") do |f|
  f.puts("Hello, world!")  # while file is open
end

This general technique is possible in all languages where passing down higher order functions is cheap, such as Ruby and Smalltalk. Smalltalk takes it to another level by expressing all control flow as method invocations, including loops and ifTrue:.

I’m not sure to what extent the Smalltalks use higher order functions for cleanup purposes as Ruby does. More imperative Lisps like CL expose mostly macros to users, but these will in turn use closure-passing unwinding primitives under the hood.

There is a technical twist here in that languages need special guarantees for these so-called downward funargs to be cheap.

A downward funarg is a higher-order function which is only passed down the stack but does not outlive the stack frame in which it was created. The captured variables referenced by a downward funarg can happily stay on the stack. This is in contrast to upward funargs which outlive their own stack frame and which require invisible heap allocations or similar tricks in the language.

setUp() and tearDown() in (JUnit, xUnit, …)

In the xUnit family of unit testing frameworks, tests are part of classes which get executed by a test runner. Each test class may optionally override setUp() and tearDown() hooks which get executed around every test execution, independent of whether that test succeeded or not.

In languages which provide a sane way of cleanup, I think that overriding the tearDown() hook should not be necessary and probably discouraged. I suspect the same is true for code in setUp(), but that is more related to ambient shared state which becomes tricky to modify when multiple tests rely on a subset of it in implicit ways.

Discussion

These approaches are quite different to each other. In practice, the programming language is often already fixed, and then only a limited number of options exists.

In general, I found that in practice it helps to have setup and cleanup close to each other, so that it’s easy to keep them in sync. Go’s defer really shines there.

It’s even nicer to have them represented by the same construct as with Python’s with or C++’s RAII idiom. With these, the usage becomes safer, but you trade that for more effort in implementing classes that are with- and RAII-enabled. (In Python, this is somewhat mitigated with the contextlib module.)

Downward funargs are both safe, easy to use and simple to implement, but have a little overhead. That’s fair in most scripting cases, but wouldn’t work for the Linux kernel.

In terms of reasoning about performance, the Linux kernel has the approach where it’s most obvious what this translates to at the machine level, but the same amount of control is usually not needed in user space programs.

At a higher level, I believe it pays off to care about symmetric setup and cleanup steps as part of the same function wherever possible. It makes it easier to track where setups and cleanups are done across the codebase, and refactoring to such a design reduces state that has to be tracked across multiple functions.

Update (2020-01-13)

Thanks for all the positive feedback, particularly to everyone who pointed out related language features:

C#’s using: Thanks, Mogens Heller Grabe for the longer explanation in the comments below, as well as the users hateful and niklasjansson on HN who pointed out documentation.
Java try-with-resources: cpt1138 and quantified on HN
Rust’s Drop trait: jupp0r on HN
Zig’s defer and errdefer: apta on HN

For Haskell, lgas mentioned the Bracket pattern, and dwohnitmok is mentioning linear types. Superficially this looks similar to dynamic-unwind and friends in Scheme and Lisp (see above), but my Haskell-fu is too weak to judge it. I suspect it would be hard to compare programming idioms between Haskell’s lazy purely functional evaluation and imperative programs.

Mutation Testing works

Mon, 30 Dec 2019 23:53:19 +0100

Disclaimer: These are Google topics, but based on publicly available resources. I’m writing this not because I get paid for it (I don’t), but because I am truly excited about it, it’s now public, and I hope this will find more wide-spread adoption.

Goran Petrović and Marko Ivanković have published a nice paper on the “State of Mutation Testing at Google” a while ago¹.

I’m a fan of this, because Mutation Testing spots real test coverage issues which regular line-based coverage won’t catch. The approach presented in the paper runs mutation testing in the context of a code review, which narrows the surfaced issues to the code you’re already working on, and keeps them actionable.

Example

To get an idea how the approach in the paper works in practice: Let’s assume we send a code review introducing a line such as if (a == b || b == 1) {, but we didn’t bother testing the new code properly. With mutation testing enabled, the infrastructure would now catch us and report something like this (from page 2 in the paper):

Changing this 1 line to

  if (a != b || b == 1) {

does not cause any test exercising them to fail.

Consider adding test cases that fail when the code is mutated to
ensure those bugs would be caught.

Mutants ran because goranpetrovic is whitelisted.

How did it figure that out?

Mutation Testing runs on a proposed change during a code review, so the problem is defined (a) through a bunch of changes to production code and (b) implicitly through the tests exercising that code.

The mutation testing infrastructure now attempts the following:

Purposefully modify program logic in the code under test (e.g. negate an if condition)²
Check if any of the tests starts failing.

We messed up the code under test, so with good “logical coverage”, one of the tests should now have failed. But if all tests have still passed, we apparently have changed logic that wasn’t properly tested.

Mutation Testing is underrated

Even though it looks like it’s originally coming from a more academic corner, Mutation Testing works in practice. The examples in the paper are not made up, and they saved me from real bugs.

I hope that there will be more implementations in the wild at some point. It’s worth the effort.

Google started publishing more material on internal developer tooling in the recent years. I ♥ it because I can now point to papers rather than to Google PR when people ask about it. :) ↩︎
The list of implemented mutations is at the top of page 4 in the paper. ↩︎

Wireguard is in the Linux kernel

Mon, 09 Dec 2019 22:59:00 +0200

I’m very happy to hear that Wireguard has made it into the Linux kernel and will be part of Linux 5.6!

Wireguard was quite a relief for me after wrestling with OpenVPN before, for multiple reasons. I’m a happy user for about a year now, and I can wholeheartedly recommend it.

Wireguard is really pushing the state of the art forward for VPNs:

high performance (it’s part of the kernel)
very simple setup
- no wrestling with OpenSSL or X.509 certificates
- key management¹ comparable to OpenSSH, keys fit one line
excellent apps for Android and iOS
- full config files can be provided as QR codes
- it does not drain your battery (a mostly stateless protocol)
support for many platforms apart from Linux through a user space implementation

An example configuration

To get an idea, this is a /etc/wireguard/wg0.conf configuration file very similar² to the one I use on my Laptop right now:

[Interface]
PrivateKey = WG8r5DNvD2KlZORhJ2XgzW3lWO8i5GJqZBePt98EgUY=
Address = 192.168.23.10/32
DNS = 192.168.23.1

[Peer]
PublicKey = 6qzH9hJbyPFp+GJJoxsBaPhUEl4mVKTGNP433xLWhBc=
PresharedKey = LiWmdHZN/Jizhv1h0qTGeslci2yZIyrkEDjrx3bUomE=
AllowedIPs = 0.0.0.0/0, ::/0
Endpoint = vpn.gnoack.org:9999

This contains our own device’s private key and configured IP in the VPN, as well as a list of peer hosts with their public keys and the IP addresses they are making available.

Note the symmetry: A server-side configuration looks the same, but with more peer entries.

The command wg-quick up wg0 brings the configuration up as a new device and sets appropriate routes.

For further reference, Wireguard’s own quickstart page has a better introduction than this one here (with a video).

Disclaimer: I’m not affiliated with Wireguard, but I received a big stack of stickers³ from Jason Donenfeld after a talk once. Congratulations on the big step forward, and thanks for the great software!

Key management can hardly be simpler than that:
```
$ wg genkey > beuys.gnoack.org
$ wg pubkey < beuys.gnoack.org
tw6MlpAFMoQInDC402FndO8Z49/H4cT11BYOHDRkcys=
$ wg genpsk > psk
```
This is a breeze compared to the OpenSSL dance required to get OpenVPN running. ↩︎
These are example values, of course. ↩︎
That was really nice. As everyone knows, stickers beat Bitcoin and Ethereum hands down as an underground hacker currency. :) They were quite popular in the sticker exchange where I placed them. 🐉 ↩︎

Go: Goroutines and Apis

Mon, 18 Nov 2019 21:30:53 +0100

Goroutines are a unusual and powerful programming language feature, so they are a tempting toy to play with, and they get a bit overused.

There is some indication that the following Go principle holds true:

Strive to provide synchronous APIs,
let the caller start goroutines.

To put this advice into a more concrete code example:

Do this:

func (t *type) Run(ctx context.Context) {
    // Implementation of background task
}

Instead of this:

func (t *type) Start() {
    t.wg.Add(1)
    go func() {
      // Implementation of background task
      t.wg.Done()
    }
}

func (t *type) Cancel() {
    // Somehow cancel the background task
    t.wg.Wait()
}

Upsides for Run():

The caller gets to decide how the “background” code gets run.
- Spawning Goroutines is easy for the caller too.
- go statements move towards the application’s top-level. Goroutine coordination becomes simpler to reason about when goroutines are started in fewer places.
- Callers can also run it without goroutine and just block on the call, if there is no other work to be done.
Background task cancellation is provided through the context.
Waiting for the background task has a trivial API (when the function returns), and can be done with the mechanism the caller prefers (waitgroups, channels, …)
It’s on the safe side API-wise: There is no Close() method which callers can forget to call (leaking resources).

Another great discussion of this was in the Go Time: On application design podcast (starting around minute 47, Mat Ryer’s explanation really resonated with me)

Addendum: In the comments, Jacob is pointing out the talk “Ways To Do Things” by Peter Bourgon (slides), who also appeared on the above “Go Time” episode. The talk describes the Run() style quite clearly and in more detail. Thanks for the excellent pointer, Jacob!

Go: Error handling

Fri, 18 Oct 2019 20:00:00 +0200

I love Go for many reasons, but this part is still itching me: I postulate that this Go idiom is a burden on our mental capacity:

if err != nil {
        return nil
}

Error handling is at a tension between two different developer needs.

On the one hand, error handling is very annoying and distracting when working on a constructively formulated use case.
On the other hand, not dealing with it correctly means that the program blows up in unknown ways when errors happen: data may be lost, I/O may be half done, features may stop working without the developers noticing.

Go’s explicitness about error handling makes sure that errors are not falling through the cracks as easily as they might do in other languages where errors are implicitly propagated¹. While many developers love Go’s explicitness, it is rather verbose for error handling, which has drawbacks:

(1) Humans are pattern matching machines. Whenever we see text whose shape resembles the error checking idiom, we have trained ourselves to quickly look away, at the more important code next to it. But bugs can hide even in repetitive parts of the code. If we’re not looking at the idiom anyway, the try() proposal might be the better option, as it de-duplicates the check.

(2) The error handling pattern takes much visual space. The idiom is necessarily interleaved with the code on the happy path. In many cases, the flow of reading a longer function is interrupted multiple times with the same 3-line code snippet.

Two different hats

I believe that reasoning about constructive code and reasoning about error handling code are two separate states of mind, and switching back and forth between these is straining and tedious².

A consequence of this is: It might be more productive to spend less time on errors during an initial implementation, and to then address the error handling aspect in a separate pass over the code. It would be better if we could have two views of the code for these different modes of operation:

For the feature-focused mode, we need a view of the code where the error handling stays explicit³, but goes out of the way visually in order to make the constructive code more prominent.
For the error-focused mode, an error-only view is not really possible because errors happen only as a result of constructive operations, but in many editors, an interactive search for the keywords return and try(?) will highlight the possible exit points quickly.

I think this would have been well solved with Robert Griesemer’s try() proposal. I’m still sad it didn’t go forward, but I’m glad the Go team is so careful about it.

Just because we have all already invested so much time into accepting this verbose way of error handling, that doesn’t mean it’s a good thing⁴.

This is particularly true for languages with exceptions where all function calls may implicitly throw exceptions and bubble them up the stack. ↩︎
It’s the same as trying to write a scientific article straight in LaTeX form: You can somehow do it, but the first formula will derail your train of thought so hard that the article will probably suffer from it. It’s a better option to write the article in a different medium and then do the translation to LaTeX in a separate step. ↩︎
Explicitness is a stated goal in Go, after all. ↩︎
The sunk cost effect is strong for programming languages! But then again, it’s much worse for C++. :) ↩︎

Dot Space Space

Fri, 30 Aug 2019 09:17:30 +0200

I have a confession to make. I used to put two spaces after each sentence, as it used to be done by typewriter typists before computers [citation needed].

The moment when I started to form this habit was around 2010. I was young and easy to impress, and I was picking up more advanced Emacs use working on the TeX source for my diploma thesis. While editing these large chunks of free-form text was when I discovered the fill-paragraph function, otherwise known as M-q, for automatically breaking text at the correct column width. This is very useful for editing text files, so it quickly became a habit to press M-q often. I’m still using it right now, in fact.

Now after a while of using this feature, I started to observe that this didn’t work exactly how I thought. Readjusting text where one line ends with a full stop, it would surprisingly add two spaces in the middle after breaking.

This text:

Hello world.
This is a text.

Would turn into that text after a press of M-q:

Hello world.  This is a text.

Then somewhere in Emacs documentation, I found key bindings jump back and forth between sentences (M-a, M-e), and these also rely on double-spaces to tell apart sentence ends from abbreviations (e.g. as in Dr. Seuss).

I did not end up picking up these shortcuts, but it was enough for me to rationalize the double spaces. In reality of course, I only started with that habit so that M-q would not break the otherwise consistent spacing style accidentally.

I used this for about 10 years now, and of course it didn’t elude me that I was the only one doing that, but it’s hard to put down old habits.

Recently, I found what I think is the remedy - M-q is actually configurable in Emacs, and the old habit of pressing M-q will now just fix my sentences to single-spaces again going forward.

This is what is needed:

(setq sentence-end-double-space nil)

What do we learn from this story? I’m not sure. Maybe that old habits are hard to shake. Some quirks creep in without you noticing and a tiny change might fix them.

M-q

Notes on Plan9's 9P protocol

Fri, 22 Mar 2019 12:26:34 +0100

The 9P protocol is the protocol for network file systems used in Plan9. Plan9 is not a widely used operating system, but it’s widely considered more true to the Unix spirit than Unix is, coming from some of the same people who made Unix as well.

Overview

To get a rough idea of the 9P protocol, consider a system where all the original core Unix syscalls (open, close, read, write, …) are lifted to be remote procedure calls. These can be sent over the network, making file systems in Plan9 network transparent.

9P’s RPCs are documented in Plan9’s man page section 5 (mirror). The supported methods are:

Session control
- version to negotiate the protocol version.
- attach and auth for authenticating with the remote end and opening a session.
- flush to cancel ongoing requests.
Basic file operations:
- open
- clunk (equivalent to close)
- read
- write
Working with directories
- walk to descend a directory hierarchy.
- stat and wstat for querying and modifying a directory entries (e.g. to rename files).
- remove for removing file entries.

In addition, there is an error response which may be returned as response to any of these requests.

The handle to a file is called “file ID” (fid) in Plan9 lingo. This can be thought of as a file descriptor.

The 9P protocol does away with a bunch of complexities that Unix, and particularly Linux has started to have. Some of the non-supported features are:

There is no mmap support in the protocol. This can’t be done over the network. Presumably it can be emulated within limits if needed.
No ioctl. Control-style operations are often done through reads and writes to special purpose files. There are often textual command languages.
There is a more blurry line between different file types as in Unix, where you can distinguish named pipes, device files and regular files.

Playing with Plan9port

An easy way to play with 9P is to install Plan9port, a Linux userspace port of Plan9’s core tools and wiring, which comes as installable package with many Linux distributions.

Plan9port includes the Plan9 compiler, the acme text editor and other things, including source code. Many of these tools are exposing 9P file systems over Unix sockets in the /tmp/ns.$USER.$DISPLAY directory. You can get your exact location by running the namespace tool:

$ namespace
/tmp/ns.me:0
$

For example, you might want to start up the factotum authentication service and have a peek at the created file system. Plan9port’s 9p tool lets you do simple accesses from the command line:

$ /usr/lib/plan9/bin/factotum
$ ls $(namespace)
factotum
$ 9p ls factotum
confirm
conv
ctl
log
needkey
proto
rpc
$

There are also public 9P services, such as the Plan9 “sources” repository. After the Bell Labs server went down, these continue to be mirrored by services such as sources.9p.io. The 9fs script is a wrapper for quickly mounting this remote service, but needs a little modification for convenience:

$ cat $PLAN9/bin/9fs \
  | sed -e 's/sources.cs.bell-labs.com/sources.9p.io/' > ~/9fs
$ chmod +x ~/9fs
$ ~/9fs sources
$ 9p ls -l sources
d-rwxrwxr-x M 0 9grid  9grid       0 Jul 15  2018 9grid
--rw-rw-r-- M 0 bootes sys         0 Jun 24  2013 _sources
d-rwxrwxr-x M 0 adm    sys         0 Jul 15  2018 adm
d-rwxrwxr-x M 0 sys    sys         0 Oct 28 19:18 contrib
d-rwxrwxr-x M 0 bootes sys         0 Jul 15  2018 dist
d-rwxrwxr-x M 0 sys    sys         0 Jul 15  2018 extra
--rw-rw-r-- M 0 bootes adm   9142602 Jan 23  2015 lsr
d-rwxrwxr-x M 0 geoff  nix         0 Jul 15  2018 nix
d-rwxrwxrwx M 0 glenda sys         0 Jul 15  2018 patch
d-rwxrwxr-x M 0 sys    sys         0 Jul 15  2018 plan9
d-rwxrwxr-x M 0 sys    sys         0 Jul 15  2018 wiki
d-rwxrwxr-x M 0 xen    xen         0 Jul 15  2018 xen
$

💡 Hint: To make browsing more convenient, you may also use the FUSE file system which comes with Plan9port:
$ 9pfuse unix\!/tmp/ns.$USER.:0/sources /tmp/sources
$ ls /tmp/sources
9grid  contrib  extra  nix    plan9     wiki
adm    dist     lsr    patch  _sources  xen
$

Looking under the hood

Let’s first read a small file from the Plan9 sources repo:

$ 9p read sources/plan9/NOTICE
Copyright © 2002 Lucent Technologies Inc.
All Rights Reserved
$

To look under the hood of 9P, we can use the 9p tool’s -D option to print all requests and responses.

$ 9p -D read sources/plan9/NOTICE
<- Tversion tag 0 msize 8192 version '9P2000'
-> Rversion tag 65535 msize 8192 version '9P2000'
<- Tauth tag 0 afid 0 uname me aname <nil>
-> Rerror tag 0 ename authentication rejected
<- Tattach tag 0 fid 0 afid -1 uname me aname
-> Rattach tag 0 qid (0000000000000002 0 d)
<- Twalk tag 0 fid 0 newfid 1 nwname 2 0:plan9 1:NOTICE
-> Rwalk tag 0 nwqid 2 0:(0000000000081aa0 78 d) 1:(0000000000081d2d 1 )
<- Topen tag 0 fid 1 mode 0
-> Ropen tag 0 qid (0000000000081d2d 1 ) iounit 8168
<- Tread tag 0 fid 1 offset 0 count 4096
-> Rread tag 0 count 63 '436f7079 72696768 7420c2a9 20323030 32204c75 ...'
Copyright © 2002 Lucent Technologies Inc.
All Rights Reserved
<- Tread tag 0 fid 1 offset 63 count 4096
-> Rread tag 0 count 0 ''
<- Tclunk tag 0 fid 1
-> Rclunk tag 0
$

The 9p tool opens the Unix socket at /tmp/ns.me.:0/sources to talk to the backend service.

The steps are:

Version negotiation: Both sides confirm the 9P variant they want to speak.
Authentication: The auth call is normally supposed to return an open file ID, over which to speak an authentication protocol. In this case, the service permits unauthenticated read accesses, so this is returning an error.
Attaching to the service, retrieving the desired root directory, in this case it’s just the root. (Note, the client picks which FID it would like to use for identifying files.)
Walking down to the desired file from the root FID 0, yielding the new FID 1.
Opening the FID for reading.
Calling read until nothing is returned any more.
Closing the FID with clunk.

Some resources

How to review code

Thu, 14 Mar 2019 13:00:06 +0200

A code review that doesn’t go right. (image by Manu Cornet, CC BY-NC-ND 3.0

I’ve reviewed over 3000 code changes so far in my career, and gone through about same amount of reviews as change author. When reviewing code, here is the golden rule I follow:

Any code change that’s a strict improvement should be approved.

Most incoming code reviews are both improvements in some aspects and make things worse in other aspects.

Understand the change

It’s tempting to glance over a change and only comment on small technical details or style issues, but this will eventually lead to worse code. In order to give meaningful feedback, understand the change to the extent where you could have authored it yourself.

Blocking issues to look out for

New code is unclear. Ask: “I do not understand this”. Insist that it’s clarified in the final submitted code rather than in a review comment.
New code is missing tests. Insist.
New code does not fit into surrounding code in terms of style or common libraries used. Insist. Link to a canonical style guide whenever you can.
Pre-existing tests were changed willy-nilly until they run green. This often takes the form of only a few lines added to every existing test case. These drive-by fixes accumulate easily over the course of some changes, until the core idea behind the test cases is obstructed. Insist on refactoring the tests.
New code introduces duplication with existing code. Ask to deduplicate. (This is particularly often seen in tests.)
The change is increasing code complexity.
- Is there a better solution you can suggest?
- Does the change’s benefit outweigh the additional complexity?
The change introduces dependencies in places where they don’t belong.

Non-blocking issues to look out for

It’s fine to ask for additional changes that are not strictly required, but always make it clear that they are optional.

“I would have done it differently.” is a gut reaction almost all reviewers have sometimes. But it’s a shortcut heuristic the mind takes which usually has a more solid technical background.

Learn the specific technical background to your gut reactions, so you can give concrete advice.

If you can’t find a specific technical problem, consider asking why it was done in that particular way. There are often be good reasons which only need to be clarified a bit in the code.
Pre-existing issues. Point out if pre-existing issues can be fixed in the same change, but explicitly mark it as optional. Even when not done immediately, this discussion helps to form a common understanding which pre-existing patterns in the code should be avoided and kept.

Taking technical debt: A game of trust

Sometimes you will want to approve newly introduced issues and postpone their fixes until later (Technical debt). Insist that the author will fix these issues after the submission.

In these cases, code reviews are a game of trust. A strategy that works for me is: Assume the best from people you don’t know yet, but don’t let them pass a second time if they didn’t pay off the technical debt.

Good ways to track technical debt issues are (1) a good memory for names, (2) a shared bug tracker and (3) TODO markers in the code (for smaller things).

Behaviors to avoid as a reviewer

Do not sit on a code review. Even if you can’t do the review, explain immediately why you can’t do it (yet) and redirect to someone better suited if possible.
Never insist that reviewees fix pre-existing issues. Being code reviewer is a position of power, but it’s not a free pass to blackmail others into doing your work.
Don’t block legitimate changes for corporate politics reasons. This should go without saying. Do not use code reviews to unjustly influence project priorities.

Parting words

I hope this collection of code reviewing tips will help someone to have more constructive code reviews. Remember:

The reviewer’s job is not to stand in the way, but to enable the code submitter to do their best work.

Feedback is always appreciated. :)

Update 2019-09-05: My colleague Max Kanat-Alexander has compiled and open sourced Google’s guidelines for code reviews at https://google.github.io/eng-practices/. It’s a longer document compiled from the input of many engineers and has a lot of good advice. I can wholeheartedly endorse this.

Procedurally generated trees

Wed, 23 Jan 2019 11:35:00 +0200

These trees are procedurally regenerated on every page load.

Basic algorithm

At the heart of this algorithm, there is a recursive function which draws a tree branch (as a rotated black rectangle) and its sub-branches. The function calls itself recursively for the two sub-branches, using smaller branch sizes. When the width is finally too small to continue for the branch we’re looking at, we draw a leaf and stop there.

To randomize the tree shape, we’re picking two weights from 0 to 1 for the two branches, which are adding up to 1. The weights are used to determine the sub-branch sizes and angles.

The two sub-branches should together have the same surface area as the parent when you saw through them. The surface area is split according to weight.
Big sub-branches (with big weight) grow straight ahead, while small ones grow more to the side, up to 45°.
Very small sub-branches with too small weights are cut off and not drawn anymore.

Background

The Javascript library I’m using is Paperjs, which I discovered through the PaperJS talk at 35C3 by bleeptrack.

Internet of Things: Wifi Radio

Wed, 09 Jan 2019 12:05:01 +0100

Update: This setup stopped working after a few firmware updates; there were apparently some software updates after the catalog of internet radio stations broke down.

Maybe they also read my bug report about the broken encoding and found my blog post. I didn’t investigate much; superficially it looked like the radio is just ignoring the returned IP address if it’s a local one.

Have you ever wondered what your Internet-connected devices are doing behind your back?

My internet radio had some problems with displaying special characters in the list of radio stations. When I checked the network traffic on Wireshark, I found that the protocol for connecting back was unencrypted, so I made it talk through a proxy which modifies the responses on the fly to fix up wrong character encoding in the server responses.

Basic setup:

The router giving out DHCP leases makes clients talk to its own local DNS service.
The DNS service rewires the hostname for the radio API to the local Raspberry Pi’s IP
The Raspberry Pi acts as a reverse HTTP proxy for the actual domain and modifies requests along the way.

Reverse proxies are simple to write in Go: use httputil.NewSingleHostReverseProxy and then set the ReverseProxy.ModifyResponse property on the returned object to an appropriate modification function such as:

func Modify(resp *http.Response) error {
        b, err := ioutil.ReadAll(resp.Body)
        resp.Body.Close()
        if err != nil {
                return err
        }
        b = ModifyBody(b)  // Actual replacements done here.
        resp.Body = ioutil.NopCloser(bytes.NewReader(b))
        resp.ContentLength = int64(len(b))
        resp.Header.Set("Content-Length", strconv.Itoa(len(b)))
        return nil
}

In my implementation, I’m replacing some wrongly encoded characters with correct ones in the server response, and for demo purposes and some silly entertainment, I’m changing the German word for Switzerland to “Swizzle”:

Shared base fixture

Tue, 06 Nov 2018 07:44:44 +0100

A diagram of how different tests share common test setup with a shared base fixture

Q: How do you know you can trust your test set-up?
A: You reuse a well-known test set up in every test.

Example code

def test_simple():
  req = set_up_working_request()
  resp = invoke(req)
  assert resp.status == OK

def test_invalid_page_size():
  req = set_up_working_request()
  req.page_size = -5
  resp = invoke(req)
  assert resp.status == INVALID_ARGUMENT

In the test,

We call set_up_working_request() and we know this results in “working” behavior. Then we purposefully inject a defective negative page size into the working request.
We exercise the system under test in the same way.
We expect a different result.

Because the only difference in set up between test_simple and test_invalid_page_size was in the differing page size, we can easily reason that the different response status is resulting from just that.

This works well together with these techniques:

Structure your test methods with separate set up, exercise and verify phases.¹
Use explicit helper functions for set up and possibly name them according to expected behavior such as “working” in the example above.
Keep the shared set up code in test methods as short as possible, so that it’s easy to spot the equal parts.
1:N: Use one base scenario (e.g. “working”) and use its set up as a base for multiple derived scenarios.

This is called the “Four-Phase test” in the xUnit test pattern book. (Meszaros07, p358) ↩︎

No error left behind

Mon, 03 Sep 2018 22:36:41 +0200

Which design guidelines should software follow in order to run reliably? Which guidelines do you use in your day-to-day work?

Error handling is a significant complexity in real-world projects, but programming textbooks are often giving very little advice on how to think about it at the higher level. This article tries to fill that gap:

We will define what an error is.
We will find out that each error has a human stakeholder.
We will discuss technical mechanisms to bring errors to human attention.
The final section contains various examples of how to apply this.

To handle the error cases in your software, you should ask yourself who is the human stakeholder who would be able to address it, and then find a suitable mechanism to escalate the error in the direction of that stakeholder.

I’ve cross-checked this approach with multiple people from various software backgrounds, but I’m still happy to receive your feedback.

What is an error?

Definition: An error is when the program is operating outside the intended path of execution.

Running programs within the intended path of execution is a goal by definition. To track and reach this goal, we need to make it observable whether the program is running healthily:

Rule: All errors should be observable from the outside.

Observable in this context means that it’s possible to bring the error to a human’s attention (e.g. end user, system administrator, developer) in an automated fashion.

The outside is defined by the bounds of your program. The error reports need to be made accessible to the surrounding environment, so that stakeholders can access them.

Error stakeholders

The stakeholders for an error are the people involved in the software lifecycle, such as: End users, developers, system administrators, network administrators etc.

Rule: For each error, there is a corresponding stakeholder.

Different errors have different stakeholders who can address the issue.

Is the error cause within the program itself, then the developer is the stakeholder.

Is the error cause in the user input or interaction (e.g. invalid input), then the user is the stakeholder.

Is the error cause in the execution environment, then the operator of that environment is the stakeholder (e.g. sysadmin, network admin, SRE).

Mechanisms

Routing errors to the stakeholders

Routing error indicators to the right stakeholders is partially done in code, and partially within the deployment. Particularly, displaying errors to end users is usually done in the program code itself. On the other hand, metrics collection set-ups can be monitored by developers and administrators alike.

There are two main strategies to make errors observable:

Propagate the error upwards towards the initiator of the erroring operation.
Propagate the error to the side (monitoring and logging).

At least one of these should be used for any error.

A function can propagate errors upwards or to the side,
at least one of these should be used.

Propagating an error upwards

Propagating “upwards” moves the responsibility of error handling to a higher level. In general, higher level software has more context for the failing operation, so they are often in a better position to route the error in the right direction.

Propagating errors upwards can take many forms.

Function level: Propagate to the calling procedure
- Return an error code
- Raise an exception
- Propagate a lower-level exception
Process level: Crash the process
- Exit the program with an error status (e.g. Unix exit())
- Abort the program (e.g. Unix abort())
Network level: Return the error in a network response
User level: Show the error to the user
- e.g. in a UI dialog

Note: Remember to convert the error into a representation which fits the level of abstraction that the caller expects. Many languages support nesting (wrapping) errors so that the full context is retained.

Propagating an error to the side

Sometimes passing the error upwards is not an option or not sufficient, for example because you want to transparently recover or hide a subsystem failure from the user.

Propagating errors to the side can be one of the following:

Increment a counter which can be monitored from the outside. (e.g. using Prometheus or Linux kernel stats counters)
Write the error to an error collection service.
- For server side software: Record stack traces and notify developers about them.
- For client side software: Ask the user to send bug reports when errors are encountered.

Make sure that all the relevant error propagation sinks are monitored by the right stakeholders during the program’s operation.

Systems like Prometheus are built with monitoring and alerting in mind. It’s often a good idea to alert on symptoms close to your business needs and then use collected metrics on other errors for further analysis¹.

Warning: Textual logging is not an error handling strategy. Logs are not meant for machine consumption, so it’s hard to have automated monitoring based on them.

Warning: Ignoring errors is not an error handling strategy either.

What if my case is not a fit?

If your case it not a fit, it’s possible that the condition at hand might not be an error to begin with, but is maybe only a “corner case” which may happen in normal operation (e.g. a lookup key was not found).

Consider switching the way that the condition is propagated. The following options are alternatives where otherwise errors may be used.

Null/empty object: Returning empty lists, sets, Optional<T> types, or other Null Objects. Note that this is not the same as null.
Separate channel: Return the non-error conditions through a separate channel (e.g. don’t use an exception, but return specially-built objects for these conditions). Example: A linter tool’s output is produced as part of the expected operation and is therefore not an error within that context.
Null: Returning null, nil or similar is idiomatic in some languages too, but may lead to follow-up errors² when the value is dereferenced.

If none of these worked for you, I’d love to hear from you, so I can correct my understanding.

Sometimes the most elegant solution is to change the API to make the error impossible³. For example, replace a dynamic check with one that’s statically guaranteed by the type system, or change to idempotent operation semantics, so that multiple invocations do not conflict with each other.

Examples

These are all written from the perspective of a piece of code detecting an error.

Recovery through redundancy

Example

A disk in a software RAID is failing.

Stakeholder

The system administrator, as soon as the percentage of recovered executions > threshold. They can then investigate the cause of network issues, exchange hard drives or similar.

Mechanism

Recover by retrying the operation on a different disk.
Increment an error counter indicating the failure cause. When incremented in a monitorable system, this counter can trigger an alert.

Similar cases

A TCP network packet is lost. (Difference: Retry has less control over the network routing)

Recovery through omission

Example

A web server uses multiple database backends. One of the non-essential backends starts to return errors.

Stakeholder

The operator of the failing subsystem.

Mechanism

Omit the backend’s response. (Treat it as if it didn’t exist.)
Increment an error counter indicating the failure cause, so that the subsystem operator can be alerted.

Delegating the decision to the calling procedure

Example

The Unix open() function is asked to open a given filename for reading, but the file doesn’t exist.

Stakeholder

We don’t know who is responsible for passing the wrong filename, but somewhere in the call chain, someone is going to know where that filename comes from.

Mechanism

Report the error to the caller with the documented ENOENT error code.

Similar cases

Mistakes in passed input values are best handled by the code which passed them, and which can judge why they happened.

Reporting upwards and sideways at the same time

Example

A web server’s servlets are returning errors in the form of HTTP status codes.

Stakeholders

The requester is a stakeholder because they care about the request.
The web server’s operator is a stakeholder because they care about keeping overall error fractions within reasonable bounds.

Mechanisms

Propagate status via HTTP, upwards to the caller who has more context.
Count HTTP statuses keyed by relevant criteria (such as servlet), e.g. in Prometheus, for the web server operator.

A good background read is the Site Reliability Engineering Book, section “Symptoms vs. Causes” ↩︎
Wikipedia: Tony Hoare, section “Apologies and retractions” ↩︎
The book Philosophy of Software Design by John Ousterhout has an entire chapter on the idea of defining errors away. The author has also given a tech talk. ↩︎

There are my core dumps!

Sun, 02 Oct 2016 16:45:00 +0100

As noted before, with systemd, your core dumps disappear into one of systemd’s log sinks, instead of being stored to the current directory. By now, they are recoverable by normal users again.

TL;DR: Run coredumpctl gdb to invoke gdb with the most recent core dump.

Related bug about disappearing coredumps: https://bugs.freedesktop.org/show_bug.cgi?id=54288

Scenario: A simple program that crashes with a segfault:

~/dumpcore$ cat dumpcore.c
#include <unistd.h>
void dump_core()   { int* boom = NULL; *boom = 2000; }
void its_time_to() { dump_core(); }
int main()         { its_time_to(); }
~/dumpcore$ cc -g -o dumpcore dumpcore.c
~/dumpcore$ ./dumpcore
Segmentation fault (core dumped)

Core dumps get written to systemd’s core dumps directory (see man coredumpctl), and can be retrieved with the coredumpctl tool:

~/dumpcore$ coredumpctl list
TIME                            PID   UID   GID SIG PRESENT EXE
...
Sun 2016-10-02 14:44:14 CEST   3857  1000   100  11 * /home/me/dumpcore/dumpcore

By default, coredumpctl seems to use the most recent core dump, so that you can easily inspect the most recent core dump with:

~/dumpcore$ coredumpctl gdb
           PID: 3857 (dumpcore)
           UID: 1000 (me)
           GID: 100 (users)
        Signal: 11 (SEGV)
     Timestamp: Sun 2016-10-02 14:44:14 CEST (1min 3s ago)
  Command Line: ./dumpcore
    Executable: /home/me/dumpcore/dumpcore
 Control Group: /user.slice/user-1000.slice/user@1000.service/gnome-terminal-server.service
          Unit: user@1000.service
     User Unit: user@1000.service
         Slice: user-1000.slice
     Owner UID: 1000 (me)
       Boot ID: 73902fbce4e241f5bab007aad96895eb
    Machine ID: 3ff4d30a94914397acd6af26eed15114
      Hostname: limbo
      Coredump: /var/lib/systemd/coredump/core.dumpcore.1000.73902fbce4e241f5bab007aad96895eb.3857.1475412254000000000000.lz4
       Message: Process 3857 (dumpcore) of user 1000 dumped core.

                Stack trace of thread 3857:
                #0  0x00000000004004b6 dump_core (dumpcore)
                #1  0x00000000004004cd its_time_to (dumpcore)
                #2  0x00000000004004de main (dumpcore)
                #3  0x00007f89ba351291 __libc_start_main (libc.so.6)
                #4  0x00000000004003da _start (dumpcore)

GNU gdb (GDB) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/me/dumpcore/dumpcore...done.
[New LWP 3857]

warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
Core was generated by `./dumpcore'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00000000004004b6 in dump_core () at dumpcore.c:2
2	void dump_core()   { int* boom = NULL; *boom = 2000; }
(gdb) bt
#0  0x00000000004004b6 in dump_core () at dumpcore.c:2
#1  0x00000000004004cd in its_time_to () at dumpcore.c:3
#2  0x00000000004004de in main () at dumpcore.c:4
(gdb)

GPG with the CCID Driver

Mon, 15 Aug 2016 12:00:00 +0100

Remark: I have not had this problem in years, and it’s probably long resolved.

Unless you’re using Debian, where people care about this, chances are that setting up GPG Smartcards with your Linux distribution is an adventure. There are two ways to do it, CCID and PCSCLite, the latter of which runs a background service pcscd – as root, at least on Arch, while GnuPG’s built-in CCID driver accesses the reader directly through the USB device, but took me a lot longer to figure out how to use.

Problem

You’re getting a “card error” when trying to access the card. The same thing works as root.

$ gpg --card-status
gpg: selecting openpgp failed: Card error
gpg: OpenPGP card not available: Card error
$

When trying the same as root, you can access the card.

Solution

Add a udev rule that changes file permissions for you, as documented on https://wiki.archlinux.org/index.php/GnuPG#Smartcard_not_detected

Add your user to a new user group that can read smartcards, and give that group read-write access to the device when it’s plugged in, by creating a file /etc/udev/rules.d with the rule:

ACTION=="add", SUBSYSTEM=="usb", ENV{ID_VENDOR_ID}=="08e6", ENV{ID_MODEL_ID}=="3438", MODE="660", GROUP="scard"

(Figure out the values for the vendor and model ID using lsusb, they are printed separated with a colon.)

Hexiamonds

Sun, 05 Jun 2016 12:00:00 +0100

I made 3D models for Hexiamond puzzle pieces, which can be downloaded here (STL format).

Each Hexiamond piece consists of 6 triangles, which yields 12 different pieces when you try out all combinations.

3d model of a Hexiamond

More background information and some interesting puzzles to try can be found on this nice page by David Goodger: http://puzzler.sourceforge.net/docs/hexiamonds.html

Python-like generator functions

Sun, 16 Nov 2014 19:41:00 +0100

Python-like generator functions, implemented as a library:

Generator functions implemented as a library

This is possible through two simple tricks:

The language represents stack frames as objects on the heap.
There’s a native function for grabbing the current stack frame.

Turns out replacing the calling stack frame is really the only thing you need in order to implement coroutines and generics.

The implementation is only 23 lines long. On my screen, including the screenshot, this weblog article is already longer up until here.

Blog on blog.gnoack.org

Landlock Path Walk Inversion Experiment

Objective

Hypothesis

Implementation

Measurements

Effects on Landlock code complexity

Effects on Performance

Noteworthy results:

Conclusion

Full results

fs_bench

net_bench

scoped_bench

Shell Tricks: Extract text between two regexes

Landlock Multithreaded Policy Enforcement

The old workaround

Programming language support

C

Go

References

Nutella is in space

Parameterized testing: You want Inputs and Outputs

She sells Z-Shells at the Z-Shore

Martin Luther King Day

FOSDEM 2026: It is happening!

Google Big Sleep: Linux Vulnerabilities

Lore links in mutt

Listening on an ephemeral TCP port

The classic TCP client/server interaction

Some less known fun facts

Servers that listen on random ports

Clients that use a specific port number

Reference

Integer overflow checking with C23

Further references

Landlock Your Vibe Coding

Argument passing adventures

Fantastic new ways of passing arguments

TIOCL_SELMOUSEREPORT

What’s the takeaway here?

I broke Emacs mouse support on the console

References

Git without a hosted platform

How it’s done

What if I need more?

Dan Bernstein on sandboxing

Choice of sandboxing mechanisms

Sandboxing should be a developer task

Networking

Conclusion

Upcoming conferences 2025 Q1

Talk: Update on Landlock: IOCTL support

Talk Summary

In other news

Git: Multirepo patch set

1. Pull both original repositories in the same local repository

Example

2. Format a patch set that spans two otherwise unrelated branches

3. Final hand-editing

Upcoming talk at LSS Europe 2024

Landlock IOCTL control in Linux 6.10

LLM-generated docs are usually worthless

Opening /proc/self/fd/1 is not the same as dup(1)

Introduction

This feature is mis-documented

/dev/fd/* behave different on other Unixes

Part 1: An experiment!

Duplicating the file descriptor using dup(2)

Duplicating the file descriptor through /proc

Other file types

TCP Sockets: Can not be reopened through /proc

Pipes: Can be reopened through /proc

Part 2: What is really happening

Where did the no_open pointer come from?

Summary

Go-Landlock: Networking support

Demo

Go API

Public tweets are not public, haha!

`/dev/fd/*` behave different on other Unixes

Duplicating the file descriptor through `/proc`

TCP Sockets: Can not be reopened through `/proc`

Pipes: Can be reopened through `/proc`

Where did the `no_open` pointer come from?

Avoid `TestMain`, use setup helper functions