Programming for fun and profithttps://slar.se/2024-03-24T00:00:00+01:00A blog about software engineering, programming languages and technical tinkeringConfiguring touchpad tap in Sway2024-03-24T00:00:00+01:002024-03-24T00:00:00+01:00Simon Larséntag:slar.se,2024-03-24:/configuring-touchpad-in-sway.html<p>One thing that didn't carry over from my i3 setup <a href="https://slar.se/wayland-one-month-later.html">when moving over to
Sway (i.e. Wayland)</a> was the touchpad
configuration. Input devices was a display server concern under X.Org, so the
window manager had nothing to do with it. Notably, now click-on-tap is not
working for me …</p><p>One thing that didn't carry over from my i3 setup <a href="https://slar.se/wayland-one-month-later.html">when moving over to
Sway (i.e. Wayland)</a> was the touchpad
configuration. Input devices was a display server concern under X.Org, so the
window manager had nothing to do with it. Notably, now click-on-tap is not
working for me, which is massively annoying.</p>
<p>If you're just here for the touchpad click-on-tap configuration, it looks like this.</p>
<div class="highlight"><pre><span></span><code><span class="nx">input</span><span class="w"> </span><span class="s">"type:touchpad"</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">dwt</span><span class="w"> </span><span class="nx">enabled</span>
<span class="w"> </span><span class="nx">dwtp</span><span class="w"> </span><span class="nx">enabled</span>
<span class="w"> </span><span class="nx">tap</span><span class="w"> </span><span class="nx">enabled</span>
<span class="w"> </span><span class="nx">tap_button_map</span><span class="w"> </span><span class="nx">lrm</span>
<span class="p">}</span>
</code></pre></div>
<p>But if you want to improve your understanding of how to configure input devices
such as touchpads and mice in a Wayland compositor, then read on!</p>
<h1>Configuring a libinput device in Sway</h1>
<p>We need to add a libinput<sup id="sf-configuring-touchpad-in-sway-1-back"><a href="#sf-configuring-touchpad-in-sway-1" class="simple-footnote" title="libinput is a stack for common input devices, such as keyboards, touchpads and mice. Read more in the Wayland wiki.">1</a></sup>
configuration into the Sway config file, which is usually located at
<code>~/.config/sway/config</code>. The Sway wiki <a href="https://github.com/swaywm/sway/wiki#libinput-config-options">has a small example for libinput
devices</a> but it
doesn't really say all that much. Generally, an input configuration should look
like this.</p>
<div class="highlight"><pre><span></span><code>input <selector> {
option value
}
</code></pre></div>
<p>For the selector, we can use either the device identifier, or the type of the
device. I prefer the device type, i.e. <code>"type:touchpad"</code>, as I use the same
configuration file for multiple devices that don't share touchpad
identifiers<sup id="sf-configuring-touchpad-in-sway-2-back"><a href="#sf-configuring-touchpad-in-sway-2" class="simple-footnote" title="Note that the type selector will select all devices of that type. If you have multiple devices that identify as touchpads, you may want to be more specific and use the identifier.">2</a></sup>.</p>
<p>You can run <code>swaymsg -t get_inputs</code> to get information about your devices. You
will get a ton of output, but somewhere you'll find a device that has <code>Type:
Touchpad</code>. Mine looks like this.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>swaymsg<span class="w"> </span>-t<span class="w"> </span>get_inputs
...
Input<span class="w"> </span>device:<span class="w"> </span>SYNA1D31:00<span class="w"> </span>06CB:CD48<span class="w"> </span>Touchpad
<span class="w"> </span>Type:<span class="w"> </span>Touchpad
<span class="w"> </span>Identifier:<span class="w"> </span><span class="m">1739</span>:52552:SYNA1D31:00_06CB:CD48_Touchpad
<span class="w"> </span>Product<span class="w"> </span>ID:<span class="w"> </span><span class="m">52552</span>
<span class="w"> </span>Vendor<span class="w"> </span>ID:<span class="w"> </span><span class="m">1739</span>
<span class="w"> </span>Libinput<span class="w"> </span>Send<span class="w"> </span>Events:<span class="w"> </span>enabled
...
</code></pre></div>
<p>We can however get a whole lot more information about each device with the
<code>--raw</code> option<sup id="sf-configuring-touchpad-in-sway-3-back"><a href="#sf-configuring-touchpad-in-sway-3" class="simple-footnote" title="When piping swaymsg output to another command, --raw is implicit. I use it here explicitly for clarity.">3</a></sup>. Combining that with
<code>jq</code><sup id="sf-configuring-touchpad-in-sway-4-back"><a href="#sf-configuring-touchpad-in-sway-4" class="simple-footnote" title="If you don't know how to use jq, do yourself the biggest favor of the year and learn it.">4</a></sup> we can get only touchpads out of the command. On all my
devices, I only have a single touchpad device. It should look something like
this.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>swaymsg<span class="w"> </span>-t<span class="w"> </span>get_inputs<span class="w"> </span>--raw<span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span><span class="s1">'.[] | select(.type=="touchpad")'</span>
<span class="o">{</span>
<span class="w"> </span><span class="s2">"identifier"</span>:<span class="w"> </span><span class="s2">"1739:52552:SYNA1D31:00_06CB:CD48_Touchpad"</span>,
<span class="w"> </span><span class="s2">"name"</span>:<span class="w"> </span><span class="s2">"SYNA1D31:00 06CB:CD48 Touchpad"</span>,
<span class="w"> </span><span class="s2">"vendor"</span>:<span class="w"> </span><span class="m">1739</span>,
<span class="w"> </span><span class="s2">"product"</span>:<span class="w"> </span><span class="m">52552</span>,
<span class="w"> </span><span class="s2">"type"</span>:<span class="w"> </span><span class="s2">"touchpad"</span>,
<span class="w"> </span><span class="s2">"scroll_factor"</span>:<span class="w"> </span><span class="m">1</span>.0,
<span class="w"> </span><span class="s2">"libinput"</span>:<span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="s2">"send_events"</span>:<span class="w"> </span><span class="s2">"enabled"</span>,
<span class="w"> </span><span class="s2">"tap"</span>:<span class="w"> </span><span class="s2">"disabled"</span>,
<span class="w"> </span><span class="s2">"tap_button_map"</span>:<span class="w"> </span><span class="s2">"lrm"</span>,
<span class="w"> </span><span class="s2">"tap_drag"</span>:<span class="w"> </span><span class="s2">"enabled"</span>,
<span class="w"> </span><span class="s2">"tap_drag_lock"</span>:<span class="w"> </span><span class="s2">"disabled"</span>,
<span class="w"> </span><span class="s2">"accel_speed"</span>:<span class="w"> </span><span class="m">0</span>.0,
<span class="w"> </span><span class="s2">"accel_profile"</span>:<span class="w"> </span><span class="s2">"adaptive"</span>,
<span class="w"> </span><span class="s2">"natural_scroll"</span>:<span class="w"> </span><span class="s2">"disabled"</span>,
<span class="w"> </span><span class="s2">"left_handed"</span>:<span class="w"> </span><span class="s2">"disabled"</span>,
<span class="w"> </span><span class="s2">"click_method"</span>:<span class="w"> </span><span class="s2">"button_areas"</span>,
<span class="w"> </span><span class="s2">"middle_emulation"</span>:<span class="w"> </span><span class="s2">"disabled"</span>,
<span class="w"> </span><span class="s2">"scroll_method"</span>:<span class="w"> </span><span class="s2">"two_finger"</span>,
<span class="w"> </span><span class="s2">"dwt"</span>:<span class="w"> </span><span class="s2">"enabled"</span>,
<span class="w"> </span><span class="s2">"dwtp"</span>:<span class="w"> </span><span class="s2">"enabled"</span>
<span class="w"> </span><span class="o">}</span>
<span class="o">}</span>
</code></pre></div>
<p>This gives us everything we need to configure the touchpad, and we can also see
if our configuration takes effect correctly. The options shown under <code>libinput</code>
is what we have to play with. The names aren't necessarily self-explanatory, but
you can find decent descriptions of them in the <code>sway-input(5)</code><sup id="sf-configuring-touchpad-in-sway-5-back"><a href="#sf-configuring-touchpad-in-sway-5" class="simple-footnote" title="Run man 5 sway-input to get the given page. See my article on man page sections if the 5 is confusing to you.">5</a></sup>man
page under the <code>LIBINPUT CONFIGURATION</code> section.</p>
<p>I'm going to set the options that are crucial for how I use a touchpad,
regardless of what their defaults are. These options are:</p>
<div class="highlight"><pre><span></span><code><span class="nx">input</span><span class="w"> </span><span class="s">"type:touchpad"</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">tap</span><span class="w"> </span><span class="nx">enabled</span><span class="w"> </span><span class="err">#</span><span class="w"> </span><span class="nx">enables</span><span class="w"> </span><span class="nx">click</span><span class="o">-</span><span class="nx">on</span><span class="o">-</span><span class="nx">tap</span>
<span class="w"> </span><span class="nx">tap_button_map</span><span class="w"> </span><span class="nx">lrm</span><span class="w"> </span><span class="err">#</span><span class="w"> </span><span class="nx">tap</span><span class="w"> </span><span class="nx">with</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="nx">finger</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">left</span><span class="w"> </span><span class="nx">click</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="nx">fingers</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">right</span><span class="w"> </span><span class="nx">click</span><span class="p">,</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="nx">fingers</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">middle</span><span class="w"> </span><span class="nx">click</span>
<span class="w"> </span><span class="nx">dwt</span><span class="w"> </span><span class="nx">enabled</span><span class="w"> </span><span class="err">#</span><span class="w"> </span><span class="nx">disable</span><span class="w"> </span><span class="p">(</span><span class="nx">touchpad</span><span class="p">)</span><span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="nx">typing</span>
<span class="w"> </span><span class="nx">dwtp</span><span class="w"> </span><span class="nx">enabled</span><span class="w"> </span><span class="err">#</span><span class="w"> </span><span class="nx">disable</span><span class="w"> </span><span class="p">(</span><span class="nx">touchpad</span><span class="p">)</span><span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="nx">track</span><span class="w"> </span><span class="nx">pointing</span>
<span class="p">}</span>
</code></pre></div>
<p>Putting this in my <code>~/.config/sway/config</code><sup id="sf-configuring-touchpad-in-sway-6-back"><a href="#sf-configuring-touchpad-in-sway-6" class="simple-footnote" title="To see exactly how I use this in my Sway config, refer to my config repository.">6</a></sup>,
reloading the environment and then checking the settings shows that <code>tap</code> is
now enabled.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>swaymsg<span class="w"> </span>reload
$<span class="w"> </span>swaymsg<span class="w"> </span>-t<span class="w"> </span>get_inputs<span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span><span class="s1">'.[] | select(.type=="touchpad")'</span>
<span class="o">{</span>
<span class="w"> </span><span class="s2">"identifier"</span>:<span class="w"> </span><span class="s2">"1739:52552:SYNA1D31:00_06CB:CD48_Touchpad"</span>,
<span class="w"> </span><span class="o">[</span>...<span class="o">]</span>
<span class="w"> </span><span class="s2">"libinput"</span>:<span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="o">[</span>...<span class="o">]</span>
<span class="w"> </span><span class="s2">"tap"</span>:<span class="w"> </span><span class="s2">"enabled"</span>,
<span class="w"> </span><span class="s2">"tap_button_map"</span>:<span class="w"> </span><span class="s2">"lrm"</span>,
<span class="w"> </span><span class="o">[</span>...<span class="o">]</span>
<span class="w"> </span><span class="s2">"dwt"</span>:<span class="w"> </span><span class="s2">"enabled"</span>,
<span class="w"> </span><span class="s2">"dwtp"</span>:<span class="w"> </span><span class="s2">"enabled"</span>
<span class="w"> </span><span class="o">}</span>
<span class="o">}</span>
</code></pre></div>
<p>And that's that, click-on-tap now works again!</p>
<h1>Summary</h1>
<p>In this article, I outlined how to configure a touchpad in Sway. The principles
shown here carry over to any kind of libinput device, and should also be
applicable in most other Wayland compositors.</p><ol class="simple-footnotes"><li id="sf-configuring-touchpad-in-sway-1">libinput is a stack for common input devices, such
as keyboards, touchpads and mice. Read more <a href="https://wayland.freedesktop.org/libinput/doc/latest/what-is-libinput.html">in the Wayland
wiki</a>. <a href="#sf-configuring-touchpad-in-sway-1-back" class="simple-footnote-back">↩</a></li><li id="sf-configuring-touchpad-in-sway-2">Note that the type selector will select <em>all</em> devices of that
type. If you have multiple devices that identify as touchpads, you may want to
be more specific and use the identifier. <a href="#sf-configuring-touchpad-in-sway-2-back" class="simple-footnote-back">↩</a></li><li id="sf-configuring-touchpad-in-sway-3">When piping <code>swaymsg</code> output to another command, <code>--raw</code> is
implicit. I use it here explicitly for clarity. <a href="#sf-configuring-touchpad-in-sway-3-back" class="simple-footnote-back">↩</a></li><li id="sf-configuring-touchpad-in-sway-4">If you don't know how to use <code>jq</code>, do yourself the biggest favor of the
year and learn it. <a href="#sf-configuring-touchpad-in-sway-4-back" class="simple-footnote-back">↩</a></li><li id="sf-configuring-touchpad-in-sway-5">Run <code>man 5
sway-input</code> to get the given page. See my article on <a href="https://slar.se/man-page-numbering.html">man page
sections</a> if the <code>5</code> is confusing to you. <a href="#sf-configuring-touchpad-in-sway-5-back" class="simple-footnote-back">↩</a></li><li id="sf-configuring-touchpad-in-sway-6">To see exactly how I use this in
my Sway config, refer to <a href="https://github.com/slarse/config/commit/21b524a956bd942424aa944fa70cce01dd005ea8">my config
repository</a>. <a href="#sf-configuring-touchpad-in-sway-6-back" class="simple-footnote-back">↩</a></li></ol>Syntax highlight anything with Tree-sitter2024-03-11T00:00:00+01:002024-03-11T00:00:00+01:00Simon Larséntag:slar.se,2024-03-11:/syntax-highlight-anything-with-tree-sitter.html<p>As of my previous post on <a href="https://slar.se/comment-and-uncomment-code-in-neovim.html">extending NeoVim for commenting and uncommenting
code blocks</a>, I'm on something of a
NeoVim extension streak. The flavor of the week is syntax highlighting. I've been
using the highly customizable
<a href="https://github.com/nvim-treesitter/nvim-treesitter">nvim-treesitter</a> for the
past few years. This depends on there being a
<a href="https://tree-sitter.github.io/tree-sitter/">Tree-sitter</a> parser …</p><p>As of my previous post on <a href="https://slar.se/comment-and-uncomment-code-in-neovim.html">extending NeoVim for commenting and uncommenting
code blocks</a>, I'm on something of a
NeoVim extension streak. The flavor of the week is syntax highlighting. I've been
using the highly customizable
<a href="https://github.com/nvim-treesitter/nvim-treesitter">nvim-treesitter</a> for the
past few years. This depends on there being a
<a href="https://tree-sitter.github.io/tree-sitter/">Tree-sitter</a> parser for whatever
language you're working with. Usually there is, but sometimes you run into
languages that are esoteric enough that there aren't any parsers available. And
then you're SOL on the whole syntax highlighting part. Unless, of course, you
write a parser of your own. Which naturally is what we'll do.</p>
<p>This is the first article in a series on working with Tree-sitter to syntax
highlight anything. Although this article series is intended for a
NeoVim-inclined audience, this first part has nothing to do with NeoVim and can
be enjoyed by anyone interested in Tree-sitter or parsers in general. In the
still under construction second part, we'll dive into working with and utilizing
Tree-sitter in NeoVim.</p>
<blockquote>
<p><strong>Companion repository:</strong> The complete parser developed in this article is
available in a companion repository at
<a href="https://github.com/slarse/tree-sitter-mds">slarse/tree-sitter-mds</a></p>
</blockquote>
<h1>What's this Tree-sitter thing?</h1>
<p><a href="https://tree-sitter.github.io/tree-sitter/">Tree-sitter</a> is a parser generator
tool for the modern era. What's a parser generator, you ask? Well, it's exactly
what it sounds like: a tool to generate a parser! By defining the syntax of a
language in a way the parser generator understands, it can generate a parser
for you that can take source code of that language and transform it into a
syntax tree.</p>
<p>Parser generators aren't a new concept. <a href="https://en.wikipedia.org/wiki/GNU_Bison">GNU
Bison</a> has been around for 38 years at
this point, and <a href="https://en.wikipedia.org/wiki/Yacc">YACC</a> had been around for a
decade by the time Bison arrived. So, not at all a new concept. What's novel
with Tree-sitter is both how easy it is to define a grammar as long as you know
a little bit of JavaScript and the fact that <a href="https://tree-sitter.github.io/tree-sitter/syntax-highlighting">syntax highlighting is a built-in
feature</a>.</p>
<p>You can do a lot of cool things with Tree-sitter's syntax trees, but in this
article our focus is on syntax highlighting.</p>
<blockquote>
<p>This article assumes that you have access to a <code>bash</code>-like shell (such as
<code>bash</code> or <code>zsh</code>). Commands intended to be executed in a shell are prefixed
with <code>$</code>. Lines that follow a <code>$</code>-prefixed line but are not prefixed are
output lines.</p>
</blockquote>
<h1>Working example: Markdown Simple</h1>
<p>As a driving example for this article series, we'll look at a tiny subset of
Markdown, which should be familiar to most anyone in the target audience.
Specifically, we want to be able to highlight the following containing
headings, paragraphs, inline code and code blocks.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># This is a heading</span>
<span class="n">This</span><span class="w"> </span><span class="n">little</span><span class="w"> </span><span class="n">paragraph</span><span class="w"> </span><span class="n">of</span><span class="w"> </span><span class="n">text</span><span class="w"> </span><span class="n">with</span><span class="w"> </span><span class="err">`</span><span class="n">inline</span><span class="w"> </span><span class="n">code</span><span class="err">`</span>
<span class="err">```</span>
<span class="k">const</span><span class="w"> </span><span class="n">words</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="s2">"javascript"</span><span class="p">,</span><span class="w"> </span><span class="s2">"code"</span><span class="p">,</span><span class="w"> </span><span class="s2">"highlighting"</span><span class="p">];</span>
<span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="k">const</span><span class="w"> </span><span class="n">word</span><span class="w"> </span><span class="n">of</span><span class="w"> </span><span class="n">words</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">console</span><span class="o">.</span><span class="n">log</span><span class="p">(</span><span class="n">word</span><span class="p">);</span>
<span class="p">}</span>
<span class="err">```</span>
<span class="c1"># This is another heading</span>
<span class="n">Another</span><span class="w"> </span><span class="n">paragraph</span><span class="w"> </span><span class="n">with</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="c1"># in the middle.</span>
</code></pre></div>
<p><strong>Store these sample file contents in <code>test.mds</code> for later use</strong>!</p>
<p>I call this subset <em>Markdown Simple</em> and choose to write it in files with the
<code>.mds</code> file extension. At the end of this article, we'll have fully functioning
syntax highlighting for <code>.mds</code> files, including properly highlighted JavaScript
code in the multiline code blocks!</p>
<h1>Getting started with creating Tree-sitter parsers</h1>
<p>Tree-sitter's docs have a good <a href="https://tree-sitter.github.io/tree-sitter/creating-parsers#project-setup">Project
setup</a>
section, but I found that it lacks a few key ingredients for our purposes. I
will offer an augmented version here.</p>
<p>As a pre-requisite, you must have decently up-to-date versions <code>node</code> and <code>npm</code>
installed. With that, we can get started creating a project directory.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mkdir<span class="w"> </span>tree-sitter-mds
$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>tree-sitter-mds
$<span class="w"> </span>npm<span class="w"> </span>init
</code></pre></div>
<p>The <code>npm init</code> command will prompt you for a bunch of stuff. If you don't know
what to choose, just go with the defaults, it hardly matters for the rest of
this article. Now, you need the <code>tree-sitter-cli</code> application, which you can
also get with <code>npm</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>npm<span class="w"> </span>--save-dev<span class="w"> </span>tree-sitter-cli
</code></pre></div>
<p>Then setup a configuration file for <code>tree-sitter-cli</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>npx<span class="w"> </span>tree-sitter<span class="w"> </span>init-config
</code></pre></div>
<p>Note where the configuration file is written to. You may need to edit it later
for your project to be located by Tree-sitter.</p>
<h2>Baby's first grammar rule</h2>
<p>As previously mentioned, a parser generator reads a grammar that defines some
language and spits out a parser for it. In Tree-sitter, we define grammar rules
in JavaScript, in a file called <code>grammar.js</code> in the root of your project. It
should have the following structure.</p>
<div class="highlight"><pre><span></span><code><span class="nx">module</span><span class="p">.</span><span class="nx">exports</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">grammar</span><span class="p">({</span>
<span class="w"> </span><span class="nx">name</span><span class="o">:</span><span class="w"> </span><span class="s1">'markdownsimple'</span><span class="p">,</span>
<span class="w"> </span><span class="nx">rules</span><span class="o">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">// grammar rules go in here</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">});</span>
</code></pre></div>
<p>If you now try to generate a parser you should get an error about there not being any rules.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>npx<span class="w"> </span>tree-sitter<span class="w"> </span>generate
<span class="o">[</span>stdin<span class="o">]</span>:409
<span class="w"> </span>throw<span class="w"> </span>new<span class="w"> </span>Error<span class="o">(</span><span class="s2">"Grammar must have at least one rule."</span><span class="o">)</span><span class="p">;</span>
</code></pre></div>
<p>Let's define a very simple grammar just to be able to parse the file.</p>
<div class="highlight"><pre><span></span><code><span class="nx">rules</span><span class="o">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">source_file</span><span class="o">:</span><span class="w"> </span><span class="nx">$</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">text</span><span class="p">,</span>
<span class="w"> </span><span class="nx">text</span><span class="o">:</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="o">/</span><span class="p">(.</span><span class="o">|</span><span class="err">\</span><span class="nx">n</span><span class="p">)</span><span class="o">+</span><span class="err">/,</span>
<span class="p">}</span>
</code></pre></div>
<p>Put the above rules into your <code>grammar.js</code> file and then generate and run the
parser.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>npx<span class="w"> </span>tree-sitter<span class="w"> </span>generate
$<span class="w"> </span>npx<span class="w"> </span>tree-sitter<span class="w"> </span>parse<span class="w"> </span>test.mds
</code></pre></div>
<p>This should produce a simple syntax tree.</p>
<div class="highlight"><pre><span></span><code><span class="p">(</span><span class="nf">source_file</span><span class="w"> </span><span class="p">[</span><span class="nv">0,</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">13,</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span>
<span class="w"> </span><span class="p">(</span><span class="nf">text</span><span class="w"> </span><span class="p">[</span><span class="nv">0,</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">13,</span><span class="w"> </span><span class="mi">0</span><span class="p">]))</span>
</code></pre></div>
<p>Let's dissect the grammar in detail so we can improve the syntax tree
granularity. The root node is the <code>source_file</code>. Every syntax tree is rooted in
this node, and there is only one of them for each parsed file. The rule
definition may look a bit strange at first sight.</p>
<div class="highlight"><pre><span></span><code><span class="nx">source_file</span><span class="o">:</span><span class="w"> </span><span class="nx">$</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">text</span>
</code></pre></div>
<p>Any non-trivial grammar is built by composing rules. Any rule that references
another rule is a so-called
<a href="https://en.wikipedia.org/wiki/Terminal_and_nonterminal_symbols"><em>non-terminal</em></a>.
The above rule says that the <code>source_file</code> consists of precisely one <code>text</code>
node. So <code>source_file</code> is a non-terminal that references <code>text</code>.</p>
<p>We then define the <code>text</code> rule using regex.</p>
<div class="highlight"><pre><span></span><code><span class="nx">_</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="o">/</span><span class="p">(.</span><span class="o">+|</span><span class="err">\</span><span class="nx">n</span><span class="p">)</span><span class="o">/</span><span class="p">,</span>
</code></pre></div>
<p>This regex just captures any character (<code>.</code>) or line feeds (<code>\n</code>). As the rule
does not reference any other rules, it is a so-called
<a href="https://en.wikipedia.org/wiki/Terminal_and_nonterminal_symbols"><em>terminal</em></a>.</p>
<blockquote>
<p>Note: Tree-sitter supports <a href="https://en.wikipedia.org/wiki/LR_parser"><code>LR(1)</code>
grammars</a>, which <a href="https://tree-sitter.github.io/tree-sitter/creating-parsers#the-grammar-dsl">limits the kinds of
regex
expressions</a>
that can be used. The finer details of this is beyond the scope of this
article.</p>
</blockquote>
<p>In summary, our grammar currently defines a <code>source_file</code> as a single <code>text</code>
node. Of course, we could immediately define <code>source_file</code> as the regex that
currently resides in <code>text</code> at this point; I just wanted to illustrate the
concept of terminal and non-terminal rules as it will become important in the
next part.</p>
<h2>Let there be color</h2>
<p>The goal of this article is to achieve nice syntax highlighting, so let's get
started immediately. We can highlight a file using the <code>highlight</code> command,
but it won't work out of the box.</p>
<p>If you run the below command, you should get an error message.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>npx<span class="w"> </span>tree-sitter<span class="w"> </span>highlight<span class="w"> </span>test.mds
No<span class="w"> </span>language<span class="w"> </span>found<span class="w"> </span><span class="k">for</span><span class="w"> </span>path<span class="w"> </span><span class="sb">`</span>test.mds<span class="sb">`</span>
If<span class="w"> </span>a<span class="w"> </span>language<span class="w"> </span>should<span class="w"> </span>be<span class="w"> </span>associated<span class="w"> </span>with<span class="w"> </span>this<span class="w"> </span>file<span class="w"> </span>extension,<span class="w"> </span>please<span class="w"> </span>ensure<span class="w"> </span>the<span class="w"> </span>path<span class="w"> </span>to<span class="w"> </span><span class="sb">`</span>test.mds<span class="sb">`</span><span class="w"> </span>is<span class="w"> </span>inside<span class="w"> </span>one<span class="w"> </span>of<span class="w"> </span>the<span class="w"> </span>following<span class="w"> </span>directories<span class="w"> </span>as<span class="w"> </span>specified<span class="w"> </span>by<span class="w"> </span>your<span class="w"> </span><span class="s1">'config.json'</span>:
<span class="w"> </span><span class="m">1</span>.<span class="w"> </span>/home/slarse/github<span class="w"> </span>
<span class="w"> </span><span class="m">2</span>.<span class="w"> </span>/home/slarse/src<span class="w"> </span>
<span class="w"> </span><span class="m">3</span>.<span class="w"> </span>/home/slarse/source<span class="w"> </span>
<span class="w"> </span><span class="m">4</span>.<span class="w"> </span>/home/slarse/projects<span class="w"> </span>
<span class="w"> </span><span class="m">5</span>.<span class="w"> </span>/home/slarse/dev<span class="w"> </span>
<span class="w"> </span><span class="m">6</span>.<span class="w"> </span>/home/slarse/git
If<span class="w"> </span>the<span class="w"> </span>directory<span class="w"> </span>that<span class="w"> </span>contains<span class="w"> </span>the<span class="w"> </span>relevant<span class="w"> </span>grammar<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="sb">`</span>test.mds<span class="sb">`</span><span class="w"> </span>is<span class="w"> </span>not<span class="w"> </span>listed<span class="w"> </span>above,<span class="w"> </span>please<span class="w"> </span>add<span class="w"> </span>the<span class="w"> </span>directory<span class="w"> </span>to<span class="w"> </span>the<span class="w"> </span>list<span class="w"> </span>of<span class="w"> </span>directories<span class="w"> </span><span class="k">in</span><span class="w"> </span>your<span class="w"> </span>config<span class="w"> </span>file,<span class="w"> </span>located<span class="w"> </span>at<span class="w"> </span>/home/slarse/.config/tree-sitter/config.json
</code></pre></div>
<p>To resolve this we need to add some information to <code>package.json</code>. Put the
following in there.</p>
<blockquote>
<p>Tip: It's easy to break the <code>package.json</code> file with a misplaced comma. To
ensure that you don't break the syntax of <code>package.json</code>, run <code>npm i</code> until it
doesn't complain anymore.</p>
</blockquote>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="nt">"tree-sitter"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"scope"</span><span class="p">:</span><span class="w"> </span><span class="s2">"source.markdownsimple"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"file-types"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"mds"</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">"highlights"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"queries/highlights.scm"</span>
<span class="w"> </span><span class="p">]</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">],</span>
</code></pre></div>
<p>Now run the <code>highlight</code> command again. If you get the <em>same</em> error as before,
you need to add the directory that contains you project's directory to
Tree-sitter's <code>config.json</code> file. For example, my project is located in
<code>/home/slarse/projects/tree-sitter/tree-sitter-mds</code>, so I add the path
<code>/home/slarse/projects/tree-sitter</code> to the list of paths at the top of
<code>/home/slarse/.config/tree-sitter/config.json</code>.</p>
<p>With that fixed, you should get a new error.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>npx<span class="w"> </span>tree-sitter<span class="w"> </span>highlight<span class="w"> </span>test.mds
Failed<span class="w"> </span>to<span class="w"> </span><span class="nb">read</span><span class="w"> </span>query<span class="w"> </span>file<span class="w"> </span><span class="s2">"queries/highlights.scm"</span>
Caused<span class="w"> </span>by:
<span class="w"> </span>No<span class="w"> </span>such<span class="w"> </span>file<span class="w"> </span>or<span class="w"> </span>directory<span class="w"> </span><span class="o">(</span>os<span class="w"> </span>error<span class="w"> </span><span class="m">2</span><span class="o">)</span>
</code></pre></div>
<p>This means your project was correctly located. We now need to define <em>queries</em>
for our highlight. Queries allow us to select nodes in the syntax tree and
assign pre-defined semantic meaning to them. Put the following content in the
file <code>queries/highlights.scm</code> in your project directory.</p>
<div class="highlight"><pre><span></span><code><span class="p">(</span><span class="nf">text</span><span class="p">)</span><span class="w"> </span><span class="nv">@string</span>
</code></pre></div>
<blockquote>
<p>I won't go into detail on queries in this article. You can find all captures
(e.g. <code>@string</code>) available <a href="https://github.com/nvim-treesitter/nvim-treesitter/blob/master/CONTRIBUTING.md#highlights">over
here</a>,
and you can read more about queries as a whole <a href="https://tree-sitter.github.io/tree-sitter/using-parsers#pattern-matching-with-queries">over
here</a>.</p>
</blockquote>
<p>This says that any <code>text</code> node should be considered a <code>string</code>. Running the
highlight command again, you ought to get some colored output.</p>
<p><img alt="Simple syntax higlighting" src="https://slar.se/photos/tree_sitter_highlighting/first_highlighta.jpg"></p>
<p>Not all that impressive; as all nodes with any content are <code>text</code> nodes
everything is just highlighted as a string. In my Tree-sitter configuration
the <code>string</code> color happens happens to be a shade of green, but it may be
different for you. For more granular highlighting, we need a more granular
syntax tree to perform queries against.</p>
<h1>Refining the grammar</h1>
<p>The basic pieces of the puzzle are now in place. We have a rudimentary syntax
tree allowing for an equally rudimentary syntax highlighting. It's now time to
start refining the grammar to capture different components. With some crude
annotations in <code>()</code>, we want a structure like this.</p>
<div class="highlight"><pre><span></span><code><span class="p">(</span><span class="n">section</span><span class="p">)</span>
<span class="c1"># This is a heading (heading)</span>
<span class="n">This</span><span class="w"> </span><span class="n">little</span><span class="w"> </span><span class="n">paragraph</span><span class="w"> </span><span class="n">of</span><span class="w"> </span><span class="n">text</span><span class="w"> </span><span class="n">with</span><span class="w"> </span><span class="err">`</span><span class="n">inline</span><span class="w"> </span><span class="n">code</span><span class="err">`</span><span class="w"> </span><span class="p">(</span><span class="n">paragraph</span><span class="w"> </span><span class="p">(</span><span class="n">text</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="n">inlineCode</span><span class="w"> </span><span class="p">(</span><span class="n">codeText</span><span class="p">)))</span>
<span class="err">```</span><span class="w"> </span><span class="p">(</span><span class="n">codeBlock</span><span class="w"> </span><span class="p">(</span><span class="n">codeText</span><span class="p">))</span>
<span class="k">const</span><span class="w"> </span><span class="n">words</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="s2">"javascript"</span><span class="p">,</span><span class="w"> </span><span class="s2">"code"</span><span class="p">,</span><span class="w"> </span><span class="s2">"highlighting"</span><span class="p">];</span>
<span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="k">const</span><span class="w"> </span><span class="n">word</span><span class="w"> </span><span class="n">of</span><span class="w"> </span><span class="n">words</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">console</span><span class="o">.</span><span class="n">log</span><span class="p">(</span><span class="n">word</span><span class="p">);</span>
<span class="p">}</span>
<span class="err">```</span>
<span class="c1"># This is another heading (heading)</span>
<span class="n">And</span><span class="w"> </span><span class="n">this</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">another</span><span class="w"> </span><span class="n">little</span><span class="w"> </span><span class="n">paragraph</span><span class="o">.</span><span class="w"> </span><span class="p">(</span><span class="n">paragraph</span><span class="w"> </span><span class="p">(</span><span class="n">text</span><span class="p">))</span>
</code></pre></div>
<p>Let's start off only with capturing the additional <code>section</code> and <code>heading</code> nodes
with new rules.</p>
<div class="highlight"><pre><span></span><code><span class="nx">rules</span><span class="o">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">source_file</span><span class="o">:</span><span class="w"> </span><span class="nx">$</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">repeat</span><span class="p">(</span><span class="nx">$</span><span class="p">.</span><span class="nx">section</span><span class="p">),</span>
<span class="w"> </span><span class="nx">section</span><span class="o">:</span><span class="w"> </span><span class="nx">$</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">seq</span><span class="p">(</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">heading</span><span class="p">,</span>
<span class="w"> </span><span class="nx">repeat</span><span class="p">(</span><span class="nx">$</span><span class="p">.</span><span class="nx">text</span><span class="p">),</span>
<span class="w"> </span><span class="p">),</span>
<span class="w"> </span><span class="nx">heading</span><span class="o">:</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="o">/</span><span class="err">#</span><span class="p">.</span><span class="o">+</span><span class="err">/,</span>
<span class="w"> </span><span class="nx">text</span><span class="o">:</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="o">/</span><span class="p">(.</span><span class="o">|</span><span class="err">\</span><span class="nx">n</span><span class="p">)</span><span class="o">+</span><span class="err">/,</span>
<span class="p">}</span>
</code></pre></div>
<p>Our non-terminals <code>source_file</code> and <code>section</code> bring in ways of composing rules
with the <code>repeat()</code> and <code>seq()</code> composers. <code>repeat(rule)</code> means "repeat rule
zero or more times, while <code>seq(rules...)</code> means "match these rules in this
order". So a <code>source_file</code> node is zero or more <code>section</code> nodes, and a <code>section</code>
node is a <code>heading</code> followed by zero or more <code>text</code> nodes.</p>
<p>Generating and running the parser again, this lands us with a slightly more
granular syntax tree where we separately capture the section's heading.</p>
<div class="highlight"><pre><span></span><code><span class="p">(</span><span class="nf">source_file</span><span class="w"> </span><span class="p">[</span><span class="nv">0,</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">13,</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span>
<span class="w"> </span><span class="p">(</span><span class="nf">section</span><span class="w"> </span><span class="p">[</span><span class="nv">0,</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">13,</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span>
<span class="w"> </span><span class="p">(</span><span class="nf">heading</span><span class="w"> </span><span class="p">[</span><span class="nv">0,</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">0,</span><span class="w"> </span><span class="mi">19</span><span class="p">])</span>
<span class="w"> </span><span class="p">(</span><span class="nf">text</span><span class="w"> </span><span class="p">[</span><span class="nv">0,</span><span class="w"> </span><span class="mi">19</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">13,</span><span class="w"> </span><span class="mi">0</span><span class="p">])))</span>
</code></pre></div>
<p>Clearly there's a bug here: we only have one section, yet our source file
contains two. The reason for this is that <code>text</code> is too liberal, it captures <code>#</code>
characters anywhere on the line. We must adjust the regex to disallow <code>#</code>
directly after line feeds.</p>
<div class="highlight"><pre><span></span><code><span class="nx">text</span><span class="o">:</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="o">/</span><span class="p">([</span><span class="o">^</span><span class="err">#</span><span class="p">]</span><span class="o">|</span><span class="p">[</span><span class="o">^</span><span class="err">\</span><span class="nx">n</span><span class="p">]</span><span class="err">#</span><span class="p">)</span><span class="o">+</span><span class="err">/,</span>
</code></pre></div>
<p>Here we say that <code>text</code> is either not a <code>#</code>, or it's a non-linefeed character
followed by <code>#</code>. Generating and parsing again we should now have our basic
section structure mapped out.</p>
<div class="highlight"><pre><span></span><code><span class="p">(</span><span class="nf">source_file</span><span class="w"> </span><span class="p">[</span><span class="nv">0,</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">13,</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span>
<span class="w"> </span><span class="p">(</span><span class="nf">section</span><span class="w"> </span><span class="p">[</span><span class="nv">0,</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">11,</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span>
<span class="w"> </span><span class="p">(</span><span class="nf">heading</span><span class="w"> </span><span class="p">[</span><span class="nv">0,</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">0,</span><span class="w"> </span><span class="mi">19</span><span class="p">])</span>
<span class="w"> </span><span class="p">(</span><span class="nf">text</span><span class="w"> </span><span class="p">[</span><span class="nv">0,</span><span class="w"> </span><span class="mi">19</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">11,</span><span class="w"> </span><span class="mi">0</span><span class="p">]))</span>
<span class="w"> </span><span class="p">(</span><span class="nf">section</span><span class="w"> </span><span class="p">[</span><span class="nv">11,</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">13,</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span>
<span class="w"> </span><span class="p">(</span><span class="nf">heading</span><span class="w"> </span><span class="p">[</span><span class="nv">11,</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">11,</span><span class="w"> </span><span class="mi">25</span><span class="p">])</span>
<span class="w"> </span><span class="p">(</span><span class="nf">text</span><span class="w"> </span><span class="p">[</span><span class="nv">11,</span><span class="w"> </span><span class="mi">25</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">13,</span><span class="w"> </span><span class="mi">0</span><span class="p">])))</span>
</code></pre></div>
<p>Great! We can now also adjust the query in <code>queries/highlights.scm</code> to only
highlight headings.</p>
<div class="highlight"><pre><span></span><code><span class="p">(</span><span class="nf">heading</span><span class="p">)</span><span class="w"> </span><span class="nv">@string</span>
</code></pre></div>
<p>This should land you with the following highlighting.</p>
<p><img alt="Syntax highlighting only on headings" src="https://slar.se/photos/tree_sitter_highlighting/second_highlighta.jpg"></p>
<p>Nice!</p>
<h1>Capturing inline code and code blocks</h1>
<p>The only thing left for us to capture is inline code and code blocks. This is
however where it starts to get a little bit complicated, so let's take each of
these in turn.</p>
<h2>Capturing code blocks</h2>
<p>Code blocks are relatively simple to capture in isolation. A rule for code
blocks could look as follows.</p>
<div class="highlight"><pre><span></span><code><span class="nx">codeBlock</span><span class="o">:</span><span class="w"> </span><span class="nx">$</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">seq</span><span class="p">(</span>
<span class="w"> </span><span class="s1">'```'</span><span class="p">,</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">codeText</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'```'</span><span class="p">,</span>
<span class="p">),</span>
<span class="nx">codeText</span><span class="o">:</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="o">/</span><span class="p">[</span><span class="o">^</span><span class="sb">`]*/,</span>
</code></pre></div>
<p>We could inline <code>codeText</code> into the <code>codeBlock</code>, but making it a separate rule
will make it easier for us to highlight the content of the code block without
affecting the backticks around it. Note that <code>codeText</code> is not quite the same as
<code>text</code>; the former allows any <em>zero or more</em> characters except for backticks.
This will of course make it impossible to use backticks inside of a <code>codeBlock</code>,
which is a simplification we're going with here. Extending the rule to allow
for backticks within a <code>codeBlock</code> node is left as an exercise to the reader.</p>
<p>We can now extend our <code>section</code> rule to contain <code>codeBlock</code>s.</p>
<div class="highlight"><pre><span></span><code><span class="nx">section</span><span class="o">:</span><span class="w"> </span><span class="nx">$</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">seq</span><span class="p">(</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">heading</span><span class="p">,</span>
<span class="w"> </span><span class="nx">repeat</span><span class="p">(</span>
<span class="w"> </span><span class="nx">choice</span><span class="p">(</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">text</span><span class="p">,</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">codeBlock</span><span class="p">,</span>
<span class="w"> </span><span class="p">)</span>
<span class="w"> </span><span class="p">),</span>
<span class="p">),</span>
</code></pre></div>
<p>If you put this in and run it, you may be a bit surprised to see that the output
doesn't change from before. The syntax tree still consists of two <code>section</code>s
containing one <code>heading</code> and <code>text</code> each. This is because <code>text</code> is consuming
the backticks that should delimit a <code>codeBlock</code>. We'll address this at the same
time as we define our paragraphs in the next part.</p>
<h2>Paragraphs with text and inline code</h2>
<p>We want to define grammar rules such that the parser can recognize <em>paragraphs</em>
that are composed of one or more <code>text</code> and/or <code>inlineCode</code> nodes. The
<code>inlineCode</code> rule is quite simple: a backtick, followed by zero or more
characters that are anything but a backtick (i.e. <code>codeText</code>), followed by a
backtick.</p>
<div class="highlight"><pre><span></span><code><span class="nx">inlineCode</span><span class="o">:</span><span class="w"> </span><span class="nx">$</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">seq</span><span class="p">(</span>
<span class="w"> </span><span class="s1">'`'</span><span class="p">,</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">codeText</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'`'</span><span class="p">,</span>
<span class="p">),</span>
</code></pre></div>
<p>We also need to alter the <code>text</code> terminal to not allow backticks, because
otherwise it will just consume them like it did the <code>#</code> before. This will also
allow our <code>codeBlock</code> rule to actually consume something.</p>
<div class="highlight"><pre><span></span><code><span class="nx">text</span><span class="o">:</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="o">/</span><span class="p">([</span><span class="o">^</span><span class="err">#</span><span class="sb">`]|[^\n]#)+/,</span>
</code></pre></div>
<p>Now we can define the <code>paragraph</code> non-terminal which, again, should be one or
more <code>inlineCode</code> or <code>text</code>.</p>
<div class="highlight"><pre><span></span><code><span class="nx">paragraph</span><span class="o">:</span><span class="w"> </span><span class="nx">$</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">repeat1</span><span class="p">(</span>
<span class="w"> </span><span class="nx">choice</span><span class="p">(</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">inlineCode</span><span class="p">,</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">text</span><span class="p">,</span>
<span class="w"> </span><span class="p">),</span>
<span class="p">),</span>
</code></pre></div>
<p>And then we update the <code>section</code> to repeat <code>paragraph</code> instead of <code>text</code>.</p>
<div class="highlight"><pre><span></span><code><span class="nx">section</span><span class="o">:</span><span class="w"> </span><span class="nx">$</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">seq</span><span class="p">(</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">heading</span><span class="p">,</span>
<span class="w"> </span><span class="nx">repeat</span><span class="p">(</span>
<span class="w"> </span><span class="nx">choice</span><span class="p">(</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">paragraph</span><span class="p">,</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">codeBlock</span><span class="p">,</span>
<span class="w"> </span><span class="p">)</span>
<span class="w"> </span><span class="p">),</span>
<span class="p">),</span>
</code></pre></div>
<p>We end up with the following rule set.</p>
<div class="highlight"><pre><span></span><code><span class="nx">rules</span><span class="o">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">source_file</span><span class="o">:</span><span class="w"> </span><span class="nx">$</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">repeat</span><span class="p">(</span><span class="nx">$</span><span class="p">.</span><span class="nx">section</span><span class="p">),</span>
<span class="w"> </span><span class="nx">section</span><span class="o">:</span><span class="w"> </span><span class="nx">$</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">seq</span><span class="p">(</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">heading</span><span class="p">,</span>
<span class="w"> </span><span class="nx">repeat</span><span class="p">(</span>
<span class="w"> </span><span class="nx">choice</span><span class="p">(</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">paragraph</span><span class="p">,</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">codeBlock</span><span class="p">,</span>
<span class="w"> </span><span class="p">)</span>
<span class="w"> </span><span class="p">),</span>
<span class="w"> </span><span class="p">),</span>
<span class="w"> </span><span class="nx">paragraph</span><span class="o">:</span><span class="w"> </span><span class="nx">$</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">repeat1</span><span class="p">(</span>
<span class="w"> </span><span class="nx">choice</span><span class="p">(</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">inlineCode</span><span class="p">,</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">text</span><span class="p">,</span>
<span class="w"> </span><span class="p">),</span>
<span class="w"> </span><span class="p">),</span>
<span class="w"> </span><span class="nx">inlineCode</span><span class="o">:</span><span class="w"> </span><span class="nx">$</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">seq</span><span class="p">(</span>
<span class="w"> </span><span class="s1">'`'</span><span class="p">,</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">codeText</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'`'</span><span class="p">,</span>
<span class="w"> </span><span class="p">),</span>
<span class="w"> </span><span class="nx">codeBlock</span><span class="o">:</span><span class="w"> </span><span class="nx">$</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">seq</span><span class="p">(</span>
<span class="w"> </span><span class="s1">'```'</span><span class="p">,</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">codeText</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'```'</span><span class="p">,</span>
<span class="w"> </span><span class="p">),</span>
<span class="w"> </span><span class="nx">codeText</span><span class="o">:</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="o">/</span><span class="p">[</span><span class="o">^</span><span class="sb">`]*/,</span>
<span class="sb"> heading: _ => /#.+/,</span>
<span class="sb"> text: _ => /([^#`</span><span class="p">]</span><span class="o">|</span><span class="p">[</span><span class="o">^</span><span class="err">\</span><span class="nx">n</span><span class="p">]</span><span class="err">#</span><span class="p">)</span><span class="o">+</span><span class="err">/,</span>
<span class="p">}</span>
</code></pre></div>
<p>If you try this out, you'll find that it doesn't work, Tree-sitter won't even
generate a parser for you. Instead, it complains about a conflict. Oh, dear.</p>
<h2>Resolving a conflict in the grammar</h2>
<p>A conflict in the grammar occurs when there are multiple ways for Tree-sitter
to interpret the rules you've provided it with. In the case of our <code>paragraph</code>
rule, Tree-sitter has the following to say.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>npx<span class="w"> </span>tree-sitter<span class="w"> </span>generate
Unresolved<span class="w"> </span>conflict<span class="w"> </span><span class="k">for</span><span class="w"> </span>symbol<span class="w"> </span>sequence:
<span class="w"> </span>heading<span class="w"> </span>paragraph_repeat1<span class="w"> </span>•<span class="w"> </span><span class="s1">'`'</span><span class="w"> </span>…
Possible<span class="w"> </span>interpretations:
<span class="w"> </span><span class="m">1</span>:<span class="w"> </span>heading<span class="w"> </span><span class="o">(</span>paragraph<span class="w"> </span>paragraph_repeat1<span class="o">)</span><span class="w"> </span>•<span class="w"> </span><span class="s1">'`'</span><span class="w"> </span>…
<span class="w"> </span><span class="m">2</span>:<span class="w"> </span>heading<span class="w"> </span><span class="o">(</span>paragraph_repeat1<span class="w"> </span>paragraph_repeat1<span class="w"> </span>•<span class="w"> </span>paragraph_repeat1<span class="o">)</span>
Possible<span class="w"> </span>resolutions:
<span class="w"> </span><span class="m">1</span>:<span class="w"> </span>Specify<span class="w"> </span>a<span class="w"> </span>left<span class="w"> </span>or<span class="w"> </span>right<span class="w"> </span>associativity<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="sb">`</span>paragraph<span class="sb">`</span>
<span class="w"> </span><span class="m">2</span>:<span class="w"> </span>Add<span class="w"> </span>a<span class="w"> </span>conflict<span class="w"> </span><span class="k">for</span><span class="w"> </span>these<span class="w"> </span>rules:<span class="w"> </span><span class="sb">`</span>paragraph<span class="sb">`</span>
</code></pre></div>
<p>This is a little bit hard to interpret if you haven't worked a lot with parsers
before. The short of it is that our use of the <code>paragraph</code> rule is ambiguous due
to the <code>paragraph</code> being repeated in the <code>section</code> rule, and then is itself a
repetition. For example, how should the first paragraph of the first section be
parsed, as shown below, be parsed?</p>
<div class="highlight"><pre><span></span><code>This little paragraph of text with `inline code`
</code></pre></div>
<p>Is it one paragraph containing <code>text</code> and <code>inlineCode</code>, i.e. <code>(paragraph (text)
(inlineCode))</code>? Or is it perhaps two paragraphs, one with the <code>text</code> and one
with the <code>inlineCode</code>, i.e. <code>(paragraph (text))</code> and <code>(paragraph (inlineCode))</code>?
It's impossible to tell based on the rules we've defined.</p>
<p>To get rid of this ambiguity, we can use
<a href="https://tree-sitter.github.io/tree-sitter/creating-parsers#using-associativity"><em>associativity</em></a>.
As associativity is a bit of a brain twister the first time around, we won't
bother trying to fully understand it here. We'll just try both and see what
happens. We can make a rule left- or right-associative by wrapping it in
<code>prec.left()</code> or <code>prec.right()</code>, respectively. Doing that in turn, we get the
following two parse results for this part of the tree.</p>
<div class="highlight"><pre><span></span><code><span class="c1">; left-associative result</span>
<span class="p">(</span><span class="nf">paragraph</span><span class="w"> </span><span class="p">(</span><span class="nf">text</span><span class="p">))</span>
<span class="p">(</span><span class="nf">paragraph</span><span class="w"> </span><span class="p">(</span><span class="nf">inlineCode</span><span class="p">))</span>
<span class="c1">; right-associative result</span>
<span class="p">(</span><span class="nf">paragraph</span><span class="w"> </span><span class="p">(</span><span class="nf">text</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">inlineCode</span><span class="p">))</span>
</code></pre></div>
<p>The right-associative version looks like what we sketched out before embarking
down this road, so we'll adjust the rule accordingly.</p>
<div class="highlight"><pre><span></span><code><span class="nx">paragraph</span><span class="o">:</span><span class="w"> </span><span class="nx">$</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">prec</span><span class="p">.</span><span class="nx">right</span><span class="p">(</span><span class="nx">repeat1</span><span class="p">(</span>
<span class="w"> </span><span class="nx">choice</span><span class="p">(</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">inlineCode</span><span class="p">,</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">text</span><span class="p">,</span>
<span class="w"> </span><span class="p">),</span>
<span class="p">)),</span>
</code></pre></div>
<p>And with that, we can now create a syntax tree with a sufficient level of
granularity to achieve the highlighting we set out to. If you generate the
parser again and run it, you should see a tree like this.</p>
<div class="highlight"><pre><span></span><code><span class="p">(</span><span class="nf">source_file</span><span class="w"> </span><span class="p">[</span><span class="nv">0,</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">13,</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span>
<span class="w"> </span><span class="p">(</span><span class="nf">section</span><span class="w"> </span><span class="p">[</span><span class="nv">0,</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">9,</span><span class="w"> </span><span class="mi">3</span><span class="p">]</span>
<span class="w"> </span><span class="p">(</span><span class="nf">heading</span><span class="w"> </span><span class="p">[</span><span class="nv">0,</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">0,</span><span class="w"> </span><span class="mi">19</span><span class="p">])</span>
<span class="w"> </span><span class="p">(</span><span class="nf">paragraph</span><span class="w"> </span><span class="p">[</span><span class="nv">0,</span><span class="w"> </span><span class="mi">19</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">1,</span><span class="w"> </span><span class="mi">48</span><span class="p">]</span>
<span class="w"> </span><span class="p">(</span><span class="nf">text</span><span class="w"> </span><span class="p">[</span><span class="nv">0,</span><span class="w"> </span><span class="mi">19</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">1,</span><span class="w"> </span><span class="mi">35</span><span class="p">])</span>
<span class="w"> </span><span class="p">(</span><span class="nf">inlineCode</span><span class="w"> </span><span class="p">[</span><span class="nv">1,</span><span class="w"> </span><span class="mi">35</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">1,</span><span class="w"> </span><span class="mi">48</span><span class="p">]</span>
<span class="w"> </span><span class="p">(</span><span class="nf">codeText</span><span class="w"> </span><span class="p">[</span><span class="nv">1,</span><span class="w"> </span><span class="mi">36</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">1,</span><span class="w"> </span><span class="mi">47</span><span class="p">])))</span>
<span class="w"> </span><span class="p">(</span><span class="nf">codeBlock</span><span class="w"> </span><span class="p">[</span><span class="nv">1,</span><span class="w"> </span><span class="mi">48</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">9,</span><span class="w"> </span><span class="mi">3</span><span class="p">]</span>
<span class="w"> </span><span class="p">(</span><span class="nf">codeText</span><span class="w"> </span><span class="p">[</span><span class="nv">3,</span><span class="w"> </span><span class="mi">3</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">9,</span><span class="w"> </span><span class="mi">0</span><span class="p">])))</span>
<span class="w"> </span><span class="p">(</span><span class="nf">section</span><span class="w"> </span><span class="p">[</span><span class="nv">9,</span><span class="w"> </span><span class="mi">3</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">13,</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span>
<span class="w"> </span><span class="p">(</span><span class="nf">heading</span><span class="w"> </span><span class="p">[</span><span class="nv">9,</span><span class="w"> </span><span class="mi">3</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">11,</span><span class="w"> </span><span class="mi">25</span><span class="p">])</span>
<span class="w"> </span><span class="p">(</span><span class="nf">paragraph</span><span class="w"> </span><span class="p">[</span><span class="nv">11,</span><span class="w"> </span><span class="mi">25</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">13,</span><span class="w"> </span><span class="mi">0</span><span class="p">]</span>
<span class="w"> </span><span class="p">(</span><span class="nf">text</span><span class="w"> </span><span class="p">[</span><span class="nv">11,</span><span class="w"> </span><span class="mi">25</span><span class="p">]</span><span class="w"> </span><span class="nb">-</span><span class="w"> </span><span class="p">[</span><span class="nv">13,</span><span class="w"> </span><span class="mi">0</span><span class="p">]))))</span>
</code></pre></div>
<p>We still only have a single highlight query, though, so next up is improving
that.</p>
<h2>Improving the highlight queries</h2>
<p>We now have a significantly more granular syntax tree to work with. As a first
attempt, let's color inline code including backticks with one color, code block
backticks with one color and finally the code blocks themselves with another
color.</p>
<div class="highlight"><pre><span></span><code><span class="p">(</span><span class="nf">heading</span><span class="p">)</span><span class="w"> </span><span class="nv">@string</span>
<span class="p">(</span><span class="nf">inlineCode</span><span class="p">)</span><span class="w"> </span><span class="nv">@property</span>
<span class="p">(</span><span class="nf">codeBlock</span><span class="p">)</span><span class="w"> </span><span class="nv">@punctuation</span><span class="o">.</span><span class="nv">delimiter</span>
<span class="p">(</span><span class="nf">codeBlock</span><span class="w"> </span><span class="p">(</span><span class="nf">codeText</span><span class="p">)</span><span class="w"> </span><span class="nv">@module</span><span class="p">)</span>
</code></pre></div>
<p>Here we have our first non-trivial use of queries, as we have two queries
referencing the same type of node: <code>codeBlock</code>. The first query marks the
entire <code>codeBlock</code> with <code>@punctuation.delimiter</code>, which is dark grey in my color
scheme. Then we mark <code>codeText</code> <em>inside</em> the <code>codeBlock</code> as a <code>@module</code>, which
is yellow in my color scheme. The latter query is more specific than the former,
and so takes precedence. The result is a significant improvement over our
starting point.</p>
<p><img alt="Syntax highlighting on headings, inline code and code blocks" src="https://slar.se/photos/tree_sitter_highlighting/third_highlighta.jpg"></p>
<p>But it's still not <em>great</em>. Having yellow JavaScript code is a marginal
improvement over just white code. This is where Tree-sitter's killer syntax
highlighting feature comes into play:
<a href="https://tree-sitter.github.io/tree-sitter/syntax-highlighting#language-injection">injections</a></p>
<h2>Injecting a JavaScript parser</h2>
<p>Injections are super cool. They allow us to succinctly state that certain nodes
should be parsed with some other parser than the one we're currently using. Just
like with highlights, injections are specified with queries. What makes this
even cooler is that the author of some given parser doesn't need to have thought
about this; you can tack it on afterwards due to queries being completely
separate from the parser itself.</p>
<p>First of all, we need to extend our <code>package.json</code> with an <code>injections</code> key.</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="nt">"tree-sitter"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"scope"</span><span class="p">:</span><span class="w"> </span><span class="s2">"source.markdownsimple"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"file-types"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"mds"</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">"highlights"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"queries/highlights.scm"</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">"injections"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"queries/injections.scm"</span>
<span class="w"> </span><span class="p">]</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">],</span>
</code></pre></div>
<p>Then we add a file at <code>queries/injections.scm</code> with the following query.</p>
<div class="highlight"><pre><span></span><code><span class="p">(((</span><span class="nf">codeBlock</span><span class="w"> </span><span class="p">(</span><span class="nf">codeText</span><span class="p">)</span><span class="w"> </span><span class="nv">@injection</span><span class="o">.</span><span class="nv">content</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="o">#</span><span class="k">set!</span><span class="w"> </span><span class="nv">injection</span><span class="o">.</span><span class="nv">language</span><span class="w"> </span><span class="s">"javascript"</span><span class="p">))</span>
</code></pre></div>
<p>Unfortunately, the highlight queries will interfere a bit with this injection,
so we also need to reduce our <code>queries/highlights.scm</code> file to the following.</p>
<div class="highlight"><pre><span></span><code><span class="p">(</span><span class="nf">heading</span><span class="p">)</span><span class="w"> </span><span class="nv">@string</span>
<span class="p">(</span><span class="nf">inlineCode</span><span class="p">)</span><span class="w"> </span><span class="nv">@property</span>
</code></pre></div>
<p>But with that change, highlighting is now a lot more interesting. Although,
admittedly, the color scheme could definitely use some work.</p>
<p><img alt="Syntax highlighting on headings, inline code and JavaScript injection in code blocks" src="https://slar.se/photos/tree_sitter_highlighting/final_highlighta.jpg"></p>
<p>With that, we've achieved the highlighting goals we set out to at the beginning
of the article.</p>
<h1>Summary</h1>
<p>In this article, we explored the basics of Tree-sitter and how to apply it to
provide syntax highlighting for a simple markup language. The full source code
for the project can be found
<a href="https://github.com/slarse/tree-sitter-mds">in the companion repository</a>.
There are still numerous shortcomings with this implementation, such as the fact
that <code>codeBlock</code>s cannot contain backticks, or that empty lines between
paragraphs don't actually produce multiple paragraphs in the syntax tree. But
the point of this article wasn't to produce a perfect Markdown Simple parser,
but rather illustrate some fundamental concepts of creating parsers with
Tree-sitter. And that, I think, has been achieved.</p>
<p>In the next part of this article series, we'll explore working with Tree-sitter
in NeoVim!</p>
<p>The final versions of the grammar and queries are inlined below for your
convenience.</p>
<div class="highlight"><pre><span></span><code><span class="c1">// grammar.js</span>
<span class="nx">module</span><span class="p">.</span><span class="nx">exports</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">grammar</span><span class="p">({</span>
<span class="w"> </span><span class="nx">name</span><span class="o">:</span><span class="w"> </span><span class="s1">'markdownsimple'</span><span class="p">,</span>
<span class="w"> </span><span class="nx">rules</span><span class="o">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">source_file</span><span class="o">:</span><span class="w"> </span><span class="nx">$</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">repeat</span><span class="p">(</span><span class="nx">$</span><span class="p">.</span><span class="nx">section</span><span class="p">),</span>
<span class="w"> </span><span class="nx">section</span><span class="o">:</span><span class="w"> </span><span class="nx">$</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">seq</span><span class="p">(</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">heading</span><span class="p">,</span>
<span class="w"> </span><span class="nx">repeat</span><span class="p">(</span>
<span class="w"> </span><span class="nx">choice</span><span class="p">(</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">paragraph</span><span class="p">,</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">codeBlock</span><span class="p">,</span>
<span class="w"> </span><span class="p">)</span>
<span class="w"> </span><span class="p">),</span>
<span class="w"> </span><span class="p">),</span>
<span class="w"> </span><span class="nx">paragraph</span><span class="o">:</span><span class="w"> </span><span class="nx">$</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">prec</span><span class="p">.</span><span class="nx">right</span><span class="p">(</span><span class="nx">repeat1</span><span class="p">(</span>
<span class="w"> </span><span class="nx">choice</span><span class="p">(</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">inlineCode</span><span class="p">,</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">text</span><span class="p">,</span>
<span class="w"> </span><span class="p">),</span>
<span class="w"> </span><span class="p">)),</span>
<span class="w"> </span><span class="nx">inlineCode</span><span class="o">:</span><span class="w"> </span><span class="nx">$</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">seq</span><span class="p">(</span>
<span class="w"> </span><span class="s1">'`'</span><span class="p">,</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">codeText</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'`'</span><span class="p">,</span>
<span class="w"> </span><span class="p">),</span>
<span class="w"> </span><span class="nx">codeBlock</span><span class="o">:</span><span class="w"> </span><span class="nx">$</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">seq</span><span class="p">(</span>
<span class="w"> </span><span class="s1">'```'</span><span class="p">,</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">codeText</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'```'</span><span class="p">,</span>
<span class="w"> </span><span class="p">),</span>
<span class="w"> </span><span class="nx">codeText</span><span class="o">:</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="o">/</span><span class="p">[</span><span class="o">^</span><span class="sb">`]*/,</span>
<span class="sb"> heading: _ => /#.+/,</span>
<span class="sb"> text: _ => /([^#`</span><span class="p">]</span><span class="o">|</span><span class="p">[</span><span class="o">^</span><span class="err">\</span><span class="nx">n</span><span class="p">]</span><span class="err">#</span><span class="p">)</span><span class="o">+</span><span class="err">/,</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">});</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="c1">; queries/highlights.scm</span>
<span class="p">(</span><span class="nf">heading</span><span class="p">)</span><span class="w"> </span><span class="nv">@string</span>
<span class="p">(</span><span class="nf">inlineCode</span><span class="p">)</span><span class="w"> </span><span class="nv">@property</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="c1">; queries/injections.scm</span>
<span class="p">(((</span><span class="nf">codeBlock</span><span class="w"> </span><span class="p">(</span><span class="nf">codeText</span><span class="p">)</span><span class="w"> </span><span class="nv">@injection</span><span class="o">.</span><span class="nv">content</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="o">#</span><span class="k">set!</span><span class="w"> </span><span class="nv">injection</span><span class="o">.</span><span class="nv">language</span><span class="w"> </span><span class="s">"javascript"</span><span class="p">))</span>
</code></pre></div>Extending NeoVim for commenting and uncommenting code blocks2024-02-29T00:00:00+01:002024-02-29T00:00:00+01:00Simon Larséntag:slar.se,2024-02-29:/comment-and-uncomment-code-in-neovim.html<p>I've been using some variation of Vim for going on a decade now, yet I've never
bothered with an efficient way of commenting out code. It just isn't something
that I do very often. But I saw a colleague comment out and uncomment lines of
code like a breeze in …</p><p>I've been using some variation of Vim for going on a decade now, yet I've never
bothered with an efficient way of commenting out code. It just isn't something
that I do very often. But I saw a colleague comment out and uncomment lines of
code like a breeze in VS Code, so obviously I had to also have that capability.
And then choose to not use it. Let's see how you can get that option, too.</p>
<blockquote>
<p>This article contains Lua code for configuring NeoVim. To follow along, paste
the Lua code presented into any <code>.lua</code> file and run <code>:luafile %</code> with the
file in the current buffer. This makes the commands defined within available
in your current NeoVim session.</p>
</blockquote>
<h1>Commenting out code the hard way</h1>
<p>A simple no-preparation way of commenting out code in any variation of Vim is to
simply select a few lines of code in visual mode and then make a substitution.
Let's say we're writing Python where the inline comment character is <code>#</code>, then
commenting out a line would look like this.</p>
<div class="highlight"><pre><span></span><code><span class="p">:</span>s<span class="sr">/^/</span>#/
</code></pre></div>
<p>The <code>^</code> character is the "start of line" character, so here we're simply
replacing the start of line with <code>#</code>. And don't worry, the start of line isn't
actually a character; there'll be a new start of line just in front of the <code>#</code>
after the substitution. Select a range of lines in visual mode and run the same
substitution and you'll comment out the entire range.</p>
<p>If you do this rarely it's fine. But if you do it regularly it becomes rather
cumbersome. Luckily for us, NeoVim is easy to extend.</p>
<h1>Defining a command to comment out code</h1>
<p>To make the above substitution a bit less of an effort, we can define a command
for it. First, we need to define a function that executes the code. A very
simple first effort would look like this.</p>
<div class="highlight"><pre><span></span><code><span class="kr">function</span> <span class="nf">comment_out</span><span class="p">()</span>
<span class="n">vim</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">nvim_command</span><span class="p">(</span><span class="s2">"s:^:#:"</span><span class="p">)</span>
<span class="n">vim</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">nvim_command</span><span class="p">(</span><span class="s2">"noh"</span><span class="p">)</span>
<span class="kr">end</span>
</code></pre></div>
<p>There are a couple things to note about this function.</p>
<ol>
<li><code>vim.api.nvim_command</code> synchronously executes a command.</li>
<li>We use <code>:</code> instead of <code>/</code> as the substitution delimiter in anticipation of
the C-style <code>//</code> inline comment.</li>
<li>We clear the search highlight with <code>noh</code> after doing the substitution.<ul>
<li>Figuring out what happens if you don't do that is left as an exercise to
the reader.</li>
</ul>
</li>
</ol>
<p>So that's our function for performing the substitution, and if we simply place
our cursor on a line we wish to comment out we can execute it like so.</p>
<div class="highlight"><pre><span></span><code><span class="p">:</span><span class="k">lua</span> comment_out<span class="p">()</span>
</code></pre></div>
<p>It should insert a <code>#</code> character at the start of the line. But this isn't
terribly ergonomic, it'd be nicer if we could just call a native NeoVim command.
Fortunately, defining such a command is pretty simple.</p>
<div class="highlight"><pre><span></span><code><span class="n">vim</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">nvim_create_user_command</span><span class="p">(</span><span class="s2">"CommentOut"</span><span class="p">,</span> <span class="n">comment_out</span><span class="p">,</span> <span class="p">{})</span>
</code></pre></div>
<p>This exposes the <code>:CommentOut</code> command, and we can now invoke our commenting out
function like so.</p>
<div class="highlight"><pre><span></span><code><span class="p">:</span>CommentOut
</code></pre></div>
<p>Still, this feels like some amount of effort, so let's add a couple of handy
keybindings to do this. I like <code><leader>co</code> which is a mnemonic for <strong>c</strong>omment
<strong>o</strong>ut.</p>
<div class="highlight"><pre><span></span><code><span class="n">vim</span><span class="p">.</span><span class="n">keymap</span><span class="p">.</span><span class="n">set</span><span class="p">(</span><span class="s2">"v"</span><span class="p">,</span> <span class="s2">"<leader>co"</span><span class="p">,</span> <span class="s2">":CommentOut<CR>"</span><span class="p">)</span> <span class="c1">-- visual mode keymap</span>
<span class="n">vim</span><span class="p">.</span><span class="n">keymap</span><span class="p">.</span><span class="n">set</span><span class="p">(</span><span class="s2">"n"</span><span class="p">,</span> <span class="s2">"<leader>co"</span><span class="p">,</span> <span class="s2">":CommentOut<CR>"</span><span class="p">)</span> <span class="c1">-- normal mode keymap</span>
</code></pre></div>
<p>Now we can simply invoke the commenting out routing with <code><leader>co</code>, or
whatever else you've decided to put in there. There are however two big problems
left to address:</p>
<ol>
<li>The <code>:CommentOut</code> command doesn't work with a range<ul>
<li>If you try to select a range of lines and run the function, you will
encounter an error saying <code>E481: No range allowed</code></li>
</ul>
</li>
<li>The commenting character is hard-coded as <code>#</code><ul>
<li>That approach works great if you only work with a single language, but if
you like me work daily with multiple languages that have different line
comment styles it's not ideal</li>
</ul>
</li>
</ol>
<p>Let's address these in turn.</p>
<h2>Adding support for range selection</h2>
<p>Being able to run our little <code>:CommentOut</code> command only on one line at a time
is hardly useful, so let's make our command compatible with range selection.
First we need to modify the creation of the command, which by default does
not allow ranges.</p>
<div class="highlight"><pre><span></span><code><span class="n">vim</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">nvim_create_user_command</span><span class="p">(</span><span class="s2">"CommentOut"</span><span class="p">,</span> <span class="n">comment_out</span><span class="p">,</span> <span class="p">{</span> <span class="n">range</span> <span class="o">=</span> <span class="kc">true</span> <span class="p">})</span>
</code></pre></div>
<p>With this modification, you'll be able to execute <code>:CommentOut</code> when a range is
selected, but only the first line of the selection will actually be commented
out. We need to add a few lines of code to the <code>comment_out()</code> function to
actually act on the entire range. While we're at it, we'll also make the
function <code>local</code> as it doesn't need to be accessible outside the module anymore.</p>
<div class="highlight"><pre><span></span><code><span class="kd">local</span> <span class="kr">function</span> <span class="nf">comment_out</span><span class="p">(</span><span class="n">opts</span><span class="p">)</span>
<span class="kd">local</span> <span class="n">start</span> <span class="o">=</span> <span class="nb">math.min</span><span class="p">(</span><span class="n">opts</span><span class="p">.</span><span class="n">line1</span><span class="p">,</span> <span class="n">opts</span><span class="p">.</span><span class="n">line2</span><span class="p">)</span>
<span class="kd">local</span> <span class="n">finish</span> <span class="o">=</span> <span class="nb">math.max</span><span class="p">(</span><span class="n">opts</span><span class="p">.</span><span class="n">line1</span><span class="p">,</span> <span class="n">opts</span><span class="p">.</span><span class="n">line2</span><span class="p">)</span>
<span class="n">vim</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">nvim_command</span><span class="p">(</span><span class="n">start</span> <span class="o">..</span> <span class="s2">","</span> <span class="o">..</span> <span class="n">finish</span> <span class="o">..</span> <span class="s2">"s:^:#:"</span><span class="p">)</span>
<span class="n">vim</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">nvim_command</span><span class="p">(</span><span class="s2">"noh"</span><span class="p">)</span>
<span class="kr">end</span>
</code></pre></div>
<p>Options for NeoVim commands are passed in via the <code>opts</code> table, and the line
numbers of the selection are stored in <code>line1</code> and <code>line2</code>. Other available
options are <a href="https://neovim.io/doc/user/api.html#nvim_create_user_command()">detailed in the NeoVim API
docs</a>.</p>
<p>Note that depending on where you start the selection, either line may be the one
at the top and bottom, respectively, so we do some rudimentary math to get the
start and finish of our substitution in the right order.</p>
<p>Selecting a few lines of code and running <code>:CommentOut</code> (or using the
keybind for it) now actually comments out those lines! Let's now figure out how
to select the correct kind of line comment for the given file type.</p>
<h2>Choosing line comment style by filetype</h2>
<p>While there's probably a super clever way of accomplishing this, I went with
something really rudimentary: a table with line comment styles by filetype. It
looks something like this:</p>
<div class="highlight"><pre><span></span><code><span class="kd">local</span> <span class="n">non_c_line_comments_by_filetype</span> <span class="o">=</span> <span class="p">{</span>
<span class="n">lua</span> <span class="o">=</span> <span class="s2">"--"</span><span class="p">,</span>
<span class="n">python</span> <span class="o">=</span> <span class="s2">"#"</span><span class="p">,</span>
<span class="n">sql</span> <span class="o">=</span> <span class="s2">"--"</span><span class="p">,</span>
<span class="p">}</span>
<span class="kd">local</span> <span class="n">default_line_comment</span> <span class="o">=</span> <span class="s2">"//"</span>
<span class="kd">local</span> <span class="kr">function</span> <span class="nf">comment_out</span><span class="p">(</span><span class="n">opts</span><span class="p">)</span>
<span class="kd">local</span> <span class="n">line_comment</span> <span class="o">=</span> <span class="n">non_c_line_comments_by_filetype</span><span class="p">[</span><span class="n">vim</span><span class="p">.</span><span class="n">bo</span><span class="p">.</span><span class="n">filetype</span><span class="p">]</span> <span class="ow">or</span> <span class="n">default_line_comment</span>
<span class="kd">local</span> <span class="n">start</span> <span class="o">=</span> <span class="nb">math.min</span><span class="p">(</span><span class="n">opts</span><span class="p">.</span><span class="n">line1</span><span class="p">,</span> <span class="n">opts</span><span class="p">.</span><span class="n">line2</span><span class="p">)</span>
<span class="kd">local</span> <span class="n">finish</span> <span class="o">=</span> <span class="nb">math.max</span><span class="p">(</span><span class="n">opts</span><span class="p">.</span><span class="n">line1</span><span class="p">,</span> <span class="n">opts</span><span class="p">.</span><span class="n">line2</span><span class="p">)</span>
<span class="n">vim</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">nvim_command</span><span class="p">(</span><span class="n">start</span> <span class="o">..</span> <span class="s2">","</span> <span class="o">..</span> <span class="n">finish</span> <span class="o">..</span> <span class="s2">"s:^:"</span> <span class="o">..</span> <span class="n">line_comment</span> <span class="o">..</span> <span class="s2">":"</span><span class="p">)</span>
<span class="n">vim</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">nvim_command</span><span class="p">(</span><span class="s2">"noh"</span><span class="p">)</span>
<span class="kr">end</span>
</code></pre></div>
<p>Basically, we default to the <code>//</code> line comment style <em>unless</em> we find a match in
the <code>non_c_line_comments_by_filetype</code> table. We use <code>vim.bo.filetype</code> to get the
current buffer's filetype. Then we do some more string concatenation in the
substitution command, and that's pretty much that. If you need other line
comment styles, just add them to the table.</p>
<p>If you now run the <code>:CommentOut</code> command in a Python file, it'll use <code>#</code> as the
comment style. But if you do it in a SQL file, it'll use <code>--</code>, and in a Go file
it'll use <code>//</code>.</p>
<h1>Uncommenting code</h1>
<p>We now have a neat way of commenting out code, but what about <em>uncommenting</em>
code? Let's straight up copy <code>comment_out()</code> and just replace the substitution
expression with something that removes a line comment start.</p>
<div class="highlight"><pre><span></span><code><span class="kd">local</span> <span class="kr">function</span> <span class="nf">uncomment</span><span class="p">(</span><span class="n">opts</span><span class="p">)</span>
<span class="kd">local</span> <span class="n">line_comment</span> <span class="o">=</span> <span class="n">non_c_line_comments_by_filetype</span><span class="p">[</span><span class="n">vim</span><span class="p">.</span><span class="n">bo</span><span class="p">.</span><span class="n">filetype</span><span class="p">]</span> <span class="ow">or</span> <span class="s2">"//"</span>
<span class="kd">local</span> <span class="n">start</span> <span class="o">=</span> <span class="nb">math.min</span><span class="p">(</span><span class="n">opts</span><span class="p">.</span><span class="n">line1</span><span class="p">,</span> <span class="n">opts</span><span class="p">.</span><span class="n">line2</span><span class="p">)</span>
<span class="kd">local</span> <span class="n">finish</span> <span class="o">=</span> <span class="nb">math.max</span><span class="p">(</span><span class="n">opts</span><span class="p">.</span><span class="n">line1</span><span class="p">,</span> <span class="n">opts</span><span class="p">.</span><span class="n">line2</span><span class="p">)</span>
<span class="n">vim</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">nvim_command</span><span class="p">(</span><span class="n">start</span> <span class="o">..</span> <span class="s2">","</span> <span class="o">..</span> <span class="n">finish</span> <span class="o">..</span> <span class="s2">"s:^"</span> <span class="o">..</span> <span class="n">line_comment</span> <span class="o">..</span> <span class="s2">"::"</span><span class="p">)</span>
<span class="n">vim</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">nvim_command</span><span class="p">(</span><span class="s2">"noh"</span><span class="p">)</span>
<span class="kr">end</span>
<span class="n">vim</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">nvim_create_user_command</span><span class="p">(</span><span class="s2">"Uncomment"</span><span class="p">,</span> <span class="n">uncomment</span><span class="p">,</span> <span class="p">{</span> <span class="n">range</span> <span class="o">=</span> <span class="kc">true</span> <span class="p">})</span>
<span class="n">vim</span><span class="p">.</span><span class="n">keymap</span><span class="p">.</span><span class="n">set</span><span class="p">(</span><span class="s2">"v"</span><span class="p">,</span> <span class="s2">"<leader>uc"</span><span class="p">,</span> <span class="s2">":Uncomment<CR>"</span><span class="p">)</span>
<span class="n">vim</span><span class="p">.</span><span class="n">keymap</span><span class="p">.</span><span class="n">set</span><span class="p">(</span><span class="s2">"n"</span><span class="p">,</span> <span class="s2">"<leader>uc"</span><span class="p">,</span> <span class="s2">":Uncomment<CR>"</span><span class="p">)</span>
</code></pre></div>
<blockquote>
<p>Note: The keybinding is a mnemonic for <strong>u</strong>n<strong>c</strong>omment.</p>
</blockquote>
<p>This mostly works. If you first run <code>:CommentOut</code> and then <code>:Uncomment</code>, the
latter cancels out the former. But there are two notable shortcomings.</p>
<ol>
<li>If you run <code>:Uncomment</code> on a line that deos not start with a line comment
start you encounter an error saying <code>Pattern not found:</code></li>
<li>The line must <em>start</em> with a line comment start; leading whitespace causes
the substitution to fail</li>
<li>If you have a formatter that indents line comments, you'll be in trouble!</li>
</ol>
<p>The first issue is simple to resolve: we wrap the call to <code>vim.api.nvim_comand</code>
in a <em>protected call</em> using the <a href="https://www.lua.org/pil/8.4.html">pcall() function</a>.
This allows us to handle errors, but in fact all we want is to prevent any error
from bubbling up to the surface; we don't really care if the substitution fails
or not as a failure simply indicates there was no comment to uncomment.</p>
<p>The second issue requires a little bit more thought. We want to be able to
uncomment lines even if there's leading whitespace to be give the command some
more flexibility. With <code>#</code> as the line comment, a first attempt could look like
this.</p>
<div class="highlight"><pre><span></span><code><span class="p">:</span>s:^\s\{<span class="p">-</span>\}#::
</code></pre></div>
<p><code>^</code> is the line start meta character, <code>\s</code> denotes any whitespace except for
linebreaks, and <code>\{-\}</code> means "zero or more`. This mostly works, but it removes
both the leading whitespace and the comment character, effectively dedenting the
line. We need to <em>capture</em> the whitespace and put it back after removing the
line comment start.</p>
<div class="highlight"><pre><span></span><code><span class="p">:</span>s:^\<span class="p">(</span>\s\{<span class="p">-</span>\}\<span class="p">)</span>#:\<span class="m">1</span>:
</code></pre></div>
<p>The escapes makes the pattern a bit difficult to read, but all we've done here
is to wrap the "zero or more whitespace" in a capture group (denoted by
parentheses), and then we reference that capture group in the replacement part
of the expression with <code>\1</code>.</p>
<p>Putting all of this together, we end up with the following <code>uncomment()</code>
function.</p>
<div class="highlight"><pre><span></span><code><span class="kd">local</span> <span class="kr">function</span> <span class="nf">uncomment</span><span class="p">(</span><span class="n">opts</span><span class="p">)</span>
<span class="kd">local</span> <span class="n">line_comment</span> <span class="o">=</span> <span class="n">non_c_line_comments_by_filetype</span><span class="p">[</span><span class="n">vim</span><span class="p">.</span><span class="n">bo</span><span class="p">.</span><span class="n">filetype</span><span class="p">]</span> <span class="ow">or</span> <span class="s2">"//"</span>
<span class="kd">local</span> <span class="n">start</span> <span class="o">=</span> <span class="nb">math.min</span><span class="p">(</span><span class="n">opts</span><span class="p">.</span><span class="n">line1</span><span class="p">,</span> <span class="n">opts</span><span class="p">.</span><span class="n">line2</span><span class="p">)</span>
<span class="kd">local</span> <span class="n">finish</span> <span class="o">=</span> <span class="nb">math.max</span><span class="p">(</span><span class="n">opts</span><span class="p">.</span><span class="n">line1</span><span class="p">,</span> <span class="n">opts</span><span class="p">.</span><span class="n">line2</span><span class="p">)</span>
<span class="nb">pcall</span><span class="p">(</span><span class="n">vim</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">nvim_command</span><span class="p">,</span> <span class="n">start</span> <span class="o">..</span> <span class="s2">","</span> <span class="o">..</span> <span class="n">finish</span> <span class="o">..</span> <span class="s2">"s:^</span><span class="se">\\</span><span class="s2">(</span><span class="se">\\</span><span class="s2">s</span><span class="se">\\</span><span class="s2">{-</span><span class="se">\\</span><span class="s2">}</span><span class="se">\\</span><span class="s2">)"</span> <span class="o">..</span> <span class="n">line_comment</span> <span class="o">..</span> <span class="s2">":</span><span class="se">\\</span><span class="s2">1:"</span><span class="p">)</span>
<span class="n">vim</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">nvim_command</span><span class="p">(</span><span class="s2">"noh"</span><span class="p">)</span>
<span class="kr">end</span>
</code></pre></div>
<p>Now it should work pretty well!</p>
<h1>Summary and full code</h1>
<p>That's pretty much it for commenting and uncommenting code, at least as far as
my semi-imagined needs for it go. This represents among the first non-trivial
extensions I've made to NeoVim using only its API and made me realise just how
much power I've left untapped for so many years. I will undoubtedly return with
more blog posts on extending NeoVim in the future; it's just too much fun not
to.</p>
<p>The full code can be found below. You can just copy and paste it into your root
<code>init.lua</code> file and it should work without any further tweaks. For a more
segregated placement, you can draw inspiration from <a href="https://github.com/slarse/config/commit/fe1ab5bcd24f4a1ad0a111b67a652de1c330a18d">my NeoVim
configuration</a>.</p>
<div class="highlight"><pre><span></span><code><span class="kd">local</span> <span class="n">non_c_line_comments_by_filetype</span> <span class="o">=</span> <span class="p">{</span>
<span class="n">lua</span> <span class="o">=</span> <span class="s2">"--"</span><span class="p">,</span>
<span class="n">python</span> <span class="o">=</span> <span class="s2">"#"</span><span class="p">,</span>
<span class="n">sql</span> <span class="o">=</span> <span class="s2">"--"</span><span class="p">,</span>
<span class="p">}</span>
<span class="kd">local</span> <span class="kr">function</span> <span class="nf">comment_out</span><span class="p">(</span><span class="n">opts</span><span class="p">)</span>
<span class="kd">local</span> <span class="n">line_comment</span> <span class="o">=</span> <span class="n">non_c_line_comments_by_filetype</span><span class="p">[</span><span class="n">vim</span><span class="p">.</span><span class="n">bo</span><span class="p">.</span><span class="n">filetype</span><span class="p">]</span> <span class="ow">or</span> <span class="s2">"//"</span>
<span class="kd">local</span> <span class="n">start</span> <span class="o">=</span> <span class="nb">math.min</span><span class="p">(</span><span class="n">opts</span><span class="p">.</span><span class="n">line1</span><span class="p">,</span> <span class="n">opts</span><span class="p">.</span><span class="n">line2</span><span class="p">)</span>
<span class="kd">local</span> <span class="n">finish</span> <span class="o">=</span> <span class="nb">math.max</span><span class="p">(</span><span class="n">opts</span><span class="p">.</span><span class="n">line1</span><span class="p">,</span> <span class="n">opts</span><span class="p">.</span><span class="n">line2</span><span class="p">)</span>
<span class="n">vim</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">nvim_command</span><span class="p">(</span><span class="n">start</span> <span class="o">..</span> <span class="s2">","</span> <span class="o">..</span> <span class="n">finish</span> <span class="o">..</span> <span class="s2">"s:^:"</span> <span class="o">..</span> <span class="n">line_comment</span> <span class="o">..</span> <span class="s2">":"</span><span class="p">)</span>
<span class="n">vim</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">nvim_command</span><span class="p">(</span><span class="s2">"noh"</span><span class="p">)</span>
<span class="kr">end</span>
<span class="kd">local</span> <span class="kr">function</span> <span class="nf">uncomment</span><span class="p">(</span><span class="n">opts</span><span class="p">)</span>
<span class="kd">local</span> <span class="n">line_comment</span> <span class="o">=</span> <span class="n">non_c_line_comments_by_filetype</span><span class="p">[</span><span class="n">vim</span><span class="p">.</span><span class="n">bo</span><span class="p">.</span><span class="n">filetype</span><span class="p">]</span> <span class="ow">or</span> <span class="s2">"//"</span>
<span class="kd">local</span> <span class="n">start</span> <span class="o">=</span> <span class="nb">math.min</span><span class="p">(</span><span class="n">opts</span><span class="p">.</span><span class="n">line1</span><span class="p">,</span> <span class="n">opts</span><span class="p">.</span><span class="n">line2</span><span class="p">)</span>
<span class="kd">local</span> <span class="n">finish</span> <span class="o">=</span> <span class="nb">math.max</span><span class="p">(</span><span class="n">opts</span><span class="p">.</span><span class="n">line1</span><span class="p">,</span> <span class="n">opts</span><span class="p">.</span><span class="n">line2</span><span class="p">)</span>
<span class="nb">pcall</span><span class="p">(</span><span class="n">vim</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">nvim_command</span><span class="p">,</span> <span class="n">start</span> <span class="o">..</span> <span class="s2">","</span> <span class="o">..</span> <span class="n">finish</span> <span class="o">..</span> <span class="s2">"s:^</span><span class="se">\\</span><span class="s2">(</span><span class="se">\\</span><span class="s2">s</span><span class="se">\\</span><span class="s2">{-</span><span class="se">\\</span><span class="s2">}</span><span class="se">\\</span><span class="s2">)"</span> <span class="o">..</span> <span class="n">line_comment</span> <span class="o">..</span> <span class="s2">":</span><span class="se">\\</span><span class="s2">1:"</span><span class="p">)</span>
<span class="n">vim</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">nvim_command</span><span class="p">(</span><span class="s2">"noh"</span><span class="p">)</span>
<span class="kr">end</span>
<span class="n">vim</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">nvim_create_user_command</span><span class="p">(</span><span class="s2">"CommentOut"</span><span class="p">,</span> <span class="n">comment_out</span><span class="p">,</span> <span class="p">{</span> <span class="n">range</span> <span class="o">=</span> <span class="kc">true</span> <span class="p">})</span>
<span class="n">vim</span><span class="p">.</span><span class="n">keymap</span><span class="p">.</span><span class="n">set</span><span class="p">(</span><span class="s2">"v"</span><span class="p">,</span> <span class="s2">"<leader>co"</span><span class="p">,</span> <span class="s2">":CommentOut<CR>"</span><span class="p">)</span>
<span class="n">vim</span><span class="p">.</span><span class="n">keymap</span><span class="p">.</span><span class="n">set</span><span class="p">(</span><span class="s2">"n"</span><span class="p">,</span> <span class="s2">"<leader>co"</span><span class="p">,</span> <span class="s2">":CommentOut<CR>"</span><span class="p">)</span>
<span class="n">vim</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">nvim_create_user_command</span><span class="p">(</span><span class="s2">"Uncomment"</span><span class="p">,</span> <span class="n">uncomment</span><span class="p">,</span> <span class="p">{</span> <span class="n">range</span> <span class="o">=</span> <span class="kc">true</span> <span class="p">})</span>
<span class="n">vim</span><span class="p">.</span><span class="n">keymap</span><span class="p">.</span><span class="n">set</span><span class="p">(</span><span class="s2">"v"</span><span class="p">,</span> <span class="s2">"<leader>uc"</span><span class="p">,</span> <span class="s2">":Uncomment<CR>"</span><span class="p">)</span>
<span class="n">vim</span><span class="p">.</span><span class="n">keymap</span><span class="p">.</span><span class="n">set</span><span class="p">(</span><span class="s2">"n"</span><span class="p">,</span> <span class="s2">"<leader>uc"</span><span class="p">,</span> <span class="s2">":Uncomment<CR>"</span><span class="p">)</span>
</code></pre></div>
<p>Happy editing!</p>Adding guardrails to psql for PostgreSQL2024-02-24T00:00:00+01:002024-02-24T00:00:00+01:00Simon Larséntag:slar.se,2024-02-24:/psql-guardrails.html<p>I've been using <code>psql</code> for many years to interface with PostgreSQL databases.
It's simple, pretty much always available as it's usually bundled with
PostgreSQL and just does what it's supposed to. It does, however, have some
pretty dangerous defaults. Not only are writes allowed, but it also
automatically commits any …</p><p>I've been using <code>psql</code> for many years to interface with PostgreSQL databases.
It's simple, pretty much always available as it's usually bundled with
PostgreSQL and just does what it's supposed to. It does, however, have some
pretty dangerous defaults. Not only are writes allowed, but it also
automatically commits any statements executed outside of explicit transactions.
Let's fix that.</p>
<h1>Configuring <code>psql</code> with <code>.psqlrc</code></h1>
<p>The <a href="https://www.postgresql.org/docs/current/app-psql.html#APP-PSQL-FILES-PSQLRC"><code>.psqlrc</code>
file</a>
can be used to set defaults for new connections made with <code>psql</code>. It should be
placed in your home directory (i.e. <code>~/.psqlrc</code>) and most often contains <code>\set</code>
and <code>SET</code> commands.</p>
<p>The difference between <code>\set</code> and <code>SET</code> commands is a bit subtle. A <a href="https://www.postgresql.org/docs/current/sql-set.html"><code>SET</code>
command</a> sets a session
variable inside of the PostgreSQL server, whereas a <a href="https://www.postgresql.org/docs/current/app-psql.html#APP-PSQL-META-COMMAND-SET"><code>\set</code>
command</a>
sets configuration in the <code>psql</code> client itself.</p>
<h2>Making the default transaction read-only</h2>
<p>PostgreSQL executes all statements as transactions. If you don't start an
explicit transaction for a statement, <a href="https://www.postgresql.org/docs/current/tutorial-transactions.html">PostgreSQL automatically wraps the
statement in a
transaction</a>.
By default transactions are both readable and writable, which isn't necessarily
desirable. To make transactions read-only by default, you can set the following
session variable inside of PostgreSQL.</p>
<div class="highlight"><pre><span></span><code><span class="k">SET</span><span class="w"> </span><span class="n">default_transaction_read_only</span><span class="o">=</span><span class="ss">"on"</span><span class="p">;</span>
</code></pre></div>
<p>You can do this in any PostgreSQL session. It can also be configured on the
database server, but here we assume it isn't. If you after having done that try
to execute a statement that writes to the database, you'll get an error.</p>
<div class="highlight"><pre><span></span><code><span class="n">test_db</span><span class="o">=#</span><span class="w"> </span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">test_table</span><span class="w"> </span><span class="p">(</span><span class="n">id</span><span class="w"> </span><span class="nb">serial</span><span class="p">);</span>
<span class="n">ERROR</span><span class="p">:</span><span class="w"> </span><span class="n">cannot</span><span class="w"> </span><span class="k">execute</span><span class="w"> </span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="k">read</span><span class="o">-</span><span class="k">only</span><span class="w"> </span><span class="k">transaction</span>
</code></pre></div>
<p>However, it's still possible to create an explicit transaction that's allowed
to write, so it doesn't hinder you if you do need to write.</p>
<div class="highlight"><pre><span></span><code><span class="n">test_db</span><span class="o">=#</span><span class="w"> </span><span class="k">BEGIN</span><span class="w"> </span><span class="k">READ</span><span class="w"> </span><span class="k">WRITE</span><span class="p">;</span>
<span class="k">BEGIN</span>
<span class="n">test_db</span><span class="o">=*#</span><span class="w"> </span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">test_table</span><span class="w"> </span><span class="p">(</span><span class="n">id</span><span class="w"> </span><span class="nb">serial</span><span class="p">);</span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span>
<span class="n">test_db</span><span class="o">=*#</span><span class="w"> </span><span class="k">COMMIT</span><span class="p">;</span>
<span class="k">COMMIT</span>
</code></pre></div>
<p>To set this in <code>.psqlrc</code>, simply add the exact same line you'd use in an
interactive session to <code>~/.psqlrc</code>. When you connect to a database, any <code>SET</code>
command in <code>.psqlrc</code> is automatically executed.</p>
<blockquote>
<p><strong>Pitfall:</strong> <code>SET</code> commands in <code>.psqlrc</code> are only executed when you
first establish a connection to the database server. If you <em>switch</em> database
after having connected with <code>\c</code>, the <code>SET</code> command is <em>not</em> executed again.
Keep that in mind.</p>
</blockquote>
<h2>Disabling autocommit</h2>
<p><a href="https://www.postgresql.org/docs/current/app-psql.html#APP-PSQL-FILES-PSQLRC"><code>psql</code> automatically
commits</a>
the implicit transaction created when you execute a statement outside of an
explicit transaction. This can be altered such that <code>psql</code> implicitly creates a
transaction without automatically committing it by setting the <code>AUTOCIMMIT</code> variable to <code>off</code>. This is a <code>psql</code> variable, so we use <code>\set</code> to set it.</p>
<div class="highlight"><pre><span></span><code><span class="err">\</span><span class="k">set</span><span class="w"> </span><span class="n">AUTOCOMMIT</span><span class="w"> </span><span class="k">false</span>
</code></pre></div>
<p>To validate that it's working as expected, you should look for the <code>*</code> in the
<code>psql</code> prompt.</p>
<div class="highlight"><pre><span></span><code><span class="n">test_db</span><span class="o">=#</span><span class="w"> </span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">test_table</span><span class="w"> </span><span class="p">(</span><span class="n">id</span><span class="w"> </span><span class="nb">serial</span><span class="p">);</span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span>
<span class="n">test_db</span><span class="o">=*#</span><span class="w"> </span><span class="k">ROLLBACK</span><span class="p">;</span><span class="w"> </span><span class="c1">-- <=== * in the prompt signifies transaction</span>
<span class="k">ROLLBACK</span>
</code></pre></div>
<p>Combined with <code>default_transaction_read_only</code>, this automatically creates a
read-only transaction that's not immediately committed, putting two layers of
protection between you and an errenous write. Unlike setting
<code>default_transaction_read_only</code> however, disabling <code>AUTOCOMMIT</code> persists even if
you switch database with <code>\c</code>.</p>
<h1>Summary</h1>
<p>When connecting to any database that you for some reason don't want to
accidentally write to, you should have guardrails. Many graphical PostgreSQL
clients that I've seen my colleagues use have such guardrails by default in
that an explicit commit must be issued by clicking a button. <code>psql</code> does not, so
setting up some yourself is imperative.</p>
<p>With <code>AUTOCOMMIT</code> disabled and <code>default_transaction_read_only</code> set to <code>on</code>, you
should however be about as safe as can be.</p>First impressions of Wayland on Arch Linux2024-02-04T00:00:00+01:002024-02-04T00:00:00+01:00Simon Larséntag:slar.se,2024-02-04:/wayland-one-month-later.html<p>Wayland appears to be the future of window systems in the world of Linux,
replacing the aging X.Org window system. The <a href="https://wiki.archlinux.org/title/Xorg">Arch Wiki article on
X.Org</a> refers to it as "the alternative
and successor [of X.Org]" and Canonical even dropped its own <a href="https://wiki.ubuntu.com/Wayland">Mir display
server in favor …</a></p><p>Wayland appears to be the future of window systems in the world of Linux,
replacing the aging X.Org window system. The <a href="https://wiki.archlinux.org/title/Xorg">Arch Wiki article on
X.Org</a> refers to it as "the alternative
and successor [of X.Org]" and Canonical even dropped its own <a href="https://wiki.ubuntu.com/Wayland">Mir display
server in favor of Wayland on Ubuntu</a>.
So when it became time for me to configure a new laptop in the beginning of
January, I decided to do give Wayland a go. Here's what I think of it so far.</p>
<h1>What the heck is a window system?</h1>
<p>First of all, what the heck is a window system? In short, it's the component
that renders graphical windows to your screen and communicates the user's input
to the underlying operating system. The X Window System, or just X for short, is
the reigning king of window systems in UNIX-like operating systems (excepting
macOS which uses <a href="https://en.wikipedia.org/wiki/Quartz_Compositor">Quartz</a>). You
will often see it referred to as X.Org; this is simply the most commonly used
<a href="https://www.x.org/wiki/">open source implementation of X</a>.</p>
<p>I could try to outline what X.Org's problems are and how Wayland solves many of
them, but I simply don't have the level of understanding necessary to do so
confidently. One thing to note is that Wayland is first and foremost a
<em>protocol</em> for the communication between a display server and applications. A
display server can implement the Wayland protocol and is then referred to as a
Wayland <em>compositor</em>.</p>
<p>If you want to learn more, <a href="https://linuxiac.com/xorg-x11-wayland-linux-display-servers-and-protocols-explained/">here's a brief article on the
subjecp</a>.</p>
<h1>Wayland first impressions with Sway</h1>
<p>I recently setup Wayland on a brand new work computer, and going from a clean
slate setup was really effortless. I chose <a href="https://swaywm.org/">Sway</a> as my
compositor and started it. And it just kind of worked. Outside of <a href="https://github.com/slarse/config/blob/d8ba6671a2245be75c5fa3b4fdfef3df61cc93a0/sway/config">configuring
Sway to my
liking</a>,
I didn't really have to do much of anything for it to work.</p>
<p>On my private Dell XPS 15 <a href="https://slar.se/xps-15-9520-arch-linux-intel-dri2.html">where I'm already running
X.Org</a>, setting up
Wayland was similarly effortless. I just installed Sway, and even though it
doesn't support <a href="https://github.com/swaywm/sway/wiki#nvidia-users">NVIDIA's proprietary
drivers</a>, running Sway with
the <code>--unsuported-gpu</code> option just worked.</p>
<p>All-in-all, setup was a breeze and I did not encounter any problems at all.</p>
<h2>Upsides of Sway</h2>
<p>I immediately noticed some improvements in my desktop experience. The first came
when connecting an external display, which Sway just found and started
outputting an image to. In almost a decade of using X.Org on just shy of a half
dozen devices, that has never happened before without effort on my part.</p>
<p>I was also treated with a complete lack of screen tearing when scrolling
websites and documents, which I had a lot of under X.Org. I'm sure there's a way
to configure that away with X and I never bothered to look into it more than
stating that it wasn't as easy as toggling a flag, but it was just for free with
Wayland. Now that it's gone I would have a hard time going back to it.</p>
<p>Finally, most things just kind of work. It's a <em>big</em> accolade to note that over
a whole month of use, I haven't encountered a single show-stopping problem. But
I have definitely encountered quite a few lesser ones.</p>
<h2>Downsites of Sway</h2>
<p>Pretty much all of the downsides I've encountered have to do with the fact that
a lot of applications I've used over the years were written for X.Org. This
means that many applications run under the the
<a href="https://wayland.freedesktop.org/xserver.html">XWayland</a> compatibility layer,
which nullifies the performance improvements of Wayland's architecture. I've
especially noted that startup times for GUI apps that run under XWayland are
slow compared to running X.Org directly.</p>
<p>Another problem is that apps directly related to X.Org typically just do not
function. For example, I've been using the <a href="https://imagemagick.org/script/import.php"><code>import</code> command from
<code>imagemagick</code></a> to grab snippeted
screenshots for nigh on a decade, and it doesn't work at all with Wayland as it
expects to work with an X.Org backend. Similarly,
<a href="https://wiki.archlinux.org/title/Redshift">Redshift</a> which I've been using for
years to adjust color temperature does not support Wayland. It results in a whole
lot of <a href="https://www.reddit.com/r/wayland/comments/si49y1/redshift_alternatives_for_wayland/?rdt=39794"><insert program> alternative for
Wayland</a>
kinds of searches, but they usually bear fruit. I found <a href="https://sr.ht/~emersion/grim/"><code>grim</code> as a replacement
for <code>import</code></a> and <a href="https://gitlab.com/chinstrap/gammastep"><code>gammastep</code> as a replacement
for <code>redshift</code></a> that way.</p>
<p>The incompatibility with Redshift does bring up another interesting point,
namely that Wayland is still neither complete nor stable. <a href="https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/14">Color management is for
example still not
standardized</a>.
In some respects, stepping into the land of Way feels a bit like using a beta
product.</p>
<h1>Conclusions</h1>
<p>Will I keep using Wayland? Definitely. I've only encountered a few very minor
problems with application compatibility, and most I've found alternatives for.
I'm even a little bit excited to find some need completely unfulfilled in the
land of Wayland applications, because that would give me a good reason to
implement it myself.</p>Dependabot's dependency grouping is awesome2024-01-29T00:00:00+01:002024-01-29T00:00:00+01:00Simon Larséntag:slar.se,2024-01-29:/dependabots-dependency-grouping.html<p>I've been using GitHub's Dependabot since it was <a href="https://github.blog/2020-06-01-keep-all-your-packages-up-to-date-with-dependabot/">released around 4 years
ago</a>,
and to a large extent, it's been great. Except for one thing: the sheer amount
of pull requests Dependabot would open for dependency updates. For some of my
repositories it became more of a chore to keep …</p><p>I've been using GitHub's Dependabot since it was <a href="https://github.blog/2020-06-01-keep-all-your-packages-up-to-date-with-dependabot/">released around 4 years
ago</a>,
and to a large extent, it's been great. Except for one thing: the sheer amount
of pull requests Dependabot would open for dependency updates. For some of my
repositories it became more of a chore to keep up with Dependabot's pull request
spamming than to just manually update dependencies every once in a while.</p>
<p>Well. Turns out that's been fixed.</p>
<h1>Dependabot's big problem: Pull request spam</h1>
<p>Since Dependabot was released, it's only mode of operation has been to open one
pull request per dependency update. As you can imagine, that leads to a whole
bunch of pull requests being opened. I would routinely get dozens of dependency
pull requests a week for some of my more dependency heavy repositories. Needless
to say, that's just not manageable.</p>
<p>One of my projects, <a href="https://github.com/kth-assert/spork">Spork</a>, which is
mostly in maintenance mod is one such example where I just gave up on keeping up
with the updates. It ended up looking like this:</p>
<p><img alt="Spork dependency spam" src="https://slar.se/photos/dependabot_dependency_grouping/pr-spama.jpg"></p>
<p>What a mess. But there's light at the end of the tunnel.</p>
<h1>The fix: Grouped dependencies</h1>
<p>To address that problem GitHub recently released
<a href="https://github.blog/changelog/2023-08-24-grouped-version-updates-for-dependabot-are-generally-available/">Grouped Version Updates</a>
as a feature for Dependabot. In short, this means that you can get Dependabot
to open pull requests containing multiple dependency updates, grouped in three
different ways.</p>
<h2>A configuration example</h2>
<p>For my project <a href="https://github.com/kth-assert/spork">Spork</a>, I decided to group
dependencies in three groups:</p>
<ol>
<li>Updates to GitHub Actions dependencies</li>
<li>Updates to production dependencies</li>
<li>Updates to development dependencies</li>
</ol>
<p>Configuring this was really straightforward, all it took was <a href="https://github.com/ASSERT-KTH/spork/commit/ac3051e07bc6632c4f41f74feb60ffc555fce1cd">this
commit</a>.</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span> directory: "/"
<span class="w"> </span> schedule:
<span class="w"> </span> interval: "daily"
<span class="gi">+ groups:</span>
<span class="gi">+ actions-deps:</span>
<span class="gi">+ patterns:</span>
<span class="gi">+ - "*"</span>
<span class="w"> </span> - package-ecosystem: "maven"
<span class="w"> </span> directory: "/"
<span class="w"> </span> schedule:
<span class="w"> </span> interval: "daily"
<span class="gi">+ groups:</span>
<span class="gi">+ dev-deps:</span>
<span class="gi">+ dependency-type: "development"</span>
<span class="gi">+ prod-deps:</span>
<span class="gi">+ dependency-type: "production"</span>
</code></pre></div>
<p>In full, the configuration looks like so.</p>
<div class="highlight"><pre><span></span><code><span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">2</span>
<span class="nt">updates</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">package-ecosystem</span><span class="p">:</span><span class="w"> </span><span class="s">"github-actions"</span>
<span class="w"> </span><span class="nt">directory</span><span class="p">:</span><span class="w"> </span><span class="s">"/"</span>
<span class="w"> </span><span class="nt">schedule</span><span class="p">:</span>
<span class="w"> </span><span class="nt">interval</span><span class="p">:</span><span class="w"> </span><span class="s">"daily"</span>
<span class="w"> </span><span class="nt">groups</span><span class="p">:</span>
<span class="w"> </span><span class="nt">actions-deps</span><span class="p">:</span>
<span class="w"> </span><span class="nt">patterns</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">"*"</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">package-ecosystem</span><span class="p">:</span><span class="w"> </span><span class="s">"maven"</span>
<span class="w"> </span><span class="nt">directory</span><span class="p">:</span><span class="w"> </span><span class="s">"/"</span>
<span class="w"> </span><span class="nt">schedule</span><span class="p">:</span>
<span class="w"> </span><span class="nt">interval</span><span class="p">:</span><span class="w"> </span><span class="s">"daily"</span>
<span class="w"> </span><span class="nt">groups</span><span class="p">:</span>
<span class="w"> </span><span class="nt">dev-deps</span><span class="p">:</span>
<span class="w"> </span><span class="nt">dependency-type</span><span class="p">:</span><span class="w"> </span><span class="s">"development"</span>
<span class="w"> </span><span class="nt">prod-deps</span><span class="p">:</span>
<span class="w"> </span><span class="nt">dependency-type</span><span class="p">:</span><span class="w"> </span><span class="s">"production"</span>
</code></pre></div>
<p>For each package ecosystem, you must group on <em>something</em>. For
<code>github-actions</code>, I just wanted a single group, and the way to achieve such an
"everything group" is to simply use a wildcard pattern. For <code>maven</code>, I ended up
splitting by the <code>dependency-type</code> attribute instead, which can be either
<code>development</code> or <code>production</code>.</p>
<p>But that's just two ways to group, you said there were three ways? Very
attentive of you! In addition to the <code>pattern</code> and <code>dependency-type</code> groupings,
you can also group by <code>update-type</code>, which can be <code>patch</code>, <code>minor</code> or <code>major</code>
and only really works for dependencies that comply with <a href="https://semver.org/">Semantic
Versioning</a></p>
<h2>Effects of the configuration</h2>
<p>Within minutes, Dependabot had closed all* of the single dependency pull requests
and created two new pull requests. One PR contained the GitHub Actions updates,
which I could immediately merge.</p>
<blockquote>
<p>*For reasons I don't quite get at this point,
<a href="https://github.com/ASSERT-KTH/spork/pull/458">one single-dependency PR for <code>gumtree-spoon-ast-diff</code> remained
open</a></p>
</blockquote>
<p><img alt="Merged GitHub Actions dependencies" src="https://slar.se/photos/dependabot_dependency_grouping/actions_depsa.jpg"></p>
<p>Note that the <code>actions-deps</code> name, which is one of the keys in the YAML config
file, ends up being written out in the pull request title. Another PR was opened
for the grouped production dependencies, but it did not pass CI.</p>
<p><img alt="Production dependencies PR" src="https://slar.se/photos/dependabot_dependency_grouping/prod-depsa.jpg"></p>
<p>This showcases the one downside with grouped dependency updates: there's no
indication as to which dependency caused the CI failure. If you scroll up to the
first image of this article, you'll similarly see the one benefit of single
dependency pull requests: the update for <code>jgit</code> is failing.</p>
<blockquote>
<p>Note: As stated above, <code>gumtree-spoon-ast-diff</code> wasn't included in the grouped
dependency update for some reason, so we'll ignore that it's also failing.</p>
</blockquote>
<p>So I decided to try to get Dependabot to open a new pull request without <code>jgit</code>.</p>
<h2>Ignoring certain dependencies in a group</h2>
<p>I knew that <code>jgit</code> was the issue in this case and decided to ignore it. It
seemed like ignoring minor version updates was the best path forward, so I
issued an <code>@dependabot ignore <dependency_name> minor version</code> command.</p>
<p><img alt="Ignore minor version updates of jgit" src="https://slar.se/photos/dependabot_dependency_grouping/ignore_minora.jpg"></p>
<p>This didn't quite have the effect I was looking for, as Dependabot then
proceeded to
<a href="https://github.com/ASSERT-KTH/spork/pull/485">open a new pull request</a>
with the previous minor version of <code>jgit</code>, which is also incompatible with
Spork. At this point I decided to bring out the big guns and ignore <code>jgit</code>
altoghether for now, as I knew I needed to put in some manual work to update it
anyway.</p>
<p><img alt="Ignore all version updates of jgit" src="https://slar.se/photos/dependabot_dependency_grouping/ignorea.jpg"></p>
<p>This finally resulted in what I wanted, a new PR without a bump to <code>jgit</code> that
does not break the build. Thus, it's ready to merge.</p>
<p><img alt="Production dependency PR now passing checks" src="https://slar.se/photos/dependabot_dependency_grouping/green-prod-pra.jpg"></p>
<p>Of course, this comes with the caveat that I need to remember to <a href="https://github.com/ASSERT-KTH/spork/issues/487">unignore <code>jgit</code>
when I get around to updating it</a>.
But I have 8 dependency updates with not too much effort, which is kind of cool!</p>
<h1>Closing thoughts</h1>
<p>I had somewhat given up on keeping dependencies up to date in my lesser loved
projects. With grouped dependency updates, I feel like it will be possible for
me to get back to keeping dependencies in good shape even for projects that I no
longer actively work on.</p>
<p>The one downside is that it's no longer evident which dependency update breaks
the build. However, over many years of maintaining projects built in various
languages and technologies, experience tells me that it's way more common for
dependencies to update just fine than for them to break something, so I think
it's a tradeoff well worth taking. As patterns of which dependencies break the
build more often start to become apparent, one can also refine the groupings and
separate dependencies that break often from those that rarely or never do so.</p>
<p>To summarize, grouped dependency updates is a killer feature that makes
Dependabot viable again. Everyone should use it, and it's honestly a shame that
backwards compatibility demands makes it such that groups will probably never be
the default.</p>A new dark theme for the blog!2023-10-25T00:00:00+02:002023-10-25T00:00:00+02:00Simon Larséntag:slar.se,2023-10-25:/new-dark-theme.html<p>If you've <em>ever</em> visited my blog before, you'll notice it looks a bit
different. I finally took the time to make a dark theme! Only 5 or so years late
to the party. But late or not, I have a new theme to show off, so let's have a
look …</p><p>If you've <em>ever</em> visited my blog before, you'll notice it looks a bit
different. I finally took the time to make a dark theme! Only 5 or so years late
to the party. But late or not, I have a new theme to show off, so let's have a
look!</p>
<h1>Before and after</h1>
<p>I only put a couple of hours into trying out different colors and contrasts, so
this is still a bit of a work in progress. But I can't say I'm not pleased.
Let's have a look at what it used to look like.</p>
<p><img alt="The site with the old light theme" src="./images/new_dark_theme/before.png" style="max-width: 95%;"></p>
<p>So, that doesn't look terrible in any way, but it doesn't fit the "programmer
aesthetic". It doesn't fit <em>my</em> aesthetic. The new theme is much more "me", and
although you should already be seeing it all around here, here's the same image
as above with the new theme.</p>
<p><img alt="The site with the new dark theme" src="./images/new_dark_theme/after.png" style="max-width: 95%;"></p>
<p>Much better! There are still a few rough edges, and I keep finding details where
the old theme isn't being overridden correctly (because this is all just <em>on top
of</em> Bootstrap), but it's all mostly there.</p>
<h1>Parts of the puzzle</h1>
<p>There are really only two parts to this. I'm still using the same
<a href="https://github.com/robulouski/voidy-bootstrap"><code>voidy-bootstrap</code> theme</a> as
before, but I rewrote some of the CSS with the somewhat Halloween-looking
experience you're looking at right now. Which is fitting, considering Halloween
is just around the corner!</p>
<p>The second part is the fantastic
<a href="https://draculatheme.com/pygments"><code>dracula</code> theme for Pygments</a>
which gives the neat colorization of code snippets. That I find to be the single
largest improvement.</p>
<h1>Enjoy!</h1>
<p>That's all there is to it, enjoy the new theme!</p>What does the number in a man page mean?2023-10-24T00:00:00+02:002023-10-24T00:00:00+02:00Simon Larséntag:slar.se,2023-10-24:/man-page-numbering.html<p>If you open a <code>man</code> page on a *NIX system (such as a Linux distro), you'll
always see a number next to the subject of the <code>man</code> page. Like <code>GIT(1)</code>,
<code>SUDO(8)</code> or <code>open(n)</code>. What's that thing in parentheses? To cut a long
story short, it's the <em>section …</em></p><p>If you open a <code>man</code> page on a *NIX system (such as a Linux distro), you'll
always see a number next to the subject of the <code>man</code> page. Like <code>GIT(1)</code>,
<code>SUDO(8)</code> or <code>open(n)</code>. What's that thing in parentheses? To cut a long
story short, it's the <em>section</em> the <code>man</code> page belongs to. Let's discover
what that means.</p>
<h1><code>man</code> pages are divided into sections</h1>
<p><code>man</code> pages act as a system reference manual on any *NIX system. All <code>man</code> pages
have a heading containing the name of the page, its section and a very short
description. Something like this.</p>
<div class="highlight"><pre><span></span><code><name>(<#section>) <description> <name>(<#section>)
</code></pre></div>
<p>As a concrete example, here's the first line you get if you execute <code>man git</code>.</p>
<div class="highlight"><pre><span></span><code>GIT(1) Git Manual GIT(1)
</code></pre></div>
<p>The name is self-explanatory, is is the short description, but the section
number is not so transparent in its meaning. To find out what it means,
we can actually consult the <code>man</code> page for the <code>man</code> program itself.</p>
<div class="highlight"><pre><span></span><code><span class="n">$</span><span class="w"> </span><span class="n">man</span><span class="w"> </span><span class="n">man</span>
<span class="n">MAN</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="n">Manual</span><span class="w"> </span><span class="n">pager</span><span class="w"> </span><span class="n">utils</span><span class="w"> </span><span class="n">MAN</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="p">[...]</span>
<span class="n">The</span><span class="w"> </span><span class="n">table</span><span class="w"> </span><span class="n">below</span><span class="w"> </span><span class="n">shows</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">section</span><span class="w"> </span><span class="n">numbers</span><span class="w"> </span><span class="n">of</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">manual</span><span class="w"> </span><span class="n">followed</span>
<span class="n">by</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">types</span><span class="w"> </span><span class="n">of</span><span class="w"> </span><span class="n">pages</span><span class="w"> </span><span class="n">they</span><span class="w"> </span><span class="n">contain</span><span class="p">.</span>
<span class="mi">1</span><span class="w"> </span><span class="n">Executable</span><span class="w"> </span><span class="n">programs</span><span class="w"> </span><span class="n">or</span><span class="w"> </span><span class="n">shell</span><span class="w"> </span><span class="n">commands</span>
<span class="mi">2</span><span class="w"> </span><span class="n">System</span><span class="w"> </span><span class="n">calls</span><span class="w"> </span><span class="p">(</span><span class="n">functions</span><span class="w"> </span><span class="n">provided</span><span class="w"> </span><span class="n">by</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">kernel</span><span class="p">)</span>
<span class="mi">3</span><span class="w"> </span><span class="n">Library</span><span class="w"> </span><span class="n">calls</span><span class="w"> </span><span class="p">(</span><span class="n">functions</span><span class="w"> </span><span class="n">within</span><span class="w"> </span><span class="n">program</span><span class="w"> </span><span class="n">libraries</span><span class="p">)</span>
<span class="mi">4</span><span class="w"> </span><span class="n">Special</span><span class="w"> </span><span class="n">files</span><span class="w"> </span><span class="p">(</span><span class="n">usually</span><span class="w"> </span><span class="n">found</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="p">)</span>
<span class="mi">5</span><span class="w"> </span><span class="n">File</span><span class="w"> </span><span class="n">formats</span><span class="w"> </span><span class="n">and</span><span class="w"> </span><span class="n">conventions</span><span class="p">,</span><span class="w"> </span><span class="n">e</span><span class="p">.</span><span class="n">g</span><span class="p">.</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">passwd</span>
<span class="mi">6</span><span class="w"> </span><span class="n">Games</span>
<span class="mi">7</span><span class="w"> </span><span class="n">Miscellaneous</span><span class="w"> </span><span class="p">(</span><span class="n">including</span><span class="w"> </span><span class="n">macro</span><span class="w"> </span><span class="n">packages</span><span class="w"> </span><span class="n">and</span><span class="w"> </span><span class="n">conventions</span><span class="p">),</span>
<span class="w"> </span><span class="n">e</span><span class="p">.</span><span class="n">g</span><span class="p">.</span><span class="w"> </span><span class="n">man</span><span class="p">(</span><span class="mi">7</span><span class="p">),</span><span class="w"> </span><span class="n">groff</span><span class="p">(</span><span class="mi">7</span><span class="p">),</span><span class="w"> </span><span class="n">man</span><span class="o">-</span><span class="n">pages</span><span class="p">(</span><span class="mi">7</span><span class="p">)</span>
<span class="mi">8</span><span class="w"> </span><span class="n">System</span><span class="w"> </span><span class="n">administration</span><span class="w"> </span><span class="n">commands</span><span class="w"> </span><span class="p">(</span><span class="n">usually</span><span class="w"> </span><span class="n">only</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">root</span><span class="p">)</span>
<span class="mi">9</span><span class="w"> </span><span class="n">Kernel</span><span class="w"> </span><span class="n">routines</span><span class="w"> </span><span class="p">[</span><span class="n">Non</span><span class="w"> </span><span class="n">standard</span><span class="p">]</span>
<span class="p">[...]</span>
</code></pre></div>
<p>This sectioning makes it possible to have multiple <code>man</code> pages with the same
name, but in different sections. By default, you'll get only one result when
executing <code>man <name></code>, and which one you get is dependent on a pre-defined
search order. Which, of course, we can also find in the <code>man</code> page for <code>man</code>.</p>
<div class="highlight"><pre><span></span><code><span class="n">The</span><span class="w"> </span><span class="n">order</span><span class="w"> </span><span class="n">of</span><span class="w"> </span><span class="n">sections</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">search</span><span class="w"> </span><span class="n">may</span><span class="w"> </span><span class="n">be</span><span class="w"> </span><span class="n">overridden</span><span class="w"> </span><span class="n">by</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">envi</span><span class="err">‐</span>
<span class="n">ronment</span><span class="w"> </span><span class="n">variable</span><span class="w"> </span><span class="o">$</span><span class="n">MANSECT</span><span class="w"> </span><span class="ow">or</span><span class="w"> </span><span class="n">by</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">SECTION</span><span class="w"> </span><span class="n">directive</span><span class="w"> </span><span class="ow">in</span>
<span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">man_db</span><span class="o">.</span><span class="n">conf</span><span class="o">.</span><span class="w"> </span><span class="n">By</span><span class="w"> </span><span class="n">default</span><span class="w"> </span><span class="n">it</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">follows</span><span class="p">:</span>
<span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="mi">1</span><span class="n">p</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">l</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="mi">3</span><span class="n">p</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">0</span><span class="n">p</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="mi">3</span><span class="n">type</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="mi">9</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="mi">7</span>
</code></pre></div>
<p>In my personal experience, the default search order most often gives you what
you want. But sometimes it doesn't, and then you need to figure out how to find
the man page you're after.</p>
<h1>Selecting man pages from different sections</h1>
<p>To select a page from a particular section, you specify the section before the
name. For example, as indicated by the excerpt from <code>MAN(1)</code> above, there's also
a <code>man(7)</code>. We can get it like so.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>man<span class="w"> </span><span class="m">7</span><span class="w"> </span>man
man<span class="o">(</span><span class="m">7</span><span class="o">)</span><span class="w"> </span>Miscellaneous<span class="w"> </span>Information<span class="w"> </span>Manual<span class="w"> </span>man<span class="o">(</span><span class="m">7</span><span class="o">)</span>
<span class="o">[</span>...<span class="o">]</span>
</code></pre></div>
<p>But what if you don't actually know which section the man page you're looking
for is in, you only know that the one you're looking at isn't the one? Then you
can use <code>whatis</code>. For example, I have four different man pages named <code>open</code> on
my machine.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>whatis<span class="w"> </span>open
open<span class="w"> </span><span class="o">(</span><span class="m">2</span><span class="o">)</span><span class="w"> </span>-<span class="w"> </span>open<span class="w"> </span>and<span class="w"> </span>possibly<span class="w"> </span>create<span class="w"> </span>a<span class="w"> </span>file
open<span class="w"> </span><span class="o">(</span>3p<span class="o">)</span><span class="w"> </span>-<span class="w"> </span>open<span class="w"> </span>file
open<span class="w"> </span><span class="o">(</span>3perl<span class="o">)</span><span class="w"> </span>-<span class="w"> </span>perl<span class="w"> </span>pragma<span class="w"> </span>to<span class="w"> </span><span class="nb">set</span><span class="w"> </span>default<span class="w"> </span>PerlIO<span class="w"> </span>layers<span class="w"> </span><span class="k">for</span><span class="w"> </span>input<span class="w"> </span>and<span class="w"> </span>output
open<span class="w"> </span><span class="o">(</span>n<span class="o">)</span><span class="w"> </span>-<span class="w"> </span>Open<span class="w"> </span>a<span class="w"> </span>file-based<span class="w"> </span>or<span class="w"> </span><span class="nb">command</span><span class="w"> </span>pipeline<span class="w"> </span>channel
</code></pre></div>
<p>Here you can also see a couple of section numbers that look a bit different,
namely <code>3perl</code> and <code>n</code>. These don't belong to any of the standard sections, but
you can open them just the same. For example, <code>man 3perl open</code> would open the
<code>open</code> page from the custom <code>3perl</code> section.</p>
<blockquote>
<p>Note: If you don't see all available man pages when running <code>whatis</code> for a
particular page, your <code>man</code> database is out-of-date, you may want to run
<code>mandb</code> manually (or whatever is equivalent on your system) to rebuild the
search index.</p>
</blockquote>
<h1>And that's all!</h1>
<p>Now you should hopefully be a bit more confident in finding the man page you're
looking for. And if at any point you forget what I've written here, almost all
of the information is available just a <code>man man</code> away!</p>The sheer insanity of interfaces and nil in Go2023-10-21T00:00:00+02:002023-10-21T00:00:00+02:00Simon Larséntag:slar.se,2023-10-21:/the-sheer-insanity-of-interfaces-and-nil-in-go.html<p>If you've only dabbled briefly in Go, you might think that its <code>nil</code> is
analogous to the good ol' <a href="https://en.wikipedia.org/wiki/Tony_Hoare#Apologies_and_retractions">"billion dollar mistake"</a>
known as <code>null</code>. I thought so, too, up until just a few weeks ago when I
decided to make a pass through Thorsten Ball's neat little book
<a href="https://slar.se/book-review-writing-an-interpreter-in-go.html">Writing …</a></p><p>If you've only dabbled briefly in Go, you might think that its <code>nil</code> is
analogous to the good ol' <a href="https://en.wikipedia.org/wiki/Tony_Hoare#Apologies_and_retractions">"billion dollar mistake"</a>
known as <code>null</code>. I thought so, too, up until just a few weeks ago when I
decided to make a pass through Thorsten Ball's neat little book
<a href="https://slar.se/book-review-writing-an-interpreter-in-go.html">Writing an Interpreter in Go</a>.
That's when I first cut myself real bad on <code>nil</code>, or more specificaly, an
interface with a <code>nil</code> value.</p>
<h1>The sensible kind of <code>nil</code></h1>
<p>First let's look at <code>nil</code> behaving like we'd expect. Here's a code snippet of a
"parser" that somewhat shows the situation I had.</p>
<div class="highlight"><pre><span></span><code><span class="kn">package</span><span class="w"> </span><span class="nx">main</span>
<span class="kn">import</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="s">"fmt"</span>
<span class="w"> </span><span class="s">"strings"</span>
<span class="p">)</span>
<span class="kd">type</span><span class="w"> </span><span class="nx">IfExpression</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">raw</span><span class="w"> </span><span class="kt">string</span>
<span class="p">}</span>
<span class="kd">func</span><span class="w"> </span><span class="nx">parseIf</span><span class="p">(</span><span class="nx">s</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="nx">IfExpression</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">strings</span><span class="p">.</span><span class="nx">HasPrefix</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span><span class="w"> </span><span class="s">"if"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="o">&</span><span class="nx">IfExpression</span><span class="p">{</span><span class="nx">raw</span><span class="p">:</span><span class="w"> </span><span class="nx">s</span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span>
<span class="p">}</span>
</code></pre></div>
<p>So, we have a parse function that returns a pointer to an <code>IfExpression</code> struct
if the input is an <code>if</code> expression (with an incredibly loose definition of what
constitutes an <code>if</code> expression). Let's add a <code>main</code> function and try it out.</p>
<div class="highlight"><pre><span></span><code><span class="kd">func</span><span class="w"> </span><span class="nx">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">ifExpr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">parseIf</span><span class="p">(</span><span class="s">"if a == b"</span><span class="p">)</span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">"ifExpr"</span><span class="p">,</span><span class="w"> </span><span class="nx">ifExpr</span><span class="p">,</span><span class="w"> </span><span class="nx">ifExpr</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">nil</span><span class="p">)</span>
<span class="w"> </span><span class="nx">notIfExpr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">parseIf</span><span class="p">(</span><span class="s">"not an expression"</span><span class="p">)</span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">"notIfExpr"</span><span class="p">,</span><span class="w"> </span><span class="nx">notIfExpr</span><span class="p">,</span><span class="w"> </span><span class="nx">notIfExpr</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">nil</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div>
<p>Running this results in completely sensible output; <code>ifExpr</code> is not <code>nil</code> while
<code>notIfExpr</code> is.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>go<span class="w"> </span>run<span class="w"> </span>main.go
<span class="p">&</span><span class="o">{</span><span class="k">if</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="o">==</span><span class="w"> </span>b<span class="o">}</span><span class="w"> </span><span class="nb">false</span>
<nil><span class="w"> </span><span class="nb">true</span>
</code></pre></div>
<p>Say what you want about <code>nil</code>, this at least makes complete sense and is
intuitively understandable. But when we throw interfaces into the mix,
that goes way out the window.</p>
<h1>The completely insane kind of <code>nil</code></h1>
<p>As my parser came along, it turned out I needed multiple kinds of expressions,
and so had to abstract the parse function to be able to return multiple kinds
of expressions. So let's generalize the expression in an interface and add
another one.</p>
<div class="highlight"><pre><span></span><code><span class="kd">type</span><span class="w"> </span><span class="nx">Expression</span><span class="w"> </span><span class="kd">interface</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">Raw</span><span class="p">()</span><span class="w"> </span><span class="kt">string</span>
<span class="p">}</span>
<span class="kd">type</span><span class="w"> </span><span class="nx">IfExpression</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">raw</span><span class="w"> </span><span class="kt">string</span>
<span class="p">}</span>
<span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">ie</span><span class="w"> </span><span class="o">*</span><span class="nx">IfExpression</span><span class="p">)</span><span class="w"> </span><span class="nx">Raw</span><span class="p">()</span><span class="w"> </span><span class="kt">string</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">ie</span><span class="p">.</span><span class="nx">raw</span><span class="w"> </span><span class="p">}</span>
<span class="kd">type</span><span class="w"> </span><span class="nx">ForExpression</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">raw</span><span class="w"> </span><span class="kt">string</span>
<span class="p">}</span>
<span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">fe</span><span class="w"> </span><span class="o">*</span><span class="nx">ForExpression</span><span class="p">)</span><span class="w"> </span><span class="nx">Raw</span><span class="p">()</span><span class="w"> </span><span class="kt">string</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">fe</span><span class="p">.</span><span class="nx">raw</span><span class="w"> </span><span class="p">}</span>
</code></pre></div>
<p>And then we adapt the parsing as well.</p>
<div class="highlight"><pre><span></span><code><span class="kd">func</span><span class="w"> </span><span class="nx">parse</span><span class="p">(</span><span class="nx">s</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="nx">Expression</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">ifExpr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">parseIf</span><span class="p">(</span><span class="nx">s</span><span class="p">);</span><span class="w"> </span><span class="nx">ifExpr</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">ifExpr</span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">parseFor</span><span class="p">(</span><span class="nx">s</span><span class="p">)</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
<span class="kd">func</span><span class="w"> </span><span class="nx">parseFor</span><span class="p">(</span><span class="nx">s</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="nx">ForExpression</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">strings</span><span class="p">.</span><span class="nx">HasPrefix</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span><span class="w"> </span><span class="s">"for"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="o">&</span><span class="nx">ForExpression</span><span class="p">{</span><span class="nx">raw</span><span class="p">:</span><span class="w"> </span><span class="nx">s</span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span>
<span class="p">}</span>
<span class="kd">func</span><span class="w"> </span><span class="nx">parseIf</span><span class="p">(</span><span class="nx">s</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="nx">IfExpression</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">strings</span><span class="p">.</span><span class="nx">HasPrefix</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span><span class="w"> </span><span class="s">"if"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="o">&</span><span class="nx">IfExpression</span><span class="p">{</span><span class="nx">raw</span><span class="p">:</span><span class="w"> </span><span class="nx">s</span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span>
<span class="p">}</span>
</code></pre></div>
<p>Now, we have a more generalized <code>parse</code> function that tries to parse an <code>if</code>
expression, and falls back on parsing a <code>for</code> expression if it turns out not to
be an <code>if</code>. We then adapt our <code>main</code> function to make use of this.</p>
<div class="highlight"><pre><span></span><code><span class="kd">func</span><span class="w"> </span><span class="nx">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">ifExpr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">parse</span><span class="p">(</span><span class="s">"if a == b"</span><span class="p">)</span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">"ifExpr"</span><span class="p">,</span><span class="w"> </span><span class="nx">ifExpr</span><span class="p">,</span><span class="w"> </span><span class="nx">ifExpr</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">nil</span><span class="p">)</span>
<span class="w"> </span><span class="nx">forExpr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">parse</span><span class="p">(</span><span class="s">"for a in b"</span><span class="p">)</span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">"forExpr"</span><span class="p">,</span><span class="w"> </span><span class="nx">forExpr</span><span class="p">,</span><span class="w"> </span><span class="nx">forExpr</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">nil</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div>
<p>And at first glance, this seems to work as expected.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>go<span class="w"> </span>run<span class="w"> </span>main.go<span class="w"> </span>
ifExpr<span class="w"> </span><span class="p">&</span><span class="o">{</span><span class="k">if</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="o">==</span><span class="w"> </span>b<span class="o">}</span><span class="w"> </span><span class="nb">false</span>
forExpr<span class="w"> </span><span class="p">&</span><span class="o">{</span><span class="k">for</span><span class="w"> </span>a<span class="w"> </span><span class="k">in</span><span class="w"> </span>b<span class="o">}</span><span class="w"> </span><span class="nb">false</span>
</code></pre></div>
<p>But what happens when you try to parse something that is neither an <code>if</code> nor a
<code>for</code>? We add the following to <code>main</code> to find out.</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="nx">notAnExpr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">parse</span><span class="p">(</span><span class="s">"not an expression"</span><span class="p">)</span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">"notAnExpr"</span><span class="p">,</span><span class="w"> </span><span class="nx">notAnExpr</span><span class="p">,</span><span class="w"> </span><span class="nx">notAnExpr</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">nil</span><span class="p">)</span>
</code></pre></div>
<p>Clearly, as <code>"not an expression"</code> is neither an <code>if</code> expression nor a <code>for</code>
expression, both <code>parseIf</code> and <code>parseFor</code> will return <code>nil</code>, and as such <code>parse</code>
should return <code>nil</code>. But that's not <em>really</em> what happens.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>go<span class="w"> </span>run<span class="w"> </span>main.go<span class="w"> </span>
ifExpr<span class="w"> </span><span class="p">&</span><span class="o">{</span><span class="k">if</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="o">==</span><span class="w"> </span>b<span class="o">}</span><span class="w"> </span><span class="nb">false</span>
forExpr<span class="w"> </span><span class="p">&</span><span class="o">{</span><span class="k">for</span><span class="w"> </span>a<span class="w"> </span><span class="k">in</span><span class="w"> </span>b<span class="o">}</span><span class="w"> </span><span class="nb">false</span>
notAnExpr<span class="w"> </span><nil><span class="w"> </span><span class="nb">false</span>
</code></pre></div>
<p>What? <code>notAnExpr</code> is <code><nil></code>, but it does not compare true to a literal<code>nil</code>?
What does that mean?</p>
<h1>The confusing type-and-value composition of interfaces</h1>
<p>In Go, an interface is represented at runtime as a
<a href="https://go.dev/doc/faq#nil_error">type and a value</a>. You can think of it as a
struct with two fields.</p>
<div class="highlight"><pre><span></span><code><span class="kd">type</span><span class="w"> </span><span class="nx">Interface</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">type</span><span class="w"> </span><span class="kt">string</span>
<span class="w"> </span><span class="nx">value</span><span class="w"> </span><span class="kd">interface</span><span class="p">{}</span>
<span class="p">}</span>
</code></pre></div>
<p>So, when <code>parseIf(s)</code> returns <code>nil</code>, the <code>parse(s)</code> function's return statement
wraps the <code>nil</code> value into something like this.</p>
<div class="highlight"><pre><span></span><code><span class="nx">Interface</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">Type</span><span class="p">:</span><span class="w"> </span><span class="s">"IfExpression"</span><span class="p">,</span>
<span class="w"> </span><span class="nx">Value</span><span class="p">:</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span>
<span class="p">}</span>
</code></pre></div>
<p>Armed with this knowledge, we can actually check for <code>nil</code> using
<a href="https://pkg.go.dev/reflect">runtime reflection</a>.</p>
<div class="highlight"><pre><span></span><code><span class="kd">func</span><span class="w"> </span><span class="nx">IsNil</span><span class="p">(</span><span class="nx">value</span><span class="w"> </span><span class="kd">interface</span><span class="p">{})</span><span class="w"> </span><span class="kt">bool</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">reflect</span><span class="p">.</span><span class="nx">ValueOf</span><span class="p">(</span><span class="nx">value</span><span class="p">).</span><span class="nx">IsNil</span><span class="p">()</span>
<span class="p">}</span>
<span class="kd">func</span><span class="w"> </span><span class="nx">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">ifExpr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">parse</span><span class="p">(</span><span class="s">"if a == b"</span><span class="p">)</span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">"ifExpr"</span><span class="p">,</span><span class="w"> </span><span class="nx">ifExpr</span><span class="p">,</span><span class="w"> </span><span class="nx">IsNil</span><span class="p">(</span><span class="nx">ifExpr</span><span class="p">))</span>
<span class="w"> </span><span class="nx">forExpr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">parse</span><span class="p">(</span><span class="s">"for a in b"</span><span class="p">)</span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">"forExpr"</span><span class="p">,</span><span class="w"> </span><span class="nx">forExpr</span><span class="p">,</span><span class="w"> </span><span class="nx">IsNil</span><span class="p">(</span><span class="nx">forExpr</span><span class="p">))</span>
<span class="w"> </span><span class="nx">notAnExpr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">parse</span><span class="p">(</span><span class="s">"a == b"</span><span class="p">)</span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">"notAnExpr"</span><span class="p">,</span><span class="w"> </span><span class="nx">notAnExpr</span><span class="p">,</span><span class="w"> </span><span class="nx">IsNil</span><span class="p">(</span><span class="nx">notAnExpr</span><span class="p">))</span>
<span class="p">}</span>
</code></pre></div>
<p>And running this, we now seem to have a functioning way to check for <code>nil</code>
interfaces.</p>
<div class="highlight"><pre><span></span><code><span class="nv">ifExpr</span><span class="w"> </span><span class="o">&</span>{<span class="k">if</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="nv">b</span>}<span class="w"> </span><span class="nv">false</span>
<span class="nv">forExpr</span><span class="w"> </span><span class="o">&</span>{<span class="k">for</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="nv">in</span><span class="w"> </span><span class="nv">b</span>}<span class="w"> </span><span class="nv">false</span>
<span class="nv">notAnExpr</span><span class="w"> </span><span class="o"><</span><span class="nv">nil</span><span class="o">></span><span class="w"> </span><span class="nv">true</span>
</code></pre></div>
<p>But do we really? Of course not.</p>
<h1>Actually, interfaces can also be "completely" <code>nil</code></h1>
<p>Let's make this even more confusing, and refactor our <code>parse</code> function a bit.
Instead of "falling back" on <code>parseFor</code>, we simply return <code>nil</code> explicitly if
we can't parse any known expression.</p>
<div class="highlight"><pre><span></span><code><span class="kd">func</span><span class="w"> </span><span class="nx">parse</span><span class="p">(</span><span class="nx">s</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="nx">Expression</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">ifExpr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">parseIf</span><span class="p">(</span><span class="nx">s</span><span class="p">);</span><span class="w"> </span><span class="nx">ifExpr</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">ifExpr</span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">forExpr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">parseFor</span><span class="p">(</span><span class="nx">s</span><span class="p">);</span><span class="w"> </span><span class="nx">forExpr</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">forExpr</span><span class="w"> </span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">nil</span>
<span class="p">}</span>
</code></pre></div>
<p>Now let's run it again.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>go<span class="w"> </span>run<span class="w"> </span>main.go<span class="w"> </span>
ifExpr<span class="w"> </span><span class="p">&</span><span class="o">{</span><span class="k">if</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="o">==</span><span class="w"> </span>b<span class="o">}</span><span class="w"> </span><span class="nb">false</span>
forExpr<span class="w"> </span><span class="p">&</span><span class="o">{</span><span class="k">for</span><span class="w"> </span>a<span class="w"> </span><span class="k">in</span><span class="w"> </span>b<span class="o">}</span><span class="w"> </span><span class="nb">false</span>
panic:<span class="w"> </span>reflect:<span class="w"> </span>call<span class="w"> </span>of<span class="w"> </span>reflect.Value.IsNil<span class="w"> </span>on<span class="w"> </span>zero<span class="w"> </span>Value
</code></pre></div>
<p>Suddenly, <code>notAnExpr</code> <em>is</em> <code>nil</code>, causing our check to panic. What in the world
is going on here? As it turns out, explicitly returning <code>nil</code> from a function
that returns an interface is semantically different to returning a <a href="https://go.dev/doc/faq#nil_error">variable
contaning <code>nil</code> value, and typed with something implementing the
interface</a>.</p>
<div class="highlight"><pre><span></span><code><span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="c1">// actually returns nil</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">value</span><span class="w"> </span><span class="o">*</span><span class="nx">ForExpression</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="kc">nil</span>
<span class="k">return</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="c1">// returns an interface with type *ForExpression and value nil</span>
</code></pre></div>
<p>So a way around this is of course to always return <code>nil</code> in the form of a typed
variable. But I'm not satisfied with that, because even if I'm very strict
about it myself, others may not be. And I may also just make an honest mistake.
So let's tweak our <code>IsNil()</code> function to account for true <code>nil</code>.</p>
<div class="highlight"><pre><span></span><code><span class="kd">func</span><span class="w"> </span><span class="nx">IsNil</span><span class="p">(</span><span class="nx">value</span><span class="w"> </span><span class="kd">interface</span><span class="p">{})</span><span class="w"> </span><span class="kt">bool</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">value</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="nx">reflect</span><span class="p">.</span><span class="nx">ValueOf</span><span class="p">(</span><span class="nx">value</span><span class="p">).</span><span class="nx">IsNil</span><span class="p">()</span>
<span class="p">}</span>
</code></pre></div>
<p>If you run this, you'll see that the runtime panic is replaced with the
following.</p>
<div class="highlight"><pre><span></span><code>notAnExpr <nil> true
</code></pre></div>
<p>So we're done? Unfortunately, not quite.</p>
<h1>Structs are a problem</h1>
<p>Our <code>IsNil()</code> handles true <code>nil</code> and interfaces with <code>nil</code> values. But,
unfortunately, it does not handle structs.</p>
<div class="highlight"><pre><span></span><code><span class="nx">value</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">IfExpression</span><span class="p">{</span><span class="nx">raw</span><span class="p">:</span><span class="w"> </span><span class="s">"if"</span><span class="p">}</span>
<span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="nx">IsNil</span><span class="p">(</span><span class="nx">value</span><span class="p">))</span><span class="w"> </span><span class="c1">// panic: reflect: call of reflect.Value.IsNil on struct Value</span>
</code></pre></div>
<p>Poop. It appears we missed a critical part of
<a href="https://pkg.go.dev/reflect#Value.IsNil"><code>IsNil()</code>'s documentation</a>.
And by missed, I of course mean that we didn't read it, but here's the relevant
part.</p>
<blockquote>
<p>IsNil reports whether its argument v is nil. The argument must be a chan,
func, interface, map, pointer, or slice value; if it is not, IsNil panics.</p>
</blockquote>
<p>So, we need to check if the value is of any of these <code>nil</code>-able types,
and otherwise assume that it is not <code>nil</code>.</p>
<div class="highlight"><pre><span></span><code><span class="kd">func</span><span class="w"> </span><span class="nx">IsNil</span><span class="p">(</span><span class="nx">value</span><span class="w"> </span><span class="kd">interface</span><span class="p">{})</span><span class="w"> </span><span class="kt">bool</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">value</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="nx">reflected</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">reflect</span><span class="p">.</span><span class="nx">ValueOf</span><span class="p">(</span><span class="nx">value</span><span class="p">)</span>
<span class="w"> </span><span class="k">switch</span><span class="w"> </span><span class="nx">reflected</span><span class="p">.</span><span class="nx">Kind</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">case</span>
<span class="w"> </span><span class="nx">reflect</span><span class="p">.</span><span class="nx">Chan</span><span class="p">,</span>
<span class="w"> </span><span class="nx">reflect</span><span class="p">.</span><span class="nx">Func</span><span class="p">,</span>
<span class="w"> </span><span class="nx">reflect</span><span class="p">.</span><span class="nx">Interface</span><span class="p">,</span>
<span class="w"> </span><span class="nx">reflect</span><span class="p">.</span><span class="nx">Map</span><span class="p">,</span>
<span class="w"> </span><span class="nx">reflect</span><span class="p">.</span><span class="nx">Ptr</span><span class="p">,</span>
<span class="w"> </span><span class="nx">reflect</span><span class="p">.</span><span class="nx">Slice</span><span class="p">:</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">reflected</span><span class="p">.</span><span class="nx">IsNil</span><span class="p">()</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">false</span>
<span class="p">}</span>
</code></pre></div>
<p>And with this, the panic is abated.</p>
<div class="highlight"><pre><span></span><code><span class="nx">value</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">IfExpression</span><span class="p">{</span><span class="nx">raw</span><span class="p">:</span><span class="w"> </span><span class="s">"if"</span><span class="p">}</span>
<span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="nx">IsNil</span><span class="p">(</span><span class="nx">value</span><span class="p">))</span><span class="w"> </span><span class="c1">// false</span>
</code></pre></div>
<p>This will work so long as Go doesn't add some other <code>nil</code>-able type. Given how
conservative the language is, such an addition to the core types seems an
unlikely prospect. But of course, it could happen, and then this function would
return <code>false</code> incorrectly in some cases.</p>
<h1>Footguns</h1>
<p>All languages have <a href="https://en.wiktionary.org/wiki/footgun">footguns</a>. Some
languages, like C++ and JavaScript, are seemingly built exclusively of them.
Go is a small-ish and conservative language, and so naturally has fewer such
contraptions. But how <code>nil</code> is handled differently in different contexts and how
the type of a variable can change the actual value that's returned from a
function, is a big one.</p>
<p>Of course, all of this may just be symptoms of my not knowing Go very well, and
this whole article could just be the frustrated ramblings of an apprentice. But,
I'm an experienced programmer, and this confused me quite a bit. That alone
should be sufficient evidence that some design choices made here aren't optimal.</p>Book Review: Writing an Interpreter in Go2023-10-14T00:00:00+02:002023-10-14T00:00:00+02:00Simon Larséntag:slar.se,2023-10-14:/book-review-writing-an-interpreter-in-go.html<p>I love programming languages, both using them and implementing them. As such, I
found the concept of learning Go by creating a programming language to be just
delightful. And, to put it briefly, it was. Let's talk about Thorsten Ball's
book on interpreters. In Go.</p>
<div class="highlight"><pre><span></span><code><span class="n">Writing</span><span class="w"> </span><span class="n">an</span><span class="w"> </span><span class="n">Interpreter</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="k">Go …</span></code></pre></div><p>I love programming languages, both using them and implementing them. As such, I
found the concept of learning Go by creating a programming language to be just
delightful. And, to put it briefly, it was. Let's talk about Thorsten Ball's
book on interpreters. In Go.</p>
<div class="highlight"><pre><span></span><code><span class="n">Writing</span><span class="w"> </span><span class="n">an</span><span class="w"> </span><span class="n">Interpreter</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="k">Go</span>
<span class="k">by</span><span class="w"> </span><span class="n">Thorsten</span><span class="w"> </span><span class="n">Ball</span>
<span class="nl">Publisher</span><span class="p">:</span><span class="w"> </span><span class="n">Thorsten</span><span class="w"> </span><span class="n">Ball</span>
<span class="nl">ISBN</span><span class="p">:</span><span class="w"> </span><span class="mi">9783982016115</span>
</code></pre></div>
<blockquote>
<p>You can buy the book directly from Thorsten Ball's website at
<a href="https://interpreterbook.com/">https://interpreterbook.com/</a>.</p>
</blockquote>
<h1>The book in a nutshell</h1>
<p><em>Writing an Interpreter in Go</em> is precisely what it sounds like; a practical
guide to writing a fully functioning interpreter in Go. You go from source
code, to tokens, to abstract syntax tree and tree evaluation. The parser
is based on the pretty fascinating
<a href="https://en.wikipedia.org/wiki/Operator-precedence_parser#Pratt_parsing"><em>Pratt parsing</em></a>
technique, while the evaluation is based on simple tree-walking. This relative
simplicity allows Ball to cram a whole lot of functionality into a very concise
200 pages worth of book. The result is impressive, and I am somewhat astonished
by just how much content is actually in here, and how well written said content
is.</p>
<h1>What I liked</h1>
<p>There is so much to like about this book. First of all, it is wonderfully
standalone. I wrote around 3500 lines of code based entirely on descriptions
and code samples from the book's 200 printed pages. Although the source code
for the interpreter is available as part of the book, I never needed to
reference it.</p>
<p>Something else I just barely needed to reference was documentation for Go
itself. While this book doesn't teach you programming and is therefore not for
complete beginners, it does somewhat teach you Go by example. There are little
to no explanations for how Go works, but due to how simple Go is I think the
examples speak for themselves. A caveat to that is that I have done some Go
programming in the past, and I think that a complete beginner to Go should
probably go through the interactive <a href="https://go.dev/tour/">A Tour of Go</a>
tutorial first. The only thing that I personally needed to reference was some
details around how interfaces and <code>nil</code> work together in Go (it's <a href="https://go.dev/doc/faq#nil_error">really quite
unintuitive</a>). It should also be noted that
there is nothing about concurrency in this book, which being a major selling
point of Go entails that this book omits some important parts of the language.
I do however find that completely reasonable given the scope of the book.</p>
<p>The book is also for the most part organized in a way that fit me very well.
In the first few parts of the book, the implementation is given up front and
then tests are added to verify the behavior. The majority of the later parts of
the book are however laid out in the opposite order, with descriptions of
functionality and tests being followed by the actual implementation. This
allowed me to read the tests and the descriptions to get a good idea of the
intended behavior, and then try my own hand at coming up with a solution.
Comparing my solution to the author's after the fact was a great way to cement
the knowledge.</p>
<p>The source code exhibits a splendid balance between proper design and being
simple enough to put into a book this short. There were several design decisions
that I did not fully agree with, especially with the parsing of strings, but I
can also see that certain simplifications had to be made to fit the format. For
the most part, I think these simplifications are well chosen.</p>
<p>Concepts are explained clearly and intuitively. Pratt parsing is by far the most
involved topic in the book, and I think Ball presents it in a very digestible
fashion. I needed to run through it a couple of times and mentally step through
the code, but when it clicked I found no fault in how the concepts were
explained. At no point did I need to reference external sources to understand
something.</p>
<p>The last part of the book adds in a few extra features, such as arrays and hash
maps. All of these extra features are implemented in a very satisfying loop,
going all the way from the lexing to the finished evaluation in one short
chapter for each feature. Adding one feature at a time in this way really aids
in understanding the workings of each part of the interpreter. And that, in
fact, is a perfect segue to the one part of this book that I did not entirely
like.</p>
<h1>What I didn't like</h1>
<p>I actually started this book around five years ago, but never finished it. In
fact, I got just barely halfway due to what I perceive as the one flaw of this
otherwise fantastic piece of literature: it has a rather slow start. It's not
until halfway through that you actually get to evaluation and thereby create a
complete path from source code to output. The fantastic pacing of the last
quarter of the book where you add features one at a time highlights that the
first half isn't as satisfying as it probably could be. While I recognize that
the amount of groundwork to put down before getting to the exciting parts is a
tough one to make, I think the book would have been better off closing the path
from source code to evaluation earlier. For instance, I don't see any need to
parse function definitions before evaluating arithmetic expressions.</p>
<h1>Conclusions</h1>
<p>This is a great book. It's clear, concise and packed full of learnings. Due to
the way the last quarter of the book is laid out, adding feature upon feature
from start to finish, I was able to quite effortlessly add features completely
of my own design by the time I finished the book. The only complaint I have
about the book is that not more of it is laid out in that fashion, because it's
just so good.</p>
<p>If you've dipped your toes in Go and want an exciting project to learn the
language better, I think this is a great way to do it. It takes a little while
before the book gets going for real, but when it does, it really takes off.</p>Fix 3D graphics in Arch Linux on Dell XPS 15 95202022-12-26T00:00:00+01:002022-12-26T00:00:00+01:00Simon Larséntag:slar.se,2022-12-26:/xps-15-9520-arch-linux-intel-dri2.html<p>I recently got myself a Dell XPS 15 9520 to replace my aging laptop. It's
highly Linux-compatible and just following the <a href="https://wiki.archlinux.org/title/Installation_guide">official installation
guide</a> got me 90% of the
way of having a well-behaved Arch Linux laptop. One big thing wasn't working,
though: 3D graphics!</p>
<h2>Symptoms: glitchy 2D graphics and …</h2><p>I recently got myself a Dell XPS 15 9520 to replace my aging laptop. It's
highly Linux-compatible and just following the <a href="https://wiki.archlinux.org/title/Installation_guide">official installation
guide</a> got me 90% of the
way of having a well-behaved Arch Linux laptop. One big thing wasn't working,
though: 3D graphics!</p>
<h2>Symptoms: glitchy 2D graphics and unusable 3D graphics</h2>
<p>2D graphics worked mostly fine after installing <code>xf86-video-intel</code>, <code>nvidia</code> and
<code>nvidia-prime</code>. Websites, most desktop applications as well as 2D games like
Terraria worked without issue. Anything with the slightest hint of 3D would
however glitch out terribly with Intel graphics and just be completely black
with 3D graphics.</p>
<p>Some applications, such as Steam, wouldn't render properly unless in fullscreen.
3D benchmark software like <code>fire</code> (from <code>mesa-demo</code>) and <code>glmark2</code> wouldn't
update properly with Intel and would again be black with NVIDIA.</p>
<h2>Solution: Using DRI2 for the Intel driver</h2>
<p>The solution I found to this problem was to <a href="https://wiki.archlinux.org/title/Intel_graphics#DRI3_issues">use
DRI2 for the Intel
driver</a>. I have no
idea of why this works, but it does. You can configure the Intel driver to use
DRI2 by creating a configuration file for X.</p>
<div class="highlight"><pre><span></span><code># /etc/X11/xorg.conf.d/20-intel.conf
Section "Device"
Identifier "Intel Graphics"
Driver "intel"
Option "DRI" "2"
EndSection
</code></pre></div>
<blockquote>
<p><strong>Note:</strong> The <a href="https://wiki.archlinux.org/title/Intel_graphics#Issues_with_selecting_Qt_elements_within_Plasma_Desktop_on_Alder_Lake.2FUHD_770">modesetting driver also works</a>, but it interferes with PRIME offloading s.t. the NVIDIA GPU is always active. This butchers battery life and so wasn't acceptable to me.</p>
</blockquote>
<h2>Need more tweaks?</h2>
<p>This wasn't the only tweak I put in to get my XPS 15 in tip-top shape, although
it was the most crucial one. I've got a bunch of other tweaks listed <a href="https://slar.se/pages/wiki/arch-linux-on-xps-15-9520.html">over on
my Wiki page for the XPS 15</a>.</p>Book Review: CPython Internals2022-11-20T00:00:00+01:002022-11-20T00:00:00+01:00Simon Larséntag:slar.se,2022-11-20:/book-review-cpython-internals.html<p>About a month ago I signed myself up to do a talk at a Python meetup hosted by
<a href="https://hiq.se/">HiQ</a>. I brazenly set my topic as <em>Under the Hood of CPython</em>,
thinking I had sufficient understanding of its inner workings to produce a
riveting talk. As I started preparing the talk …</p><p>About a month ago I signed myself up to do a talk at a Python meetup hosted by
<a href="https://hiq.se/">HiQ</a>. I brazenly set my topic as <em>Under the Hood of CPython</em>,
thinking I had sufficient understanding of its inner workings to produce a
riveting talk. As I started preparing the talk, I came to the gut-wrenching
conclusion that my knowledge was too shallow, I simply didn't know enough of
the details to put together an in-depth talk on the subject. Thankfully, I knew
where to turn to for the details I needed: Anthony Shaw's <em>CPython Internals</em>
book. Here's what I think of it.</p>
<div class="highlight"><pre><span></span><code>CPython Internals
by Anthony Shaw
Released May 2021
Publisher(s): Real Python (realpython.com)
ISBN: 9781775093344
</code></pre></div>
<blockquote>
<p>Confused about the difference between Python and CPython? See <a href="https://slar.se/the-difference-between-python-and-cpython.html">The difference
between Python and
CPython</a>.</p>
</blockquote>
<h1>The book in a nutshell</h1>
<p><em>CPython Internals</em> gives you a guided tour of the CPython project, from
parsing source code to compiling bytecode to interpreting said bytecode. The
book is meant to serve as a starting point for budding CPython contributors or
Python developers that simply want to learn a bit more about the reference
implementation. It highlights the most important files in the project for each
of the respective parts and guides you through their execution. Throughout the
book we also get to follow along with a worked example of extending the language
with an "almost equal" operator, written as <code>~=</code>.</p>
<p>The book concludes with three concrete ways in which you can use the knowledge
you've attained: 1) creating C extensions, 2) improving existing Python programs
by leveraging knowledge of the internals and 3) contributing to the CPython
project. While point 2) is potentially a little bit vague, points 1) and 3) are
concrete and well described.</p>
<p>Before writing the book, Anthony wrote an in-depth article on the same topic.
You can find it over on <a href="https://realpython.com/cpython-source-code-guide/">Real
Python</a>. It is something of
an appetizer for the book, but stands strong on its own. Having a brief look at
that article will give you a better understanding of what the book is about
than anything I could write here.</p>
<h1>What I liked</h1>
<p>I went from surface-level understanding of the CPython project to being pretty
confident about where to poke around to do what just from reading this book.
It's comprehensive in scope and the worked example of the <code>~=</code> operator helps <em>a
lot</em> in facilitating an understanding for how to extend CPython with your own
silly things.</p>
<p>I also appreciated the nods to other respectable sources. The <a href="https://devguide.python.org/">Python
Developer's Guide</a> is a great resource for quickly
refreshing how to do something (but going from 0 knowledge about the project
it's a bit to terse). Luciano Ramalho's book <a href="https://www.oreilly.com/library/view/fluent-python-2nd/9781492056348/">Fluent Python 2nd
ed</a> is
also noted as an excellent reference on the Python object model, which I
absolutely agree with.</p>
<p>There is enough context in each chapter that you don't really need much
pre-existing understanding of any of the subjects. If you can read Python code
and have a little bit of experience with reading C code, you're all good to go.
Concepts relevant to the book such as parallelism and memory management are
explained both on an abstract level and in how they are implemented in CPython.
The book is to a great extent a standalone resource and it should be very
approachable even to developers without much experience. There's even an
appendix at the end to explain what little you need to know about the C
programming language to be able to understand the code samples.</p>
<h1>What I didn't like</h1>
<p>There was nothing about the book that I thought was <em>bad</em>, but for me
personally, I would have preferred less explanation of fundamental concepts
(such as threading), and more in-depth details on CPython itself. That being
said, I think Anthony overall has made good calls on the tradeoffs between depth
and approachability. Given that the book is meant to be a starting point for
CPython development as opposed to a complete reference, I think that this
nit-pick of mine is nothing more than personal preference, and perhaps that I'm
slightly outside the target audience of the book. It's clearly meant to contain
everything you'd need to know to go from zero to hero, and it's a lot easier to
skip over content you feel is redundant than it is to find content you didn't
know you needed.</p>
<h1>Conclusions</h1>
<p>CPython Internals saved my neck. I had three weeks to go from a shallow
understanding of the CPython project to being able to explain it to others in a
~30-minute talk, and <a href="https://github.com/slarse/talks/tree/80503786d4587abdadba9913c55c7d133269f3a2/under-the-hood-of-cpython">I made
it</a>.
Without this book I wouldn't have. It was so approachable that I could use it
for "Sunday reading" before bedtime and in spare minutes on public transport.</p>
<p>I highly recommend this book to anyone who's interested in the CPython project.
It's not necessary to have future CPython contributions as a goal to get a lot
out of this book, I found it incredibly interesting in its own right. The fact
that Anthony has managed to pack so much information, with so much context
(recall that I thought there was too much of that) in less than 400 pages is
nothing short of spectacular.</p>The difference between Python and CPython2022-11-19T00:00:00+01:002022-11-19T00:00:00+01:00Simon Larséntag:slar.se,2022-11-19:/the-difference-between-python-and-cpython.html<p>At one point or another, every Python developer or hobbyist encounters the word
<em>CPython</em>. For example, the <a href="https://docs.python.org/3/library/dis.html">dis
module</a> states that it exposes an
"implementation detail of the CPython interpreter". What does that mean?
Some ask themselves how CPython differs from Python, and then
they move on with their lives …</p><p>At one point or another, every Python developer or hobbyist encounters the word
<em>CPython</em>. For example, the <a href="https://docs.python.org/3/library/dis.html">dis
module</a> states that it exposes an
"implementation detail of the CPython interpreter". What does that mean?
Some ask themselves how CPython differs from Python, and then
they move on with their lives without ever getting to the bottom of it. But if
you're one of the curious ones who couldn't put that thought down, this article
is for you.</p>
<h1>Python is a language specification</h1>
<p>Python is a programming language. A programming language is an abstract concept.
It's fundamentally a set of rules saying what you're allowed to write in the
language, the <em>syntax</em> of the language, and another set of rules saying what
should happen if you execute some (syntactically valid) code, the <em>semantics</em> of
the language.</p>
<p>For example, take the <a href="https://docs.python.org/3/reference/simple_stmts.html#the-pass-statement"><code>pass</code>
statement</a>.
The syntax of the <code>pass</code> statement is incredibly simple, it's just the keyword
<code>pass</code>:</p>
<div class="highlight"><pre><span></span><code><span class="nv">pass_stmt</span><span class="w"> </span><span class="o">::=</span><span class="w"> </span><span class="s">"pass"</span>
</code></pre></div>
<p>The semantics are equally simple.</p>
<blockquote>
<p>pass is a null operation — when it is executed, nothing happens. It is useful
as a placeholder when a statement is required syntactically, but no code needs
to be executed, [...]</p>
</blockquote>
<p>For example:</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">funtion_that_does_nothing</span><span class="p">():</span>
<span class="k">pass</span>
</code></pre></div>
<p>The pass statement could be <em>implemented</em> in any number of ways. It could
correspond to a NOOP instruction that when executed does nothing. It could also
correspond to <em>nothing</em>, i.e. no instruction is generated for it. There are
really endless possibilities in producing the semantic behavior described by the
Python language reference.</p>
<h1>CPython is an implementation of the Python language specification</h1>
<p>CPython is an implementation of the Python language specification. In fact, it
is the <a href="https://wiki.python.org/moin/CPython"><em>reference</em> implementation</a>,
meaning that its runtime semantics are the law. It is necessarily the case that
not all runtime semantics are precisely described in the Python documentation,
and then whatever CPython does is considered the desired behavior. Unless, of
course, it is identified as a bug in the runtime.</p>
<p>When it comes to the <code>pass</code> statement, CPython takes the "nothing" approach of
not generating an instruction. This has less overhead than generating and
executing a NOOP instruction. However, as the <em>effect</em> of those two approaches
are the same, it would be equally correct to take the NOOP instruction approach.
So CPython is a reference for what happens when you execute some code, not <em>how</em>
that happens.</p>
<p>There are <a href="https://wiki.python.org/moin/PythonImplementations?action=show&redirect=implementation">other
implementations</a>
of the Python programming language, such as <a href="https://www.pypy.org/">PyPy</a>. It is fundamentally different
to CPython in a variety of ways. For one, PyPy is implemented in Python, whereas
CPython is to a large extent implemented in C. PyPy also employs a JIT
(just-in-time) compiler that can compile hot code to machine code to
substantially speed up running times, often making it <a href="https://speed.pypy.org/">substantially faster than
CPython</a>.</p>
<h1>Closing words</h1>
<p>The difference between Python and CPython is that the former is a language
specification while the latter is an implementation of said language
specification. However, in most everyday conversations among developers, they
are one and the same. There is also the fact that CPython being the reference
implementation makes it part of the specification of the language. Python as a
programming language and CPython as an implementation are to some extent
mutually dependent: you can't really define one without the other.</p>
<p>Distinguishing at all between the two may seem nit-picky, and in the vast
majority of cases I'd argue it is. But when module documentation mentions
implementation details of CPython or you find mentions of other implementations
altogether, the distinction becomes important for it all to make sense.
For me personally, just the urge to understand why two different words were used
for seemingly the same thing was enough of a justification to dig into it, but
I'm also hoping that this article brings more sense into the Python worlds of
others.</p>The pre Python 2.5 ternary operator hack2022-11-06T00:00:00+01:002022-11-06T00:00:00+01:00Simon Larséntag:slar.se,2022-11-06:/python-24-ternary.html<p>The modern day ternary operator is well-known to most Pythonistas:</p>
<div class="highlight"><pre><span></span><code><span class="o"><</span><span class="n">expr_if_true</span><span class="o">></span> <span class="k">if</span> <span class="o"><</span><span class="n">condition</span><span class="o">></span> <span class="k">else</span> <span class="o"><</span><span class="n">expr_if_false</span><span class="o">></span>
</code></pre></div>
<p>It's officially known as a <em>conditional expression</em> and was introduced
back in Python 2.5 with <a href="https://www.regular-expressions.info/atomic.html">PEP
308</a>. Some like it, some
don't, and while discussing it with a colleague of mine he mentioned that …</p><p>The modern day ternary operator is well-known to most Pythonistas:</p>
<div class="highlight"><pre><span></span><code><span class="o"><</span><span class="n">expr_if_true</span><span class="o">></span> <span class="k">if</span> <span class="o"><</span><span class="n">condition</span><span class="o">></span> <span class="k">else</span> <span class="o"><</span><span class="n">expr_if_false</span><span class="o">></span>
</code></pre></div>
<p>It's officially known as a <em>conditional expression</em> and was introduced
back in Python 2.5 with <a href="https://www.regular-expressions.info/atomic.html">PEP
308</a>. Some like it, some
don't, and while discussing it with a colleague of mine he mentioned that there
"used to be something a whole lot worse" around the code bases written by
developers favoring ternary operators. He couldn't remember what it looked like
as these events are <a href="https://peps.python.org/pep-0356/">15 years in the past</a>,
but a few days later he came back to me with a code snippet like this:</p>
<div class="highlight"><pre><span></span><code><span class="p">[</span><span class="s2">"nope"</span><span class="p">,</span> <span class="s2">"yep"</span><span class="p">][</span><span class="kc">False</span><span class="p">]</span>
</code></pre></div>
<p>What?</p>
<h2>No ternary operator you say?</h2>
<p>Developers are creative and opinionated. Sometimes this mix leads to
monstrosities, such as when creative developers who really liked ternaries
created this pattern:</p>
<div class="highlight"><pre><span></span><code><span class="p">[</span><span class="o"><</span><span class="n">expr_if_false</span><span class="o">></span><span class="p">,</span> <span class="o"><</span><span class="n">expr_if_true</span><span class="o">></span><span class="p">][</span><span class="o"><</span><span class="n">condition</span><span class="o">></span><span class="p">]</span>
</code></pre></div>
<p>You're reading that right. It's a list with two elements, the first of which
is returned if the condition is <code>False</code> and the second if it's <code>True</code>. Here is
an example:</p>
<div class="highlight"><pre><span></span><code><span class="n">condition</span> <span class="o">=</span> <span class="kc">True</span>
<span class="n">message_modern</span> <span class="o">=</span> <span class="s2">"success"</span> <span class="k">if</span> <span class="n">condition</span> <span class="k">else</span> <span class="s2">"failure"</span>
<span class="nb">print</span><span class="p">(</span><span class="n">message_modern</span><span class="p">)</span> <span class="c1"># success</span>
<span class="n">message_old</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"failure"</span><span class="p">,</span> <span class="s2">"success"</span><span class="p">][</span><span class="n">condition</span><span class="p">]</span>
<span class="nb">print</span><span class="p">(</span><span class="n">message_old</span><span class="p">)</span> <span class="c1"># success</span>
</code></pre></div>
<p>This will make perfect sense to a C programmer: <code>True</code> and <code>1</code> are
interchangeable, as are <code>False</code> and <code>0</code>. And that's precisely how this works,
the underlying <a href="https://github.com/python/cpython/blob/2db55e0c0069a928775fa819973a76f840c5ab5a/Objects/listobject.c#L242-L255">function that implements the list index
access</a>
performs some rudimentary checks and then just uses the provided index as an
offset to the base pointer of the list.</p>
<h2>But the hack isn't quite the same</h2>
<p>At first glance, it may seem like the old hack with the list is functionally
equivalent to the modern ternary operator. It isn't quite, though, because it
lacks one very important property: lazy evaluation of the branches. In short,
the ternary operator only evaluates the branch that it returns. Here's an
example:</p>
<div class="highlight"><pre><span></span><code><span class="n">condition</span> <span class="o">=</span> <span class="kc">False</span>
<span class="n">result_modern</span> <span class="o">=</span> <span class="mi">1</span> <span class="o">/</span> <span class="mi">0</span> <span class="k">if</span> <span class="n">condition</span> <span class="k">else</span> <span class="mi">42</span>
<span class="nb">print</span><span class="p">(</span><span class="n">result_modern</span><span class="p">)</span> <span class="c1"># 42</span>
<span class="n">result_old</span> <span class="o">=</span> <span class="p">[</span><span class="mi">42</span><span class="p">,</span> <span class="mi">1</span> <span class="o">/</span> <span class="mi">0</span><span class="p">][</span><span class="n">condition</span><span class="p">]</span> <span class="c1"># raises ZeroDivisionError: division by zero</span>
<span class="nb">print</span><span class="p">(</span><span class="n">result_old</span><span class="p">)</span>
</code></pre></div>
<p>The ternary operator does not result in a crash on division by zero as it does
not evaluate the expression in the true-branch, whereas the old hack crashes
immediately as it first evaluates both expressions and then returns one of them.
To fully emulate the modern behavior we'd need something like this:</p>
<div class="highlight"><pre><span></span><code><span class="n">condition</span> <span class="o">=</span> <span class="kc">False</span>
<span class="n">result_old</span> <span class="o">=</span> <span class="p">[</span><span class="k">lambda</span><span class="p">:</span> <span class="mi">42</span><span class="p">,</span> <span class="k">lambda</span><span class="p">:</span> <span class="mi">1</span> <span class="o">/</span> <span class="mi">0</span><span class="p">][</span><span class="n">condition</span><span class="p">]()</span>
<span class="nb">print</span><span class="p">(</span><span class="n">result_old</span><span class="p">)</span> <span class="c1"># 42</span>
</code></pre></div>
<p>Here we get lazy evaluation by virtue of wrapping the two branches in lambdas,
and then executing the lambda that's returned. I think we can all agree that's
not a pretty sight.</p>
<h2>Conditional expressions are a good thing</h2>
<p>Regardless of your stance on using ternary operators (or conditional
expressions, as they're called in Python), it's probably a good thing they
exist. Otherwise creative and opinionated programmers get around to hacks to
emulate the behavior that end up being completely unreadable to others.</p>Why I write a blog that nobody reads (and you should, too)2022-10-29T00:00:00+02:002022-10-29T00:00:00+02:00Simon Larséntag:slar.se,2022-10-29:/why-i-write-a-blog-that-nobody-reads-and-you-should-too.html<p>I don't write this blog in hopes that it will get a lot of traction. I don't
market it, I don't try to optimize it for SEO and I I don't even tell my
friends and coworkers about it. Although I would of course be delighted to hear
someone who …</p><p>I don't write this blog in hopes that it will get a lot of traction. I don't
market it, I don't try to optimize it for SEO and I I don't even tell my
friends and coworkers about it. Although I would of course be delighted to hear
someone who found something useful in my blog, I really write it for myself.
And what could I possibly get out of that? Glad you asked.</p>
<h2>Expressing thoughts and ideas is hard</h2>
<p>I'm a software engineer. My job is mostly about coming up with neat solutions to
tricky problems. Coming up with the solution in my head is however just part of
the fight as I also need to be able to concisely express the ideas to my
colleagues. Even harder is to convey the heart of a technical problem or
solution to a non-technical person, which happens more often than you might
think. For example, a manager might come around and inquire about why something
still isn't working as it should or why project X got so expensive.</p>
<p>Writing a technical blog about helps me practice this skill. I put quite a bit
of care into most blog posts I write and I also often look back at earlier posts
to consider how I can improve the writing.</p>
<h2>Retaining knowledge is also hard</h2>
<p>Another reason for why I write this blog is to retain knowledge. Writing about
something you've just learned is a great way to further cement that knowledge.
As I put a large amount of care into my posts and want to be sure that I get
things right, I also often pick up bits and pieces that I missed when attaining
the knowledge I'm writing about.</p>
<p>As I noted in <a href="https://slar.se/introducing-my-wiki.html">the post about my new
Wiki</a>, my strive toward high quality
posts does unfortunately mean that I don't publish blog posts all that often.
That's where the <a href="https://slar.se/pages/wiki.html">Wiki</a> comes in. In there I
simply write things down without any particular care about quality. I can
thereby help myself retain knowledge about things that I don't yet understand
well enough to write a blog post about, or that I simply don't have the time
to write about.</p>
<h2>Putting yourself out there is perhaps even harder</h2>
<p>Exposing your work to the public is not something most people are comfortable
with. It's easy to feel self-conscious about the things you create and be afraid
of potential backlash or negative feedback. Putting up blog posts strengthens my
confidence when it comes to communicating and expressing myself, even if I don't
typically get any direct feedback (be that positive or negative) due to not
having comments on this site. Similarly, putting up <a href="https://github.com/slarse">my projects on
GitHub</a> strengthens my confidence in my technical
skills.</p>
<p>Having published my work for many years I now feel little to no anxiety about
publishing my work. Rather, I find it enjoyable to put something out in the open
where others may find it, even if few ever do.</p>Introducing my Wiki!2022-10-24T00:00:00+02:002022-10-24T00:00:00+02:00Simon Larséntag:slar.se,2022-10-24:/introducing-my-wiki.html<p>A while ago, I decided that I wanted to have a low-effort way of recording what
I learn out in the open. When I publish a blog post I feel compelled to keep
the standard of the writing on a fairly high level. I've started dozens of
articles that I …</p><p>A while ago, I decided that I wanted to have a low-effort way of recording what
I learn out in the open. When I publish a blog post I feel compelled to keep
the standard of the writing on a fairly high level. I've started dozens of
articles that I never published because I couldn't massage them into a state
that meeting my own standards. And that's not even counting the hundreds
of articles that I never even started because it seemed like too much of an
effort to record a tiny learning experience.</p>
<p>This strongly goes against the <em>Record What You Learn</em> an <em>Share What You
Learn</em> patterns from <a href="https://slar.se/book-review-apprenticeship-patterns.html">Apprenticeship
Patterns</a>; two patterns
that I value highly. Enter the Wiki!</p>
<h1>My public Wiki</h1>
<p>The <a href="https://slar.se/pages/wiki.html">Wiki</a> can be reached from the menu bar at the
top of the page. Here, I jot down quick notes about what I learn, what I have
yet to learn and anything else that I feel is worth recording. The quality of
the writing will range from poor to worse, and it'll be haphazardly organized,
at least initially. I hope to turn this into a wealth of knowledge both for
myself and for others in the coming years.</p>
<h2>Why not use an established Wiki software?</h2>
<p>Although there are several open source Wikis out there, such as
<a href="https://xwiki.org">XWiki</a>, they're usually quite heavy and require a backend.
This site is a <a href="https://en.wikipedia.org/wiki/Static_web_page">static site</a>,
which is very cheap to host and requires no particular security considerations
due to the lack of a backend. I quite like those properties, and I'm currently
adapting the templates for this site to better suit my Wiki ambitions. This
first rendition is <em>rough</em>, but it suits my current needs, and improving upon
it as my Wiki grows wll be an interesting challenge.</p>
<h2>Searching the Wiki</h2>
<p>The <a href="https://slar.se/pages/wiki.html">Wiki</a> is indexed along with the rest of the site
and can be searched using the <a href="https://slar.se/pelican-search.html">search bar</a> on
the <a href="https://slar.se/index.html">home page</a>. In other words, you can search for anything on
this website through that single search bar!</p>
<h2>Going forward</h2>
<p>I intend to keep improving the wiki. As the amount of content grows, I expect to
have to improve the layout, cross-document linking and other related features.
I'm very excited to finally get around to this as I've been thinking about it
for a long time. Even though I do hope others will find it useful, it's mostly
for my own gain, and I'm confident it will serve me well.</p>Book Review: The Rust Programming Language2022-08-10T00:00:00+02:002022-08-10T00:00:00+02:00Simon Larséntag:slar.se,2022-08-10:/book-review-the-rust-programming-language.html<p>I've been learning Rust on and off for the past few months, and <em>The Rust
Programming Language</em> has been my <a href="https://slar.se/learning-a-new-programming-language.html">primary learning
resource</a> during this time.
It's a great introduction to the language, and is even <a href="https://doc.rust-lang.org/stable/book/">freely available
online</a>.</p>
<p>Here, I'm reviewing the 2018 print version. Although a little bit …</p><p>I've been learning Rust on and off for the past few months, and <em>The Rust
Programming Language</em> has been my <a href="https://slar.se/learning-a-new-programming-language.html">primary learning
resource</a> during this time.
It's a great introduction to the language, and is even <a href="https://doc.rust-lang.org/stable/book/">freely available
online</a>.</p>
<p>Here, I'm reviewing the 2018 print version. Although a little bit out of date by
now, there's nothing that's become obsolete, so I can still recommend even this
version. For the most up to date version, the online one is however the way to
go.</p>
<div class="highlight"><pre><span></span><code>The Rust Programming Language
by Steve Klabnik, Carol Nichols
Released June 2018
Publisher(s): No Starch Press
ISBN: 9781593278281
</code></pre></div>
<h1>The book in a nutshell</h1>
<p><em>The Rust Programming Language</em> is really a book where the title perfectly
captures what the book is about. It teaches Rust mostly by practical examples,
and for the most part the examples are self-contained and executable. For some
concepts, examples are either missing or more illustrative than executable, but
these are few and far between.</p>
<p>The book also to a large extent explains programming concepts, and has a rather
elaborate section on concurrency. It however isn't on a beginner level, and
doesn't go out of its way to intuitively describe what a variable is, or how
looping works. I would rate this a great book for someone who is already
somewhat familiar with programming concepts, but it's not the best resource for
getting started with programming as a whole. It's the perfect "second language
book".</p>
<h1>What I liked</h1>
<p>This book is incredibly well written and organized. Concepts to be learned are
first presented on a high level, and then the authors drill into the details.
But not too far into the details; at several points we are referred to other
resources to acquire a deeper understanding for certain concepts. It's also a
great standalone resource for learning Rust as it covers surrounding tooling
like <code>rustup</code> and <code>cargo</code> in addition to the language itself. You can learn Rust
to a decent level of proficiency from this book alone.</p>
<p>The well thought out pacing of the book carries over to the code samples, which
are excellent through-and-through. There are some rather tricky concepts to get
your head around in Rust compared to other programming languages, and the code
samples are crucial in getting the point across. These are also presented in a
top down fashion. By that, I mean that the higher level code is presented first
containing calls to yet to be defined functions that are presented later. </p>
<p>As a crude example, imagine that we want to present a program that adds two
numbers and prints the result. That may look like so:</p>
<div class="highlight"><pre><span></span><code><span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">sum</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">add</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="p">);</span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"{}"</span><span class="p">,</span><span class="w"> </span><span class="n">sum</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div>
<p>And then we define the <code>add</code> function after having presented the high-level idea
we want to implement:</p>
<div class="highlight"><pre><span></span><code><span class="k">fn</span> <span class="nf">add</span><span class="p">(</span><span class="n">lhs</span>: <span class="kt">i32</span><span class="p">,</span><span class="w"> </span><span class="n">rhs</span>: <span class="kt">i32</span><span class="p">)</span><span class="w"> </span>-> <span class="kt">i32</span> <span class="p">{</span>
<span class="w"> </span><span class="n">lhs</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">rhs</span>
<span class="p">}</span>
</code></pre></div>
<p>This top down approach to presenting code samples really helps in getting a good
idea for <em>what</em> needs to be done before <em>how</em> it is actually implemented. I
strongly prefer this approach to a bottom up one, where you start with the low
level <em>how</em> before getting to the high level <em>what</em>.</p>
<p>The book ends with a project on building a multithreaded web server, which is
meant to solidify many of the concepts taught throughout the book. It's a great
way to close out a great book.</p>
<h1>What I didn't like</h1>
<p>This is my first book review where I can't come up with something that I overtly
did not like about a book. There are things the book lacks, such as more
beginner-friendly introductions to core programming concepts, but I feel that's
by design rather than thoughtless omission. The book would simply be way too
long if it had to include such things as well.</p>
<p>Perhaps I will find something to be annoyed with as I revisit this in the
future, but as it stands I am completely pleased with my reading experience.</p>
<h1>Conclusions</h1>
<p>This is a great book to learn the Rust programming language. It's not
appropriate for absolute beginners, but I think that may be simply a consequence
of Rust being designed to tackle rather advanced problems. I would not
recommend a budding programmer to start out with Rust, and so it seems
completely natural to me that the official learning resource doesn't cater toward
such a crowd. That being said, you don't need to be a seasoned programmer
to get value out of this book, as concepts are explained with quite a lot of
"backstory". You don't need to be overly familiar with the problems Rust
attempts to solve (memory safety, for example) as the book clearly exemplifies
the problems before outlining the solutions.</p>
<p>As <em>The Rust Programming Language</em> is <a href="https://doc.rust-lang.org/stable/book/">freely available
online</a>, I whole-heartedly recommend it
for those looking to dive into Rust. I see no good reason to go looking
elsewhere for resources when there's such a great one staring you right in the
face. This is <em>the</em> starting place for a prospective Rustacean!</p>Book Review: 97 Things Every Programmer Should Know2022-07-11T21:00:48+02:002022-07-11T21:00:48+02:00Simon Larséntag:slar.se,2022-07-11:/book-review-97-things.html<p><em>97 Things Every Programmer Should Know</em> is a collection of short essays by
experienced programmers. And by short, I mean <em>short</em>: 1-3 fairly tiny pages a
piece. If you're on a journey to become a software engineer then this book will
give you a crash course in terminology you should …</p><p><em>97 Things Every Programmer Should Know</em> is a collection of short essays by
experienced programmers. And by short, I mean <em>short</em>: 1-3 fairly tiny pages a
piece. If you're on a journey to become a software engineer then this book will
give you a crash course in terminology you should be familiar with. Even as a
practicing software engineer there is wisdom to be found in this book, but a
novice will undoubtedly get more out of reading it.</p>
<div class="highlight"><pre><span></span><code><span class="mf">97</span><span class="w"> </span><span class="n">Things</span><span class="w"> </span><span class="n">Every</span><span class="w"> </span><span class="n">Programmer</span><span class="w"> </span><span class="n">Should</span><span class="w"> </span><span class="n">Know</span>
<span class="n">by</span><span class="w"> </span><span class="n">Kevlin</span><span class="w"> </span><span class="n">Henney</span>
<span class="n">Released</span><span class="w"> </span><span class="n">February</span><span class="w"> </span><span class="mf">2010</span>
<span class="n">Publisher</span><span class="p">(</span><span class="n">s</span><span class="p">):</span><span class="w"> </span><span class="n">O</span><span class="err">'</span><span class="n">Reilly</span><span class="w"> </span><span class="n">Media</span><span class="p">,</span><span class="w"> </span><span class="n">Inc</span><span class="mf">.</span>
<span class="n">ISBN</span><span class="p">:</span><span class="w"> </span><span class="mf">9780596809485</span>
</code></pre></div>
<h1>The book in a nutshell</h1>
<p>As I mentioned in the introduction, <em>97 Things Every Programmer Should Know</em> is
a collection of very short essays by practicing professionals. The essays treat
a wide variety of themes that are relevant to a practicing software engineer,
ranging from hard technical skills such as the <em>Don't Repeat Yourself</em> principle
and <em>Single Responsibility Principle</em>, to softer skills such as interacting with
managers and fostering good relationships with your colleagues. Most of the
essays are focused on the technical side of things, however, and there is good
variety in the technical topics. While a lot of the essays are about how to
write good code, there is also a healthy amount of recommendations for tooling
to use, such as static code analysers, automated test suites, version control
systems and more.</p>
<p>There really isn't much more to say regarding what the book is about. It's like
a collection of very well-written blog posts on programming-related topics. So
if you're reading this blog, chances are good you're going to enjoy this book.</p>
<h1>What I liked</h1>
<p>In some ways, <em>97 Things</em> has a lot in common with <a href="https://slar.se/book-review-apprenticeship-patterns.html">Apprenticeship
Patterns</a> that I reviewed last week.
They're both about how you improve as a software engineer. The difference is
that <em>97 Things</em> is a lot more to the point and more concrete. For the most
part, it presents tips and tricks that you can apply immediately and see
benefits from just as fast. For the budding software engineer, it's an excellent
pool of topics to diversify your skill set. That's with an emphasis on <em>topics</em>,
though. This book presents a brief introduction to a wide variety of topics, but
it dives deeply into none of them. I think this is a great strength of the book,
as it means you will never get stuck on some "boring" topic that doesn't
interest you.</p>
<p>While you can absolutely draw connections between the essays, they are written
as standalone pieces of work. This makes <em>97 Things</em> a perfect book to read on
the go, when you might just have a few minutes or so to read.</p>
<p>Out of all 97 things, I recall only a handful that I didn't find genuinely
helpful or insightful. Some essays even contradict each other, which actually
gives a nice perspective on the fact that a <em>lot</em> of best practices are, to a
large degree, opinions. A notable example I recall is about automatic code
formatting, where one essay discourages its use and a few others encourage it.
I think this is a great benefit of having so many different authors. You don't
just get one person's opinion.</p>
<h1>What I didn't like</h1>
<p>As with many other books on how to be a good software engineer, there is a slight
tint of workaholism over some of the essays. I especially found an essay by
Robert C. Martin on what it means to be a professional programmer to send this
message. It's the same theme I found a little bit disturbing with
<a href="https://slar.se/book-review-apprenticeship-patterns.html">Apprenticeship Patterns</a>. I can't say I
disagree with the message; I truly believe in the craftsmanship approach to
software engineering, the path of lifelong learning. But at the same time I
don't think it is for everyone, and I think it should be possible to treat
programming as "just a job".</p>
<p>A minor inconvenience is that the essays are ordered alphabetically, where I
would have preferred them to be ordered by theme. Finding a particular essay of
which you recall the theme but not the title is needlessly difficult. I just now
suffered through it trying to find the aforementioned essay by Robert C. Martin.</p>
<p>But as a package, I find little to dislike about the book. Even the workaholism
part is effectively counteracted by an entire essay dedicated to sending the
message "work smarter not harder".</p>
<h1>Conclusions</h1>
<p>This is another book that I wish I'd read years ago. I strongly recommend it as
a read for <em>any</em> software engineer at the beginning of their career or student
(self-taught or at a seat of learning) who is preparing for their career. Being
such a light and quick read, I find no good reason not to spend the few hours it
takes to read through the book and be exposed to a whole lot of different ideas.</p>Full text search with pelican-search!2022-07-10T21:58:53+02:002022-07-10T21:58:53+02:00Simon Larséntag:slar.se,2022-07-10:/pelican-search.html<p>I've been meaning to take this blog into more of the Wiki direction. Or rather
some unholy mixture of a Wiki and a blog. An important step in that direction is
full text search, which I've now got!</p>
<p>This blog is powered by <a href="https://blog.getpelican.com/">Pelican</a>, and quite
recently a new plugin …</p><p>I've been meaning to take this blog into more of the Wiki direction. Or rather
some unholy mixture of a Wiki and a blog. An important step in that direction is
full text search, which I've now got!</p>
<p>This blog is powered by <a href="https://blog.getpelican.com/">Pelican</a>, and quite
recently a new plugin called
<a href="https://github.com/pelican-plugins/search">pelican-search</a> popped up.
It's quite simple: it adds the search bar you see on the <a href="https://slar.se/index.html">home
page</a>, which taps into the powerful <a href="https://stork-search.net/">Stork</a>
search library.</p>
<p>It all sums up to the super fast search bar that all visitors of this website
now have at their disposal. Pretty neat!</p>The art of learning from the less experienced2022-07-07T22:31:50+02:002022-08-07T23:59:50+02:00Simon Larséntag:slar.se,2022-07-07:/learning-from-the-less-experienced.html<p>Software engineering is a lifelong journey of learning. Regardless of how
dedicated you are in your learning, there will never come a point where you
have learned it all. As such, it's important to use all learning resources
available to us. As is evident from the themes on my blog …</p><p>Software engineering is a lifelong journey of learning. Regardless of how
dedicated you are in your learning, there will never come a point where you
have learned it all. As such, it's important to use all learning resources
available to us. As is evident from the themes on my blog, I'm very partial to
books and other written media. Indeed, this blog post is such written media.
But perhaps the best source of learning for a software engineer is simply other
software engineers.</p>
<p>Now, learning new things from those more experienced than you isn't that much a
leap of the imagination. Of course you're going to try to soak up anything you
can learn from your seasoned team lead, or that database expert who does magical
things with SQL queries, or anyone else you identify as being highly proficient
in something that interests you. No, there is no real challenge there, barring
the fact that these experienced engineers may not have the inclination to teach
you. But this blog post is about your disposition as a learner, so let's stick
to the topic. And actually get to the topic to begin with: learning from the
less experienced.</p>
<h1>A student can teach their teacher</h1>
<p>When I attended university, I worked many years as a teaching assistant in
introductory computer science classes. During one class I held, there was at
some point a part of an assignment that required flipping the value of a
boolean variable. Something like: given a boolean variable <code>isOdd</code>, define a
new boolean variable <code>isEven</code> that is <code>true</code> if <code>isOdd</code> is <code>false</code>, and <code>false</code>
if <code>isOdd</code> is <code>true</code>. One student presented a solution like the one below.</p>
<div class="highlight"><pre><span></span><code><span class="kt">boolean</span><span class="w"> </span><span class="n">isEven</span><span class="p">;</span>
<span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">isOdd</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">true</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">isEven</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">false</span><span class="p">;</span>
<span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">isEven</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">true</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>Being an enthusiastic but still rather fresh teaching assistant with not all
that much programming experience, I said it was a viable solution but it would
be more concise to use a ternary operator.</p>
<div class="highlight"><pre><span></span><code><span class="kt">boolean</span><span class="w"> </span><span class="n">isEven</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">isOdd</span><span class="w"> </span><span class="o">?</span><span class="w"> </span><span class="kc">false</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">;</span>
</code></pre></div>
<p>The students sat back in awe at my incredibly simple solution to the problem.
That is, until 5 seconds passed and another student had a bright idea: "why not
just negate <code>isOdd</code>?". What the student meant was the following:</p>
<div class="highlight"><pre><span></span><code><span class="kt">boolean</span><span class="w"> </span><span class="n">isEven</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">!</span><span class="n">isOdd</span><span class="p">;</span>
</code></pre></div>
<p>Not only is this solution the most concise, it also more clearly represents the
concept the task asked for. Something is "even" precisely if it is "not odd",
after all. I managed to humble myself enough to commend the student for a well
thought out solution.</p>
<h1>First year students routinely taught me new things</h1>
<p>I taught the first year computer science courses for four years. I expanded my
skills exponentially during this time. And yet, every year there would be new
first year students that knew something I did not, or had some insight I
lacked. While these events definitely became less frequent as I gained more
experience, they never ceased. I'm confident that I could go back there now and
teach the same courses again, and there would be a student or two with
something to teach me.</p>
<p>The great insight that I gained from this is that regardless of how far ahead
you are of someone else, you are doing both yourself and them a disservice by
not being open to let them teach you things. It's also incredibly hard to
determine if someone is less experienced than you are. In the <code>!isOdd</code> scenario
outlined above, I was in a position of authority relative to the student who
had the best solution, but it's not unlikely that student had done a lot more
programming than I had, given that I didn't start until I was in university.</p>
<h1>The great challenge in keeping an open mind in disagreement</h1>
<p>The reason I could so easily swallow my pride and commend the student with the
<code>!isOdd</code> solution is not that I at that time was particularly humble. I simply
agreed with the solution the student had in mind, it fit my mental model. I've
since been in situations where someone I've viewed as less experienced (and more
importantly, <em>less proficient</em>) than myself has come with a suggestion that I've
fundamentally disagreed with. In such scenarios keeping an open mind is a
lot more difficult, and all I can do is try to the best of my abilities. I will
argue my point, and I can argue fiercely, but I also try my absolute hardest not
to dismiss their point outright, and hear out their arguments. I also try not to
let my predetermined view of their experience and proficiency taint my
judgement. Sometimes I succeed on the spot, and sometimes I succeed in hindsight
when reflecting over a past conversation. And most assuredly, sometimes I
simply fail.</p>
<p>My point with all of this storytelling really boils down to one piece of advise:
avoid leaning on your impressions of someone's experience and proficiency when
evaluating their arguments for some point. I guarantee they know things you don't.
Like me, you are unlikely to always succeed, but you'll benefit from the times
you do. Not to mention that the other party of the argument will most often
appreciate you letting them make their case. Perhaps that's actually the more
important part of thes story. But it sure does not hurt that there's something
in it for you as well.</p>Book Review: Apprenticeship Patterns2022-07-02T13:00:48+02:002022-07-02T13:27:48+02:00Simon Larséntag:slar.se,2022-07-02:/book-review-apprenticeship-patterns.html<p><em>Apprenticeship Patterns: Guidance for the Aspiring Software Craftsman</em> is
fundamentally a book about lifelong learning. It is about treating software
engineering as a craft you may never master; in fact it may never have been
mastered before. Perhaps it simply cannot be mastered in the traditional sense
of the word …</p><p><em>Apprenticeship Patterns: Guidance for the Aspiring Software Craftsman</em> is
fundamentally a book about lifelong learning. It is about treating software
engineering as a craft you may never master; in fact it may never have been
mastered before. Perhaps it simply cannot be mastered in the traditional sense
of the word. But that doesn't mean that we should not aspire to master it, that
we should not embark on <em>The Long Road</em>. So how can a book help with that?</p>
<div class="highlight"><pre><span></span><code><span class="nv">Apprenticeship</span><span class="w"> </span><span class="nv">Patterns</span>:<span class="w"> </span><span class="nv">Guidance</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">Aspiring</span><span class="w"> </span><span class="nv">Software</span><span class="w"> </span><span class="nv">Craftsman</span>
<span class="nv">by</span><span class="w"> </span><span class="nv">Dave</span><span class="w"> </span><span class="nv">Hoover</span>,<span class="w"> </span><span class="nv">Adewale</span><span class="w"> </span><span class="nv">Oshineye</span>
<span class="nv">Released</span><span class="w"> </span><span class="nv">October</span><span class="w"> </span><span class="mi">2009</span>
<span class="nv">Publisher</span><span class="ss">(</span><span class="nv">s</span><span class="ss">)</span>:<span class="w"> </span><span class="nv">O</span><span class="err">'Reilly Media, Inc.</span>
<span class="err">ISBN: 9780596518387</span>
</code></pre></div>
<h1>The book in a nutshell</h1>
<p>The core tenet of the book is that software engineering is a <em>craft</em>. A craft in
the same sense that blacksmithing is, or carpentry, or the construction of
musical instruments. The authors provide a concise reasoning for why this is so.</p>
<blockquote>
<p>Software engineering is a craft precisely because we don't understand it well
enough to make it a codified discipline like science or engineering.</p>
</blockquote>
<p>One can agree or disagree with this statement, but it is hard to argue against
the notion that software engineering is something that requires a high degree of
<em>skill</em>. That's not skill as in technical proficiency, although that is part of
it. It's skill as in everything that encompasses a successful software engineer,
of which technical proficiency is important but far from being the skill in it's
own right.</p>
<p>As the title suggests, <em>Apprenticeship Patterns</em> is comprised of a series of
<em>patterns</em>. These are meant to help you hone your skill as a software craftsman.
Each pattern is composed of four parts:</p>
<ul>
<li>Context: A generalized context to put the pattern in perspective.</li>
<li>Problem: A concrete problem statement.</li>
<li>Solution: One or more suggestions for solving or alleviating the problem.</li>
<li>Action: A concrete exercise to practice the solution.</li>
</ul>
<p>The patterns are really <em>contextual habits</em>; given a situation X it is
appropriate to do Y. A (perhaps <em>the</em>) core pattern is <em>The Long Road</em>, which
boils down to the fact that mastering a craft is a lifelong process. It
challenges the notion that the quickest way to success as measured in notoriety
or material wealth is what one should aspire for. The solution is a lengthy
affair, but it importantly suggests that climbing the corporate ladder through
quick promotions or similar easily takes you away from the actual crafting of
software, thus diverting you from The Long Road. You should be prepared to work
as a developer for many years to come. Many of the other patterns of the book
complement The Long Road, such as <em>Stay In The Trenches</em> which specifically
deals with the problem of success being rewarded with promotions.</p>
<p>Another pattern that struck home with me is <em>Record What You Learn</em>. Quite
unsurprisingly, it simply suggests that one should keep a record of the things
you have learned and plan to learn in the future. This book review is in fact a
direct application of that pattern.</p>
<p>There are many, many more patterns in the book. Some of my favorite ones include:</p>
<ul>
<li><em>Be The Worst</em>: Place yourself in situations where you are surrounded by
craftsmen that are more skilled than you in some area, and that you can learn
from. Avoid becoming complacent.</li>
<li><em>Share What You Learn</em>: Put your learning experiences out in the open, for
anyone to find.</li>
<li><em>Kindred Spirits</em>: Surround yourself with others passionate to learn the
things you wish to dive into.</li>
<li><em>Use The Source</em>: Read open source code. A lot.</li>
<li><em>Read Constantly</em>: Read books. A lot.</li>
</ul>
<p>I'm not going to list all the patterns that resonated with me, and I can't even
do these few ones justice with just this brief explanation of them. You really
need to read the book with each context, problem, solution and action for the
respective patterns to get the full picture.</p>
<h1>What I liked</h1>
<p>This book is a very easy read. I like to have at least one <em>soft</em> book to read
when my mind wanders too much for me to take in a technical book, and
<em>Apprenticeship Patterns</em> perfectly fits that bill. It took me a couple of weeks
with a few pages a night to get through it. It's easy to pick it up, read about
a pattern or two, and then put it down. It's well organized and easy to refer
back to after completion. The patterns are presented with just enough context to
make them understandable, yet the authors do not dwell on things for too long.
It is a concise book that is still easy to comprehend.</p>
<p>I also found that it is a highly inspirational book, and approaching software
engineering as a craft really speaks to me. Many of the patterns of this book
are quite obvious to me and I practice several of them already, yet there are
many patterns I think I should practice that I don't. Record What You Learn is
the perfect example. I've been thinking for years that I should do so, but never
really got around to doing it other than a sporadic blog post once or twice a
year. After reading the book, I have newfound motivation to apply many of
the patterns, as this book review is tangible evidence of. In the future, I
intend to always finish up a book with a book review. Hopefully I will improve
in writing them with time, as I am not all that happy with how this review
turned out. But Share What You Learn tells me I should post this anyway, and I
really do agree with that.</p>
<p>I also think this is a terrific book to read as a budding software engineer. It
is called <em>Apprenticeship Patterns</em>, after all. I wish I would have read it
years ago, and I wish I was already practicing many of the patterns. But better
late than never, and as I will discuss in the next section there is a reason for
me to be somewhat thankful for not reading it earlier.</p>
<h1>What I didn't like</h1>
<p>While I think this is a good handbook for mastering a craft, I also think the
approach is potentially unhealthy if applied without moderation. I have
personally struggled a lot with finding a balance between improving my skills as
a software and just living a life separate from software engineering. Or
computers and technology in general. I found the book truly inspirational, so
much so that it prompted be to spend two hours of my Saturday morning writing
this book review. To be completely honest, I am still tweaking the
aforementioned balance. I would not say that I am struggling anymore, but I
cannot deny that it is still a work in progress.</p>
<h1>Conclusions</h1>
<p>This book struck home with me, and I wholeheartedly recommend it. But that is a
recommendation with a caveat, as I don't think this is a book for everyone.
It's really all in the title, this book is for the aspiring software
<em>craftsman</em>. I don't think you <em>have</em> to approach software engineering as a
craft that you devote yourself to, and the authors actually allude to this as
well. It's one approach, and it's a great boon to the field that some take it.
But for others, working with software can be "just a job", and not their
passion. That is fine, a job can be just a means to provide for yourself. And
if that's you, then I say skip this book. But if you do have a passion for
building great software, I think this is a book you don't want to miss.</p>Eleven Table Tennis: A VR masterpiece2022-06-12T00:00:00+02:002022-06-12T00:00:00+02:00Simon Larséntag:slar.se,2022-06-12:/eleven-table-tennis-a-vr-masterpiece.html<p>As a software engineer, I sit and stand still for large chunks of the day.
Unfortunately, I'm also a big fan of playing video games, which traditionally
falls into the same category of physical exercise. Much to avoid repetitive
stress injury and everything else that comes along with sitting at …</p><p>As a software engineer, I sit and stand still for large chunks of the day.
Unfortunately, I'm also a big fan of playing video games, which traditionally
falls into the same category of physical exercise. Much to avoid repetitive
stress injury and everything else that comes along with sitting at a computer
day in and day out, I've in the past year started to devote more time to
virtual reality (VR) gaming. In some VR games you actually get to move around,
and I'm writing this article to fawn over my favorite experience yet: <a href="https://elevenvr.com/en/">Eleven
Table Tennis</a>.</p>
<h3>Table tennis is an excellent fit for VR</h3>
<p>There are two things about table tennis that makes it very well suited to
VR. First, the playing field is small enough that it can fit into a
fair-sized living room. Granted, fitting a full-sized table tennis table into
a smaller apartment can be somewhat inconvenient, but when playing in VR you
actually only need the space for <em>half</em> of the table. That is not to say that
I don't occasionally physically assault some furniture that was unfortunate
enough to come in my way, but most of the time it works well even though my
playing space is only roughly 2.5x2.5 meters.</p>
<p>Second, the real-life feedback of hitting a table tennis
ball with a racket is rather realistically reproduced by a vibration in a
motion controller. There's just not a lot of feedback from those tiny and
hollow table tennis balls, so even though most motion controllers can't produce
particularly strong vibrations, it's still enough. I've played both with
a Valve Index and an Oculus Rift CV1, and both provide a perfectly serviceable
experience.</p>
<h3>Eleven Table Tennis delivers on that premise</h3>
<p>So the actual game of table tennis is really well suited to a VR treatment.
Eleven Table Tennis takes this premise and delivers an almost realistic physics
engine that for the most part just <em>feels right</em>. Granted, there are the odd
"wait, what?..." moments from time to time, and I especially feel like some
serves can be delivered in a way that's not really possible in real life. But
overall, it feels true to life.</p>
<p>While playing against bots in Eleven Table Tennis is a perfectly fine challenge,
what really sets the game apart is a stellar online multiplayer. I jump into the
game both in early mornings before work and in late evenings before bed, and
it rarely takes me more than 30 seconds to find an opponent. While waiting,
you can keep playing against the bots as well, so there's never an idle moment.</p>
<p>As icing on the cake, playing table tennis in VR is a really decent workout. I'm
over the moon about how I can combine my love for video games with physical
exercise. There are games that do that even better, such as <a href="https://www.beatsaber.com/">Beat
Saber</a>, but I find the competitive nature of table
tennis to be much more engaging and worthwhile.</p>
<p>If you're a VR fan and don't have this game yet, do yourself a favor and get
it. Just ensure that you don't have any expensive furniture in your immediate
play area.</p>Learning a new programming language as a practicing software engineer2022-06-12T00:00:00+02:002022-06-12T00:00:00+02:00Simon Larséntag:slar.se,2022-06-12:/learning-a-new-programming-language.html<p>When it comes to programming languages, I consider myself something of a
polyglot. To me, learning a new language is one of the most enjoyable
things in all of software engineering. This is especially true when you
start to venture into new paradigms; going from procedural to functional,
functional to …</p><p>When it comes to programming languages, I consider myself something of a
polyglot. To me, learning a new language is one of the most enjoyable
things in all of software engineering. This is especially true when you
start to venture into new paradigms; going from procedural to functional,
functional to logical and logical to constraint-based. Exploring programming
languages is one of the best way to improve your craft, even if you don't
end up using that language in your daily work.</p>
<p>Having learned (to varying degrees of proficiency) a good few languages over
the years, I figured it's about time I share my process for learning a new one.
And when I say learn here, I mean <em>really</em> learn the language and become
proficient in it, as opposed to learning just enough to become dangerous.
There is a time and place for the latter as well, but that is not the focus
of this article. This article is also not about learning your <em>first</em>
programming language, as that's a process where you need to also learn
programming as a skill.</p>
<p>I'm currently in the process of learning the Rust programming language, and in
this article I'll take you through how I go about learning it. My process, which
I will detail throughout this article, is as follows:</p>
<ol>
<li>Identify learning resources</li>
<li>Start learning</li>
<li>Build a project</li>
<li>Maintain knowledge</li>
</ol>
<h1>Step 1: Identify learning resources</h1>
<p>The first thing I do is to identify the learning resources that I intend to use.
This does not have to be an exhaustive list of resources, as sometimes I stumble
upon more of them as I go along, but I always put in some amount of research
into which learning resources are the most well received by others. I divide
these into <em>primary</em> and <em>secondary</em> learning resources, where the primary ones
guide me through the learning experience while the secondary ones are
supplementary and can be skipped altogether if time is tight.</p>
<h2>Primary learning resources</h2>
<p>My primary learning resources are one or more structured overviews of the
language I'm about to learn. I usually only aim for one or two primary resources,
and in the vast majority of cases I go with books. The reason I prefer books
over things like video courses is that I find books to be better suited to
active learning. It is incredibly easy for me to watch a video and just start
thinking about something else. In fact, that's something I do even when reading
a book, but to a lesser extent.</p>
<p>Sometimes, finding a good primary learning resource is easy. In the case of
Rust, the community maintains a book called <a href="https://doc.rust-lang.org/stable/book/"><em>The Rust Programming
Language</em></a>, which is the recommended
starting point for newcomers. After that, I intend to continue with
<a href="https://www.oreilly.com/library/view/programming-rust-2nd/9781492052586/"><em>Programming
Rust</em></a>,
which appears to be more in-depth. In some cases, identifying a good primary
resource may be far from trivial, and requires a fair amount of looking around.
But for Rust it did not take me much time at all to find what I was looking for.</p>
<p>I am aware that many prefer video resources over books nowadays. If that fits
your learning style, then that's all good. Books are just where it's at for me
personally. Sites like <a href="https://udemy.com">Udemy</a> and
<a href="https://pluralsight.com">Pluralsight</a> provide an ample selection of courses
suitable as primary learning resources for most programming languages in use
today. In my experience, such courses do however often cater to people learning
<em>programming</em>, rather than software engineers just looking to add a new language
to their repertoire.</p>
<p>An important note on my primary learning resource is that I never follow more
than one at any one time. That's essentially what makes it a primary resource.
So right now, I'm going with <em>The Rust Programming Language</em> first and will then
move on to <em>Programming Rust</em>, but I don't read them at the same time.</p>
<h2>Secondary learning resources</h2>
<p>Secondary learning resources are complementary, both to primary learning
resources and other secondary learning resources. I most often have at one or
two secondary resources that I utilize in parallel with a primary one. My
favourite sources of secondary learning resources are conference talks and
podcasts.</p>
<p>For Rust, I found a rather high-quality podcast called <a href="https://newrustacean.com/">New
Rustacean</a>, which features the host's journey of
learning the Rust programming language. It's a nice complement to my reading,
and I enjoy being able to listen to it on the go. If you're about to learn
Python, see my post on <a href="https://slar.se/awesome-python-podcasts.html">Awesome Python
Podcasts</a> for some inspiration.</p>
<p>Talks are most easily found on YouTube. Many of today's popular programming
languages have at least one yearly conference, while a lot of them have many
more. Python as various incarnations of <a href="https://pycon.org/">PyCon</a>, Rust has
<a href="https://rustconf.com/">RustConf</a> and C++ has <a href="https://cppcon.org/">CppCon</a>, and
you'll find that pretty much any language has a yearly conference named
<code><Language>Con(f)</code>. Some put all talks on YouTube, while others are a bit
less easy to get a hold of.</p>
<p>There is one more detail about secondary learning resources that I think is
worth noting, namely that they keep being useful to me long after I've stopped
investing time in primary learning resources. Secondary learning resources can
also enter my radar when I'm already a well-rounded programmer in a given
language. I will touch more on this when I discuss maintaining knowledge of a
programming language.</p>
<h1>Step 2: Start to learn</h1>
<p>In the initial step of learning, I simply sit down with my chosen primary
learning resource and consume only that. As I'm a book person, this entails
sitting down for a nice read. Although a very common advice is to learn actively
by trying out examples and writing your own code as soon as possible, I
personally prefer to just read for a while before I start dabbling with code
myself. Sometimes I'll try a small example or other that looks extra
interesting, but I generally wait with any serious amount of programming until
Step 3 of my learning process. The goal of Step 2 is not really to learn the
language, but to get a good overview of it.</p>
<p>I often start using my chosen secondary resources immediately after my first
sitting with a primary resource, at least if there's at least one podcast lined
up. What I mostly seek from secondary resources is to get an idea of the
community surrounding the programming language, what the ecosystem (e.g
libraries and frameworks) looks like and keep up to date with the development of
the core language. More on this in Step 4.</p>
<h1>Step 3: Build a project</h1>
<p>As soon as I feel like I have touched all subjects I need to sit down and start
up a small project, that's exactly what I do. This point usually rolls around
when I know how to create a project in the language, how to write unit tests and
how to write documentation. I'll often have a particular project in mind, and
then I may also know about a few additional things I need to learn about before
I get going.</p>
<p>My chosen project for learning Rust is to create a compiler and runtime
environment for a statically typed Python-like programming language that I call
<em>Rusthon</em>. This is an open source project and you can find it over at <a href="https://github.com/slarse/rusthon">Rusthon's
GitHub page</a> if you're interested.</p>
<p>Like I mentioned in Step 2, this is where the advice of active learning comes
into play. It's virtually impossible to become proficient with a new language
you're confined to following along with examples and solving tiny problems
without context. You need something larger to work on, I really do think this is
essential.</p>
<h1>Step 4: Maintain knowledge</h1>
<p>Once I've learned a language well enough to become proficient in it, I need to
maintain that knowledge and also keep up-to-date with developments in the
community and ecosystem. These are two rather separate concerns. Maintaining a
working knowledge of the language requires one to write code in it. For this I
keep using my project, or find new things to do. I often try to get involved in
open source projects, such as <a href="https://github.com/repobee/repobee">RepoBee</a>
(Python) and <a href="https://github.com/inria/spoon">Spoon</a> (Java) to make meaningful
contributions to the community.</p>
<p>Keeping up-to-date with the community and ecosystem is where I mostly keep using
secondary learning resources, mostly podcasts but also talks from conferences.
This may be less important in older languages with less vibrant ecosystems, but
in "newer" languages like Java, Python and Rust, knowing your way around the
package ecosystem and keeping up-to-date with new language features is rather
important.</p>
<h1>Final thoughts</h1>
<p>Learning a new programming language is something I enjoy greatly. I go about it
in a rather rigorous fashion where my goal is to become really proficient. I
<em>immerse</em> myself in the language, its community and its ecosystem. This also
entails that I spend some amount of effort in maintaining my knowledge and grasp
on the language, leading to the selection of languages I call myself proficient
in being small. When I learn a new language it often effectively replaces
something already in my repertoire. Rust is a contender to replace C, although I
can't say I've been all that thorough in maintaining my skills in C in the past
few years.</p>
<p>It is also worth considering that learning a new language without maintaining
your skills in it for any considerable amount of time can still be worthwhile. I
think this is especially true when exploring different paradigms, as I alluded
to in the introduction of this article. I learned Haskell many years back but
did not maintain a working knowledge of it for very long. Yet, the concepts that
I learned from being forced to code in a purely functional fashion are valuable
to me to this day. It was thanks to programming in Haskell that I really grasped
the concepts of recursion and higher-order functions, and understanding that has
greatly benefited me in all languages I've practiced since.</p>
<p>To summarize, learning a new language can be greatly beneficial. Even if you
don't intend to add it to your repertoire, it can still be a worthwhile effort
to step out of your comfort zone and explore new programming paradigms. The
things you learn from programming in one language can benefit your programming
in another, and make you a more well-rounded software engineer.</p>RepoBee at ITiCSE and SIGCSE 2021!2021-09-09T21:57:00+02:002021-09-09T21:57:00+02:00Simon Larséntag:slar.se,2021-09-09:/repobee-at-iticse-and-sigcse-2021.html<p>Another year, and a few new papers published on my favorite project:
<a href="https://repobee.org">RepoBee</a>. We made a hat-trick and appeared at
ITiCSE 2021 for a third year in a row (see <a href="https://slar.se/repobee-and-simon-at-iticse-2019.html">ITiCSE 2019</a>
and <a href="https://slar.se/repobee-at-iticse-2020.html">ITiCSE 2020</a>), but this time we managed to sneak
in two papers. One paper that I presented …</p><p>Another year, and a few new papers published on my favorite project:
<a href="https://repobee.org">RepoBee</a>. We made a hat-trick and appeared at
ITiCSE 2021 for a third year in a row (see <a href="https://slar.se/repobee-and-simon-at-iticse-2019.html">ITiCSE 2019</a>
and <a href="https://slar.se/repobee-at-iticse-2020.html">ITiCSE 2020</a>), but this time we managed to sneak
in two papers. One paper that I presented detailed <a href="https://doi.org/10.1145/3456565.3460031">RepoBee's double-blind peer
review</a>, and the other paper that was
presented by my colleague Tobias showcased the <a href="https://doi.org/10.1145/3456565.3460036">repobee-sanitizer
plugin</a>. Amazingly, the presentations
won <a href="https://iticse.acm.org/best-paper-award/">both "Best Presentation" awards for their
category</a>.</p>
<p>We also made an appearance at SIGCSE with a 20-minute demo and Q&A. The teaser
trailer for the demo <a href="https://repobee.org/media/repobee-demo.mp4">is available
here</a>. The demo itself was
essentially that, but less compressed and a little more elaborate.
It was followed by a nicely active Q&A, where we made some new connections.</p>
<p>All my publications can be found on the <a href="https://slar.se/pages/publications-and-essays.html">Publications
page</a>.</p>RepoBee at ITiCSE 2020!2020-10-04T09:20:00+02:002020-10-04T09:20:00+02:00Simon Larséntag:slar.se,2020-10-04:/repobee-at-iticse-2020.html<p>In 2019, <a href="https://slar.se/repobee-and-simon-at-iticse-2019.html">I presented RepoBee at the ITiCSE conference</a>
in Aberdeen. This year, ITiCSE went virtual, but RepoBee still made an
appearance in the new tools and tricks section with a small, two-page paper.
Although I let my co-author Ric Glassey deal with the virtual presentation, I'm
still quite proud …</p><p>In 2019, <a href="https://slar.se/repobee-and-simon-at-iticse-2019.html">I presented RepoBee at the ITiCSE conference</a>
in Aberdeen. This year, ITiCSE went virtual, but RepoBee still made an
appearance in the new tools and tricks section with a small, two-page paper.
Although I let my co-author Ric Glassey deal with the virtual presentation, I'm
still quite proud that I had another paper on RepoBee published at one of the
major tech education conferences.</p>
<p>If you're unaware, <a href="https://repobee.org">RepoBee</a> is a tool for managing Git
repositories in an educational context. A basic use case is for a teacher to
have a template repository, and create copies of it for students or groups of
students on GitHub or GitLab. If you've heard of <a href="https://classroom.github.com">GitHub
Classroom</a>, it's the same concept, but RepoBee is
far more powerful and customizable, at the expense of being a little more
intricate to use.</p>
<p>In the paper, Ric and I mostly discuss the need for customizable tools to tailor
to different teaching methodologies and preferences, and how RepoBee makes this
possible through plugins and by supporting both GitHub and GitLab. We also
discuss the two primary modes of RepoBee, and how they can be combined. In
<em>dictate</em> mode, RepoBee creates repositories for students based on template
repositories. In <em>discovery</em> mode, RepoBee only sets up student teams/groups,
and then the students themselves are responsible for creating the repositories,
which RepoBee can then "discover".</p>
<p>The full paper <a href="https://doi.org/10.1145/3341525.3393999">is available in the ACM digital
library</a>.</p>Essential pytest pt. 3: Rerunning failed tests (and the pytest cache)2020-10-03T19:00:00+02:002020-10-03T19:00:00+02:00Simon Larséntag:slar.se,2020-10-03:/essential-pytest-3.html<p>This is the third part of a series of small articles detailing some of the
functionality of the <a href="https://docs.pytest.org/en/latest/">pytest</a> testing
framework that I find most essential. The series assumes you know how to run
tests with <code>pytest</code> already.</p>
<p>In this third part, we'll take a super quick look at the …</p><p>This is the third part of a series of small articles detailing some of the
functionality of the <a href="https://docs.pytest.org/en/latest/">pytest</a> testing
framework that I find most essential. The series assumes you know how to run
tests with <code>pytest</code> already.</p>
<p>In this third part, we'll take a super quick look at the <code>--lf</code> flag that lets
us rerun failed tests, as well as the caching mechanism that makes it possible.</p>
<h2>Using <code>--lf</code> to rerun failed tests</h2>
<p>In this article, we'll use the test suite from <a href="https://slar.se/essential-pytest-1.html">the first
article</a>.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># test_mul.py</span>
<span class="k">def</span> <span class="nf">mul</span><span class="p">(</span><span class="n">lhs</span><span class="p">,</span> <span class="n">rhs</span><span class="p">):</span>
<span class="k">return</span> <span class="n">lhs</span> <span class="o">*</span> <span class="n">lhs</span>
<span class="k">def</span> <span class="nf">test_multiply_equal_numbers</span><span class="p">():</span>
<span class="k">assert</span> <span class="n">mul</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">5</span><span class="p">)</span> <span class="o">==</span> <span class="mi">25</span>
<span class="k">def</span> <span class="nf">test_multiply_by_zero</span><span class="p">():</span>
<span class="k">assert</span> <span class="n">mul</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span>
<span class="k">def</span> <span class="nf">test_multiply_different_numbers</span><span class="p">():</span>
<span class="k">assert</span> <span class="n">mul</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> <span class="o">==</span> <span class="mi">15</span>
</code></pre></div>
<p>Just like in that article, the implementation of <code>mul</code> is broken.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>-v<span class="w"> </span>--tb<span class="o">=</span><span class="nv">no</span>
<span class="o">==========================</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>session<span class="w"> </span><span class="nv">starts</span><span class="w"> </span><span class="o">===========================</span>
platform<span class="w"> </span>linux<span class="w"> </span>--<span class="w"> </span>Python<span class="w"> </span><span class="m">3</span>.8.5,<span class="w"> </span>pytest-6.1.0,<span class="w"> </span>py-1.9.0,<span class="w"> </span>pluggy-0.13.1
cachedir:<span class="w"> </span>.pytest_cache
rootdir:<span class="w"> </span>/home/slarse/python
collected<span class="w"> </span><span class="m">3</span><span class="w"> </span>items<span class="w"> </span>
test_mul.py::test_multiply_equal_numbers<span class="w"> </span>PASSED<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="m">33</span>%<span class="o">]</span>
test_mul.py::test_multiply_by_zero<span class="w"> </span>FAILED<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="m">66</span>%<span class="o">]</span>
test_mul.py::test_multiply_different_numbers<span class="w"> </span>FAILED<span class="w"> </span><span class="o">[</span><span class="m">100</span>%<span class="o">]</span>
<span class="o">========================</span><span class="w"> </span>short<span class="w"> </span><span class="nb">test</span><span class="w"> </span>summary<span class="w"> </span><span class="nv">info</span><span class="w"> </span><span class="o">=========================</span>
FAILED<span class="w"> </span>test_mul.py::test_multiply_by_zero<span class="w"> </span>-<span class="w"> </span>assert<span class="w"> </span><span class="nv">1</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">0</span>
FAILED<span class="w"> </span>test_mul.py::test_multiply_different_numbers<span class="w"> </span>-<span class="w"> </span>assert<span class="w"> </span><span class="nv">25</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="nv">15</span>
<span class="o">======================</span><span class="w"> </span><span class="m">2</span><span class="w"> </span>failed,<span class="w"> </span><span class="m">1</span><span class="w"> </span>passed<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span>.05s<span class="w"> </span><span class="o">=======================</span>
</code></pre></div>
<p>Note how 2 tests failed. <code>pytest</code> caches the failed tests from the last run,
which enables us to rerun them with the <code>--lf|--last-failed</code> flag. So let's do
that, and show some more traceback information while we're at it. Note that
only the failing tests are executed.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>-v<span class="w"> </span>--lf<span class="w"> </span>--tb<span class="o">=</span><span class="nv">short</span>
<span class="o">==========================</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>session<span class="w"> </span><span class="nv">starts</span><span class="w"> </span><span class="o">===========================</span>
platform<span class="w"> </span>linux<span class="w"> </span>--<span class="w"> </span>Python<span class="w"> </span><span class="m">3</span>.8.5,<span class="w"> </span>pytest-6.1.0,<span class="w"> </span>py-1.9.0,<span class="w"> </span>pluggy-0.13.1
cachedir:<span class="w"> </span>.pytest_cache
rootdir:<span class="w"> </span>/home/slarse/python
collected<span class="w"> </span><span class="m">2</span><span class="w"> </span>items<span class="w"> </span>
run-last-failure:<span class="w"> </span>rerun<span class="w"> </span>previous<span class="w"> </span><span class="m">2</span><span class="w"> </span>failures
test_mul.py::test_multiply_by_zero<span class="w"> </span>FAILED<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="m">50</span>%<span class="o">]</span>
test_mul.py::test_multiply_different_numbers<span class="w"> </span>FAILED<span class="w"> </span><span class="o">[</span><span class="m">100</span>%<span class="o">]</span>
<span class="o">================================</span><span class="w"> </span><span class="nv">FAILURES</span><span class="w"> </span><span class="o">================================</span>
_________________________<span class="w"> </span>test_multiply_by_zero<span class="w"> </span>__________________________
test_mul.py:8:<span class="w"> </span><span class="k">in</span><span class="w"> </span>test_multiply_by_zero
<span class="w"> </span>assert<span class="w"> </span>mul<span class="o">(</span><span class="m">1</span>,<span class="w"> </span><span class="m">0</span><span class="o">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">0</span>
E<span class="w"> </span>assert<span class="w"> </span><span class="nv">1</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">0</span>
E<span class="w"> </span>+1
E<span class="w"> </span>-0
____________________<span class="w"> </span>test_multiply_different_numbers<span class="w"> </span>_____________________
test_mul.py:11:<span class="w"> </span><span class="k">in</span><span class="w"> </span>test_multiply_different_numbers
<span class="w"> </span>assert<span class="w"> </span>mul<span class="o">(</span><span class="m">5</span>,<span class="w"> </span><span class="m">3</span><span class="o">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">15</span>
E<span class="w"> </span>assert<span class="w"> </span><span class="nv">25</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">15</span>
E<span class="w"> </span>+25
E<span class="w"> </span>-15
<span class="o">========================</span><span class="w"> </span>short<span class="w"> </span><span class="nb">test</span><span class="w"> </span>summary<span class="w"> </span><span class="nv">info</span><span class="w"> </span><span class="o">=========================</span>
FAILED<span class="w"> </span>test_mul.py::test_multiply_by_zero<span class="w"> </span>-<span class="w"> </span>assert<span class="w"> </span><span class="nv">1</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">0</span>
FAILED<span class="w"> </span>test_mul.py::test_multiply_different_numbers<span class="w"> </span>-<span class="w"> </span>assert<span class="w"> </span><span class="nv">25</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="nv">15</span>
<span class="o">===========================</span><span class="w"> </span><span class="m">2</span><span class="w"> </span>failed<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span>.12s<span class="w"> </span><span class="o">============================</span>
</code></pre></div>
<p>My primary use case for <code>--lf</code> is for sorting out bugs. Every time a test
passes, it is removed from the last-failed cache, and thus does not run the next
time <code>--lf</code> is specified. This way, it's easy to quickly target only failing
tests, and systematically eliminate them one by one.</p>
<blockquote>
<p><strong>Pitfall:</strong> A common mistake is to use <code>--lf</code> to eliminate the failing tests
one by one, and then forget to run all tests when the last of the initially
failing tests passes. It's entirely possible to fix the implementation such
that a test <code>A</code> passes, and then subsequently reintroduce the same problem in
addressing another test, but at that point <code>A</code> is no longer executing with
<code>--lf</code>.</p>
</blockquote>
<h2>Interacting with the cache</h2>
<p>I mentioned that the failed tests from the last run are stored in a cache. This
cache is located in the <code>.pytest_cache</code> directory of the current working
directory. There are a few flags to interact with said cache. First, you can
execute <code>pytest</code> with the <code>--cache-show</code> flag to show the current contents of
the cache.</p>
<div class="highlight"><pre><span></span><code>pytest<span class="w"> </span>--cache-show
<span class="o">==========================</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>session<span class="w"> </span><span class="nv">starts</span><span class="w"> </span><span class="o">===========================</span>
platform<span class="w"> </span>linux<span class="w"> </span>--<span class="w"> </span>Python<span class="w"> </span><span class="m">3</span>.8.5,<span class="w"> </span>pytest-6.1.0,<span class="w"> </span>py-1.9.0,<span class="w"> </span>pluggy-0.13.1
rootdir:<span class="w"> </span>/home/slarse/python
cachedir:<span class="w"> </span>/home/slarse/python/.pytest_cache
--------------------------<span class="w"> </span>cache<span class="w"> </span>values<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="s1">'*'</span><span class="w"> </span>--------------------------
cache/lastfailed<span class="w"> </span>contains:
<span class="w"> </span><span class="o">{</span><span class="s1">'test_mul.py::test_multiply_by_zero'</span>:<span class="w"> </span>True,
<span class="w"> </span><span class="s1">'test_mul.py::test_multiply_different_numbers'</span>:<span class="w"> </span>True<span class="o">}</span>
cache/nodeids<span class="w"> </span>contains:
<span class="w"> </span><span class="o">[</span><span class="s1">'test_mul.py::test_multiply_by_zero'</span>,
<span class="w"> </span><span class="s1">'test_mul.py::test_multiply_different_numbers'</span>,
<span class="w"> </span><span class="s1">'test_mul.py::test_multiply_equal_numbers'</span><span class="o">]</span>
cache/stepwise<span class="w"> </span>contains:
<span class="w"> </span><span class="o">[]</span>
<span class="o">=========================</span><span class="w"> </span>no<span class="w"> </span>tests<span class="w"> </span>ran<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span>.00s<span class="w"> </span><span class="o">==========================</span>
</code></pre></div>
<p>Here, we can for example see the contents of the last-failed cache
(<code>cache/lastfailed</code>), and the tests currently known by <code>pytest</code>
(<code>cache/nodeids</code>). It's possible to supply <code>--cache-show</code> with an optional
value, in order to show only some part of the cache. For example,
<code>--cache-show=lastfailed</code> shows only the last-failed cache contents.</p>
<p>On occasion, the cache may get into an inconsistent state, typically due to
strange interactions by the user. This has happened to me on several occasions,
with tests simply not executing as I expect them to. At that point, supplying
the <code>--cache-clear</code> flag to a test run will clear the cache. Alternatively, you
may simply remove the <code>.pytest_cache</code> directory.</p>
<h2>Summary</h2>
<p>Being able to execute only the failing tests from the previous test run is a
very handy feature when addressing bugs, both saving time in test execution and
limiting the amount of output shown to the user. It's however important to
remember to execute all tests after the last failing test passes, so as to check
for regressions. One should also be aware that the functionality hinges on
caching in the <code>.pytest_cache</code> directory, which on rare occasions may need to be
cleared.</p>Essential pytest pt. 2: Selecting tests to run2020-10-03T14:00:00+02:002020-10-03T14:00:00+02:00Simon Larséntag:slar.se,2020-10-03:/essential-pytest-2.html<p>This is the second part of a series of small articles detailing some of the
functionality of the <a href="https://docs.pytest.org/en/latest/">pytest</a> testing
framework that I find most essential. The series assumes you know how to run
tests with <code>pytest</code> already.</p>
<p>In this second part, we'll take a look at the <code>-k</code> and …</p><p>This is the second part of a series of small articles detailing some of the
functionality of the <a href="https://docs.pytest.org/en/latest/">pytest</a> testing
framework that I find most essential. The series assumes you know how to run
tests with <code>pytest</code> already.</p>
<p>In this second part, we'll take a look at the <code>-k</code> and <code>-m</code> options to control
which tests in the test suite are executed.</p>
<h2>The test suite</h2>
<p>In this article, we'll use the test suite from <a href="https://slar.se/essential-pytest-1.html">the first
article</a>.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># test_mul.py</span>
<span class="k">def</span> <span class="nf">mul</span><span class="p">(</span><span class="n">lhs</span><span class="p">,</span> <span class="n">rhs</span><span class="p">):</span>
<span class="k">return</span> <span class="n">lhs</span> <span class="o">*</span> <span class="n">rhs</span>
<span class="k">def</span> <span class="nf">test_multiply_equal_numbers</span><span class="p">():</span>
<span class="k">assert</span> <span class="n">mul</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">5</span><span class="p">)</span> <span class="o">==</span> <span class="mi">25</span>
<span class="k">def</span> <span class="nf">test_multiply_by_zero</span><span class="p">():</span>
<span class="k">assert</span> <span class="n">mul</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span>
<span class="k">def</span> <span class="nf">test_multiply_different_numbers</span><span class="p">():</span>
<span class="k">assert</span> <span class="n">mul</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> <span class="o">==</span> <span class="mi">15</span>
</code></pre></div>
<p>Note that <code>mul</code> is now correctly implemented, so all tests will pass.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>-v
<span class="o">==========================</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>session<span class="w"> </span><span class="nv">starts</span><span class="w"> </span><span class="o">===========================</span>
platform<span class="w"> </span>linux<span class="w"> </span>--<span class="w"> </span>Python<span class="w"> </span><span class="m">3</span>.8.5,<span class="w"> </span>pytest-6.1.0,<span class="w"> </span>py-1.9.0,<span class="w"> </span>pluggy-0.13.1
cachedir:<span class="w"> </span>.pytest_cache
rootdir:<span class="w"> </span>/home/slarse/python
collected<span class="w"> </span><span class="m">3</span><span class="w"> </span>items<span class="w"> </span>
mul.py::test_multiply_equal_numbers<span class="w"> </span>PASSED<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="m">33</span>%<span class="o">]</span>
mul.py::test_multiply_by_zero<span class="w"> </span>PASSED<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="m">66</span>%<span class="o">]</span>
mul.py::test_multiply_different_numbers<span class="w"> </span>PASSED<span class="w"> </span><span class="o">[</span><span class="m">100</span>%<span class="o">]</span>
<span class="o">===========================</span><span class="w"> </span><span class="m">3</span><span class="w"> </span>passed<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span>.01s<span class="w"> </span><span class="o">============================</span>
</code></pre></div>
<p>Now, let's learn how to run subsets of these tests, without modifying the
source code.</p>
<h2>Using the <code>-k</code> option to select tests by substring matching</h2>
<p>The <code>-k</code> option is wonderful, and allows us to select a subset of tests to
execute based on simple substring matching. The simplest use of <code>-k</code> is to
provide it with a whitespace-less string. Any test with a name that <em>contains</em>
that string will be executed. To be clear, the name of a test is e.g.
<code>mul.py::test_multiply_equal_numbers</code>, that is to say, the qualified path to it.</p>
<p>As a simple example, we can select only the test that multiplies by zero like
so.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>-v<span class="w"> </span>-k<span class="w"> </span><span class="nv">zero</span>
<span class="o">==========================</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>session<span class="w"> </span><span class="nv">starts</span><span class="w"> </span><span class="o">===========================</span>
platform<span class="w"> </span>linux<span class="w"> </span>--<span class="w"> </span>Python<span class="w"> </span><span class="m">3</span>.8.5,<span class="w"> </span>pytest-6.1.0,<span class="w"> </span>py-1.9.0,<span class="w"> </span>pluggy-0.13.1
cachedir:<span class="w"> </span>.pytest_cache
rootdir:<span class="w"> </span>/home/slarse/python
collected<span class="w"> </span><span class="m">3</span><span class="w"> </span>items<span class="w"> </span>/<span class="w"> </span><span class="m">2</span><span class="w"> </span>deselected<span class="w"> </span>/<span class="w"> </span><span class="m">1</span><span class="w"> </span>selected<span class="w"> </span>
test_mul.py::test_multiply_by_zero<span class="w"> </span>PASSED<span class="w"> </span><span class="o">[</span><span class="m">100</span>%<span class="o">]</span>
<span class="o">====================</span><span class="w"> </span><span class="m">1</span><span class="w"> </span>passed,<span class="w"> </span><span class="m">2</span><span class="w"> </span>deselected<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span>.05s<span class="w"> </span><span class="o">=====================</span>
</code></pre></div>
<p>Note that 2 tests were deselected. It is also possible to create logical
expressions using <code>not</code>, <code>or</code> and <code>and</code>. <code>not</code> simply inverts the condition:
any test that does <em>not</em> match the substring is executed.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>-v<span class="w"> </span>-k<span class="w"> </span><span class="s1">'not zero'</span>
<span class="o">==========================</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>session<span class="w"> </span><span class="nv">starts</span><span class="w"> </span><span class="o">===========================</span>
platform<span class="w"> </span>linux<span class="w"> </span>--<span class="w"> </span>Python<span class="w"> </span><span class="m">3</span>.8.5,<span class="w"> </span>pytest-6.1.0,<span class="w"> </span>py-1.9.0,<span class="w"> </span>pluggy-0.13.1
cachedir:<span class="w"> </span>.pytest_cache
rootdir:<span class="w"> </span>/home/slarse/python
collected<span class="w"> </span><span class="m">3</span><span class="w"> </span>items<span class="w"> </span>/<span class="w"> </span><span class="m">1</span><span class="w"> </span>deselected<span class="w"> </span>/<span class="w"> </span><span class="m">2</span><span class="w"> </span>selected<span class="w"> </span>
test_mul.py::test_multiply_equal_numbers<span class="w"> </span>PASSED<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="m">50</span>%<span class="o">]</span>
test_mul.py::test_multiply_different_numbers<span class="w"> </span>PASSED<span class="w"> </span><span class="o">[</span><span class="m">100</span>%<span class="o">]</span>
<span class="o">====================</span><span class="w"> </span><span class="m">2</span><span class="w"> </span>passed,<span class="w"> </span><span class="m">1</span><span class="w"> </span>deselected<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span>.05s<span class="w"> </span><span class="o">=====================</span>
</code></pre></div>
<p>With <code>or</code>, we can select tests that match any of a number of substrings.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>-v<span class="w"> </span>-k<span class="w"> </span><span class="s1">'equal or different'</span>
<span class="o">==========================</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>session<span class="w"> </span><span class="nv">starts</span><span class="w"> </span><span class="o">===========================</span>
platform<span class="w"> </span>linux<span class="w"> </span>--<span class="w"> </span>Python<span class="w"> </span><span class="m">3</span>.8.5,<span class="w"> </span>pytest-6.1.0,<span class="w"> </span>py-1.9.0,<span class="w"> </span>pluggy-0.13.1
/usr/bin/python
cachedir:<span class="w"> </span>.pytest_cache
rootdir:<span class="w"> </span>/home/slarse/python
collected<span class="w"> </span><span class="m">3</span><span class="w"> </span>items<span class="w"> </span>/<span class="w"> </span><span class="m">1</span><span class="w"> </span>deselected<span class="w"> </span>/<span class="w"> </span><span class="m">2</span><span class="w"> </span>selected<span class="w"> </span>
test_mul.py::test_multiply_equal_numbers<span class="w"> </span>PASSED<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="m">50</span>%<span class="o">]</span>
test_mul.py::test_multiply_different_numbers<span class="w"> </span>PASSED<span class="w"> </span><span class="o">[</span><span class="m">100</span>%<span class="o">]</span>
<span class="o">====================</span><span class="w"> </span><span class="m">2</span><span class="w"> </span>passed,<span class="w"> </span><span class="m">1</span><span class="w"> </span>deselected<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span>.06s<span class="w"> </span><span class="o">=====================</span>
</code></pre></div>
<p>Finally, <code>and</code> allows us to select tests that match multiple substrings.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>-v<span class="w"> </span>-k<span class="w"> </span><span class="s1">'multiply and equal'</span>
<span class="o">==========================</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>session<span class="w"> </span><span class="nv">starts</span><span class="w"> </span><span class="o">===========================</span>
platform<span class="w"> </span>linux<span class="w"> </span>--<span class="w"> </span>Python<span class="w"> </span><span class="m">3</span>.8.5,<span class="w"> </span>pytest-6.1.0,<span class="w"> </span>py-1.9.0,<span class="w"> </span>pluggy-0.13.1
/usr/bin/python
cachedir:<span class="w"> </span>.pytest_cache
rootdir:<span class="w"> </span>/home/slarse/python
collected<span class="w"> </span><span class="m">3</span><span class="w"> </span>items<span class="w"> </span>/<span class="w"> </span><span class="m">2</span><span class="w"> </span>deselected<span class="w"> </span>/<span class="w"> </span><span class="m">1</span><span class="w"> </span>selected<span class="w"> </span>
test_mul.py::test_multiply_equal_numbers<span class="w"> </span>PASSED<span class="w"> </span><span class="o">[</span><span class="m">100</span>%<span class="o">]</span>
<span class="o">====================</span><span class="w"> </span><span class="m">1</span><span class="w"> </span>passed,<span class="w"> </span><span class="m">2</span><span class="w"> </span>deselected<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span>.05s<span class="w"> </span><span class="o">=====================</span>
</code></pre></div>
<p>And that's pretty much all there is to the <code>-k</code> option. It's extremely useful
when test suites grow in size, and I use it daily.</p>
<h2>Using the <code>-m</code> option to select by marker</h2>
<p>With <code>-m</code>, we can select tests by <em>markers</em>. You can mark a test function (or
class) by placing a decorator above it.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># test_mul.py</span>
<span class="kn">import</span> <span class="nn">pytest</span>
<span class="k">def</span> <span class="nf">mul</span><span class="p">(</span><span class="n">lhs</span><span class="p">,</span> <span class="n">rhs</span><span class="p">):</span>
<span class="k">return</span> <span class="n">lhs</span> <span class="o">*</span> <span class="n">rhs</span>
<span class="nd">@pytest</span><span class="o">.</span><span class="n">mark</span><span class="o">.</span><span class="n">normcase</span>
<span class="k">def</span> <span class="nf">test_multiply_equal_numbers</span><span class="p">():</span>
<span class="k">assert</span> <span class="n">mul</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">5</span><span class="p">)</span> <span class="o">==</span> <span class="mi">25</span>
<span class="nd">@pytest</span><span class="o">.</span><span class="n">mark</span><span class="o">.</span><span class="n">edgecase</span>
<span class="k">def</span> <span class="nf">test_multiply_by_zero</span><span class="p">():</span>
<span class="k">assert</span> <span class="n">mul</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span>
<span class="nd">@pytest</span><span class="o">.</span><span class="n">mark</span><span class="o">.</span><span class="n">normcase</span>
<span class="k">def</span> <span class="nf">test_multiply_different_numbers</span><span class="p">():</span>
<span class="k">assert</span> <span class="n">mul</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> <span class="o">==</span> <span class="mi">15</span>
</code></pre></div>
<p>Note that we must actually import the <code>pytest</code> module to be able to mark tests
with <code>@pytest.mark.x</code>. Now, we can run all tests marked with e.g. <code>normcase</code>
like so.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>-v<span class="w"> </span>-m<span class="w"> </span><span class="nv">normcase</span>
<span class="o">==========================</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>session<span class="w"> </span><span class="nv">starts</span><span class="w"> </span><span class="o">===========================</span>
platform<span class="w"> </span>linux<span class="w"> </span>--<span class="w"> </span>Python<span class="w"> </span><span class="m">3</span>.8.5,<span class="w"> </span>pytest-6.1.0,<span class="w"> </span>py-1.9.0,<span class="w"> </span>pluggy-0.13.1
cachedir:<span class="w"> </span>.pytest_cache
rootdir:<span class="w"> </span>/home/slarse/python
collected<span class="w"> </span><span class="m">3</span><span class="w"> </span>items<span class="w"> </span>/<span class="w"> </span><span class="m">1</span><span class="w"> </span>deselected<span class="w"> </span>/<span class="w"> </span><span class="m">2</span><span class="w"> </span>selected<span class="w"> </span>
test_mul.py::test_multiply_equal_numbers<span class="w"> </span>PASSED<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="m">50</span>%<span class="o">]</span>
test_mul.py::test_multiply_different_numbers<span class="w"> </span>PASSED<span class="w"> </span><span class="o">[</span><span class="m">100</span>%<span class="o">]</span>
<span class="o">============================</span><span class="w"> </span>warnings<span class="w"> </span><span class="nv">summary</span><span class="w"> </span><span class="o">============================</span>
test_mul.py:6
<span class="w"> </span>/home/slarse/python/test_mul.py:6:<span class="w"> </span>PytestUnknownMarkWarning:<span class="w"> </span>Unknown<span class="w"> </span>pytest.mark.normcase<span class="w"> </span>-<span class="w"> </span>is<span class="w"> </span>this<span class="w"> </span>a<span class="w"> </span>typo?<span class="w"> </span>You<span class="w"> </span>can<span class="w"> </span>register<span class="w"> </span>custom<span class="w"> </span>marks<span class="w"> </span>to<span class="w"> </span>avoid<span class="w"> </span>this<span class="w"> </span>warning<span class="w"> </span>-<span class="w"> </span><span class="k">for</span><span class="w"> </span>details,<span class="w"> </span>see<span class="w"> </span>https://docs.pytest.org/en/stable/mark.html
<span class="w"> </span>@pytest.mark.normcase
<span class="o">[</span>...<span class="w"> </span><span class="m">2</span><span class="w"> </span>WARNINGS<span class="w"> </span>OMITTED<span class="w"> </span>...<span class="o">]</span>
--<span class="w"> </span>Docs:<span class="w"> </span>https://docs.pytest.org/en/stable/warnings.html
<span class="o">==============</span><span class="w"> </span><span class="m">2</span><span class="w"> </span>passed,<span class="w"> </span><span class="m">1</span><span class="w"> </span>deselected,<span class="w"> </span><span class="m">3</span><span class="w"> </span>warnings<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span>.01s<span class="w"> </span><span class="o">===============</span>
</code></pre></div>
<p>Note that this resulted in 3 warnings, one for each of the markings. The reason
for this is that newer versions of <code>pytest</code> want you to <em>register</em> markers, <a href="https://docs.pytest.org/en/stable/example/markers.html#registering-markers">as
described
here</a>.
The purpose of this is to avoid users misspelling markers, and registering them
will make the warnings go away.</p>
<p>As might be expected, the <code>-m</code> option also accepts logical expressions using
<code>not</code>, <code>and</code> and <code>or</code>, just like the <code>-k</code> option does. Personally, I very rarely
use <code>-m</code> when using <code>pytest</code>, but some people swear by it, which is why I wanted
to include it in this article.</p>
<h2>Trick: Grouping related tests into classes makes selection easier</h2>
<p>A trick that I like to employ is to group related tests into classes. The class
name is then incorporated into the test's name, and it becomes very easy to
select tests that are part of the same class. Here's a simple example, where I'm
testing two functions in the same module <code>test_arithmetics.py</code>:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># test_arithmetics.py</span>
<span class="k">def</span> <span class="nf">mul</span><span class="p">(</span><span class="n">lhs</span><span class="p">,</span> <span class="n">rhs</span><span class="p">):</span>
<span class="k">return</span> <span class="n">lhs</span> <span class="o">*</span> <span class="n">rhs</span>
<span class="k">def</span> <span class="nf">div</span><span class="p">(</span><span class="n">lhs</span><span class="p">,</span> <span class="n">rhs</span><span class="p">):</span>
<span class="k">return</span> <span class="n">lhs</span> <span class="o">/</span> <span class="n">rhs</span>
<span class="k">class</span> <span class="nc">TestMul</span><span class="p">:</span>
<span class="w"> </span><span class="sd">"""Tests for the mul function."""</span>
<span class="k">def</span> <span class="nf">test_multiply_equal_numbers</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">assert</span> <span class="n">mul</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">5</span><span class="p">)</span> <span class="o">==</span> <span class="mi">25</span>
<span class="k">def</span> <span class="nf">test_multiply_by_zero</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">assert</span> <span class="n">mul</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span>
<span class="k">def</span> <span class="nf">test_multiply_different_numbers</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">assert</span> <span class="n">mul</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> <span class="o">==</span> <span class="mi">15</span>
<span class="k">class</span> <span class="nc">TestDiv</span><span class="p">:</span>
<span class="w"> </span><span class="sd">"""Tests for the div function."""</span>
<span class="k">def</span> <span class="nf">test_divide_equal_numbers</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">assert</span> <span class="n">div</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span>
</code></pre></div>
<p>Note that in grouping test functions into test classes, the <code>self</code> argument
must be added. This is a little bit annoying, as I rarely if ever use the
<code>self</code> argument in a test case, but it's something that has to be done.</p>
<p>Now, I can for example run only the tests in <code>TestDiv</code> like so.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>-v<span class="w"> </span>-k<span class="w"> </span><span class="nv">TestDiv</span>
<span class="o">==========================</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>session<span class="w"> </span><span class="nv">starts</span><span class="w"> </span><span class="o">===========================</span>
platform<span class="w"> </span>linux<span class="w"> </span>--<span class="w"> </span>Python<span class="w"> </span><span class="m">3</span>.8.5,<span class="w"> </span>pytest-6.1.0,<span class="w"> </span>py-1.9.0,<span class="w"> </span>pluggy-0.13.1
cachedir:<span class="w"> </span>.pytest_cache
rootdir:<span class="w"> </span>/home/slarse/python
collected<span class="w"> </span><span class="m">4</span><span class="w"> </span>items<span class="w"> </span>/<span class="w"> </span><span class="m">3</span><span class="w"> </span>deselected<span class="w"> </span>/<span class="w"> </span><span class="m">1</span><span class="w"> </span>selected<span class="w"> </span>
test_arithmetics.py::TestDiv::test_divide_equal_numbers<span class="w"> </span>PASSED<span class="w"> </span><span class="o">[</span><span class="m">100</span>%<span class="o">]</span>
<span class="o">====================</span><span class="w"> </span><span class="m">1</span><span class="w"> </span>passed,<span class="w"> </span><span class="m">3</span><span class="w"> </span>deselected<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span>.05s<span class="w"> </span><span class="o">=====================</span>
</code></pre></div>
<p>Note that the test name that's printed above includes the class name, which is
why it is possible to select it with <code>-k</code>. Of course, grouping related tests
into modules is equally viable, as the module name (here, <code>test_arithmetics.py</code>)
is also part of the test name. I typically do both by creating one test module
per module of production code, and one test class per production code function.
This allows me to easily select tests at two levels of granularity, which comes
in very handy.</p>
<h2>Summary</h2>
<p>Selecting a subset of test cases to run is crucial to my development workflow.
When there are 100s or even 1000s of tests to run, running all of them is often
not what you want to do. My preferred way of selecting test cases is by using
the <code>-k</code> option to match substrings of test names, but the <code>-m</code> option is also
there for those that like to put marker decorators in their code. Finally,
grouping related tests into modules and classes allows for easy selection of
tests on two levels of granularity, which is something that I exploit daily.</p>Thoughts on graduating with an MSc in Computer Science and Engineering2020-09-29T00:00:00+02:002020-09-29T00:00:00+02:00Simon Larséntag:slar.se,2020-09-29:/thoughts-on-graduating-with-an-msc-in-computer-science-and-engineering.html<p>After five long years of studies (seven if you include the two years of
materials science), I've finally graduated with an <em>MSc in Computer Science and
Engineering</em> from KTH Royal Institute of Technology. I'm still awaiting my
degree certificates, but the <a href="http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-281960">thesis is
published</a> and I don't
have to do …</p><p>After five long years of studies (seven if you include the two years of
materials science), I've finally graduated with an <em>MSc in Computer Science and
Engineering</em> from KTH Royal Institute of Technology. I'm still awaiting my
degree certificates, but the <a href="http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-281960">thesis is
published</a> and I don't
have to do anything but wait. I have two weeks left of my one month off before I
start working, and I found that now would be a good time to reflect a bit on my
education.</p>
<h2>A CS degree does not an engineer make</h2>
<p>Early on in my education, it became abundantly clear to me that my CS degree
would be highly theoretical, and the practical elements were mostly toy
projects. I needed side projects, both to practice applying the theory I learned
in class, and to get experience with common software development practices such
as version control (Git) and issue management.</p>
<p>While my first few projects were toy projects, such as
<a href="https://github.com/slarse/clanim/commits/master">clanim</a>, I started my grail
project in <a href="https://github.com/repobee/repobee">RepoBee</a> fairly early during my
bachelor's (technically, I started its predecessor). This was a "real" project
for me, that I used daily and was also used by others. This gave me great
incentive to create a good product and keep working on it. As RepoBee is a
management tool for version control in education, it also came naturally to
adopt proper version control practices, as opposed to just winging it.</p>
<p>The takeaway from this is that in order to be well-equipped for work after
school, side projects are really invaluable. Not only are side projects
invaluable, but I think it's important to work on real projects. It creates
incentive to keep going, and also gives you something to show off to future
employers. If you can't come up with something yourself, then there are a
borderline innumerable amount of open-source software projects out there that
need all the help they can get. Such as RepoBee :)</p>
<h2>A CS degree gives you an exceptional theoretical foundation in computing</h2>
<p>Although I think the engineering aspects were lacking in my education, the
theoretical foundation that I now possess is nothing short of incredible. I
never thought I could learn so much about mathematics, algorithms, operating
systems, network protocols, computer security etc in only five short years. I
also learned <em>how</em> to learn, and how to do so efficiently. This is probably the
most important thing you can take with you from university.</p>
<p>I think that a lot of the theory would have been very hard for me to learn on my
own, whereas the practical engineering practices were not. As such, in
hindsight, I appreciate the heavy emphasis on theory. I've had a large amount of
use for my knowledge of algorithms, data structures and time complexities
already, and given my interest in programming languages and version control
systems, I expect this trend will carry on.</p>
<p>A lot of people will tell you that a CS degree is not worth it, that you don't
even learn the practical skills you need for engineering. This is true to an
extent, but I think it's an oversimplification and whether or not it's worth it
is highly individual. For me, the degree was <em>entirely</em> worth it. I was exposed
to subjects I would not have found on my own and was taught concepts I would
have struggled to grasp without a tutor. I also greatly enjoy learning for the
sake of it, and I like to dive deep. I like to understand how things work,
rather than just understand how to use them. A CS degree is definitely not for
everyone, but the blanket statement that it isn't worth it is simply false. For
those that enjoy learning and have a deep interest in programming, CS is the way
to go. If you want to learn the practical skills you need to land a job as fast
as possible, then it probably isn't.</p>
<h2>Teaching is learning</h2>
<p>After finishing the first year of my studies, I applied and was accepted to a
position as a teaching assistant. I would go on to work as a TA during The
remaining four years of my studies, year-round. I held exercises, worked labs,
corrected student submissions, developed coursework, and much more. This greatly
accelerated my own learning, for two reasons. First, in order to teach a
subject, you really must learn it well, and students' questions inevitably
highlight the shortcomings in your own knowledge. I received so many questions
that I could not answer that I likely would not have known I could not answer
had those questions not been asked. Second, in working as a TA I was introduced
to other more senior TAs, who were much more knowledgable than I was.
Discussions with them would lead to my learning things that I would not have
found out on my own.</p>
<p>Another benefit of working as a TA was that I got the opportunity to develop
RepoBee as a paid project, giving me another source of income during the summer
and winter breaks. I also got the opportunity to write some short research
papers, attend conferences, and connect with other faculty. If I wanted to, I
could easily launch an academic career at this point. However, even though I
enjoy science, I am more interested in practical engineering, and so an academic
career seems unlikely at this point.</p>
<p>My point here is simple: if you have the opportunity to teach, then do so! I
attribute a lot of my success to my experience as a TA, and many doors have been
opened for me as a result of working with other academics. I can't recommend it
enough.</p>
<h2>Closing thoughts</h2>
<p>My education has overall been a great experience. I've met a lot of interesting
people and done a lot of interesting things. I've taken classes, taught classes,
written software, theses and conference papers, and despaired in the face of
the odd inordinately difficult exam. Although I greatly enjoyed my time at
university, I don't want to continue with a PhD. I feel done with studies. For
the time being, I'll be working as a research engineer developing experimental
software at KTH, which seems like a nice middle ground between academics and
industry. After that, I don't really know, which is pretty exciting in
and of itself.</p>The Linux /etc/passwd file, and why it doesn't contain passwords2020-08-02T11:51:33+00:002020-08-02T12:15:33+00:00Simon Larséntag:slar.se,2020-08-02:/etc-passwd.html<p>On any Linux distribution, there's a file located at <code>/etc/passwd</code>. This file
contains information about users that exist on the system, including their
username, user id, group id and more. In this short article, I'll outline the
structure of the <code>/etc/passwd</code> file, and also illuminate why it doesn't …</p><p>On any Linux distribution, there's a file located at <code>/etc/passwd</code>. This file
contains information about users that exist on the system, including their
username, user id, group id and more. In this short article, I'll outline the
structure of the <code>/etc/passwd</code> file, and also illuminate why it doesn't
typically contain any passwords.</p>
<h2>Layout of the <code>/etc/passwd</code> file</h2>
<p>The layout of the <code>/etc/passwd</code> file is fairly simple. Each line represents a
user on the system, with different fields being separated by colons as follows:</p>
<div class="highlight"><pre><span></span><code><span class="n">name</span><span class="o">:</span><span class="n">password</span><span class="o">:</span><span class="n">UID</span><span class="o">:</span><span class="n">GID</span><span class="o">:</span><span class="n">GECOS</span><span class="o">:</span><span class="n">directory</span><span class="o">:</span><span class="n">shell</span>
</code></pre></div>
<p><code>name</code> and <code>password</code> are the username and password of the user, <code>UID</code> is the
user's numerical id, <code>GID</code> is the id of the <em>first</em> group the user belongs to,
<code>GECOS</code> is an optional comment, <code>directory</code> is the user's home directory, and
<code>shell</code> is the path to the executable that launches the user's preferred
shell. As an example, a part of my <code>/etc/passwd</code> file looks like this:</p>
<blockquote>
<p><strong>Note:</strong> You can find the groups users belong to in the <code>/etc/group</code> file.</p>
</blockquote>
<div class="highlight"><pre><span></span><code><span class="n">root</span><span class="o">:</span><span class="n">x</span><span class="o">:</span><span class="mi">0</span><span class="o">:</span><span class="mi">0</span><span class="o">::/</span><span class="n">root</span><span class="o">:/</span><span class="n">bin</span><span class="o">/</span><span class="n">bash</span>
<span class="n">slarse</span><span class="o">:</span><span class="n">x</span><span class="o">:</span><span class="mi">1000</span><span class="o">:</span><span class="mi">985</span><span class="o">::/</span><span class="n">home</span><span class="sr">/slarse:/bin/</span><span class="n">bash</span>
<span class="n">mysql</span><span class="o">:</span><span class="n">x</span><span class="o">:</span><span class="mi">970</span><span class="o">:</span><span class="mi">970</span><span class="o">:</span><span class="n">MariaDB</span><span class="o">:/</span><span class="n">var</span><span class="sr">/lib/mysql:/sbin/</span><span class="n">nologin</span>
</code></pre></div>
<p>We can see that the root user has the fields set as follows:</p>
<div class="highlight"><pre><span></span><code><span class="nv">password</span><span class="o">=</span>x
<span class="nv">UID</span><span class="o">=</span><span class="m">0</span>
<span class="nv">GID</span><span class="o">=</span><span class="m">0</span>
<span class="nv">GECOS</span><span class="o">=</span>
<span class="nv">directory</span><span class="o">=</span>/root
<span class="nv">shell</span><span class="o">=</span>/bin/bash
</code></pre></div>
<p>The user and group IDs of the root user are always 0, and it typically has its
home directory in <code>/root</code>. But is the password of root user really <code>x</code>? No, it
isn't. An <code>x</code> in the password field means that the password is located in the
<em>shadow</em> file. More on that in the next section. The entry for my own user,
slarse, is largely similar to that of the root user.</p>
<p>The entry for the mysql user is however a bit different. For starters, it has a
comment in the <code>GECOS</code> field saying <em>MariaDB</em>, which indicates that the mysql
user is actually used by the <code>MariaDB</code> fork of the <code>MySQL</code> database system. It
also has in interesting login shell, namely <code>/sbin/nologin</code>. The description of
the <code>nologin</code> program from its manpage simply reads: <em>nologin - politely refuse
a login</em>. This program simply refuses a login, regardless of what credentials
are supplied.</p>
<p>And that's pretty much it for what the <code>/etc/passwd</code> file contains. For more
details, you can read the <code>passwd (5)</code> manpage. Now, what about that shadow
file?</p>
<blockquote>
<p><strong>Hint:</strong> To access section <code>Y</code> of a manpage <code>PAGE</code>, type <code>man PAGE.Y</code> into a
terminal. For example, to access <code>passwd (5)</code>, you type <code>man passwd.5</code>.</p>
</blockquote>
<h2>The <code>/etc/shadow</code> file</h2>
<p>The <code>/etc/passwd</code> file is a so-called <em>world-readable</em>, meaning that any user on
the system can read it. Many programs use this file to map users to their ids,
for example, and so its broad accessibility is necessary. A side effect is that
storing encrypted passwords in the <code>/etc/passwd</code> file lets any user that has
access to the system read the encrypted password of any other user. In times
long past, when cracking encrypted passwords was computationally infeasible,
this wasn't really a problem. Nowadays however, cracking an encrypted password
is only a matter of (feasible) time.</p>
<blockquote>
<p><strong>Note:</strong> The <code>/etc/passwd</code> file is word-readable, but it's only writeable by
root to avoid other users tampering with it, such as by replacing an <code>x</code> with
an actual password.</p>
</blockquote>
<p>The <code>/etc/shadow</code> file presents a solution to this problem. It is readable only
by the root user, and contains the encrypted passwords of users with an <code>x</code> in
the password field of their <code>/etc/passwd</code> entry. The shadow file is technically
optional, but you will probably never find a system that doesn't use it.</p>
<p>I won't go into detail on how the shadow file is structured, as it's not a file
that's typically accessed by user space programs. If you want to know more about
it, you can read the manpage of <code>shadow (5)</code>.</p>
<p>And that's it for this article, hope you learned something!</p>Essential pytest pt. 1: Controlling the verbosity of output2020-07-31T20:07:56+00:002020-07-31T20:07:56+00:00Simon Larséntag:slar.se,2020-07-31:/essential-pytest-1.html<p>This is the first part of a series of small articles detailing some of the
functionality of the <a href="https://docs.pytest.org/en/latest/">pytest</a> testing
framework that I find most essential. The series assumes you know how to run
tests with <code>pytest</code> already.</p>
<p>In this first part, we'll take a look at the <code>-v</code> and …</p><p>This is the first part of a series of small articles detailing some of the
functionality of the <a href="https://docs.pytest.org/en/latest/">pytest</a> testing
framework that I find most essential. The series assumes you know how to run
tests with <code>pytest</code> already.</p>
<p>In this first part, we'll take a look at the <code>-v</code> and <code>--tb</code> options to control
the verbosity of the output.</p>
<h2>The test suite</h2>
<p>For the purposes of this article, I've implemented a very simple multiplication
function called <code>mul</code>, along with a few tests. Here's the entire thing, in a
file called <code>test_mul.py</code>:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># test_mul.py</span>
<span class="k">def</span> <span class="nf">mul</span><span class="p">(</span><span class="n">lhs</span><span class="p">,</span> <span class="n">rhs</span><span class="p">):</span>
<span class="k">return</span> <span class="n">lhs</span> <span class="o">*</span> <span class="n">lhs</span>
<span class="k">def</span> <span class="nf">test_multiply_equal_numbers</span><span class="p">():</span>
<span class="k">assert</span> <span class="n">mul</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">5</span><span class="p">)</span> <span class="o">==</span> <span class="mi">25</span>
<span class="k">def</span> <span class="nf">test_multiply_by_zero</span><span class="p">():</span>
<span class="k">assert</span> <span class="n">mul</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span>
<span class="k">def</span> <span class="nf">test_multiply_different_numbers</span><span class="p">():</span>
<span class="k">assert</span> <span class="n">mul</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> <span class="o">==</span> <span class="mi">15</span>
</code></pre></div>
<p>Obviously, the implementation of <code>mul</code> is broken, and running <code>pytest</code> gives
the following output.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nv">pytest</span>
<span class="o">==========================</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>session<span class="w"> </span><span class="nv">starts</span><span class="w"> </span><span class="o">===========================</span>
platform<span class="w"> </span>linux<span class="w"> </span>--<span class="w"> </span>Python<span class="w"> </span><span class="m">3</span>.8.3,<span class="w"> </span>pytest-5.4.3,<span class="w"> </span>py-1.9.0,<span class="w"> </span>pluggy-0.13.1
rootdir:<span class="w"> </span>/home/slarse/python
collected<span class="w"> </span><span class="m">3</span><span class="w"> </span>items<span class="w"> </span>
test_mul.py<span class="w"> </span>.FF<span class="w"> </span><span class="o">[</span><span class="m">100</span>%<span class="o">]</span>
<span class="o">================================</span><span class="w"> </span><span class="nv">FAILURES</span><span class="w"> </span><span class="o">================================</span>
_________________________<span class="w"> </span>test_multiply_by_zero<span class="w"> </span>__________________________
<span class="w"> </span>def<span class="w"> </span>test_multiply_by_zero<span class="o">()</span>:
><span class="w"> </span>assert<span class="w"> </span>mul<span class="o">(</span><span class="m">1</span>,<span class="w"> </span><span class="m">0</span><span class="o">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">0</span>
E<span class="w"> </span>assert<span class="w"> </span><span class="nv">1</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">0</span>
E<span class="w"> </span>+<span class="w"> </span>where<span class="w"> </span><span class="nv">1</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>mul<span class="o">(</span><span class="m">1</span>,<span class="w"> </span><span class="m">0</span><span class="o">)</span>
test_mul.py:8:<span class="w"> </span>AssertionError
____________________<span class="w"> </span>test_multiply_different_numbers<span class="w"> </span>_____________________
<span class="w"> </span>def<span class="w"> </span>test_multiply_different_numbers<span class="o">()</span>:
><span class="w"> </span>assert<span class="w"> </span>mul<span class="o">(</span><span class="m">5</span>,<span class="w"> </span><span class="m">3</span><span class="o">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">15</span>
E<span class="w"> </span>assert<span class="w"> </span><span class="nv">25</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">15</span>
E<span class="w"> </span>+<span class="w"> </span>where<span class="w"> </span><span class="nv">25</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>mul<span class="o">(</span><span class="m">5</span>,<span class="w"> </span><span class="m">3</span><span class="o">)</span>
test_mul.py:11:<span class="w"> </span><span class="nv">AssertionError</span>
<span class="o">========================</span><span class="w"> </span>short<span class="w"> </span><span class="nb">test</span><span class="w"> </span>summary<span class="w"> </span><span class="nv">info</span><span class="w"> </span><span class="o">=========================</span>
FAILED<span class="w"> </span>test_mul.py::test_multiply_by_zero<span class="w"> </span>-<span class="w"> </span>assert<span class="w"> </span><span class="nv">1</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">0</span>
FAILED<span class="w"> </span>test_mul.py::test_multiply_different_numbers<span class="w"> </span>-<span class="w"> </span>assert<span class="w"> </span><span class="nv">25</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="nv">15</span>
<span class="o">======================</span><span class="w"> </span><span class="m">2</span><span class="w"> </span>failed,<span class="w"> </span><span class="m">1</span><span class="w"> </span>passed<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span>.08s<span class="w"> </span><span class="o">=======================</span>
</code></pre></div>
<p>Let's learn how to control how much of what we see here.</p>
<h2>Using the <code>--tb</code> option to control traceback verbosity</h2>
<p>Most of what you're seeing in the output of the previous section is the
<em>traceback</em> information. While the traceback shown above is manageable as is,
consider that it stems from a single-line function and single-line tests. With
that in mind, it's actually pretty freaking verbose. We can show less of it by
using the <code>--tb</code> option. We can even shut it off completely.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>--tb<span class="o">=</span><span class="nv">no</span>
<span class="o">==========================</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>session<span class="w"> </span><span class="nv">starts</span><span class="w"> </span><span class="o">===========================</span>
platform<span class="w"> </span>linux<span class="w"> </span>--<span class="w"> </span>Python<span class="w"> </span><span class="m">3</span>.8.3,<span class="w"> </span>pytest-5.4.3,<span class="w"> </span>py-1.9.0,<span class="w"> </span>pluggy-0.13.1
rootdir:<span class="w"> </span>/home/slarse/python
collected<span class="w"> </span><span class="m">3</span><span class="w"> </span>items<span class="w"> </span>
test_mul.py<span class="w"> </span>.FF<span class="w"> </span><span class="o">[</span><span class="m">100</span>%<span class="o">]</span>
<span class="o">========================</span><span class="w"> </span>short<span class="w"> </span><span class="nb">test</span><span class="w"> </span>summary<span class="w"> </span><span class="nv">info</span><span class="w"> </span><span class="o">=========================</span>
FAILED<span class="w"> </span>test_mul.py::test_multiply_by_zero<span class="w"> </span>-<span class="w"> </span>assert<span class="w"> </span><span class="nv">1</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">0</span>
FAILED<span class="w"> </span>test_mul.py::test_multiply_different_numbers<span class="w"> </span>-<span class="w"> </span>assert<span class="w"> </span><span class="nv">25</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="nv">15</span>
<span class="o">======================</span><span class="w"> </span><span class="m">2</span><span class="w"> </span>failed,<span class="w"> </span><span class="m">1</span><span class="w"> </span>passed<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span>.02s<span class="w"> </span><span class="o">=======================</span>
</code></pre></div>
<p>This is useful when you're just trying to figure out what tests are failing, and
when test output is just entirely overwhelming. I find myself using it quite
frequently. Another useful traceback value is <code>line</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>--tb<span class="o">=</span><span class="nv">line</span>
<span class="o">==========================</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>session<span class="w"> </span><span class="nv">starts</span><span class="w"> </span><span class="o">===========================</span>
platform<span class="w"> </span>linux<span class="w"> </span>--<span class="w"> </span>Python<span class="w"> </span><span class="m">3</span>.8.3,<span class="w"> </span>pytest-5.4.3,<span class="w"> </span>py-1.9.0,<span class="w"> </span>pluggy-0.13.1
rootdir:<span class="w"> </span>/home/slarse/python
collected<span class="w"> </span><span class="m">3</span><span class="w"> </span>items<span class="w"> </span>
test_mul.py<span class="w"> </span>.FF<span class="w"> </span><span class="o">[</span><span class="m">100</span>%<span class="o">]</span>
<span class="o">================================</span><span class="w"> </span><span class="nv">FAILURES</span><span class="w"> </span><span class="o">================================</span>
/home/slarse/python/test_mul.py:8:<span class="w"> </span>assert<span class="w"> </span><span class="nv">1</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">0</span>
/home/slarse/python/test_mul.py:11:<span class="w"> </span>assert<span class="w"> </span><span class="nv">25</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="nv">15</span>
<span class="o">========================</span><span class="w"> </span>short<span class="w"> </span><span class="nb">test</span><span class="w"> </span>summary<span class="w"> </span><span class="nv">info</span><span class="w"> </span><span class="o">=========================</span>
FAILED<span class="w"> </span>test_mul.py::test_multiply_by_zero<span class="w"> </span>-<span class="w"> </span>assert<span class="w"> </span><span class="nv">1</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">0</span>
FAILED<span class="w"> </span>test_mul.py::test_multiply_different_numbers<span class="w"> </span>-<span class="w"> </span>assert<span class="w"> </span><span class="nv">25</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="nv">15</span>
<span class="o">======================</span><span class="w"> </span><span class="m">2</span><span class="w"> </span>failed,<span class="w"> </span><span class="m">1</span><span class="w"> </span>passed<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span>.03s<span class="w"> </span><span class="o">=======================</span>
</code></pre></div>
<p>This lets us see the exact lines where the test failures occurred. In this case,
it shows the lines of the assertions, but it could also for example show the
line where an exception was raised.</p>
<p>Another one that I find really useful is <code>--tb=short</code>. It shows the full
traceback, but with much less context around each function call. It won't
make much of a difference for this short a traceback, but it makes a world of
difference for deeply nested function calls.</p>
<p>There are more ways to manipulate the traceback, but these are the two I use the
most, aside from the default. To see the other options, refer to <code>pytest -h</code> and
look for the <code>--tb</code> option.</p>
<h2>Using <code>-v</code> to show more verbose test output</h2>
<p>The <code>-v</code> option controls the verbosity of test output while the tests are
running, and also the verbosity of single items in the traceback. It's really
useful when tests take a long time to run, and you want to know approximately
where you're at.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>--tb<span class="o">=</span>no<span class="w"> </span>-v
<span class="o">==========================</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>session<span class="w"> </span><span class="nv">starts</span><span class="w"> </span><span class="o">===========================</span>
platform<span class="w"> </span>linux<span class="w"> </span>--<span class="w"> </span>Python<span class="w"> </span><span class="m">3</span>.8.3,<span class="w"> </span>pytest-5.4.3,<span class="w"> </span>py-1.9.0,<span class="w"> </span>pluggy-0.13.1<span class="w"> </span>--<span class="w"> </span>/usr/bin/python
cachedir:<span class="w"> </span>.pytest_cache
rootdir:<span class="w"> </span>/home/slarse/python
collected<span class="w"> </span><span class="m">3</span><span class="w"> </span>items<span class="w"> </span>
test_mul.py::test_multiply_equal_numbers<span class="w"> </span>PASSED<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="m">33</span>%<span class="o">]</span>
test_mul.py::test_multiply_by_zero<span class="w"> </span>FAILED<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="m">66</span>%<span class="o">]</span>
test_mul.py::test_multiply_different_numbers<span class="w"> </span>FAILED<span class="w"> </span><span class="o">[</span><span class="m">100</span>%<span class="o">]</span>
<span class="o">========================</span><span class="w"> </span>short<span class="w"> </span><span class="nb">test</span><span class="w"> </span>summary<span class="w"> </span><span class="nv">info</span><span class="w"> </span><span class="o">=========================</span>
FAILED<span class="w"> </span>test_mul.py::test_multiply_by_zero<span class="w"> </span>-<span class="w"> </span>assert<span class="w"> </span><span class="nv">1</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">0</span>
FAILED<span class="w"> </span>test_mul.py::test_multiply_different_numbers<span class="w"> </span>-<span class="w"> </span>assert<span class="w"> </span><span class="nv">25</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="nv">15</span>
<span class="o">======================</span><span class="w"> </span><span class="m">2</span><span class="w"> </span>failed,<span class="w"> </span><span class="m">1</span><span class="w"> </span>passed<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span>.03s<span class="w"> </span><span class="o">=======================</span>
</code></pre></div>
<p>Note how each test is now shown on a line of its own, as opposed to just <code>.</code> and
<code>F</code> in the previous runs. The lines show up as the tests are running, and I find
it useful to track long-running tests.</p>
<p>But what about that "single-item" verbosity that I mentioned? When there are
single items in the traceback that are very large, such as a list of say 1000
elements, then pytest will truncate them by default. To demnstrate, consider
this single (pointless) test:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># test_truncation.py</span>
<span class="kn">import</span> <span class="nn">pytest</span>
<span class="k">def</span> <span class="nf">test_truncation_demonstration</span><span class="p">():</span>
<span class="k">assert</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span> <span class="o">==</span> <span class="nb">list</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">1000</span><span class="p">))</span>
</code></pre></div>
<p>Running this test will yield a traceback that looks something like this.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest
<span class="o">[</span>...<span class="w"> </span>OMITTED<span class="w"> </span>...<span class="o">]</span>
______________________<span class="w"> </span>test_truncation_demonstration<span class="w"> </span>_____________________
<span class="w"> </span>def<span class="w"> </span>test_truncation_demonstration<span class="o">()</span>:
<span class="w"> </span>><span class="w"> </span>assert<span class="w"> </span><span class="o">[</span><span class="m">0</span>,<span class="w"> </span><span class="m">1</span>,<span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="m">3</span><span class="o">]</span><span class="w"> </span><span class="o">==</span><span class="w"> </span>list<span class="o">(</span>range<span class="o">(</span><span class="m">1000</span><span class="o">))</span>
<span class="w"> </span>E<span class="w"> </span>assert<span class="w"> </span><span class="o">[</span><span class="m">0</span>,<span class="w"> </span><span class="m">1</span>,<span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="m">3</span><span class="o">]</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="o">[</span><span class="m">0</span>,<span class="w"> </span><span class="m">1</span>,<span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="m">3</span>,<span class="w"> </span><span class="m">4</span>,<span class="w"> </span><span class="m">5</span>,<span class="w"> </span>...<span class="o">]</span>
<span class="w"> </span>E<span class="w"> </span>Right<span class="w"> </span>contains<span class="w"> </span><span class="m">996</span><span class="w"> </span>more<span class="w"> </span>items,<span class="w"> </span>first<span class="w"> </span>extra<span class="w"> </span>item:<span class="w"> </span><span class="m">4</span>
<span class="w"> </span>E<span class="w"> </span>Use<span class="w"> </span>-v<span class="w"> </span>to<span class="w"> </span>get<span class="w"> </span>the<span class="w"> </span>full<span class="w"> </span>diff
<span class="w"> </span>truncation.py:4:<span class="w"> </span>AssertionError
</code></pre></div>
<p>Note how the long list has been truncated such that only the first few items are
shown. Note also how pytest is suggesting the use of <code>-v</code>. If we supply <code>-v</code>, it
actually still doesn't show the whole list.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>-v
<span class="o">[</span>...<span class="w"> </span>OMITTED<span class="w"> </span>...<span class="o">]</span>
______________________<span class="w"> </span>test_truncation_demonstration<span class="w"> </span>_____________________
<span class="w"> </span>def<span class="w"> </span>test_truncation_demonstration<span class="o">()</span>:
<span class="w"> </span>><span class="w"> </span>assert<span class="w"> </span><span class="o">[</span><span class="m">0</span>,<span class="w"> </span><span class="m">1</span>,<span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="m">3</span><span class="o">]</span><span class="w"> </span><span class="o">==</span><span class="w"> </span>list<span class="o">(</span>range<span class="o">(</span><span class="m">1000</span><span class="o">))</span>
<span class="w"> </span>E<span class="w"> </span>AssertionError:<span class="w"> </span>assert<span class="w"> </span><span class="o">[</span><span class="m">0</span>,<span class="w"> </span><span class="m">1</span>,<span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="m">3</span><span class="o">]</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="o">[</span><span class="m">0</span>,<span class="w"> </span><span class="m">1</span>,<span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="m">3</span>,<span class="w"> </span><span class="m">4</span>,<span class="w"> </span><span class="m">5</span>,<span class="w"> </span>...<span class="o">]</span>
<span class="w"> </span>E<span class="w"> </span>Right<span class="w"> </span>contains<span class="w"> </span><span class="m">996</span><span class="w"> </span>more<span class="w"> </span>items,<span class="w"> </span>first<span class="w"> </span>extra<span class="w"> </span>item:<span class="w"> </span><span class="m">4</span>
<span class="w"> </span>E<span class="w"> </span>Full<span class="w"> </span>diff:
<span class="w"> </span>E<span class="w"> </span><span class="o">[</span>
<span class="w"> </span>E<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span>E<span class="w"> </span><span class="m">1</span>,
<span class="w"> </span>E<span class="w"> </span><span class="m">2</span>,
<span class="w"> </span>E<span class="w"> </span><span class="m">3</span>,...
<span class="w"> </span>E<span class="w"> </span>
<span class="w"> </span>E<span class="w"> </span>...Full<span class="w"> </span>output<span class="w"> </span>truncated<span class="w"> </span><span class="o">(</span><span class="m">998</span><span class="w"> </span>lines<span class="w"> </span>hidden<span class="o">)</span>,<span class="w"> </span>use<span class="w"> </span><span class="s1">'-vv'</span><span class="w"> </span>to<span class="w"> </span>show
<span class="w"> </span>truncation.py:4:<span class="w"> </span>AssertionError
</code></pre></div>
<p>In fact, pytest isn't even showing more output here, it just shows the same
things more verbosely. If we stack <code>-v</code> twice, i.e. <code>-vv</code> or <code>-v -v</code>, then we
get the full output. I find <code>-v</code> on its own to be rather useless, and typically
always supply <code>-vv</code> if I want verbose output. For obvious reasons, I will not
show what that output looks like.</p>
<h2>Summary</h2>
<p>In this article, we had a look at the <code>--tb</code> and <code>-v</code> options to control output
verbosity in pytest. <code>--tb</code> controls the overall size of the traceback, and can
be supplied with values like <code>no</code> for no output at all, <code>line</code> for just a single
line of traceback, or <code>short</code> for complete traceback with shortened context.
<code>-v</code> controls the verbosity of running tests and single items in the traceback,
such as very long lists which are truncated by default. Typically, <code>-vv</code> is
required to get fully verbose output.</p>
<p>And that's about it, I hope you learned something!</p>Don't use String for method options, use an enum!2019-11-10T18:56:00+01:002019-11-10T18:56:00+01:00Simon Larséntag:slar.se,2019-11-10:/dont-use-string-for-method-options-use-an-enum.html<p>In this article, we are going to have a look at a method that accepts an option.
That is to say, it accepts an argument that somehow decides how it operates. If
you use a lot of libraries in your day-to-day programming, you're bound to come
across methods that accept …</p><p>In this article, we are going to have a look at a method that accepts an option.
That is to say, it accepts an argument that somehow decides how it operates. If
you use a lot of libraries in your day-to-day programming, you're bound to come
across methods that accept <code>String</code> values as such options, and you've probably
been infuriated by you misspelling the options, or just trying to figure out
what options there are in the first place. That suggests that there must be a
better solution, and as you may have figured out by now,
<a href="https://docs.oracle.com/javase/tutorial/java/javaOO/enum.html">enums</a> is that
solution.</p>
<blockquote>
<p><strong>Note:</strong> This article discusses enums in Java, but the very same arguments
are valid for any language that has support for enum types, or something
comparable.</p>
</blockquote>
<h2>The problem: What options do I have?</h2>
<p>Consider the following method that formats a <code>String</code> according to an option
supplied as another <code>String</code>:</p>
<blockquote>
<p>Yes, this is a somewhat contrived example, but bear with me!</p>
</blockquote>
<div class="highlight"><pre><span></span><code><span class="kd">public</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="n">String</span><span class="w"> </span><span class="nf">format</span><span class="p">(</span><span class="n">String</span><span class="w"> </span><span class="n">str</span><span class="p">,</span><span class="w"> </span><span class="n">String</span><span class="w"> </span><span class="n">option</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">switch</span><span class="w"> </span><span class="p">(</span><span class="n">option</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="s">"upper"</span><span class="p">:</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">str</span><span class="p">.</span><span class="na">toUpperCase</span><span class="p">();</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="s">"lower"</span><span class="p">:</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">str</span><span class="p">.</span><span class="na">toLowerCase</span><span class="p">();</span>
<span class="w"> </span><span class="k">default</span><span class="p">:</span>
<span class="w"> </span><span class="k">throw</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">IllegalStateException</span><span class="p">(</span><span class="s">"Internal errror, unmatched option "</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">option</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>We could then use the method something like this:</p>
<div class="highlight"><pre><span></span><code><span class="n">format</span><span class="p">(</span><span class="s">"hello"</span><span class="p">,</span><span class="w"> </span><span class="s">"upper"</span><span class="p">);</span>
</code></pre></div>
<p>Can you see any problems with this? Well first of all, if you don't have access
to the source, how do you know which options can be passed? There are an
infinite amount of strings, after all. At best, the Javadoc will say precisely
which values are valid options, <a href="https://docs.oracle.com/javase/8/docs/api/java/nio/file/Files.html#getAttribute-java.nio.file.Path-java.lang.String-java.nio.file.LinkOption...-">but that is not always the case even in the
Java standard
library</a>.
But even if all of the options are clearly documented at one point, it would be
so easy for a developer to add or remove an option, and forget to enact the
corresponding change in the Javadoc. It is also difficult to have automatic
checks that actually verify that all possible options are documented. And even
assuming that all options are properly documented at all times, the compiler
can't distinguish which <code>String</code> values are valid and which are not, so a user
misspelling an option won't know until runtime.</p>
<h2>The solution: enums!</h2>
<p>My goal here is not to explain the ins and outs of what enums are, but rather
show a use case. In short, an enum is a data type with a (typically very)
limited amount of possible values (<a href="https://docs.oracle.com/javase/tutorial/java/javaOO/enum.html">you can read more about it
here</a>).
Now, let's instead define this enum type:</p>
<div class="highlight"><pre><span></span><code><span class="kd">public</span><span class="w"> </span><span class="kd">enum</span><span class="w"> </span><span class="n">FormatOption</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">UPPER</span><span class="p">,</span>
<span class="w"> </span><span class="n">LOWER</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>And refactor the method with it:</p>
<div class="highlight"><pre><span></span><code><span class="kd">public</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="n">String</span><span class="w"> </span><span class="nf">format</span><span class="p">(</span><span class="n">String</span><span class="w"> </span><span class="n">str</span><span class="p">,</span><span class="w"> </span><span class="n">FormatOption</span><span class="w"> </span><span class="n">option</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">switch</span><span class="w"> </span><span class="p">(</span><span class="n">option</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="n">UPPER</span><span class="p">:</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">str</span><span class="p">.</span><span class="na">toUpperCase</span><span class="p">();</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="n">LOWER</span><span class="p">:</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">str</span><span class="p">.</span><span class="na">toLowerCase</span><span class="p">();</span>
<span class="w"> </span><span class="k">default</span><span class="p">:</span>
<span class="w"> </span><span class="k">throw</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">IllegalStateException</span><span class="p">(</span><span class="s">"Internal errror, unmatched option "</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">option</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>This method can then be used like:</p>
<div class="highlight"><pre><span></span><code><span class="n">format</span><span class="p">(</span><span class="s">"hello"</span><span class="p">,</span><span class="w"> </span><span class="n">FormatOption</span><span class="p">.</span><span class="na">UPPER</span><span class="p">);</span>
</code></pre></div>
<p>With this small alteration, we have eliminated <em>all</em> of the problems mentioned
before. The possible values for the <code>option</code> argument are now self-documented
in the <code>FormatOption</code> enum. Additionally, any modern IDE will kindly list the
possible values when you type <code>FormatOption.</code>, such that a programmer does not
even necessarily need to consult the documentation, assuming that the enum
values have descriptive enough names. The compiler can also distinguish between
<code>FormatOption.UPPER</code> and a misspelling such as <code>FormatOption.UPER</code>, as the
latter is not defined, so runtime errors due to invalid options is no longer a
problem.</p>
<h2>What's the catch?</h2>
<p>So what's the catch? Well, if you have many methods like this, you'll end up
with a lot of enum types. Personally, I think that's totally worth it, and you
could also nest the enums inside the classes that use them to reduce their
overall footprint in the project. The whole thing could then look like this:</p>
<div class="highlight"><pre><span></span><code><span class="kd">public</span><span class="w"> </span><span class="kd">class</span> <span class="nc">Formatter</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">public</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="kd">enum</span><span class="w"> </span><span class="n">Option</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">UPPER</span><span class="p">,</span>
<span class="w"> </span><span class="n">LOWER</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="kd">public</span><span class="w"> </span><span class="kd">static</span><span class="w"> </span><span class="n">String</span><span class="w"> </span><span class="nf">format</span><span class="p">(</span><span class="n">String</span><span class="w"> </span><span class="n">str</span><span class="p">,</span><span class="w"> </span><span class="n">Option</span><span class="w"> </span><span class="n">option</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">String</span><span class="w"> </span><span class="n">result</span><span class="p">;</span>
<span class="w"> </span><span class="k">switch</span><span class="w"> </span><span class="p">(</span><span class="n">option</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="n">UPPER</span><span class="p">:</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">str</span><span class="p">.</span><span class="na">toUpperCase</span><span class="p">();</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="n">LOWER</span><span class="p">:</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">str</span><span class="p">.</span><span class="na">toLowerCase</span><span class="p">();</span>
<span class="w"> </span><span class="k">default</span><span class="p">:</span>
<span class="w"> </span><span class="k">throw</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">IllegalStateException</span><span class="p">(</span><span class="s">"Internal errror, unmatched option "</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">option</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>Didn't add that much complexity now, did it?</p>
<h2>Summary</h2>
<p>If you have a method that you want to pass options to, <em>use enums</em>.
That's really all there is to it. Enums are quite widely used in the Java
standard library as well, such as
<a href="https://docs.oracle.com/javase/8/docs/api/java/nio/file/StandardCopyOption.html">StandardCopyOption</a>,
<a href="https://docs.oracle.com/javase/8/docs/api/java/nio/file/StandardOpenOption.html">StandardOpenOption</a> and
<a href="https://docs.oracle.com/javase/8/docs/api/java/nio/file/LinkOption.html">LinkOption</a>
in the <a href="https://docs.oracle.com/javase/8/docs/api/java/nio/file/package-summary.html"><code>java.nio.file</code>
API</a>,
which are used much in the same way as I used the enum in this article.
Hopefully, having read this article, you won't be creating any more APIs that
accept <code>String</code> options!</p>Java's Optional: Why you should prefer it over null2019-10-11T16:57:00+02:002019-10-11T16:57:00+02:00Simon Larséntag:slar.se,2019-10-11:/javas-optional-why-you-should-prefer-it-over-null.html<p>Null references are problematic, to say the least. Tony Hoare (inventor of the
null reference) even went as far to say call them his <a href="https://en.wikipedia.org/wiki/Tony_Hoare#Apologies_and_retractions">"billion dollar
mistake"</a>.
In this article, I first make a cursory exploration of why null references are
so problematic, and then have a look at Java's …</p><p>Null references are problematic, to say the least. Tony Hoare (inventor of the
null reference) even went as far to say call them his <a href="https://en.wikipedia.org/wiki/Tony_Hoare#Apologies_and_retractions">"billion dollar
mistake"</a>.
In this article, I first make a cursory exploration of why null references are
so problematic, and then have a look at Java's proposed solution: the
<a href="https://docs.oracle.com/javase/8/docs/api/java/util/Optional.html"><code>Optional<T></code> class</a>.</p>
<h1>Why null is problematic</h1>
<p>There are many reasons why null is problematic, but there are a few that are
particularly easy to illustrate. I will be using the
<a href="https://docs.oracle.com/javase/8/docs/api/java/util/Map.html#get(java.lang.Object)"><code>Map.get</code></a>
method as an example, as it returns null if the key provided to it is not in
the map. For all of the examples, assume that there is a variable <code>Map<Integer,
String> map</code> in the current scope.</p>
<h2>null circumvents the type system</h2>
<p>Java has a fairly rigorous type system (although it's not entirely sound, I may
do an article on that later!). The type system can't help with null, however.
Any variable of reference type can either contain a reference to an object of
that type, or null. This leads to a whole lot of code that looks like this:</p>
<div class="highlight"><pre><span></span><code><span class="n">String</span><span class="w"> </span><span class="n">value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">map</span><span class="p">.</span><span class="na">get</span><span class="p">(</span><span class="mi">10</span><span class="p">);</span>
<span class="n">String</span><span class="w"> </span><span class="n">valueUpper</span><span class="p">;</span>
<span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">value</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">null</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">valueUpper</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">value</span><span class="p">.</span><span class="na">toUpperCase</span><span class="p">();</span>
<span class="p">}</span>
<span class="c1">// else do something different with the knowledge that value is null</span>
</code></pre></div>
<p>While you may argue that null checks are pretty ugly, the real problem is that
they are not enforced by the type system. The above might just as well have
been written like this, and the type checker would have been none the wiser:</p>
<div class="highlight"><pre><span></span><code><span class="n">String</span><span class="w"> </span><span class="n">value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">map</span><span class="p">.</span><span class="na">get</span><span class="p">(</span><span class="mi">10</span><span class="p">);</span>
<span class="n">String</span><span class="w"> </span><span class="n">valueUpper</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">value</span><span class="p">.</span><span class="na">toUpperCase</span><span class="p">();</span>
</code></pre></div>
<p>This program could crash with a <code>NullPointerException</code>. That's not great. But
it can be even worse. What if the call to <code>value.toUpperCase()</code> occurs in an
entirely different part of the program, perhaps hours or even days after
<code>Map.get</code> returned null? Then, not only do you have a crash, but a crash that
can potentially be very difficult to diagnose.</p>
<h2>It's easy to miss that a method can return null</h2>
<p>The cause of the previous problem is often that it's not always obvious that a
method may return null. If you have a look at the <a href="https://github.com/openjdk/jdk/blob/0dbfc97c05218ffd10242901d73c0715ccb53bf3/src/java.base/share/classes/java/util/Map.java#L217-L242">documentation for
Map.get</a>,
you'll see that it says that it may return null a little here and there, and
it's pretty clear about that. But still, a careless developer might miss this,
and there's also the fact that many methods are <em>not</em> this well documented.</p>
<h2>Why not just throw an exception instead?</h2>
<p>One question you may ask is, why even return null, why not just throw an
exception? Indeed, throwing an exception may be a good solution in many cases,
but sometimes it just isn't desirable. In the case of <code>Map.get</code>, it's mostly
about efficiency. If <code>Map.get</code> were to throw an exception when the key was
missing, you'd essentially have two alternatives. </p>
<h3>Ask for forgiveness (catch the exception)</h3>
<div class="highlight"><pre><span></span><code><span class="n">String</span><span class="w"> </span><span class="n">value</span><span class="p">;</span>
<span class="k">try</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">map</span><span class="p">.</span><span class="na">get</span><span class="p">(</span><span class="mi">10</span><span class="p">);</span>
<span class="p">}</span><span class="w"> </span><span class="k">catch</span><span class="w"> </span><span class="p">(</span><span class="n">NoSuchElementException</span><span class="w"> </span><span class="n">e</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">// handle error</span>
<span class="p">}</span>
</code></pre></div>
<p>That's both ugly, and very inefficient if it is often the case that the key is
missing. Catching an exception involves a whole lot of work for the JVM, so you
really do not want to do this for an operation that you often perform.</p>
<h3>Look before you leap</h3>
<div class="highlight"><pre><span></span><code><span class="n">String</span><span class="w"> </span><span class="n">value</span><span class="p">;</span>
<span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">map</span><span class="p">.</span><span class="na">containsKey</span><span class="p">(</span><span class="mi">10</span><span class="p">))</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">map</span><span class="p">.</span><span class="na">get</span><span class="p">(</span><span class="mi">10</span><span class="p">);</span>
<span class="p">}</span>
<span class="c1">// else handle missing keys</span>
</code></pre></div>
<p>This is not too ugly, but it is inefficient as the map has to be traversed twice:
once to check if it contains the key, and once more to fetch the value
associated with the key. Throwing an exception becomes even more undesirable if
you're using
<a href="https://docs.oracle.com/javase/8/docs/api/java/util/stream/Stream.html">Streams</a>.
So, I think we can safely conclude that throwing an exception is not the
be-all-end-all solution.</p>
<h1><code>Optional<T></code> to the rescue</h1>
<p>Using the <a href="https://docs.oracle.com/javase/8/docs/api/java/util/Optional.html"><code>Optional<T></code>
class</a>, we
solve the issues discussed previously. <code>Optional</code> is simply a container for
another object that may or may not be null.
<a href="https://docs.oracle.com/javase/8/docs/api/java/util/Optional.html#get--"><code>Optional.get</code></a>
returns the contained object if present, or throws a <code>NoSuchElementException</code> if
it is not (i.e. it is null).
<a href="https://docs.oracle.com/javase/8/docs/api/java/util/Optional.html#isPresent--"><code>Optional.isPresent</code></a>
lets us check first if the value is present, to avoid an exception if it is not.
<strong>Let's pretend</strong> like there's a method <code>Map.getOptional</code> that returns an
<code>Optional<T></code> instead of just <code>T</code>. We then have several options.</p>
<blockquote>
<p><strong>Important:</strong> There is no <code>Map.getOptional</code> method, this is just
hypothetical. We'll see toward the end of the article how one can wrap
<code>Map.get</code> to create a <code>getOptional</code> method.</p>
</blockquote>
<h2>Retrieve the value without checking if it is present</h2>
<p>If you're certain that the value will be present, you may simply retrieve it
immediately.</p>
<div class="highlight"><pre><span></span><code><span class="n">Optional</span><span class="o"><</span><span class="n">String</span><span class="o">></span><span class="w"> </span><span class="n">opt</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">map</span><span class="p">.</span><span class="na">getOptional</span><span class="p">(</span><span class="mi">10</span><span class="p">);</span>
<span class="n">String</span><span class="w"> </span><span class="n">value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">opt</span><span class="p">.</span><span class="na">get</span><span class="p">();</span><span class="w"> </span><span class="c1">// `value` will either be non-null, or we crash</span>
</code></pre></div>
<p>This may at first glance just look like more work than previously, two <code>get</code>
calls instead of one. The benefit here is that the programmer is extremely
unlikely to miss the fact that the returned value may not be present, as the
return value itself has the type <code>Optional<String></code>, and the type system will
scream bloody murder if they try to use the <code>Optional<String></code> value like a
<code>String</code>. Requiring that extra <code>get</code> <em>forces</em> the programmer to make a conscious
choice of how to handle errors. Now, this may crash with a
<code>NoSuchElementException</code>, but as <code>Optional</code> values typically are not passed
around, the is unlikely to happen far from where the <code>Optional</code> was produced.</p>
<h2>Check if the object is present</h2>
<p>If you're uncertain whether the value will be present, you may simply check for
it:</p>
<div class="highlight"><pre><span></span><code><span class="n">Optional</span><span class="o"><</span><span class="n">String</span><span class="o">></span><span class="w"> </span><span class="n">opt</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">map</span><span class="p">.</span><span class="na">getOptional</span><span class="p">(</span><span class="mi">10</span><span class="p">);</span>
<span class="n">String</span><span class="w"> </span><span class="n">value</span><span class="p">;</span>
<span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">opt</span><span class="p">.</span><span class="na">isPresent</span><span class="p">())</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">opt</span><span class="p">.</span><span class="na">get</span><span class="p">();</span>
<span class="p">}</span>
</code></pre></div>
<p>This is very much like the null check we had before, but it's supported by the
type system.</p>
<h2>Use a fallback value</h2>
<p>Often when we don't want the code to crash, we have a fallback value. <code>Optional</code>
has a method to handle that.</p>
<div class="highlight"><pre><span></span><code><span class="n">String</span><span class="w"> </span><span class="n">value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">map</span><span class="p">.</span><span class="na">getOptional</span><span class="p">(</span><span class="mi">10</span><span class="p">).</span><span class="na">orElse</span><span class="p">(</span><span class="s">"Nothing here :)"</span><span class="p">);</span>
</code></pre></div>
<p>If the value is present, it is returned. Otherwise, we get <code>"Nothing here :)"</code>.
This, I think, is one of the cleanest and tidiest uses of <code>Optional</code>.</p>
<h2>What about drawbacks?</h2>
<p>Of course, <code>Optional</code> has drawbacks. As every single object is wrapped in
another object, there will be an increase in memory consumption. The extra
method call <em>may</em> result in a noticeable performance penalty, but I don't dare
say anything concrete about that without running some tests (the JVM is pretty
darn good at inlining and optimizing). Perhaps, your most performance critical
segments of code should not use <code>Optional</code>. But the vast majority of your code
is not performance critical, so most often it will be a moot point. Finally,
there's also a little bit of extra boilerplate to deal with, which also may not
be desirable.</p>
<h1>Alright, I'm sold, how do I <code>Optional</code>?</h1>
<p>We've already had a look at how to consume <code>Optional</code>s. But how do we produce
them? It's really quite easy. The three most important methods are:</p>
<ol>
<li><a href="https://docs.oracle.com/javase/8/docs/api/java/util/Optional.html#of-T-"><code>static <T> Optional.of(T value)</code></a><ul>
<li>A static method that wraps a value in an <code>Optional</code> instance. Throws an
exception if <code>value</code> is <code>null</code>.</li>
</ul>
</li>
<li><a href="https://docs.oracle.com/javase/8/docs/api/java/util/Optional.html#empty--"><code>static <T> Optional.empty()</code></a><ul>
<li>A static method that returns an empty <code>Optional</code> instance.</li>
</ul>
</li>
<li><a href="https://docs.oracle.com/javase/8/docs/api/java/util/Optional.html#ofNullable-T-"><code>static <T> Optional.ofNullable(T value)</code></a><ul>
<li>A static method that wraps a value in an <code>Optional</code>. The value may be
<code>null</code>, which essentially produces an empty <code>Optional</code>.</li>
</ul>
</li>
</ol>
<p>The <code>of</code> and <code>empty</code> methods are the ones you want to produce <code>Optional</code> values
from scratch. Here's a useless but simple example: an identity function for
integer values that is only defined for <code>n >= 0</code>. Using null, it would look
like this.</p>
<div class="highlight"><pre><span></span><code><span class="cm">/**</span>
<span class="cm"> * @param n An Integer value.</span>
<span class="cm"> * @return n iff n >= 0, otherwise null</span>
<span class="cm"> */</span>
<span class="kd">public</span><span class="w"> </span><span class="n">Integer</span><span class="w"> </span><span class="nf">id</span><span class="p">(</span><span class="n">Integer</span><span class="w"> </span><span class="n">n</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">n</span><span class="w"> </span><span class="o">>=</span><span class="w"> </span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">n</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="kc">null</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>This comes with all of the previously discussed problems associated with null
return values. Here's the equivalent method using <code>Optional</code>.</p>
<div class="highlight"><pre><span></span><code><span class="cm">/**</span>
<span class="cm"> * @param n An Integer value.</span>
<span class="cm"> * @return An Optional with n iff n >= 0, otherwise an empty Optional.</span>
<span class="cm"> */</span>
<span class="kd">public</span><span class="w"> </span><span class="n">Optional</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span><span class="w"> </span><span class="nf">id</span><span class="p">(</span><span class="n">Integer</span><span class="w"> </span><span class="n">n</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">n</span><span class="w"> </span><span class="o">>=</span><span class="w"> </span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">Optional</span><span class="p">.</span><span class="na">of</span><span class="p">(</span><span class="n">n</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">Optional</span><span class="p">.</span><span class="na">empty</span><span class="p">();</span>
<span class="p">}</span>
</code></pre></div>
<p>Notice how both the documentation, and the return value itself, clearly states
that the value returned from the method may not be present. It is more or less
impossible to miss that this method may return an empty value (as long as you
know what <code>Optional</code> is, that is).</p>
<p>The <code>ofNullable</code> method is great for wrapping existing methods that may return
null. For example, assuming that we have the <code>Map<Integer, String> map</code> field
from earlier, we can wrap its <code>get</code> method in our own <code>getOptional</code>.</p>
<div class="highlight"><pre><span></span><code><span class="kd">public</span><span class="w"> </span><span class="nf">getOptional</span><span class="p">(</span><span class="n">Integer</span><span class="w"> </span><span class="n">key</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">String</span><span class="w"> </span><span class="n">value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">map</span><span class="p">.</span><span class="na">get</span><span class="p">(</span><span class="n">key</span><span class="p">);</span><span class="w"> </span><span class="c1">// might be null!</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">Optional</span><span class="p">.</span><span class="na">ofNullable</span><span class="p">(</span><span class="n">value</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div>
<p>This lets us easily create APIs that contain two versions of, for example, a
getter method: one that returns <code>T</code>, and one that returns <code>Optional<T></code>. And
that's all the essentials. Not that hard, right?</p>
<h1>Summary</h1>
<p><code>Optional</code> solves most of the problems with null references in a mostly
elegant way. The most important thing with <code>Optional</code> is that it is a strong
form of documentation, which states both to the programmer and to the compiler
that the value asked for may be present. There's also the fact that
<code>Optional</code> provides both the null-check approach using <code>isPresent</code>, and the
exception-throwing approach by calling <code>get</code> without checking for presence. As
such, the caller of a method gets to decide which of these approaches to use,
increasing flexibility. <code>Optional</code> is also a critical part of the <a href="https://docs.oracle.com/javase/8/docs/api/java/util/stream/Stream.html">Stream
API</a>,
which would be forced to throw exceptions left and right without it (and you'd
be forced to catch them!). Although the use of <code>Optional</code> may incur a
performance penalty, it is trivial to provide two versions of performance
critical methods: one that returns an <code>Optional<T></code> and one that just returns
<code>T</code>. If you want to learn more about <code>Optional</code>, I recommend first checking out
<a href="https://docs.oracle.com/javase/8/docs/api/java/util/Optional.html">the API
documentation</a>.
I also encourage having a look at the <a href="https://github.com/openjdk/jdk/blob/a95a39a04e066548764e15bfc793a6c242a22bb7/src/java.base/share/classes/java/util/Optional.java">source
code for
<code>Optional</code></a>,
it's a surprisingly simple class that provides all of this functionality!</p>RepoBee (and Simon) at ITiCSE 2019!2019-07-22T23:02:00+02:002019-04-10T23:20:00+02:00Simon Larséntag:slar.se,2019-07-22:/repobee-and-simon-at-iticse-2019.html<p>I just got home from the <em>Innovation and Technology in Computer Science</em>
(ITiCSE) conference in lovely Aberdeen, Scotland. I was there to present a small
experience paper on developing and using <a href="https://repobee.org">RepoBee</a>, an open
source tool for generating and managing large amounts of Git repositories for
students in higher education …</p><p>I just got home from the <em>Innovation and Technology in Computer Science</em>
(ITiCSE) conference in lovely Aberdeen, Scotland. I was there to present a small
experience paper on developing and using <a href="https://repobee.org">RepoBee</a>, an open
source tool for generating and managing large amounts of Git repositories for
students in higher education. The paper is available over in the <a href="https://doi.org/10.1145/3304221.3319784">ACM Digital
Library</a>, and it's my very first
publication outside of school, so I'm quite proud of it!</p>
<p>The conference itself was also terrific. The atmosphere was very friendly, there
were many interesting talks and social activities, and I have a hard time
imagining how a conference could be more inviting to a first-time
conference-goer like me. If you have a chance to attend it, I highly recommend
doing so.</p>Git worktrees: work in parallel on multiple versions of a project2019-07-22T22:01:00+02:002019-07-22T22:01:00+02:00Simon Larséntag:slar.se,2019-07-22:/git-worktrees-work-in-parallel-on-multiple-versions-of-a-project.html<p>I've been AWOL for a month due to injury, sickness and conference-going. But
with all that finally out of the way, I have another Tip of the Week, this time
relating to Git: the <code>git worktree</code> command. With <code>git worktree</code>, you can check
out <em>multiple</em> branches at once, which is …</p><p>I've been AWOL for a month due to injury, sickness and conference-going. But
with all that finally out of the way, I have another Tip of the Week, this time
relating to Git: the <code>git worktree</code> command. With <code>git worktree</code>, you can check
out <em>multiple</em> branches at once, which is super useful for when working on major
changes where you need to view multiple versions, or maybe you're just trying a
few different solutions to a single prodlem. If you've ever found yourself
frantically switching branches, stashing changes to be able to switch branches,
and even creating copies of the repository you're working in, then this article
is for you.</p>
<h3>An example repo</h3>
<p>Let's first create an example repo. Here's a little terminal session where I
create a repository, add a README to it on the master branch, add another line
to the readme on a branch called other, and finally checking out to master.</p>
<div class="highlight"><pre><span></span><code><span class="o">[</span>~<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>mkdir<span class="w"> </span>repo
<span class="o">[</span>~<span class="o">]</span><span class="w"> </span>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>repo
<span class="o">[</span>repo<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>init
Initialized<span class="w"> </span>empty<span class="w"> </span>Git<span class="w"> </span>repository<span class="w"> </span><span class="k">in</span><span class="w"> </span>/home/slarse/repo/.git/
<span class="o">[</span>repo<span class="o">]</span><span class="w"> </span>$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Hello!"</span><span class="w"> </span>><span class="w"> </span>README.md
<span class="o">[</span>repo<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>add<span class="w"> </span>README.md<span class="w"> </span><span class="o">&&</span><span class="w"> </span>git<span class="w"> </span>commit<span class="w"> </span>-m<span class="w"> </span><span class="s1">'Add README'</span>
<span class="o">[</span>master<span class="w"> </span><span class="o">(</span>root-commit<span class="o">)</span><span class="w"> </span>6094baf<span class="o">]</span><span class="w"> </span>Add<span class="w"> </span>README
<span class="w"> </span><span class="m">1</span><span class="w"> </span>file<span class="w"> </span>changed,<span class="w"> </span><span class="m">1</span><span class="w"> </span>insertion<span class="o">(</span>+<span class="o">)</span>
<span class="w"> </span>create<span class="w"> </span>mode<span class="w"> </span><span class="m">100644</span><span class="w"> </span>README.md
<span class="o">(</span>master<span class="o">)[</span>repo<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>checkout<span class="w"> </span>-b<span class="w"> </span>other
Switched<span class="w"> </span>to<span class="w"> </span>a<span class="w"> </span>new<span class="w"> </span>branch<span class="w"> </span><span class="s1">'other'</span>
<span class="o">(</span>other<span class="o">)[</span>repo<span class="o">]</span><span class="w"> </span>$<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"There!"</span><span class="w"> </span>>><span class="w"> </span>README.md<span class="w"> </span>
<span class="o">(</span>other<span class="o">)[</span>repo<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>commit<span class="w"> </span>-am<span class="w"> </span><span class="s1">'Add new line to README'</span>
<span class="o">[</span>other<span class="w"> </span>b779dfb<span class="o">]</span><span class="w"> </span>Add<span class="w"> </span>new<span class="w"> </span>line<span class="w"> </span>to<span class="w"> </span>README
<span class="w"> </span><span class="m">1</span><span class="w"> </span>file<span class="w"> </span>changed,<span class="w"> </span><span class="m">1</span><span class="w"> </span>insertion<span class="o">(</span>+<span class="o">)</span>
<span class="o">(</span>other<span class="o">)[</span>repo<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>checkout<span class="w"> </span>master
Switched<span class="w"> </span>to<span class="w"> </span>branch<span class="w"> </span><span class="s1">'master'</span>
<span class="o">(</span>master<span class="o">)[</span>repo<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>
</code></pre></div>
<p>It's not super important how you do it, just make sure to have two branches.</p>
<h3>Adding a new worktree</h3>
<p>First of all: what is a <em>worktree</em>? Usually, you only have <em>the</em> worktree, which
is the part of a repository where you actually do your work (edit files etc).
Running <code>git worktree list</code> on most repos will show the location of this single
worktree, and what commit/branch it is checked out to.</p>
<div class="highlight"><pre><span></span><code><span class="o">(</span>master<span class="o">)[</span>repo<span class="o">]</span><span class="w"> </span>$<span class="w"> </span><span class="nb">pwd</span><span class="w"> </span><span class="c1"># just checking the current working directory</span>
/home/slarse/repo
<span class="o">(</span>master<span class="o">)[</span>repo<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>worktree<span class="w"> </span>list
/home/slarse/repo<span class="w"> </span>6094baf<span class="w"> </span><span class="o">[</span>master<span class="o">]</span><span class="w"> </span><span class="c1"># points to the cwd, checked out to master</span>
</code></pre></div>
<blockquote>
<p><strong>Note:</strong> When I run <code>git worktree list</code> after this point, it's just to show
the results of commands.</p>
</blockquote>
<p>With <code>git worktree add</code>, you can add additional worktrees checked out to
different commits. The most basic usage is <code>git worktree add <path>
<commit-ish></code>, where <code>path</code> is a path to the new worktree (i.e. where you want
to put it), and <code>commit-ish</code> is something like a commit or branch (or a few
other things that are not important for every-day use). Let's check out <code>other</code>
in a new worktree. </p>
<div class="highlight"><pre><span></span><code><span class="o">(</span>master<span class="o">)[</span>repo<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>worktree<span class="w"> </span>add<span class="w"> </span>../repo-other<span class="w"> </span>other
Preparing<span class="w"> </span>worktree<span class="w"> </span><span class="o">(</span>checking<span class="w"> </span>out<span class="w"> </span><span class="s1">'other'</span><span class="o">)</span>
HEAD<span class="w"> </span>is<span class="w"> </span>now<span class="w"> </span>at<span class="w"> </span>b779dfb<span class="w"> </span>Add<span class="w"> </span>new<span class="w"> </span>line<span class="w"> </span>to<span class="w"> </span>README
<span class="o">(</span>master<span class="o">)[</span>repo<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>worktree<span class="w"> </span>list
/home/slarse/repo<span class="w"> </span>6094baf<span class="w"> </span><span class="o">[</span>master<span class="o">]</span>
/home/slarse/repo-other<span class="w"> </span>b779dfb<span class="w"> </span><span class="o">[</span>other<span class="o">]</span>
<span class="o">(</span>master<span class="o">)[</span>repo<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>ls<span class="w"> </span>-a<span class="w"> </span>../repo-other<span class="w"> </span><span class="c1"># have a look in the new working tree</span>
.<span class="w"> </span>..<span class="w"> </span>.git<span class="w"> </span>README.md
</code></pre></div>
<p>As you can see, the new worktree has been created, and can be seen in the list
of worktrees. <code>.git</code> is usually a directory, but in the case of a non-primary
worktree, it's actually just a file with a path to the original <code>.git</code>
directory.</p>
<div class="highlight"><pre><span></span><code><span class="o">(</span>master<span class="o">)[</span>repo<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>cat<span class="w"> </span>../repo-other/.git<span class="w"> </span>
gitdir:<span class="w"> </span>/home/slarse/repo/.git/worktrees/repo-other
</code></pre></div>
<p>Like many things in Git, it's brilliantly simple. You can start working in your
new worktree like it's an entirely separate repository, with the caveat that you
can't check out to a branch that is checked out in some other worktree. That
includes checking out to other commits or branches, and even creating entirely
new branches.</p>
<h3>Moving a worktree</h3>
<p>If for some reason you need to move a worktree, you should use <code>git worktree
move</code> to make sure that all of the references are correctly changed. It's very
simple, just type <code>git worktree move <src> <dst></code>. For example, if I want to
move <code>../repo-other</code> to <code>../repo-work</code>, I do:</p>
<div class="highlight"><pre><span></span><code><span class="o">(</span>master<span class="o">)[</span>repo<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>worktree<span class="w"> </span>move<span class="w"> </span>../repo-other<span class="w"> </span>../repo-work
<span class="o">(</span>master<span class="o">)[</span>repo<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>worktree<span class="w"> </span>list
/home/slarse/repo<span class="w"> </span>6094baf<span class="w"> </span><span class="o">[</span>master<span class="o">]</span>
/home/slarse/repo-work<span class="w"> </span>b779dfb<span class="w"> </span><span class="o">[</span>other<span class="o">]</span>
</code></pre></div>
<p>That's all there is to moving worktrees. Not very exciting, and I can't recall
ever actually doing it, but I can see how it could be useful.</p>
<h3>Removing a worktree</h3>
<p>To remove a worktree, run <code>git worktree remove <path></code>.</p>
<div class="highlight"><pre><span></span><code><span class="o">(</span>master<span class="o">)[</span>repo<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>worktree<span class="w"> </span>remove<span class="w"> </span>../repo-work/
<span class="o">(</span>master<span class="o">)[</span>repo<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>worktree<span class="w"> </span>list
/home/slarse/repo<span class="w"> </span>6094baf<span class="w"> </span><span class="o">[</span>master<span class="o">]</span>
</code></pre></div>
<p>You can also just remove the directory with the worktree and the reference to it
will be removed automatically (but not necessarily immediately). Run <code>git
worktree prune</code> to trigger this removal process.</p>
<h3>The other worktree commands</h3>
<p>There are a few more <code>git worktree</code> commands that I've never felt the need to
use. Have a look at them
<a href="https://git-scm.com/docs/git-worktree">in the git-worktree documentation</a>.</p>
<h3>Summary</h3>
<p>In this short article I showcased <code>git worktree</code>. It's super useful to work in
parallel on different versions of the same project, without having to create
copies of the repository and thereby having to deal with synchronizing multiple
local copies (which can quickly get hard to manage). I find myself using this
more and more, and if you find it useful yourself I highly recommend reading up
on it more in its man-page (either with <code>man git-worktree</code> or
<a href="https://git-scm.com/docs/git-worktree">online</a>).</p>Redirecting stdout and stderr in bash2019-06-23T21:37:00+02:002019-06-23T21:37:00+02:00Simon Larséntag:slar.se,2019-06-23:/redirecting-stdout-and-stderr-in-bash.html<p>A couple of weeks ago I covered some basic I/O redirection in bash (see
<a href="https://slar.se/io-redirection-in-bash.html">I/O redirection in bash</a>). Well, there's actually
a lot more to it, so for this TOTW I thought I'd touch on a few more advanced
usages.</p>
<h2>Redirecting stderr</h2>
<p>Sometimes, you may find that part …</p><p>A couple of weeks ago I covered some basic I/O redirection in bash (see
<a href="https://slar.se/io-redirection-in-bash.html">I/O redirection in bash</a>). Well, there's actually
a lot more to it, so for this TOTW I thought I'd touch on a few more advanced
usages.</p>
<h2>Redirecting stderr</h2>
<p>Sometimes, you may find that part or all of the output of a command isn't
properly redirected. As a quick example, navigate to any directory that is <em>not</em>
a Git repository, and run <code>git status</code>. You should see something like this:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>git<span class="w"> </span>status
fatal:<span class="w"> </span>not<span class="w"> </span>a<span class="w"> </span>git<span class="w"> </span>repository<span class="w"> </span><span class="o">(</span>or<span class="w"> </span>any<span class="w"> </span>of<span class="w"> </span>the<span class="w"> </span>parent<span class="w"> </span>directories<span class="o">)</span>:<span class="w"> </span>.git
</code></pre></div>
<p>Yet, if you try to redirect it with a standard redirect, the output
is still displayed, and the file you redirect to remains empty.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>git<span class="w"> </span>status<span class="w"> </span>><span class="w"> </span>output
fatal:<span class="w"> </span>not<span class="w"> </span>a<span class="w"> </span>git<span class="w"> </span>repository<span class="w"> </span><span class="o">(</span>or<span class="w"> </span>any<span class="w"> </span>of<span class="w"> </span>the<span class="w"> </span>parent<span class="w"> </span>directories<span class="o">)</span>:<span class="w"> </span>.git
$<span class="w"> </span>cat<span class="w"> </span>output
$<span class="w"> </span>
</code></pre></div>
<p>The reason is quite simple: the output from <code>git status</code> is an error message,
which is typically output on <em>standard error</em> (stderr), while I/O redirection
operates on <em>standard output</em> (stdout) by default. When redirecting output (or
input, for that matter), one can optionally provide a file descriptor specifying
which output stream to redirect. On a typical UNIX-like system, stdout is file
descriptor 1, and stderr is file descriptor 2. So if we want to catch that
stderr output, we just need to prepend a 2 to the redirection operator.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>git<span class="w"> </span>status<span class="w"> </span><span class="m">2</span>><span class="w"> </span>output
$<span class="w"> </span>cat<span class="w"> </span>output
fatal:<span class="w"> </span>not<span class="w"> </span>a<span class="w"> </span>git<span class="w"> </span>repository<span class="w"> </span><span class="o">(</span>or<span class="w"> </span>any<span class="w"> </span>of<span class="w"> </span>the<span class="w"> </span>parent<span class="w"> </span>directories<span class="o">)</span>:<span class="w"> </span>.git
</code></pre></div>
<p>You can probably guess that if you leave the file descriptor out, it will
default to 1. In some cases, you may want to redirect both stderr and stdout to
the same file. But many programs output both on stderr and stdout, and we may
want to redirect both of them.</p>
<h2>Redirecting stderr and stdout</h2>
<p>So, we can specify a file descriptor to redirect stdout or stderr (or any other
file descriptor, really), but many programs output on both stderr and stdout,
and it's often useful to redirect both. Here's a small Python script <code>print.py</code>
that outputs on line on stdout and one on stderr.</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">sys</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"some standard output"</span><span class="p">,</span> <span class="n">file</span><span class="o">=</span><span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"some error output"</span><span class="p">,</span> <span class="n">file</span><span class="o">=</span><span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="p">)</span>
</code></pre></div>
<blockquote>
<p><strong>Note:</strong> That's Python as in Python 3.</p>
</blockquote>
<p>If we redirect stdout only, then the stderr line is still printed to the
terminal.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>python3<span class="w"> </span>print.py<span class="w"> </span><span class="m">1</span>><span class="w"> </span>stdout_output<span class="w"> </span><span class="c1"># recall that the 1 can be omitted</span>
some<span class="w"> </span>error<span class="w"> </span>output
$<span class="w"> </span>cat<span class="w"> </span>stdout_output
some<span class="w"> </span>standard<span class="w"> </span>output
</code></pre></div>
<p>Similarly, redirecting only stderr leaves the stdout output on the terminal.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>python3<span class="w"> </span>print.py<span class="w"> </span><span class="m">2</span>><span class="w"> </span>stderr_output
some<span class="w"> </span>standard<span class="w"> </span>output
$<span class="w"> </span>cat<span class="w"> </span>stderr_output
some<span class="w"> </span>error<span class="w"> </span>output
</code></pre></div>
<p>Quite intuitively, if we want to redirect both stderr and stdout to one file
each, we can simply do two redirections following one another.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>python3<span class="w"> </span>print.py<span class="w"> </span><span class="m">1</span>><span class="w"> </span>stdout_output<span class="w"> </span><span class="m">2</span>><span class="w"> </span>stderr_output
$<span class="w"> </span>cat<span class="w"> </span>stdout_output
some<span class="w"> </span>standard<span class="w"> </span>output
$<span class="w"> </span>cat<span class="w"> </span>stderr_output
some<span class="w"> </span>error<span class="w"> </span>output
</code></pre></div>
<p>There's also the possibility to redirect both stdout and stderr to the same file
using the special <code>&</code> character in place of a file descriptor.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>python3<span class="w"> </span>print.py<span class="w"> </span><span class="p">&</span>><span class="w"> </span>output
$<span class="w"> </span>cat<span class="w"> </span>output
some<span class="w"> </span>standard<span class="w"> </span>output
some<span class="w"> </span>error<span class="w"> </span>output
</code></pre></div>
<p>And with that and the previous article, I've shared pretty much everything I
feel is useful with output redirection. In some future Tip of the Week, I'm sure
I'll get into input redirecton as well, as it's much the same.</p>Technical e-books from Humble Bundle2019-06-13T09:12:00+02:002019-06-13T09:12:00+02:00Simon Larséntag:slar.se,2019-06-13:/technical-e-books-from-humble-bundle.html<p>Another Tip of the Week, in the same week as the previous one (because I've
been slacking off). This one is very simple, and rather non-technical. I simply
want to direct attention toward
<a href="https://www.humblebundle.com/">humblebundle.com</a>, and their quite frequent book
bundles. Many of these book bundles include or are entirely …</p><p>Another Tip of the Week, in the same week as the previous one (because I've
been slacking off). This one is very simple, and rather non-technical. I simply
want to direct attention toward
<a href="https://www.humblebundle.com/">humblebundle.com</a>, and their quite frequent book
bundles. Many of these book bundles include or are entirely centered around
programming topics. I've gotten tons of great books from there over the years,
and it can really pay off to keep an eye on the new bundles. Typically, the top
tier of a book bundle is around 15-20 EUR, while most books in it cost more
alone. There have been general programming bundles, hacking/security bundles,
Python bundles (a lot of them), Linux bundles, and much more. A few of the best
books I've gotten from Humble include:</p>
<ul>
<li><a href="https://www.amazon.com/Linux-Programming-Interface-System-Handbook/dp/1593272200">The Linux Programming Interface</a><ul>
<li>Got it in a 20 EUR bundle, while this book alone costs somewhere around 70.</li>
</ul>
</li>
<li><a href="https://www.amazon.com/Fluent-Python-Concise-Effective-Programming-dp-1491946008/dp/1491946008/ref=mt_paperback?_encoding=UTF8&me=&qid=">Fluent Python</a><ul>
<li>I actually already owned this book at the time of acquiring it from a
bundle, but it's by far the best book on Python I've ever read.</li>
</ul>
</li>
<li><a href="https://www.amazon.com/Flask-Web-Development-Developing-Applications-dp-1491991739/dp/1491991739/ref=mt_paperback?_encoding=UTF8&me=&qid=1560410372">Flask Web Development</a></li>
<li><a href="https://www.amazon.com/CSS-Definitive-Guide-Visual-Presentation-dp-1449393195/dp/1449393195/ref=mt_paperback?_encoding=UTF8&me=&qid=1560410482">CSS: The Definitive Guide</a></li>
</ul>
<p>And many, many more! Really, all I want to get said with this TOTW is that you
should really keep an eye on Humble Bundle, as their book bundles contain
tremendous value, and you can choose to put some (or all) of the money you pay
toward charity. And that's all there is to it!</p>
<blockquote>
<p><strong>Disclaimer:</strong> I am not sponsored by Humble in any way, I am simply a
long-time user of the service.</p>
</blockquote>I/O redirection in bash2019-06-11T23:16:00+02:002019-06-11T23:16:00+02:00Simon Larséntag:slar.se,2019-06-11:/io-redirection-in-bash.html<p>Alright, so Tip of the Week has turned somewhat into "tip every two or three
weeks". It turns out that it's pretty difficult to find the time to actually
write something every week. but I'll keep trying. With that out of the way,
let's head into the subject matter of …</p><p>Alright, so Tip of the Week has turned somewhat into "tip every two or three
weeks". It turns out that it's pretty difficult to find the time to actually
write something every week. but I'll keep trying. With that out of the way,
let's head into the subject matter of this post: <em>I/O redirection</em>.
We'll just have a look at the most basic but also most generally applicable use
of redirection: taking the output from a program and storing it in a file.</p>
<blockquote>
<p><strong>Important:</strong> Files will both be created and clobbered in this TOTW. When
trying this stuff out, first create a new directory and do everything in
there, so you don't litter your filesystem with strange files, or
accidentally overwrite something important.</p>
</blockquote>
<h2>Redirecting output</h2>
<p>To set the stage, I'll be working in a directory with the following contents:</p>
<div class="highlight"><pre><span></span><code><span class="o">[</span>tmp<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>ls
file1.txt<span class="w"> </span>file2.txt<span class="w"> </span>image1.png<span class="w"> </span>file2.txt
</code></pre></div>
<p>Redirecting output is fairly simple, and useful when you want to save the
output of some command in a file. There are two primary ways of redirecting
output: <em>appending</em> and <em>truncating</em>. Appending is the one I use the most,
so let's start with that one.</p>
<h3>Appending output redirection</h3>
<p>With <code>>></code>, we can make an appending redirect.</p>
<div class="highlight"><pre><span></span><code><span class="o">[</span>tmp<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>ls<span class="w"> </span>>><span class="w"> </span>ls_output.txt<span class="w"> </span><span class="c1"># output from ls saved to output.txt</span>
<span class="o">[</span>tmp<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>cat<span class="w"> </span>ls_output.txt<span class="w"> </span><span class="c1"># let's have a look... </span>
file1.txt
file2.txt
image1.png
image2.png
ls_output.txt
<span class="o">[</span>tmp<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>ls<span class="w"> </span>>><span class="w"> </span>ls_output.txt<span class="w"> </span><span class="c1"># append new output</span>
<span class="o">[</span>tmp<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>cat<span class="w"> </span>ls_output.txt<span class="w"> </span><span class="c1"># let's have a look again</span>
file1.txt
file2.txt
image1.png
image2.png
ls_output.txt
file1.txt
file2.txt
image1.png
image2.png
ls_output.txt
</code></pre></div>
<p>There are three things to note here. First, the <code>ls_output.txt</code> file does not
exist in the initial directory, and so it is created with the first redirect.
Note however that <code>ls_output.txt</code> is present in the first redirected output
from <code>ls</code>: <code>ls_output.txt</code> is actually created <em>before</em> <code>ls</code> is run as there
needs to be an open
<a href="https://en.wikipedia.org/wiki/File_descriptor">file descriptor</a>* to the file
pass along. </p>
<blockquote>
<p>** * ** A file descriptor can simply be thought of as a pointer to a file.
There is no need to understand file descriptors intimately to use basic I/O
redirection efficiently.</p>
</blockquote>
<p>The second redirect is then appended to the file, which at that
point already exists. And that pretty much sums up how an appending redirect
functions: it appends output to the specified file if it exists, and creates a
file with the output if it does not exist. I find that this is most often the
functionality that I want, but in some cases, you want to re-create the file
from scratch with each redirect. That can be achieved with a truncating
redirect.</p>
<blockquote>
<p><strong>Note:</strong> You may note that the output of <code>ls</code> is formatted differently when
output to the terminal, and when redirected to a file. <code>ls</code> checks whether
the stdout file descriptor points to a terminal, or something else, and
formats the output accordingly. The details are somewhat out of scope.</p>
</blockquote>
<h3>Truncating output redirection</h3>
<p>Let's assume that we start over from the initial state of the directory, before
<code>ls_output.txt</code> existed. We can then make a truncating redirect with <code>></code>.</p>
<div class="highlight"><pre><span></span><code><span class="o">[</span>tmp<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>rm<span class="w"> </span>ls_output.txt<span class="w"> </span><span class="c1"># restore initial directory state</span>
<span class="o">[</span>tmp<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>ls<span class="w"> </span>><span class="w"> </span>ls_output.txt<span class="w"> </span><span class="c1"># make a truncating redirect</span>
<span class="o">[</span>tmp<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>cat<span class="w"> </span>ls_output.txt<span class="w"> </span><span class="c1"># and inspect the results</span>
file1.txt
file2.txt
image1.png
image2.png
ls_output.txt
<span class="o">[</span>tmp<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>ls<span class="w"> </span>><span class="w"> </span>ls_output.txt<span class="w"> </span><span class="c1"># another truncating redirect</span>
<span class="o">[</span>tmp<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>cat<span class="w"> </span>ls_output.txt
file1.txt
file2.txt
image1.png
image2.png
ls_output.txt
</code></pre></div>
<p>If you did not know what truncating meant before, you can probably figure it out
now. With a single <code>></code>, the specified file is created if it does not exist, just
like with <code>>></code>, but it is entirely overwritten (truncated, clobbered) if it
already does exist. I rarely use a truncating redirect, as it is an easy thing
to accidentally truncate a file you did not mean to touch. I recommend to always
use an appending redirect, unless you have a good reason to truncate the
targeted file.</p>
<p>And that's it for this TOTW!</p>Piping commands in bash2019-05-21T00:00:00+02:002019-05-21T00:00:00+02:00Simon Larséntag:slar.se,2019-05-21:/piping-commands-in-bash.html<p>Many, many bash commands are built around and meant to be used with a
fundamental feature of the bash shell (actually, most shells), called <em>piping</em>.
Put simply, piping takes the output of one command and provides it as input to
the next. Here's a simple example of running <code>ls</code> and …</p><p>Many, many bash commands are built around and meant to be used with a
fundamental feature of the bash shell (actually, most shells), called <em>piping</em>.
Put simply, piping takes the output of one command and provides it as input to
the next. Here's a simple example of running <code>ls</code> and filtering the result with
<code>grep</code> to find all <code>.py</code> files in the current directory.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ls<span class="w"> </span><span class="c1"># just run ls </span>
file1.md<span class="w"> </span>file2.md<span class="w"> </span>file3.md<span class="w"> </span>script1.py<span class="w"> </span>script2.py
$<span class="w"> </span>ls<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s1">'\.py$'</span>
script1.py
script2.py
</code></pre></div>
<p>To be precise, the <code>|</code> (pipe) operator takes the output from the command on the
left, and provides it as input to the command on the right. Pipes can be chained
practically as much as you'd like. For example, if we want to get amount of
<code>.py</code> files in the current directory, we can pipe the output from <code>grep</code> to the
<code>wc</code> (word count) command, with the <code>-l</code> option to count lines only.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ls<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s1">'\.py$'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>wc<span class="w"> </span>-l
<span class="m">2</span>
</code></pre></div>
<p><code>wc</code> counts two lines, which is precisely the amount of <code>.py</code> files that we
found. Let's move on to I/O redirection. Piping allows you to easily compose
powerful programs from simple commands, and is a very intuitive way to work.
Next week, I'll cover I/O redirection, which is another super useful feature of
bash that's a bit more complicated.</p>Using bash aliases2019-05-06T12:19:00+02:002019-05-06T12:19:00+02:00Simon Larséntag:slar.se,2019-05-06:/using-bash-aliases.html<p>For this <em>Tip of the Week</em>, I'd like to present something that took me a while
to figure out why it was useful. That something is bash aliases, and I'll now
walk you through how to create aliases, and the two main ways in which I use
them (although I'm …</p><p>For this <em>Tip of the Week</em>, I'd like to present something that took me a while
to figure out why it was useful. That something is bash aliases, and I'll now
walk you through how to create aliases, and the two main ways in which I use
them (although I'm sure there are more use cases).</p>
<h1>Using aliases</h1>
<p>I think the <code>bash</code> manpage has a very good and concise description of what an
alias is:</p>
<blockquote>
<p>Aliases allow a string to be substituted for a word when it is used as the
first word of a simple command</p>
</blockquote>
<p>In other words, I can define a command that is substituted for some other
command. Creating an alias is very simple. The syntax looks like this:</p>
<div class="highlight"><pre><span></span><code>alias <NAME>=<COMMAND>
</code></pre></div>
<p>So for example, if I want to have a command <code>hellofile</code> that creates a file with
the text "Hello, world!", I can achieve that with the following alias.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">alias</span><span class="w"> </span><span class="nv">hellofile</span><span class="o">=</span><span class="s1">'echo "Hello, world!" > hellofile.txt'</span>
</code></pre></div>
<p>Note the single quotes around the command definition. Without them, <code>bash</code>
would interpret the alias as being only <code>echo</code>, and the rest of the line as
another command. Now, if I run the command <code>hellofile</code>, it fill be substituted
with <code>echo "Hello, world!" > hellofile.txt</code>. You should think of aliases as pure
text substitution: precisely what you put in the alias definition will be put on
the command line when you invoke it. You can view all of your current aliases
by running <code>alias</code> without any options. Now, let's have a look at some common
use cases!</p>
<h2>Specifying "default" options for commands</h2>
<p>This is probably the most common use case for aliases, and it's likely that you
already have some in play. A common one is to have <code>ls</code> aliased to <code>ls
--color=auto</code>. That is to say, the following alias is defined:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">alias</span><span class="w"> </span><span class="nv">ls</span><span class="o">=</span><span class="s1">'ls --color=auto'</span>
</code></pre></div>
<p>So if I now run e.g. <code>ls /etc</code>, the resulting command is actually <code>ls
--color=auto /etc</code>. Note how the alias does not have to be the <em>only</em> word I
type for the command, it just has to be the first one. Another command that I
use an alias for is <code>xclip</code>, which is a small utility for copying stuff. I use
it almost exclusively to copy file contents to the clipboard, but that's not the
default functionality. In order to copy to the clipboard, I must write this
rather cumbersome command.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>xclip<span class="w"> </span>-selection<span class="w"> </span>clipboard<span class="w"> </span><FILEPATH>
</code></pre></div>
<p>So I have an alias for it so I can just type <code>xclip <FILEPATH></code> to copy to the
clipboard.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">alias</span><span class="w"> </span><span class="nv">xclip</span><span class="o">=</span><span class="s1">'xclip -selection clipboard'</span>
</code></pre></div>
<p>As a side note, it may not be the best style to clobber an existing command with
an alias, but I still tend to do that for some of my most commonly used
commands. If you want to use the vanilla command, simply put it within single
quotes, which will hinder the alias from expanding (e.g. type <code>'ls'</code> to run <code>ls</code>
without <code>--color=auto</code>). Note that just defining an alias in a <code>bash</code> session
will not persist: it needs to be defined anew for each session. To have it
permanently defined, put the definition in a startup script (e.g. <code>.bashrc</code> or
<code>.bash_profile</code>).</p>
<h2>Creating throwaway commands</h2>
<p>Now, the aliases I described above are useful to have defined permanently, and
should be defined in a startup script. The second use case I have for aliases is
when I have a repetitive command that I need to type over and over in the same
session, but isn't useful in general. An example would be when I need to run
some specific Java class in a project. Let's say I need to run the class
<code>se.slar.awesome.project.Main</code> over and over. Instead of typing <code>java
se.slar.awesome.project.Main</code> over and over, I define an alias for it.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">alias</span><span class="w"> </span><span class="nv">runmain</span><span class="o">=</span><span class="s1">'java se.slar.awesome.project.Main'</span>
</code></pre></div>
<p>And then, instead of writing all of that out, or having to do some
<a href="https://slar.se/reverse-search-in-bash.html">reverse searching</a> or
<a href="https://slar.se/history-and-history-expansion-in-bash.html">history lookups</a>, I can just type <code>runmain</code>.
As defining an alias is so effortless, I tend to do it even if I know I'm just
gonna use the complex command a couple of times.</p>
<p>And that's all I wanted to cover, hope you enjoyed it and stay tuned for the
next TOTW coming next week!</p>Git local2019-04-29T22:58:00+02:002019-04-29T22:58:00+02:00Simon Larséntag:slar.se,2019-04-29:/git-local.html<p>Nowadays, Git is almost ubiquitous in software development. Most developers also
know that Git is a <em>decentralized</em> version control system, meaning that every
copy of the repository carries the full revision history, and there is no
"central" repository. A consequence of the decentralized aspect of Git is that
you can …</p><p>Nowadays, Git is almost ubiquitous in software development. Most developers also
know that Git is a <em>decentralized</em> version control system, meaning that every
copy of the repository carries the full revision history, and there is no
"central" repository. A consequence of the decentralized aspect of Git is that
you can create repositories locally, and version control documents in them
locally, without ever setting up a remote repository on e.g. GitHub or GitLab.
In this TOTW, I'll show you how to use Git locally, and also how to change your
mind and put it on e.g. GitHub at a later time.</p>
<blockquote>
<p><strong>Note:</strong> This also touches on an important and often misunderstood point: Git
and GitHub are <em>not</em> the same thing. Git is a version control system, while
GitHub is a service which allows hosting of remote repositories, issue
management etc. GitHub is also not the only service around,
<a href="https://gitlab.com">GitLab</a> and <a href="https://bitbucket.com">BitBucket</a> are two
other prominent services which host Git repositories.</p>
</blockquote>
<h3>Using Git locally</h3>
<p>How do you use Git locally, then? It's simple. Just create a directory and run
<code>git init</code> to initialize it as a Git repository. Here's an example command line
session of what it looks like.</p>
<div class="highlight"><pre><span></span><code><span class="o">[</span>~<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>mkdir<span class="w"> </span>repo
<span class="o">[</span>~<span class="o">]</span><span class="w"> </span>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>repo
<span class="o">[</span>repo<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>ls<span class="w"> </span>-a
.<span class="w"> </span>..<span class="w"> </span><span class="c1"># repo is empty</span>
<span class="o">[</span>repo<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>init
Initialized<span class="w"> </span>empty<span class="w"> </span>Git<span class="w"> </span>repository<span class="w"> </span><span class="k">in</span><span class="w"> </span>/home/slarse/repo/.git/
<span class="o">[</span>repo<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>ls<span class="w"> </span>-a
.<span class="w"> </span>..<span class="w"> </span>.git<span class="w"> </span><span class="c1"># the .git directory indicates that this is now a Git repo</span>
</code></pre></div>
<p>I often use Git to version control stuff that I have no intention of ever
putting up in a remote repository. This is useful for when you accidentally
remove stuff, or just need to try out a bunch of different ideas that you can
swap back and forth between by simply switching branches.</p>
<h3>Changing your mind (also called adding a remote)</h3>
<p>If you suddenly feel like that local repo should be put up on a hosting service
after all, maybe just to back it up, or maybe to collaborate with someone else,
it's very simple to do so. First, create an empty repository (as in completely
empty, don't initialize it with a README or license). Then copy the address to
the repository (I prefer to use SSH). Let's say I have a repo at
<code>git@github.com:slarse/superrepo.git</code>. I can then add it as a remote to my local
repo, and push my master branch to it.</p>
<div class="highlight"><pre><span></span><code><span class="o">[</span>repo<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>remote<span class="w"> </span>add<span class="w"> </span>origin<span class="w"> </span>git@github.com:slarse/superrepo.git
<span class="o">[</span>repo<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>branch
*<span class="w"> </span>master<span class="w"> </span><span class="c1"># I'm on the master branch, which is what I want to push</span>
<span class="o">[</span>repo<span class="o">]</span><span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>push<span class="w"> </span>--set-upstream<span class="w"> </span>origin<span class="w"> </span>master
Enumerating<span class="w"> </span>objects:<span class="w"> </span><span class="m">3</span>,<span class="w"> </span><span class="k">done</span>.
Counting<span class="w"> </span>objects:<span class="w"> </span><span class="m">100</span>%<span class="w"> </span><span class="o">(</span><span class="m">3</span>/3<span class="o">)</span>,<span class="w"> </span><span class="k">done</span>.
Writing<span class="w"> </span>objects:<span class="w"> </span><span class="m">100</span>%<span class="w"> </span><span class="o">(</span><span class="m">3</span>/3<span class="o">)</span>,<span class="w"> </span><span class="m">213</span><span class="w"> </span>bytes<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="m">213</span>.00<span class="w"> </span>KiB/s,<span class="w"> </span><span class="k">done</span>.
Total<span class="w"> </span><span class="m">3</span><span class="w"> </span><span class="o">(</span>delta<span class="w"> </span><span class="m">0</span><span class="o">)</span>,<span class="w"> </span>reused<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="o">(</span>delta<span class="w"> </span><span class="m">0</span><span class="o">)</span>
To<span class="w"> </span>github.com:slarse/superrepo.git
<span class="w"> </span>*<span class="w"> </span><span class="o">[</span>new<span class="w"> </span>branch<span class="o">]</span><span class="w"> </span>master<span class="w"> </span>-><span class="w"> </span>master
Branch<span class="w"> </span><span class="s1">'master'</span><span class="w"> </span><span class="nb">set</span><span class="w"> </span>up<span class="w"> </span>to<span class="w"> </span>track<span class="w"> </span>remote<span class="w"> </span>branch<span class="w"> </span><span class="s1">'master'</span><span class="w"> </span>from<span class="w"> </span><span class="s1">'origin'</span>.
</code></pre></div>
<p>Now my previously local-only repo is also in GitHub, and I can push and pull
from it as usual. That's all for this tip of the week, it's just meant to spark
an idea that took me quite a while to come up with myself!</p>History and history expansion in bash2019-04-22T11:59:00+02:002019-04-22T11:59:00+02:00Simon Larséntag:slar.se,2019-04-22:/history-and-history-expansion-in-bash.html<p>Admittedly, this TOTW is one day late, so this week there will be 2xTOTW! In
any case, the tip I want to bring up here is very much related to last week's
TOTW on <a href="https://slar.se/reverse-search-in-bash.html">Reverse search in bash</a>. Sometimes,
reverse searching just doesn't work out. You may not be quite …</p><p>Admittedly, this TOTW is one day late, so this week there will be 2xTOTW! In
any case, the tip I want to bring up here is very much related to last week's
TOTW on <a href="https://slar.se/reverse-search-in-bash.html">Reverse search in bash</a>. Sometimes,
reverse searching just doesn't work out. You may not be quite sure what you
are looking for, or there are just too many recent commands that look samey.
In such cases, using the <code>history</code> command is a good alternative. </p>
<h3><code>history</code></h3>
<p>The <code>history</code> command will display the last commands that you have entered, and
looks something like this:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">history</span>
<span class="w"> </span><span class="m">1009</span><span class="w"> </span><span class="nb">fg</span>
<span class="w"> </span><span class="m">1010</span><span class="w"> </span>git<span class="w"> </span>status
<span class="w"> </span><span class="m">1011</span><span class="w"> </span>git<span class="w"> </span>commit<span class="w"> </span>-a<span class="w"> </span>-m<span class="w"> </span><span class="s1">'Add module docstring to github_api module'</span>
<span class="w"> </span><span class="o">[</span>***OUTPUT<span class="w"> </span>TRUNCATED***<span class="o">]</span>
<span class="w"> </span><span class="m">2007</span><span class="w"> </span><span class="nb">history</span>
</code></pre></div>
<p>Each command is called an <em>event</em>, and the output is formatted as <code><event_nr>
<event></code>. Precisely how many commands are returned by the <code>history</code> is
determined by the <code>HISTSIZE</code> and <code>HISTFILESIZE</code> environment variables. Setting
these to something like <code>5000</code> and <code>10000</code>, respectively, should be manageable
even for the weakest of computers. You can also limit the output of <code>history</code>
by providing an integer argument, so e.g. <code>history 5</code> will display the last 5
commands. Now, the real power of <code>history</code> becomes apparent when using it
with <em>history expansion</em>.</p>
<h3>History expansion</h3>
<p>History expansion can be used to expand an event number into the whole command
it corresponds to. To expand an event, one simply types <code>!<event_nr></code>. For
example, looking at the <code>history</code> output above I can see that event number 1010
corresponds to <code>git status</code>. I can execute the command again with history
expansion like so:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>!1010
git<span class="w"> </span>status<span class="w"> </span><span class="c1"># Command is echoed</span>
On<span class="w"> </span>branch<span class="w"> </span>master<span class="w"> </span><span class="c1"># Output from executing the command</span>
<span class="o">[</span>***REST<span class="w"> </span>OF<span class="w"> </span>OUTPUT<span class="w"> </span>OMITTED***<span class="o">]</span>
</code></pre></div>
<p>The command is first echoed, and then executed. There are a few other ways to
specify the event number.</p>
<ul>
<li><code>!</code>: Execute the last event.<ul>
<li>I.e. type <code>!!</code> in the terminal.</li>
<li>Can be useful to re-execute a command that you realized you needed <code>sudo</code>
for with <code>sudo !!</code>.</li>
</ul>
</li>
<li><code>-n</code>: Execute the nth previous event.<ul>
<li>E.g. type <code>!-1</code> to execute the last event, <code>!-2</code> to execute the one
before that, and so on.</li>
<li>I personally don't find this very useful.</li>
</ul>
</li>
</ul>
<p>There is one more very useful feature that I often use, and that is the ability
to only print the command. This can be achieved by appending <code>:p</code> to the
history expansion command. Here is an example:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>!1011:p
git<span class="w"> </span>commit<span class="w"> </span>-a<span class="w"> </span>-m<span class="w"> </span><span class="s1">'Add module docstring to github_api module'</span>
</code></pre></div>
<p>The command can then be accessed by pressing UP-arrow or <code>ctrl-p</code>, which is
very useful if you need to do minor modifications to it. There are tons of
more ways to use history expansion, and I strongly recommend reading the man-page
on it. Type <code>man bash</code> and then search for <code>HISTORY EXPANSION</code>, or do the same
in <a href="https://linux.die.net/man/1/bash">this online bash man page</a>.</p>
<h3>Filtering history</h3>
<p>A final tip on using history expansion is to filter the output with <code>grep</code>. For
example, if I only want to find commands that include the word <code>git</code>, I can
filter the output of <code>history</code> by <em>piping</em> to <code>grep</code> with the <code>|</code> character.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">history</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>git
<span class="w"> </span><span class="m">1010</span><span class="w"> </span>git<span class="w"> </span>status
<span class="w"> </span><span class="m">1011</span><span class="w"> </span>git<span class="w"> </span>commit<span class="w"> </span>-a<span class="w"> </span>-m<span class="w"> </span><span class="s1">'Add module docstring to github_api module'</span>
</code></pre></div>
<p>I will most likely do another TOTW on piping, but the basic principle is that
<code>|</code> takes the output from the command on the left and feeds it as input to the
command on the right. That's it for this TOTW, stay tuned for the next one
coming on Sunday the 28th of April!</p>Reverse search in bash2019-04-09T23:23:00+02:002019-04-09T23:23:00+02:00Simon Larséntag:slar.se,2019-04-09:/reverse-search-in-bash.html<p>Have you ever found yourself furiously tapping the UP-arrow (or <code>ctrl+p</code>) to
find a command that's probably waaaay up there? Would you be surprised if I
told you there's a better way? When you want to re-use a command you've written
previously, and you know it's not the previous …</p><p>Have you ever found yourself furiously tapping the UP-arrow (or <code>ctrl+p</code>) to
find a command that's probably waaaay up there? Would you be surprised if I
told you there's a better way? When you want to re-use a command you've written
previously, and you know it's not the previous command, or the one before that,
your first resort should be a <em>reverse search</em>. This can be accessed with
<code>ctrl+r</code>. If you press that button combination, you should see something like this:</p>
<div class="highlight"><pre><span></span><code><span class="o">(</span>reverse-i-search<span class="o">)</span><span class="sb">`</span><span class="err">'</span>:<span class="w"> </span>
</code></pre></div>
<p>Just start typing the beginning of the command you're looking for, and most
often, it will pop up. For example, I sometimes need to re-run the
previous<code>git</code> command that I ran a while back. I then press <code>ctrl+r</code> and type
<code>git</code> to get something like this:</p>
<div class="highlight"><pre><span></span><code><span class="o">(</span>reverse-i-search<span class="o">)</span><span class="sb">`</span>git<span class="err">'</span>:<span class="w"> </span>git<span class="w"> </span>push
</code></pre></div>
<p>Note how the initial <code>git</code> before the <code>:</code> is what I've actually written here,
and the text after the <code>:</code> (in this case <code>git push</code>) is what's been found with
the reverse search. Pressing <code>tab</code> now will terminate the search and put the
result of the search on the command line for editing. Then, simply press
<code>enter</code> to execute the command as usual. You can also skip over the editing
part and press <code>enter</code> right away to execute the command as-is. Sometimes,
however, the result you get first isn't what you want (obviously, just typing
<code>git push</code> would have been faster in this case). You can then press <code>ctrl+r</code>
again to cycle to the next hit.</p>
<div class="highlight"><pre><span></span><code>(reverse-i-search)`git': git commit -a -m 'Add module docstring to github_api module'
</code></pre></div>
<p>Now there's a command that I might not want to have to type out again in its
entirety, better showing why a reverse search may be useful. That's it for this
week's TotW, check back next week for more!</p>Announcing Tip of the Week (TotW)2019-04-09T23:15:00+02:002019-04-09T23:15:00+02:00Simon Larséntag:slar.se,2019-04-09:/announcing-tip-of-the-week-totw.html<p>In order to actually get around to writing some content, I've decided to start
a little series: Tip of the Week! Every week, I'll spend 30 minutes or so
writing a very small article about some tip related to programming, Linux or
technology in general. And no, this one does …</p><p>In order to actually get around to writing some content, I've decided to start
a little series: Tip of the Week! Every week, I'll spend 30 minutes or so
writing a very small article about some tip related to programming, Linux or
technology in general. And no, this one does not count, so I still have to
write this week's TotW!</p>Migrating my blog2019-04-06T18:32:00+02:002019-04-10T23:20:00+02:00Simon Larséntag:slar.se,2019-04-06:/migrating-my-blog.html<blockquote>
<p><strong>Edit:</strong> The migration is complete! The Flask-based site has been retired
and this Pelican-based site is fully fleshed out with the old content :)</p>
</blockquote>
<p>I'm currently in the midst of migrating my old blog over here. Until I'm done,
both sites will be a bit half-baked, sorry about that!</p>Testing tips: Tests that don't test2019-03-05T22:07:56+00:002019-03-05T22:07:56+00:00Simon Larséntag:slar.se,2019-03-05:/tests-that-dont-test.html<p>Unit testing is a skill that takes some time to develop, and there are numerous
pitfalls for the beginner. As I've done my fair share of unit testing, and
taught a lot of students what I know, I've decided to share my top tips of
things to think about when …</p><p>Unit testing is a skill that takes some time to develop, and there are numerous
pitfalls for the beginner. As I've done my fair share of unit testing, and
taught a lot of students what I know, I've decided to share my top tips of
things to think about when testing. First up is one that may seem obvious, but
beginners and experienced testers alike fail with on occasion: make sure you
are actually testing something.</p>
<h2>Tests that don't test</h2>
<p>Quite often, I find tests written by students that don't actually test anything,
and will pass regardless of what the student's code is doing. Sometimes, I find
tests written by yours truly that are similarly ineffective. A test that passes
when it should not is dangerous, because it makes you feel confident about code
that isn't properly tested. On the flip side, a test that fails when it should
not is annoying and may hamper productivity, but unlike a falsely positive test,
it is highly noticeable. The devious part of tests that don't test is that they
easily slip by unnoticed, you don't often investigate a test that passes! These
tests generally come in four flavors:</p>
<ol>
<li>Not calling the function under test.</li>
<li>Copy mistakes with references/pointers.</li>
<li>Mistakes during setup.</li>
<li>Mistakes with assertions.</li>
</ol>
<p>Even though I have a few years worth of testing experience, and have written
thousands upon thousands of tests, I still make these mistakes from time to
time. Let's first go over them one by one to get a feel for what can go wrong.
After that, I'll share my techniques for catching these errors. For all of the
examples, we will look at a test case for sorting a randomly ordered list with
an in-place sorting algorithm. The implementation under test is called <code>mysort</code>.
Assume that, for all examples, a list called <code>random_list</code> with randomly ordered
elements is setup in a fixture. The tests will be written in <code>pytest</code> syntax,
but most problems and solutions are easily transferable to many other languages
and testing frameworks (e.g. JUnit in Java). Here is the test header and
docstring. Note the inclusion of the <code>random_list</code> fixture as a parameter. In the test,
it can simply be used as a list.</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">test_sort_randomly_ordered_list</span><span class="p">(</span><span class="n">random_list</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Sort a randomly ordered list and ensure that the result for</span>
<span class="sd"> ``mysort`` is the same as the built-in ``list.sort``</span>
<span class="sd"> """</span>
</code></pre></div>
<p>For brevity, the docstring will be excluded from now on. Let's get to it the,
shall we?</p>
<h3>Not calling the function under test</h3>
<p>This mistake definitely sits in the top two most common ones that I encounter. A
typical example of this is when using <em>redundant computation</em> to produce a test
oracle. That is, using some other implementation of the function under test to
compute the expected result. What I've seen happen many times is that the
student by mistake uses the other implementation for both the expected value,
and the actual value. Here's an example.</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">test_sort_randomly_ordered_list</span><span class="p">(</span><span class="n">random_list</span><span class="p">):</span>
<span class="c1"># calculate test oracle</span>
<span class="n">expected</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">random_list</span><span class="p">)</span> <span class="c1"># note the copy for later!</span>
<span class="n">expected</span><span class="o">.</span><span class="n">sort</span><span class="p">()</span>
<span class="c1"># calculate actual value, use ``sort`` by mistake</span>
<span class="c1"># should be ``mysort(random_list)``</span>
<span class="n">random_list</span><span class="o">.</span><span class="n">sort</span><span class="p">()</span>
<span class="k">assert</span> <span class="n">random_list</span> <span class="o">==</span> <span class="n">expected</span>
</code></pre></div>
<p>Obviously, this test will always pass as <code>list.sort</code> is used for both
computations. This is a very common mistake, and if made once in a test suite, I
often find it propagating elsewhere due to copy-paste errors. This kind of
mistake is applicable in most any language, and is especially easy to make if
the redundant function and the function under test have similar names and usage
(which was actually not the case here!).</p>
<h3>Copy mistakes with references/pointers</h3>
<p>Another very common issue that is often related to redundant computation is
failing to make a proper copy of a data structure. If you have a look at the
previous example, there is comment telling you to note the copy. Compare that
with this example:</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">test_sort_randomly_ordered_list</span><span class="p">(</span><span class="n">random_list</span><span class="p">):</span>
<span class="c1"># calculate test oracle</span>
<span class="n">expected</span> <span class="o">=</span> <span class="n">random_list</span> <span class="c1"># this is not a copy!</span>
<span class="n">expected</span><span class="o">.</span><span class="n">sort</span><span class="p">()</span>
<span class="c1"># calculate actual value</span>
<span class="n">mysort</span><span class="p">(</span><span class="n">random_list</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">random_list</span> <span class="o">==</span> <span class="n">expected</span>
</code></pre></div>
<p>Just assigning <code>expected = random_list</code> will not create a copy of <code>random_list</code>,
but copy the reference to the list. Therefore, both <code>expected</code> and <code>random_list</code>
reference the <em>same list</em>. The assertion is then semantically equivalent to
<code>assert random_list == random_list</code>, which is obviously true no matter what
<code>mysort</code> did with the list. This is a problem in any language that uses
references (not C++ references, but pointer-like references), such as Java and
Python, or when dealing with pointers in pretty much any language that has them.</p>
<h3>Mistakes during setup</h3>
<p>This is also fairly common, and can manifest in a variety of ways. The general
idea is that the setup is performed such that the outcome of the test is very
likely to be the same even if the production code is anything but correct. One
example would be that the supposedly randomly ordered list is actually
comprised of duplicates of a single element. Let's have a look at an incorrect
implementation of the <code>random_list</code> fixture. Note that <code>_</code> is used as a
variable name when we don't care about the value of it.</p>
<div class="highlight"><pre><span></span><code><span class="nd">@pytest</span><span class="o">.</span><span class="n">fixture</span>
<span class="k">def</span> <span class="nf">random_list</span><span class="p">():</span>
<span class="w"> </span><span class="sd">"""Generate a randomly ordered list with 100 elements."""</span>
<span class="n">lst</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span>
<span class="n">random</span><span class="o">.</span><span class="n">seed</span><span class="p">(</span><span class="mi">5234</span><span class="p">)</span> <span class="c1"># seed to make list generation deterministic</span>
<span class="n">lst</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="o">-</span><span class="mi">100</span><span class="p">,</span> <span class="mi">100</span><span class="p">))</span>
<span class="k">return</span> <span class="n">lst</span>
</code></pre></div>
<p>It is good practice to <em>seed</em> the pseudo-random generator (PRG) when testing to
make tests reproducible. A PRG is actually a deterministic function that, given
an initial state (a seed), will always produce the same sequence of numbers.
<code>random.seed(5234)</code> sets this initial state to <code>5234</code>. This fixture is actually
fairly well implemented, but has a critical error. Since the seed is set inside
the loop, before the call to <code>random.randint</code>, the latter will always produce
the same value. As the list is already sorted, <code>mysort</code> can do almost anything
but remove an element and still pass the test. This is a fairly sophisticated
error that an intermediate tester may accidentally make. There are infinite
variations on how setup may go wrong, and this is applicable to pretty much any
programming language. As a side note, the correct way to do this would of
course be to seed <em>before</em> the loop. Note that even with the correct
configuration, there is a very small chance that the random elements are
generated in ascending order.</p>
<h3>Mistakes with assertions</h3>
<p>The final issue is also common, and comes in many shapes and forms. One thing I
sometimes see is that the assertions are tautologies, such as <code>assert
random_list == random_list</code> (obviously true), and probably mostly result from
typos and unchecked auto-completion. Another common one is that assertions are
simply missing, and is most often found in tests that are large enough that a
missing line or two is not immediately apparent.</p>
<h2>Finding tests that don't test</h2>
<p>There are essentially two ways I know of to find tests that (pretty much) never
fail.</p>
<ol>
<li>Write the tests first (Test-driven development)</li>
<li>Inject errors into production code and expect tests to fail</li>
</ol>
<h3>Test-driven development (TDD)</h3>
<p>TDD involves writing the test cases before you implement the functionality.
You first write the test cases, ensure that the test cases fail, and then
implement the production code such that the tests pas. I typically use TDD
when:</p>
<ul>
<li>The functionality I need to implement is strictly defined.<ul>
<li>Fox example when implementing well-defined algorithms and data structures.</li>
</ul>
</li>
<li>I'm fixing a bug.<ul>
<li>Reproduce the bug with a test-case, then fix it!</li>
</ul>
</li>
</ul>
<p>This approach will catch many incarnations of the errors I've brought up in
this article simply because the tests should definitely not pass before the
production code is even written. There is one caveat, though. Some
practitioners of TDD think that test cases should be written even before the
function skeletons have been written, and argue that a compilation failure is
also a test failure. With that approach, you probably will not catch any of the
errors brought up here, except maybe
<a href="#not-calling-the-function-under-test">the first one</a>. My recommendation for
TDD is to write function skeletons and make sure the function can actually be
called (it's perfectly fine if it crashes after being called). <em>Then</em> write
your tests, and make sure they fail before you start implementing production
code. I don't think TDD is always practical to use, however, especially when
I'm a bit unsure of what to do and need to experiment with different APIs.
That's when the second technique comes in real handy.</p>
<h3>Inject errors into production code and expect tests to fail</h3>
<p>This is a highly useful technique that can always be performed, and I do this
almost every time I implement tests after production code. The idea is simply to
consider what your test is testing, and inject errors into the production code
such that the test should fail. <code>test_sort_randomly_ordered_list</code> is a fairly
broad test case, so we can inject fairly general errors. A simple example would
simply be to return early such that <code>mysort</code> does not sort at all. Narrower test
cases may require more sophisticated errors to be injected.</p>
<blockquote>
<p><strong>Aside: Mutation testing</strong> There is actually a whole field of testing
dedicated to this kind of error (or <em>fault</em>) injection called
<a href="https://en.wikipedia.org/wiki/Mutation_testing">mutation testing</a>. Faults are
automatically injected into production code, and the test suite is run to
determine whether the fault is found (<em>killed</em>) or not. There are frameworks
for this, such as the <a href="http://pitest.org/">Pitest</a> for Java, and
<a href="https://github.com/sixty-north/cosmic-ray">Cosmic Ray</a> for Python. In
general, it takes a <em>long</em> time to run mutation testing on a test suite, as
often the whole test suite needs to be run for a single fault. And there are
many, many possible faults.</p>
</blockquote>
<h2>Summary</h2>
<p>While I framed this as a unit testing article, these concepts are applicable to
most kinds of testing. You should always attempt to make sure that your test is
doing what it claims to be doing. A single typo may be what stands between a
test that does not test, and a test that does. This article focused on finding
tests that don't test, but there are also things you can do to <em>prevent</em> tests
that don't test from manifesting. Copy/pasting test code and then making minor
changes is for example a common source of most of the discussed errors. But
ultimately, there is no surefire way of avoiding tests that don't test, so I
strongly recommend that you actively search for them no matter what precautions
you take!</p>TornadoFX+Exposed pt. 3: Adding, editing and removing rows2018-12-30T14:50:50+00:002018-12-30T15:13:57+00:00Simon Larséntag:slar.se,2018-12-30:/tornadofxexposed-pt-3-adding-editing-and-removing-rows.html<p>Welcome to the third and final part in this article series on using TornadoFX
together with Exposed. In the previous two parts, we set up the database with a
single table and created a simple TornadoFX view with which we could view its
contents. Now, we will focus on adding …</p><p>Welcome to the third and final part in this article series on using TornadoFX
together with Exposed. In the previous two parts, we set up the database with a
single table and created a simple TornadoFX view with which we could view its
contents. Now, we will focus on adding and deleting rows to the <code>Categories</code>
table, as well as adding new ones. This part is a bit longer than the two
previous ones, but it also contains a whole lot more content.</p>
<blockquote>
<p>The full source code is available
<a href="https://github.com/slarse/tornadofx-exposed-example/tree/part_3">on GitHub</a></p>
</blockquote>
<h1>Article index</h1>
<ol>
<li><a href="https://slar.se/tornadofxexposed-pt-1-project-and-database-setup.html">Project and database setup</a></li>
<li><a href="https://slar.se/tornadofxexposed-pt-2-showing-a-database-table.html">Showing a database table</a></li>
<li>Adding, editing and removing rows -- This part!</li>
</ol>
<h1>Making the app interactive</h1>
<p>So far, all we can do with our app is view the contents of the database. That's
neat and all, but it would be even nicer if we could interact with the database
and edit its contents. What this article will address is how to:</p>
<ol>
<li>Delete rows.</li>
<li>Add rows.</li>
<li>Edit rows.</li>
</ol>
<p>Deleting rows is the simplest thing to accomplish, so let's start with that.</p>
<h2>Deleting rows</h2>
<p>Deleting a row is pretty easy. First, we'll add the desired functionality to
the controller.</p>
<div class="highlight"><pre><span></span><code><span class="kd">fun</span><span class="w"> </span><span class="nf">deleteCategory</span><span class="p">(</span><span class="n">model</span><span class="p">:</span><span class="w"> </span><span class="n">CategoryModel</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">transaction</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">model</span><span class="p">.</span><span class="na">item</span><span class="p">.</span><span class="na">delete</span><span class="p">()</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">categories</span><span class="p">.</span><span class="na">remove</span><span class="p">(</span><span class="n">model</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div>
<p>Note that <code>model.item</code> returns the backing <code>Category</code> object, on which we
simply call <code>delete()</code> to remove it from the database. Then, we also have to
update our local list by removing the model from it. Note that I assume <code>model</code>
to be in the <code>categories</code> list for the sake of simplicity, but this is a pretty
bold assumption that you probably should not make in a real application. Now,
let's put this new functionality to work: we need to add a button to the view
that calls the delete function on the currently selected row. We will slightly
alter the layout to make this happen. We change this:</p>
<div class="highlight"><pre><span></span><code><span class="kd">override</span><span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="nv">root</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">borderpane</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">categories</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">dbController</span><span class="p">.</span><span class="na">categories</span>
<span class="w"> </span><span class="n">center</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">tableview</span><span class="o"><</span><span class="n">CategoryModel</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">categoryTable</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">editModel</span>
<span class="w"> </span><span class="n">items</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">categories</span>
<span class="w"> </span><span class="n">column</span><span class="p">(</span><span class="s">"Name"</span><span class="p">,</span><span class="w"> </span><span class="n">CategoryModel</span><span class="o">::</span><span class="n">name</span><span class="p">)</span>
<span class="w"> </span><span class="n">column</span><span class="p">(</span><span class="s">"Description"</span><span class="p">,</span><span class="w"> </span><span class="n">CategoryModel</span><span class="o">::</span><span class="n">description</span><span class="p">)</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>to this:</p>
<div class="highlight"><pre><span></span><code><span class="kd">override</span><span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="nv">root</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">borderpane</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">categories</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">dbController</span><span class="p">.</span><span class="na">categories</span>
<span class="w"> </span><span class="n">center</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">vbox</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">buttonbar</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">button</span><span class="p">(</span><span class="s">"DELETE SELECTED"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">action</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="nv">model</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">categoryTable</span><span class="p">.</span><span class="na">tableView</span><span class="p">.</span><span class="na">selectedItem</span>
<span class="w"> </span><span class="k">when</span><span class="w"> </span><span class="p">(</span><span class="n">model</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kc">null</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="k">return</span><span class="nd">@action</span>
<span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">dbController</span><span class="p">.</span><span class="na">deleteCategory</span><span class="p">(</span><span class="n">model</span><span class="p">)</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">tableview</span><span class="o"><</span><span class="n">CategoryModel</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">categoryTable</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">editModel</span>
<span class="w"> </span><span class="n">items</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">categories</span>
<span class="w"> </span><span class="n">column</span><span class="p">(</span><span class="s">"Name"</span><span class="p">,</span><span class="w"> </span><span class="n">CategoryModel</span><span class="o">::</span><span class="n">name</span><span class="p">)</span>
<span class="w"> </span><span class="n">column</span><span class="p">(</span><span class="s">"Description"</span><span class="p">,</span><span class="w"> </span><span class="n">CategoryModel</span><span class="o">::</span><span class="n">description</span><span class="p">)</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>We use a <code>buttonbar</code> as we will be adding more buttons later on. The code should
be fairly easy to read: the button's action will do nothing if the currently
selected model is <code>null</code> (i.e. nothing is selected), and call the
<code>deleteCategory</code> method otherwise. You should now have a view looking something
like this:</p>
<p><img alt="Table view with a delete button" src="https://slar.se/images/tornado_exposed/view_with_delete.jpg"></p>
<p>If you first click a row and then the delete button, the row should disappear.
Now that we can delete rows, let's turn our attention to adding new rows.</p>
<h2>Adding new rows</h2>
<p>For this, we're going to add a small form to the right of the table which will
allow us to enter new rows. As before, we'll start with the controller, adding
the following method to it:</p>
<div class="highlight"><pre><span></span><code><span class="kd">fun</span><span class="w"> </span><span class="nf">addCategory</span><span class="p">(</span><span class="n">name</span><span class="p">:</span><span class="w"> </span><span class="kt">String</span><span class="p">,</span><span class="w"> </span><span class="n">description</span><span class="p">:</span><span class="w"> </span><span class="kt">String</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">transaction</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="nv">category</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Category</span><span class="p">.</span><span class="na">new</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="na">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">name</span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="na">description</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">description</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">categories</span><span class="p">.</span><span class="na">add</span><span class="p">(</span>
<span class="w"> </span><span class="n">CategoryModel</span><span class="p">().</span><span class="na">apply</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">category</span>
<span class="w"> </span><span class="p">})</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>Here, we first create a new <code>Category</code>, and then add it to the <code>categories</code>
list (wrapped in a <code>CategoryModel</code>). Now, we need to add the form to the view
so we can submit the values for <code>name</code> and <code>description</code>. First, we need to add
two new properties to the <code>CategoryEditor</code> view:</p>
<div class="highlight"><pre><span></span><code><span class="kd">var</span><span class="w"> </span><span class="nv">nameField</span><span class="p">:</span><span class="w"> </span><span class="n">TextField</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="n">singleAssign</span><span class="p">()</span>
<span class="kd">var</span><span class="w"> </span><span class="nv">descriptionField</span><span class="p">:</span><span class="w"> </span><span class="n">TextField</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="n">singleAssign</span><span class="p">()</span>
</code></pre></div>
<p>We need these to be able to access what we put in the form fields. We also
need to import <code>TextField</code></p>
<div class="highlight"><pre><span></span><code><span class="k">import</span><span class="w"> </span><span class="nn">javafx.scene.control.TextField</span>
</code></pre></div>
<p>To add the actual form, we put the following <em>after</em> the <code>center</code> element:</p>
<div class="highlight"><pre><span></span><code><span class="n">right</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">form</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">fieldset</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">field</span><span class="p">(</span><span class="s">"Name"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">textfield</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">nameField</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">this</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">fieldset</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">field</span><span class="p">(</span><span class="s">"Description"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">textfield</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">descriptionField</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">this</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">button</span><span class="p">(</span><span class="s">"ADD CATEGORY"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">action</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">dbController</span><span class="p">.</span><span class="na">addCategory</span><span class="p">(</span><span class="n">nameField</span><span class="p">.</span><span class="na">text</span><span class="p">,</span><span class="w"> </span><span class="n">descriptionField</span><span class="p">.</span><span class="na">text</span><span class="p">)</span>
<span class="w"> </span><span class="n">nameField</span><span class="p">.</span><span class="na">text</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">""</span>
<span class="w"> </span><span class="n">descriptionField</span><span class="p">.</span><span class="na">text</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">""</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>This will result in a view looking something like this:</p>
<p><img alt="View with add category form" src="https://slar.se/images/tornado_exposed/view_with_add_form.jpg"></p>
<p>Writing some stuff in the fields and clicking <code>ADD CATEGORY</code> should immediately
create a new row in the table. Not the most beautiful thing in the world, I'll
admit, but it serves its purpose for this guide. Now we only have one more
feature to add, namely editing rows.</p>
<h2>Editing rows</h2>
<p>Now we will finally see why we used a <code>TableViewEditModel</code> instead of a plain
<code>TableView</code>: the former allows us to edit rows directly in the table. To allow
for inline editing, we need to add some stuff to the view itself. Our table
view currently looks like this:</p>
<div class="highlight"><pre><span></span><code><span class="n">tableview</span><span class="o"><</span><span class="n">CategoryModel</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">categoryTable</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">editModel</span>
<span class="w"> </span><span class="n">items</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">categories</span>
<span class="w"> </span><span class="n">column</span><span class="p">(</span><span class="s">"Name"</span><span class="p">,</span><span class="w"> </span><span class="n">CategoryModel</span><span class="o">::</span><span class="n">name</span><span class="p">)</span>
<span class="w"> </span><span class="n">column</span><span class="p">(</span><span class="s">"Description"</span><span class="p">,</span><span class="w"> </span><span class="n">CategoryModel</span><span class="o">::</span><span class="n">description</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div>
<p>To enable editing, we simply add a call to <code>enableCellEditing()</code>, and call
<code>makeEditable()</code> on the columns. We'll also add <code>enableDirtyTracking()</code> to
allow us to see which cells have been edited, but not saved.</p>
<div class="highlight"><pre><span></span><code><span class="n">tableview</span><span class="o"><</span><span class="n">CategoryModel</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">categoryTable</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">editModel</span>
<span class="w"> </span><span class="n">items</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">categories</span>
<span class="w"> </span><span class="n">enableCellEditing</span><span class="p">()</span>
<span class="w"> </span><span class="n">enableDirtyTracking</span><span class="p">()</span>
<span class="w"> </span><span class="n">column</span><span class="p">(</span><span class="s">"Name"</span><span class="p">,</span><span class="w"> </span><span class="n">CategoryModel</span><span class="o">::</span><span class="n">name</span><span class="p">).</span><span class="na">makeEditable</span><span class="p">()</span>
<span class="w"> </span><span class="n">column</span><span class="p">(</span><span class="s">"Description"</span><span class="p">,</span><span class="w"> </span><span class="n">CategoryModel</span><span class="o">::</span><span class="n">description</span><span class="p">).</span><span class="na">makeEditable</span><span class="p">()</span>
<span class="p">}</span>
</code></pre></div>
<p>Now, we can edit cells by clicking them:</p>
<p><img alt="Editing a cell" src="https://slar.se/images/tornado_exposed/editing_cell.jpg"></p>
<p>And after pressing enter, we can see that the cell has been edited by the blue
triangle. The cell is <em>dirty</em>:</p>
<p><img alt="Dirty cell" src="https://slar.se/images/tornado_exposed/edited_cell.jpg"></p>
<p>However, the change won't "stick". If we restart the application, the text will
be back to what it was before we edited the cell. The reason is that the change
was never committed to the database, it was just stored in the model. Thus,
what we need now is to commit any dirty rows to the database. As always, we
start with adding the functionality we need from the controller.</p>
<div class="highlight"><pre><span></span><code><span class="kd">fun</span><span class="w"> </span><span class="nf">commitDirty</span><span class="p">(</span><span class="n">modelDirtyMappings</span><span class="p">:</span><span class="w"> </span><span class="n">Sequence</span><span class="o"><</span><span class="n">Map</span><span class="p">.</span><span class="na">Entry</span><span class="o"><</span><span class="n">CategoryModel</span><span class="p">,</span><span class="w"> </span><span class="n">TableColumnDirtyState</span><span class="o"><</span><span class="n">CategoryModel</span><span class="o">>>></span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">transaction</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">modelDirtyMappings</span><span class="p">.</span><span class="na">filter</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nb">it</span><span class="p">.</span><span class="na">value</span><span class="p">.</span><span class="na">isDirty</span><span class="w"> </span><span class="p">}.</span><span class="na">forEach</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nb">it</span><span class="p">.</span><span class="na">key</span><span class="p">.</span><span class="na">commit</span><span class="p">()</span><span class="w"> </span><span class="c1">// commit value to database</span>
<span class="w"> </span><span class="nb">it</span><span class="p">.</span><span class="na">value</span><span class="p">.</span><span class="na">commit</span><span class="p">()</span><span class="w"> </span><span class="c1">// clear dirty state</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>This function iterates over a sequence of map entries that map a model (key) to
a dirty state (value). We'll soon see that we can get such a map from the table
view. Note that committing the key <em>must</em> be done in a transaction, as it will
write to the database. The type is a bit of a mouthful, though, so let's define
a type alias for it.</p>
<div class="highlight"><pre><span></span><code><span class="k">typealias</span><span class="w"> </span><span class="n">ModelToDirtyState</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Map</span><span class="p">.</span><span class="na">Entry</span><span class="o"><</span><span class="n">CategoryModel</span><span class="p">,</span><span class="w"> </span><span class="n">TableColumnDirtyState</span><span class="o"><</span><span class="n">CategoryModel</span><span class="o">>></span>
</code></pre></div>
<p>Note that the <code>typealias</code> must be a top level declaration (i.e. you can't put
it in a class or function). And rewrite the header of <code>commitDirty</code> like this:</p>
<div class="highlight"><pre><span></span><code><span class="kd">fun</span><span class="w"> </span><span class="nf">commitDirty</span><span class="p">(</span><span class="n">modelDirtyMappings</span><span class="p">:</span><span class="w"> </span><span class="n">Sequence</span><span class="o"><</span><span class="n">ModelToDirtyState</span><span class="o">></span><span class="p">)</span>
</code></pre></div>
<p>Slightly more readable, right? Now, let's put it to use. We'll add a new button
in the button bar to execute the commit.</p>
<div class="highlight"><pre><span></span><code><span class="n">button</span><span class="p">(</span><span class="s">"COMMIT"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">action</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">dbController</span><span class="p">.</span><span class="na">commitDirty</span><span class="p">(</span><span class="n">categoryTable</span><span class="p">.</span><span class="na">items</span><span class="p">.</span><span class="na">asSequence</span><span class="p">())</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>Clicking this button when there are dirty cells will allow us to commit these
to the database. As a finishing touch, we'll add a button to reset (rollback)
dirty cells to their previous state.</p>
<div class="highlight"><pre><span></span><code><span class="n">button</span><span class="p">(</span><span class="s">"ROLLBACK"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">action</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">categoryTable</span><span class="p">.</span><span class="na">rollback</span><span class="p">()</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>Note that this does not require a transaction, as all that happens is that the
model state is reset (the DAO is unaffected). This will leave us with a final
GUI looking like this:</p>
<p><img alt="Final GUI" src="https://slar.se/images/tornado_exposed/final_gui.jpg"></p>
<h1>Closing words</h1>
<p>That was all for this series of articles on TornadoFX and exposed. This is by
no means a fully-fledged database UI, but it is a pretty good start. There are
tons of things here that need to be improved, though. Below are a few examples
off the top of my head.</p>
<ul>
<li>There is just about no error handling, everything is just assumed to work
out. For example, if a user enters a duplicate category, an unhandled
exception is raised.</li>
<li>Much of the functionality is very specific to the <code>Category</code> type, and needs
to be generalized. As a lot of this is done with generics, such
generalization is actually not trivial (as generic types are invariant by
default).</li>
<li>There is a lot of room for user error. For example, deleting a row is done
without prompting the user with something like "Are you sure you wanna do
this?". The commit/rollback functionality of editing is much more user
friendly and a step in the right direction.</li>
<li>The views are completely unstyled and look rather dull.</li>
</ul>
<p>And with that, I wish you good fortune in working with this! Of course, you are
free to use all of these examples as you see fit.</p>TornadoFX+Exposed pt. 2: Showing a database table2018-12-26T09:09:47+00:002018-12-30T15:14:09+00:00Simon Larséntag:slar.se,2018-12-26:/tornadofxexposed-pt-2-showing-a-database-table.html<p>Welcome to the second part of the TornadoFX+Exposed series of articles. In this
part, we'll take a look at how to create a TornadoFX view for the <code>Categories</code>
table. In the next part, we'll expand upon the view and make it possible to
add, edit and delete rows.</p>
<blockquote>
<p>The …</p></blockquote><p>Welcome to the second part of the TornadoFX+Exposed series of articles. In this
part, we'll take a look at how to create a TornadoFX view for the <code>Categories</code>
table. In the next part, we'll expand upon the view and make it possible to
add, edit and delete rows.</p>
<blockquote>
<p>The full source code is available
<a href="https://github.com/slarse/tornadofx-exposed-example/tree/part_2">on GitHub</a></p>
</blockquote>
<h1>Article index</h1>
<ol>
<li><a href="https://slar.se/tornadofxexposed-pt-1-project-and-database-setup.html">Project and database setup</a></li>
<li>Showing a database table -- This part!</li>
<li><a href="https://slar.se/tornadofxexposed-pt-3-adding-editing-and-removing-rows.html">Adding, editing and removing rows</a></li>
</ol>
<h1>Creating a table view</h1>
<p>To be able to view the <code>Categories</code> table, we're going to need three things:</p>
<ol>
<li>A view model to wrap <code>Category</code> instances. We can actually get away without
having a model, but having a model greatly simplifies some of the operations
we will implement in the next article.</li>
<li>A controller for interacting with the database</li>
<li>A view for displaying the data.</li>
</ol>
<p>We will do all of this in a new file called <code>categoryview.kt</code>. Let's start with
the view model, as it is by far the simplest component.</p>
<h2>An ItemViewModel wrapper for Category</h2>
<p>For this, we'll extend a utility class called <code>ItemViewModel</code> (you can read
about it in detail in the
<a href="https://edvin.gitbooks.io/tornadofx-guide/part1/11.%20Editing%20Models%20and%20Validation.html">TornadoFX guide</a>.
It will simply look like this:</p>
<div class="highlight"><pre><span></span><code><span class="k">import</span><span class="w"> </span><span class="nn">tornadofx.*</span>
<span class="kd">class</span><span class="w"> </span><span class="nc">CategoryModel</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">ItemViewModel</span><span class="o"><</span><span class="n">Category</span><span class="o">></span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="nv">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">bind</span><span class="p">(</span><span class="n">Category</span><span class="o">::</span><span class="n">name</span><span class="p">)</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="nv">description</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">bind</span><span class="p">(</span><span class="n">Category</span><span class="o">::</span><span class="n">description</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div>
<p>This is essentially a proxy for the <code>Category</code> class, acting as a middle-man
between the presentation layer and the database access layer. Any change we
make to a <code>CategoryModel</code> in the GUI will be stored in the model alone, and
will only propagate to the underlying <code>Category</code> object when we <em>commit</em> the
change(s). This is very convenient, as it allows us to buffer changes and then
commit all of them in a single database transaction, instead of having one
transaction per change. Now, let's move on to the controller.</p>
<h2>The database controller</h2>
<p>The controller is also fairly simple. Initially, it will only be able to fetch
items from the database. In the next article, we will extend the controller
with add and delete-functionality. Here's the initial version of the
controller:</p>
<div class="highlight"><pre><span></span><code><span class="k">import</span><span class="w"> </span><span class="nn">javafx.collections.ObservableList</span>
<span class="k">import</span><span class="w"> </span><span class="nn">java.sql.Connection</span>
<span class="k">import</span><span class="w"> </span><span class="nn">org.jetbrains.exposed.sql.Database</span>
<span class="k">import</span><span class="w"> </span><span class="nn">org.jetbrains.exposed.sql.transactions.transaction</span>
<span class="k">import</span><span class="w"> </span><span class="nn">org.jetbrains.exposed.sql.transactions.TransactionManager</span>
<span class="kd">class</span><span class="w"> </span><span class="nc">DBController</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">Controller</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="nv">categories</span><span class="p">:</span><span class="w"> </span><span class="n">ObservableList</span><span class="o"><</span><span class="n">CategoryModel</span><span class="o">></span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="n">lazy</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">transaction</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">Category</span><span class="p">.</span><span class="na">all</span><span class="p">().</span><span class="na">map</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">CategoryModel</span><span class="p">().</span><span class="na">apply</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">it</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}.</span><span class="na">observable</span><span class="p">()</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">init</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">Database</span><span class="p">.</span><span class="na">connect</span><span class="p">(</span><span class="s">"jdbc:sqlite:file:data.sqlite"</span><span class="p">,</span><span class="w"> </span><span class="n">driver</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"org.sqlite.JDBC"</span><span class="p">)</span>
<span class="w"> </span><span class="n">TransactionManager</span><span class="p">.</span><span class="na">manager</span><span class="p">.</span><span class="na">defaultIsolationLevel</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Connection</span><span class="p">.</span><span class="na">TRANSACTION_SERIALIZABLE</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>The <code>categories</code> property is lazily initialized to a fetch from the database,
in which all <code>Category</code> DAOs are wrapped in <code>CategoryModel</code>s. There's a bit of
a trade-off here: it's more efficient to fetch the whole table only once and
then maintain the state with any objects that are added or <code>init</code> contains
precisely the same database connection setup that we used in the first article.
Let's move on to the actual view.</p>
<h2>The table view</h2>
<p>For the table view, we're going to use a <code>TableViewEditModel</code> instead of a
plain <code>TableView</code>. The reason is that the <code>TableViewEditModel</code> has some
additional functionality, most notably the ability to edit rows directly in the
table. Again, you can read up on the details in the
<a href="https://edvin.gitbooks.io/tornadofx-guide/part1/11.%20Editing%20Models%20and%20Validation.html">TornadoFX guide</a>. Our initial attempt looks like this:</p>
<div class="highlight"><pre><span></span><code><span class="kd">class</span><span class="w"> </span><span class="nc">CategoryEditor</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">View</span><span class="p">(</span><span class="s">"Categories"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="nv">dbController</span><span class="p">:</span><span class="w"> </span><span class="n">DBController</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="n">inject</span><span class="p">()</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nv">categoryTable</span><span class="p">:</span><span class="w"> </span><span class="n">TableViewEditModel</span><span class="o"><</span><span class="n">CategoryModel</span><span class="o">></span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="n">singleAssign</span><span class="p">()</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nv">categories</span><span class="p">:</span><span class="w"> </span><span class="n">ObservableList</span><span class="o"><</span><span class="n">CategoryModel</span><span class="o">></span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="n">singleAssign</span><span class="p">()</span>
<span class="w"> </span><span class="kd">override</span><span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="nv">root</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">borderpane</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">categories</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">dbController</span><span class="p">.</span><span class="na">categories</span>
<span class="w"> </span><span class="n">center</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">tableview</span><span class="o"><</span><span class="n">CategoryModel</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">categoryTable</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">editModel</span>
<span class="w"> </span><span class="n">items</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">categories</span>
<span class="w"> </span><span class="n">column</span><span class="p">(</span><span class="s">"Name"</span><span class="p">,</span><span class="w"> </span><span class="n">CategoryModel</span><span class="o">::</span><span class="n">name</span><span class="p">)</span>
<span class="w"> </span><span class="n">column</span><span class="p">(</span><span class="s">"Description"</span><span class="p">,</span><span class="w"> </span><span class="n">CategoryModel</span><span class="o">::</span><span class="n">description</span><span class="p">)</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>There's not too much going on here. The three properties store references to
the controller, the table view, and the list of categories. The view itself is
not very eventful either, we simply fetch the categories using the controller
and initialize the table view. Note that <code>editModel</code> and <code>items</code> are properties
of the <code>TableViewEditModel</code>, where the former is a reference to the table and
the latter the property containing the items of the table (which we set to the
<code>categories</code> observable list). Later, when we wish to update the table, we
simply work with the <code>categories</code> list. Don't worry that there are some unused
references here, we will put them to use in the next article.</p>
<h2>Creating a runnable app</h2>
<p>Now, we just need to make the app runnable. That's as simple as adding the
following:</p>
<div class="highlight"><pre><span></span><code><span class="kd">class</span><span class="w"> </span><span class="nc">Kuizzy</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">App</span><span class="p">(</span><span class="n">CategoryEditor</span><span class="o">::</span><span class="n">class</span><span class="p">)</span>
<span class="kd">fun</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="n">args</span><span class="p">:</span><span class="w"> </span><span class="n">Array</span><span class="o"><</span><span class="kt">String</span><span class="o">></span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">launch</span><span class="o"><</span><span class="n">Kuizzy</span><span class="o">></span><span class="p">(</span><span class="n">args</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div>
<p>Running the main method will start the app, and you should then see a view that
looks something like this:</p>
<p><img alt="Read only view" src="https://slar.se/images/tornado_exposed/read_only_view.jpg"></p>
<p>That's pretty much it for this part. In the next and final part, we'll look
into how to add, delete and edit rows of the <code>Categories</code> table. You can find
<a href="https://slar.se/tornadofxexposed-pt-3-adding-editing-and-removing-rows.html">part 3 here.</a></p>TornadoFX+Exposed pt. 1: Project and database setup2018-12-25T22:42:36+00:002018-12-30T14:53:11+00:00Simon Larséntag:slar.se,2018-12-25:/tornadofxexposed-pt-1-project-and-database-setup.html<p>I recently got it into my head that I'd like to make a quiz game with a GUI,
which felt like a simple enough diversion during the holidays. Since I already
have this site to maintain in terms of web development, I figured that desktop
app development in Kotlin using …</p><p>I recently got it into my head that I'd like to make a quiz game with a GUI,
which felt like a simple enough diversion during the holidays. Since I already
have this site to maintain in terms of web development, I figured that desktop
app development in Kotlin using the TornadoFX framework would be a nice change
of pace. <em>Kuizzy</em>, which is what I call the project, will obviously need some
kind of data storage for questions and the like, so I settled on usind
JetBrains' framework Exposed with a sqlite database. Starting out, I had
trouble figuring out how to use TornadoFX and Exposed together, and therefore
decided to write this three-part series of articles on how I managed to make it
work. I won't dive deep into either, but rather show by example how to perform
some elementary tasks. In the end, we'll have a small piece of the database
admin part of Kuizzy up and running.</p>
<blockquote>
<p>The full source code is available
<a href="https://github.com/slarse/tornadofx-exposed-example/tree/part_1">on GitHub</a></p>
</blockquote>
<h1>Article index</h1>
<ol>
<li>Project and database setup -- This part!</li>
<li><a href="https://slar.se/tornadofxexposed-pt-2-showing-a-database-table.html">Showing a database table</a></li>
<li><a href="https://slar.se/tornadofxexposed-pt-3-adding-editing-and-removing-rows.html">Adding, editing and removing</a></li>
</ol>
<h1>Getting started</h1>
<p>In this first article, we will be concerned only with getting everything set up.
This includes getting the dependencies and setting up the database with a table.
I will assume that you know how to use Kotlin, and how to handle dependencies
(e.g. by using a build system like Gradle, or just doing it manually). These are
things that I will not explain, as there are plenty of resources for that
available elsewhere. It's also important to note that these articles are not
meant to be seen as <em>the</em> way to do this. In order to keep the articles
reasonably focused and short, I take tons of shortcuts, completely eschew error
handling and create very specialized functionality. The point of this article
series is to show you how to get started with TornadoFX+Exposed, and you are
meant to develop it further on your own.</p>
<h2>What are we aiming for?</h2>
<p>I think it helps tremendously when reading something to have the end goal in
sight. What we're shooting for here is an interface that looks something like
this:</p>
<p><img alt="Final app" src="https://slar.se/images/tornado_exposed/final_gui.jpg"></p>
<p>We will be able to create and delete rows, as well as edit rows directly in
the table. It's not
pretty and it's not very user friendly, but it conveys an idea and has all the
basic functionality required do administrate a single-table database.
Now that you have a rough idea of what we're trying to accomplish, let's
have a look at what libraries and tools we need to make it happen.</p>
<h2>Preliminaries</h2>
<p>Before we can get started, we need to make sure all dependencies are accounted
for. Here's a complete list of the libraries and frameworks I'll be using
throughout this series:</p>
<ul>
<li>Java 8 and openjfx 8<ul>
<li>Note that if you install Oracle's JDK, JavaFX is included. You only need
openjfx if you use openjdk.</li>
</ul>
</li>
<li>Kotlin 1.3.0<ul>
<li>1.2+ should work fine</li>
</ul>
</li>
<li><a href="https://mvnrepository.com/artifact/no.tornado/tornadofx/1.7.17">TornadoFX 1.7.17</a></li>
<li><a href="https://mvnrepository.com/artifact/org.jetbrains.exposed/exposed/0.11.2">Exposed 0.11.2</a></li>
<li><a href="https://mvnrepository.com/artifact/org.xerial/sqlite-jdbc/3.25.2">xerial-sqlite-jdbc 3.25.2</a><ul>
<li>I'll be using sqlite, but using any other SQL database supported by
Exposed only requires changing a line or two of code.</li>
</ul>
</li>
<li>Gradle 5.0<ul>
<li>Any reasonably up-to-date version should work. You can also just use
whatever way you see fit to handle the dependencies.</li>
</ul>
</li>
</ul>
<p>Here's my <code>build.gradle</code>:</p>
<div class="highlight"><pre><span></span><code><span class="err">plugi</span><span class="kc">ns</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">'org.je</span><span class="kc">t</span><span class="err">brai</span><span class="kc">ns</span><span class="err">.ko</span><span class="kc">tl</span><span class="err">i</span><span class="kc">n</span><span class="err">.jvm'</span><span class="w"> </span><span class="err">versio</span><span class="kc">n</span><span class="w"> </span><span class="err">'</span><span class="mf">1.3.0</span><span class="err">'</span>
<span class="p">}</span>
<span class="err">group</span><span class="w"> </span><span class="err">'se.slarse'</span>
<span class="err">versio</span><span class="kc">n</span><span class="w"> </span><span class="err">'</span><span class="mf">0.0.1</span><span class="err">'</span>
<span class="err">reposi</span><span class="kc">t</span><span class="err">ories</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">mave</span><span class="kc">n</span><span class="err">Ce</span><span class="kc">ntral</span><span class="err">()</span>
<span class="w"> </span><span class="err">jce</span><span class="kc">nter</span><span class="err">()</span>
<span class="p">}</span>
<span class="err">depe</span><span class="kc">n</span><span class="err">de</span><span class="kc">n</span><span class="err">cies</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">compile</span><span class="w"> </span><span class="err">'org.je</span><span class="kc">t</span><span class="err">brai</span><span class="kc">ns</span><span class="err">.ko</span><span class="kc">tl</span><span class="err">i</span><span class="kc">n</span><span class="p">:</span><span class="err">ko</span><span class="kc">tl</span><span class="err">i</span><span class="kc">n</span><span class="mi">-</span><span class="err">s</span><span class="kc">t</span><span class="err">dlib</span><span class="mi">-</span><span class="err">jdk</span><span class="mi">8</span><span class="err">'</span>
<span class="w"> </span><span class="err">compile</span><span class="w"> </span><span class="err">'</span><span class="kc">n</span><span class="err">o.</span><span class="kc">t</span><span class="err">or</span><span class="kc">na</span><span class="err">do</span><span class="p">:</span><span class="kc">t</span><span class="err">or</span><span class="kc">na</span><span class="err">do</span><span class="kc">f</span><span class="err">x</span><span class="p">:</span><span class="mf">1.7.17</span><span class="err">'</span>
<span class="w"> </span><span class="err">compile</span><span class="w"> </span><span class="err">'org.je</span><span class="kc">t</span><span class="err">brai</span><span class="kc">ns</span><span class="err">.exposed</span><span class="p">:</span><span class="err">exposed</span><span class="p">:</span><span class="mf">0.11.2</span><span class="err">'</span>
<span class="w"> </span><span class="err">compile</span><span class="w"> </span><span class="err">'org.xerial</span><span class="p">:</span><span class="err">sqli</span><span class="kc">te</span><span class="mi">-</span><span class="err">jdbc</span><span class="p">:</span><span class="mf">3.25.2</span><span class="err">'</span>
<span class="p">}</span>
<span class="err">compileKo</span><span class="kc">tl</span><span class="err">i</span><span class="kc">n</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">ko</span><span class="kc">tl</span><span class="err">i</span><span class="kc">n</span><span class="err">Op</span><span class="kc">t</span><span class="err">io</span><span class="kc">ns</span><span class="err">.jvmTarge</span><span class="kc">t</span><span class="w"> </span><span class="err">=</span><span class="w"> </span><span class="s2">"1.8"</span>
<span class="p">}</span>
<span class="err">compileTes</span><span class="kc">t</span><span class="err">Ko</span><span class="kc">tl</span><span class="err">i</span><span class="kc">n</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">ko</span><span class="kc">tl</span><span class="err">i</span><span class="kc">n</span><span class="err">Op</span><span class="kc">t</span><span class="err">io</span><span class="kc">ns</span><span class="err">.jvmTarge</span><span class="kc">t</span><span class="w"> </span><span class="err">=</span><span class="w"> </span><span class="s2">"1.8"</span>
<span class="p">}</span>
</code></pre></div>
<h2>Setting up the database</h2>
<p>Any quiz game worth it's salt has categories, and that's the part of the
database that we'll develop. Exposed allows us to interact with a SQL database
in two different ways: through the SQL Domain Specific Language (DSL), or
through the Data Access Object (DAO) pattern. I'll use the DAO, as I thought it
meshed nicely with TornadoFX. You can read about both of them on the
<a href="https://github.com/JetBrains/Exposed">Exposed GitHub page</a>. We will put all of
the database code in a file called <code>database.kt</code>. Let's first define the table
for categories.</p>
<div class="highlight"><pre><span></span><code><span class="k">import</span><span class="w"> </span><span class="nn">org.jetbrains.exposed.dao.*</span>
<span class="k">import</span><span class="w"> </span><span class="nn">org.jetbrains.exposed.sql.*</span>
<span class="k">import</span><span class="w"> </span><span class="nn">org.jetbrains.exposed.sql.transactions.TransactionManager</span>
<span class="k">import</span><span class="w"> </span><span class="nn">org.jetbrains.exposed.sql.transactions.transaction</span>
<span class="k">import</span><span class="w"> </span><span class="nn">java.sql.Connection</span>
<span class="kd">object</span><span class="w"> </span><span class="nc">Categories</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">IntIdTable</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="nv">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">varchar</span><span class="p">(</span><span class="s">"name"</span><span class="p">,</span><span class="w"> </span><span class="m">64</span><span class="p">).</span><span class="na">uniqueIndex</span><span class="p">()</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="nv">description</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">varchar</span><span class="p">(</span><span class="s">"description"</span><span class="p">,</span><span class="w"> </span><span class="m">128</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div>
<p>Note that we don't explicitly define the primary key, that's all handled in the
background. Along with the table (which is a singleton object), we also need to
represent rows.</p>
<div class="highlight"><pre><span></span><code><span class="kd">class</span><span class="w"> </span><span class="nc">Category</span><span class="p">(</span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="n">EntityID</span><span class="o"><</span><span class="kt">Int</span><span class="o">></span><span class="p">)</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">IntEntity</span><span class="p">(</span><span class="n">id</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">companion</span><span class="w"> </span><span class="kd">object</span><span class="w"> </span><span class="err">: </span><span class="nc">IntEntityClass</span><span class="o"><</span><span class="n">Category</span><span class="o">></span><span class="p">(</span><span class="n">Categories</span><span class="p">)</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nv">name</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="n">Categories</span><span class="p">.</span><span class="na">name</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nv">description</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="n">Categories</span><span class="p">.</span><span class="na">description</span>
<span class="w"> </span><span class="kd">override</span><span class="w"> </span><span class="kd">fun</span><span class="w"> </span><span class="nf">toString</span><span class="p">():</span><span class="w"> </span><span class="kt">String</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="s">"Category(name=\"</span><span class="si">$</span><span class="n">name</span><span class="s">\", description=\"</span><span class="si">$</span><span class="n">description</span><span class="s">\")"</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>The <code>Category</code> class is what we'll use to create DAOs, with which we can
create, modify and delete rows in the <code>Categories</code> table. Finally, let's create
the table and add some rows to it. We won't actually touch the <code>Categories</code>
object directly at all.</p>
<div class="highlight"><pre><span></span><code><span class="kd">fun</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="n">args</span><span class="p">:</span><span class="w"> </span><span class="n">Array</span><span class="o"><</span><span class="kt">String</span><span class="o">></span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">// "connect" to database file called data.sqlite in the current working directory</span>
<span class="w"> </span><span class="c1">// (creates the file if it does not exist)</span>
<span class="w"> </span><span class="n">Database</span><span class="p">.</span><span class="na">connect</span><span class="p">(</span><span class="s">"jdbc:sqlite:file:data.sqlite"</span><span class="p">,</span><span class="w"> </span><span class="n">driver</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"org.sqlite.JDBC"</span><span class="p">)</span>
<span class="w"> </span><span class="c1">// this isolation level is required for sqlite, may not be applicable to other DBMS</span>
<span class="w"> </span><span class="n">TransactionManager</span><span class="p">.</span><span class="na">manager</span><span class="p">.</span><span class="na">defaultIsolationLevel</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Connection</span><span class="p">.</span><span class="na">TRANSACTION_SERIALIZABLE</span>
<span class="w"> </span><span class="n">transaction</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">addLogger</span><span class="p">(</span><span class="n">StdOutSqlLogger</span><span class="p">)</span>
<span class="w"> </span><span class="c1">// create the table</span>
<span class="w"> </span><span class="n">SchemaUtils</span><span class="p">.</span><span class="na">create</span><span class="p">(</span><span class="n">Categories</span><span class="p">)</span>
<span class="w"> </span><span class="c1">// add some entries</span>
<span class="w"> </span><span class="n">Category</span><span class="p">.</span><span class="na">new</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"java"</span>
<span class="w"> </span><span class="n">description</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"The Java programming language"</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">Category</span><span class="p">.</span><span class="na">new</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"cpp"</span>
<span class="w"> </span><span class="n">description</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"The C++ programming language"</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="c1">// new transaction to check the results</span>
<span class="w"> </span><span class="n">transaction</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">Category</span><span class="p">.</span><span class="na">all</span><span class="p">().</span><span class="na">forEach</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">println</span><span class="p">(</span><span class="nb">it</span><span class="p">)</span><span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>Note how all interactions with the database are conducted inside of a
<code>transaction</code> (which is a function taking a lambda, that abstracts a database
transaction). You'll see this several times throughout these articles. That's
it for database setup! If you run main function, you'll get output that looks
something like this:</p>
<div class="highlight"><pre><span></span><code>SQL:<span class="w"> </span>CREATE<span class="w"> </span>TABLE<span class="w"> </span>IF<span class="w"> </span>NOT<span class="w"> </span>EXISTS<span class="w"> </span>Categories<span class="w"> </span><span class="o">(</span>id<span class="w"> </span>INTEGER<span class="w"> </span>PRIMARY<span class="w"> </span>KEY,<span class="w"> </span>name<span class="w"> </span>VARCHAR<span class="o">(</span><span class="m">64</span><span class="o">)</span><span class="w"> </span>NOT<span class="w"> </span>NULL,<span class="w"> </span>description<span class="w"> </span>VARCHAR<span class="o">(</span><span class="m">128</span><span class="o">)</span><span class="w"> </span>NOT<span class="w"> </span>NULL<span class="o">)</span>
SQL:<span class="w"> </span>CREATE<span class="w"> </span>UNIQUE<span class="w"> </span>INDEX<span class="w"> </span>Categories_name<span class="w"> </span>ON<span class="w"> </span>Categories<span class="w"> </span><span class="o">(</span>name<span class="o">)</span>
SQL:<span class="w"> </span>INSERT<span class="w"> </span>INTO<span class="w"> </span>Categories<span class="w"> </span><span class="o">(</span>description,<span class="w"> </span>name<span class="o">)</span><span class="w"> </span>VALUES<span class="w"> </span><span class="o">(</span><span class="s1">'The Java programming language'</span>,<span class="w"> </span><span class="s1">'java'</span><span class="o">)</span>
SQL:<span class="w"> </span>INSERT<span class="w"> </span>INTO<span class="w"> </span>Categories<span class="w"> </span><span class="o">(</span>description,<span class="w"> </span>name<span class="o">)</span><span class="w"> </span>VALUES<span class="w"> </span><span class="o">(</span><span class="s1">'The C++ programming language'</span>,<span class="w"> </span><span class="s1">'cpp'</span><span class="o">)</span>
Category<span class="o">(</span><span class="nv">name</span><span class="o">=</span><span class="s2">"java"</span>,<span class="w"> </span><span class="nv">description</span><span class="o">=</span><span class="s2">"The Java programming language"</span><span class="o">)</span>
Category<span class="o">(</span><span class="nv">name</span><span class="o">=</span><span class="s2">"cpp"</span>,<span class="w"> </span><span class="nv">description</span><span class="o">=</span><span class="s2">"The C++ programming language"</span><span class="o">)</span>
</code></pre></div>
<p>If you run the main function again, it will fail the unique constraint on the
<code>name</code> attribute and crash. And that's it for part 1. In the next part, we'll
look at how to create a read-only view of the <code>Categories</code> table. You can
find
<a href="https://slar.se/tornadofxexposed-pt-2-showing-a-database-table.html">part 2 here</a>.</p>Collapsing and expanding HTML elements using (mostly) CSS2018-11-14T17:57:34+00:002018-11-14T17:57:34+00:00Simon Larséntag:slar.se,2018-11-14:/collapsing-and-expanding-html-elements-using-mostly-css.html<p>Sections that collapse and expand at the click of a button is fairly ubiquitous
across the web nowadays. It's especially handy for mobile, where the display is
much smaller than your typical computer monitor. In this article, I'll walk you
through how to create a basic collapsible content-area using almost …</p><p>Sections that collapse and expand at the click of a button is fairly ubiquitous
across the web nowadays. It's especially handy for mobile, where the display is
much smaller than your typical computer monitor. In this article, I'll walk you
through how to create a basic collapsible content-area using almost only CSS,
along with a few lines of close-to-trivial JavaScript. The focus is on CSS, not
JavaScript, so you should be able to follow this even with the most rudimentary
programming experience.</p>
<h1>The fictional sidebar</h1>
<p>For this toy example, we will be creating a collapsible <code>div</code> (it could really
be just about any element) that can be collapsed and expanded by clicking a
"trigger". Just for the purpose of showing the effects more clearly, we'll do
it inside of another <code>div</code> element, which we'll imagine is a sidebar of a
website (like the sidebar with recent posts and tags on this site). Here's the
markup for the sidebar:</p>
<div class="highlight"><pre><span></span><code><span class="p"><</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"sidebar"</span><span class="p">></span>
<span class="cp"><!- our content goes in here! --></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
</code></pre></div>
<p>And the CSS:</p>
<div class="highlight"><pre><span></span><code>.sidebar {
width: 30%;
border: black solid 2px;
}
</code></pre></div>
<p>This is really <em>not</em> important for this demo, I just mention it so that you
don't wonder about some unknown HTML and CSS in the final demo. Now, let's fill
that sidebar up with some content.</p>
<div class="highlight"><pre><span></span><code><span class="p"><</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"sidebar"</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">id</span><span class="o">=</span><span class="s">"trigger"</span><span class="p">></span>Cool content heading<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">id</span><span class="o">=</span><span class="s">"content"</span><span class="p">></span>
<span class="p"><</span><span class="nt">p</span><span class="p">></span>
This content would be neat to hide and show at the click of a button!
<span class="p"></</span><span class="nt">p</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
</code></pre></div>
<p>We have 3 <code>div</code> tags in total: one is the container (sidebar) which really has
little to do with this article. The second is a heading for the content, which
will act as the trigger for showing and hiding the content. The third is the
one containing the content that we want to hide/show (a single paragraph).
Let's get to it! Note the <code>id</code> attributes on the two inner <code>div</code> tags. When I
later refer to the <code>#trigger</code>, I mean the div with <code>id="trigger"</code>, and likewise
for the <code>#content</code>. The <code>id</code> attributes serve no other purpose here, you can
remove them if you wish and everything will still work as expected.</p>
<h2>Collapsing and expanding the <code>div</code></h2>
<p>To collapse and expand <code>#content</code>, we will use two classes: <code>collapse-trigger</code>
and <code>collapse</code>. The basic idea is this:</p>
<ol>
<li>An element with the <code>collapse</code> is hidden by default.</li>
<li>If a <code>collapse</code> element follows an element with the <code>collapse-trigger</code> AND
the <code>active</code> classes, the <code>collapse</code> is visible.</li>
</ol>
<p>You can probably guess where to put the classes in the markup already:</p>
<div class="highlight"><pre><span></span><code><span class="p"><</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"sidebar"</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">id</span><span class="o">=</span><span class="s">"trigger"</span> <span class="na">class</span><span class="o">=</span><span class="s">"collapse-trigger"</span><span class="p">></span>Cool content heading<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">id</span><span class="o">=</span><span class="s">"content"</span> <span class="na">class</span><span class="o">=</span><span class="s">"collapsible"</span><span class="p">></span>
<span class="p"><</span><span class="nt">p</span><span class="p">></span>
This content would be neat to hide and show at the click of a button!
<span class="p"></</span><span class="nt">p</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
</code></pre></div>
<p>For the CSS, fulfilling point 1 above (<code>collapsible</code> hidden by default) is
simple:</p>
<div class="highlight"><pre><span></span><code><span class="p">.</span><span class="nc">collapsible</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">display</span><span class="p">:</span><span class="w"> </span><span class="kc">none</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>This will simply not display the element. But how do we fulfill the second
requirement? We can use the
<a href="https://developer.mozilla.org/en-US/docs/Web/CSS/Adjacent_sibling_selectors">adjacent sibling combinator</a>
(<code>+</code>). It's a selector combinator that allows us to match some element, only if
it is immediately preceded by some other element. For example, the selector <code>h1
+ p</code> will match any <code>p</code> tag that is immediately preceded by an <code>h1</code> tag:</p>
<div class="highlight"><pre><span></span><code><span class="nt"><h1></span>This<span class="w"> </span>is<span class="w"> </span>a<span class="w"> </span>heading<span class="nt"></h1></span>
<span class="nt"><p></span>This<span class="w"> </span>paragraph<span class="w"> </span>will<span class="w"> </span>be<span class="w"> </span>matched<span class="w"> </span>by<span class="w"> </span>"h1<span class="w"> </span>+<span class="w"> </span>p"<span class="nt"></p></span>
</code></pre></div>
<p>So, to show our <code>collapsible</code> when it is directly preceded by a
<code>collapse-trigger</code> AND <code>active</code> element, we do this:</p>
<div class="highlight"><pre><span></span><code><span class="p">.</span><span class="nc">collapse-trigger</span><span class="p">.</span><span class="nc">active</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">.</span><span class="nc">collapsible</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">display</span><span class="p">:</span><span class="w"> </span><span class="kc">block</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>We display it as a <code>block</code> here, but one could use other display modes as well,
depending on what visual effect is sought. Note that we are chaining two
classes in the left hand side of the <code>+</code> combinator, which means that an
element matches only if it's <code>class</code> attribute contains both of those classes
(and possibly more of them). The order of the classes is however not important,
i.e. <code>.active.collapse-trigger</code> would be equivalent.</p>
<p>That's actually all there is to it, as far as the CSS goes. Now, we can
collapse and expand the <code>#content</code> by opening the developer tools (<code>F12</code> in
Firefox and Chrome) and manually assigning the <code>active</code> class to <code>#trigger</code>.
But that's not very convenient in every day use. This is where we need the
tiniest bit of JavaScript to be able to toggle <code>active</code>.</p>
<h2>Toggling the <code>active</code> class with the click of a button</h2>
<p>What we want to do is to remove and add the <code>active</code> class from any
<code>collapse-trigger</code> element by clicking it. Here, we need JavaScript, because
there is no way to change the class of an element with only CSS. For every
<code>collapse-trigger</code> in the page, we need to attach an event listener that
toggles the <code>active</code> class every time the element is clicked. It can be done
like this:</p>
<div class="highlight"><pre><span></span><code><span class="kd">function</span><span class="w"> </span><span class="nx">attachCollapseTriggers</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">colTriggers</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">document</span><span class="p">.</span><span class="nx">getElementsByClassName</span><span class="p">(</span><span class="s2">"collapse-trigger"</span><span class="p">);</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="kd">var</span><span class="w"> </span><span class="nx">colTrig</span><span class="w"> </span><span class="k">of</span><span class="w"> </span><span class="nx">colTriggers</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">colTrig</span><span class="p">.</span><span class="nx">addEventListener</span><span class="p">(</span><span class="s2">"click"</span><span class="p">,</span><span class="w"> </span><span class="kd">function</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">classList</span><span class="p">.</span><span class="nx">toggle</span><span class="p">(</span><span class="s2">"active"</span><span class="p">);</span>
<span class="w"> </span><span class="p">});</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>Essentially, we use the <code>getElementsByClassName</code> DOM method to find all
elements with the <code>collapse-trigger</code> class. Then, we iterate over those
elements, and add an event listener to it. The first argument to
<code>addEventListener</code> is an event (in this case a button click). The second
argument is a callback function, for which we provide an anonymous function. If
you have a hard time wrapping your head around anonymous functions, this will
accomplish the same thing:</p>
<div class="highlight"><pre><span></span><code><span class="kd">function</span><span class="w"> </span><span class="nx">collapseTrigger</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">classList</span><span class="p">.</span><span class="nx">toggle</span><span class="p">(</span><span class="s2">"active"</span><span class="p">);</span>
<span class="p">}</span>
<span class="kd">function</span><span class="w"> </span><span class="nx">attachCollapseTriggers</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">colTriggers</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">document</span><span class="p">.</span><span class="nx">getElementsByClassName</span><span class="p">(</span><span class="s2">"collapse-trigger"</span><span class="p">);</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="kd">var</span><span class="w"> </span><span class="nx">colTrig</span><span class="w"> </span><span class="k">of</span><span class="w"> </span><span class="nx">colTriggers</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">colTrig</span><span class="p">.</span><span class="nx">addEventListener</span><span class="p">(</span><span class="s2">"click"</span><span class="p">,</span><span class="w"> </span><span class="nx">collapseTrigger</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>When the element is clicked, the <code>collapseTrigger</code> function is called. Of
course, we need to call <code>attachCollapseTriggers</code> sometime after the page has
loaded for this to take effect. And that's it for the JavaScript, clicking the
<code>#trigger</code> will now cause <code>#content</code> to collapse and expand! However, it's not
very clear to the user that the <code>#trigger</code> even can be clicked. Let's make that
just a little bit more clear by adding some visual cues.</p>
<h2>Finishing touches using the <code>::after</code> pseudo class</h2>
<p>A typical visual cue that a drop down can be expanded is a down-triangle (▼).
An up-triangle (▲) is as recognizable a cue that a menu can be collapsed. The
down-triangle should be appended to any <code>collapse-trigger</code> that is not active,
while the up-triangle should be appended to any <code>collapse-trigger</code> that also
has the <code>active</code> class. We can do that simply use the <code>::after</code> pseudo class.</p>
<div class="highlight"><pre><span></span><code><span class="p">.</span><span class="nc">collapse-trigger</span><span class="p">::</span><span class="nd">after</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">content</span><span class="p">:</span><span class="w"> </span><span class="s2">"▼"</span><span class="p">;</span>
<span class="w"> </span><span class="k">float</span><span class="p">:</span><span class="w"> </span><span class="kc">right</span><span class="p">;</span><span class="w"> </span><span class="c">/* float to the right-hand side of the content box */</span>
<span class="p">}</span>
<span class="p">.</span><span class="nc">collapse-trigger</span><span class="p">.</span><span class="nc">active</span><span class="p">::</span><span class="nd">after</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">content</span><span class="p">:</span><span class="w"> </span><span class="s2">"▲"</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>So, what happens here, exactly? When a <code>collapse-trigger</code> is not <code>active</code>, it
doesn't match the <code>.collapse-trigger.active</code> selector, so the content will
simply be the down-triangle. When a <code>collapse-trigger</code> <em>is</em> active, it will
match both selectors. However, <code>.collapse-trigger.active</code> is more specific than
<code>.collapse-trigger</code>, so it wins out, and the content will be an up-triangle.
And that's it, all done!</p>
<h1>Code listing and JSFiddle link</h1>
<p>The full code is available in the following subsections, and you can find a
<a href="https://jsfiddle.net/p3qtev5w/">JSFiddle here</a>.</p>
<h3>Markup</h3>
<div class="highlight"><pre><span></span><code><span class="cp"><!- the outer div with the class "sidebar" isn't important, it's just any container --></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"sidebar"</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"collapse-trigger"</span><span class="p">></span>Cool content heading<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"collapsible"</span><span class="p">></span>
<span class="p"><</span><span class="nt">p</span><span class="p">></span>
This content would be neat to hide and show at the click of a button!
<span class="p"></</span><span class="nt">p</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
</code></pre></div>
<h3>CSS</h3>
<div class="highlight"><pre><span></span><code><span class="c">/* the sidebar class is just an arbitrary container for this example */</span>
<span class="p">.</span><span class="nc">sidebar</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">width</span><span class="p">:</span><span class="w"> </span><span class="mi">30</span><span class="kt">%</span><span class="p">;</span>
<span class="w"> </span><span class="k">padding</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="kt">em</span><span class="p">;</span>
<span class="w"> </span><span class="k">border</span><span class="p">:</span><span class="w"> </span><span class="kc">black</span><span class="w"> </span><span class="kc">solid</span><span class="w"> </span><span class="mi">2</span><span class="kt">px</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">.</span><span class="nc">collapse-trigger</span><span class="p">::</span><span class="nd">after</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">content</span><span class="p">:</span><span class="w"> </span><span class="s2">"▼"</span><span class="p">;</span>
<span class="w"> </span><span class="k">float</span><span class="p">:</span><span class="w"> </span><span class="kc">right</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">.</span><span class="nc">collapse-trigger</span><span class="p">.</span><span class="nc">active</span><span class="p">::</span><span class="nd">after</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">content</span><span class="p">:</span><span class="w"> </span><span class="s2">"▲"</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">.</span><span class="nc">collapsible</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">display</span><span class="p">:</span><span class="w"> </span><span class="kc">none</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">.</span><span class="nc">collapse-trigger</span><span class="p">.</span><span class="nc">active</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">.</span><span class="nc">collapsible</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">display</span><span class="p">:</span><span class="w"> </span><span class="kc">block</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<h3>JavaScript</h3>
<div class="highlight"><pre><span></span><code><span class="kd">function</span><span class="w"> </span><span class="nx">attachCollapseTriggers</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">colTriggers</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">document</span><span class="p">.</span><span class="nx">getElementsByClassName</span><span class="p">(</span><span class="s2">"collapse-trigger"</span><span class="p">);</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="kd">var</span><span class="w"> </span><span class="nx">colTrig</span><span class="w"> </span><span class="k">of</span><span class="w"> </span><span class="nx">colTriggers</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">colTrig</span><span class="p">.</span><span class="nx">addEventListener</span><span class="p">(</span><span class="s2">"click"</span><span class="p">,</span><span class="w"> </span><span class="kd">function</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">classList</span><span class="p">.</span><span class="nx">toggle</span><span class="p">(</span><span class="s2">"active"</span><span class="p">);</span>
<span class="w"> </span><span class="p">});</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
<span class="nx">attachCollapseTriggers</span><span class="p">();</span>
</code></pre></div>A binary search tree in Kotlin pt. 2: Generic node2018-10-30T08:26:05+00:002018-10-30T08:26:05+00:00Simon Larséntag:slar.se,2018-10-30:/a-binary-search-tree-in-kotlin-pt-2-generic-node.html<p>Welcome to part 2 of my series on the idiomatic Kotlin binary tree! In this
part, we're gonna have a look at how to make the node representation from part 1
capable of carrying any kind of data (i.e. <em>generic</em>).</p>
<h2>Series index</h2>
<ol>
<li><a href="https://slar.se/a-binary-search-tree-in-kotlin-pt-1-representing-a-node.html">Representing a node</a></li>
<li>Generic node (this part …</li></ol><p>Welcome to part 2 of my series on the idiomatic Kotlin binary tree! In this
part, we're gonna have a look at how to make the node representation from part 1
capable of carrying any kind of data (i.e. <em>generic</em>).</p>
<h2>Series index</h2>
<ol>
<li><a href="https://slar.se/a-binary-search-tree-in-kotlin-pt-1-representing-a-node.html">Representing a node</a></li>
<li>Generic node (this part!)</li>
<li>Generic BST with insert, contains and traversal (coming soon!)</li>
</ol>
<h2>Attribution and reading recommendations</h2>
<p>In this part, we'll start working a little bit with binary tree algorithms. More
specifically, we'll complete the <code>contains</code> function from part 1. All of the
algorithms I implement in this series are based on
<a href="https://yourbasic.org/algorithms/binary-search-tree/">this article by Stefan Nilsson</a>.
If you are unfamiliar with the concepts of binary trees, I highly recommend that
you sift through that article before continuing with this one.</p>
<h2>Improving <code>contains</code></h2>
<p>Recall the <code>contains</code> function that we started working on in part 1. Before we
start working on generics, I want us to complete this function. It will make
some of the decisions about generics much more apparent. Anyway, here's
<code>contains</code> as we wrote it in part 1.</p>
<div class="highlight"><pre><span></span><code><span class="c1">// check if data is contained in node</span>
<span class="kd">fun</span><span class="w"> </span><span class="nf">contains</span><span class="p">(</span><span class="n">node</span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="p">,</span><span class="w"> </span><span class="n">data</span><span class="p">:</span><span class="w"> </span><span class="kt">Int</span><span class="p">):</span><span class="w"> </span><span class="kt">Boolean</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">when</span><span class="w"> </span><span class="p">(</span><span class="n">node</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">Empty</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="kc">false</span>
<span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">Node</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">node</span><span class="p">.</span><span class="na">data</span><span class="w"> </span><span class="c1">// note implicit cast</span>
<span class="p">}</span>
</code></pre></div>
<p>It's not a particularly useful function, as it only checks the current node.
What we'd rather have it do is check the entire subtree, in which <code>node</code> is the
root. This can quite easily be performed with recursion. The <code>Empty</code> case still
stands, if we hit an empty node, the data is not contained in the tree. If
<code>node</code> is a <code>Node</code>, on the other hand, there are three possibilities:</p>
<ol>
<li><code>data < node.data</code>, in which case we keep searching in the left subtree.</li>
<li><code>data > node.data</code>, in which case we keep searching in the right subtree.</li>
<li><code>data</code> is neither smaller or larger than <code>node.data</code>, so they are comparably
equal. If this actually means that they are equal or not is implementation
specific, but it is <em>highly</em> recommended that ordering is consistent with
<code>equals</code>. We will assume that this is the case.</li>
</ol>
<p>As we have three distinct cases, we can again use a <code>when</code> expression.</p>
<div class="highlight"><pre><span></span><code><span class="c1">// check if data is contained in the tree rooted in node</span>
<span class="kd">fun</span><span class="w"> </span><span class="nf">contains</span><span class="p">(</span><span class="n">node</span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="p">,</span><span class="w"> </span><span class="n">data</span><span class="p">:</span><span class="w"> </span><span class="kt">Int</span><span class="p">):</span><span class="w"> </span><span class="kt">Boolean</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">when</span><span class="w"> </span><span class="p">(</span><span class="n">node</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">Empty</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="kc">false</span>
<span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">Node</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="k">when</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="c1">// no-argument when so we can do arbitrary comparisons</span>
<span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">node</span><span class="p">.</span><span class="na">data</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">contains</span><span class="p">(</span><span class="n">node</span><span class="p">.</span><span class="na">left</span><span class="p">,</span><span class="w"> </span><span class="n">data</span><span class="p">)</span>
<span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="n">node</span><span class="p">.</span><span class="na">data</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">contains</span><span class="p">(</span><span class="n">node</span><span class="p">.</span><span class="na">right</span><span class="p">,</span><span class="w"> </span><span class="n">data</span><span class="p">)</span>
<span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>Note that the nested <code>when</code> expression has no argument in parentheses, allowing
us to perform more complex operations in the matchings. And that's it for the
<code>contains</code> operation. It's really quite elegant. We can quickly ammend the
<code>main</code> function to try it out:</p>
<div class="highlight"><pre><span></span><code><span class="kd">fun</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="n">args</span><span class="p">:</span><span class="w"> </span><span class="n">Array</span><span class="o"><</span><span class="kt">String</span><span class="o">></span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">// create the tree</span>
<span class="w"> </span><span class="c1">// 6</span>
<span class="w"> </span><span class="c1">// / \</span>
<span class="w"> </span><span class="c1">// 3 9</span>
<span class="w"> </span><span class="c1">// \</span>
<span class="w"> </span><span class="c1">// 4</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="nv">root</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Node</span><span class="p">(</span><span class="m">6</span><span class="p">,</span>
<span class="w"> </span><span class="n">left</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Node</span><span class="p">(</span><span class="m">3</span><span class="p">,</span>
<span class="w"> </span><span class="n">right</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Node</span><span class="p">(</span><span class="m">4</span><span class="p">)),</span>
<span class="w"> </span><span class="n">right</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Node</span><span class="p">(</span><span class="m">9</span><span class="p">))</span>
<span class="w"> </span><span class="n">println</span><span class="p">(</span><span class="s">"Search for elements in the tree"</span><span class="p">)</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="n">data</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">listOf</span><span class="p">(</span><span class="m">6</span><span class="p">,</span><span class="w"> </span><span class="m">3</span><span class="p">,</span><span class="w"> </span><span class="m">4</span><span class="p">,</span><span class="w"> </span><span class="m">9</span><span class="p">))</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">println</span><span class="p">(</span><span class="n">contains</span><span class="p">(</span><span class="n">root</span><span class="p">,</span><span class="w"> </span><span class="n">data</span><span class="p">))</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">println</span><span class="p">(</span><span class="s">"Search for elements not in the tree"</span><span class="p">)</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="n">data</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">listOf</span><span class="p">(</span><span class="m">10</span><span class="p">,</span><span class="w"> </span><span class="m">2</span><span class="p">,</span><span class="w"> </span><span class="m">1</span><span class="p">,</span><span class="w"> </span><span class="o">-</span><span class="m">12</span><span class="p">))</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">println</span><span class="p">(</span><span class="n">contains</span><span class="p">(</span><span class="n">root</span><span class="p">,</span><span class="w"> </span><span class="n">data</span><span class="p">))</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>With that out of the way, let's dive into making the whole thing generic!</p>
<h2>Getting generic</h2>
<p>Now, how do we make the node classes generic? A first attempt might be to just
change the <code>Node</code> class, and do something like this:</p>
<div class="highlight"><pre><span></span><code><span class="kd">data</span><span class="w"> </span><span class="kd">class</span><span class="w"> </span><span class="nc">Node</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="kd">val</span><span class="w"> </span><span class="nv">data</span><span class="p">:</span><span class="w"> </span><span class="n">T</span><span class="p">,</span><span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nv">left</span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Empty</span><span class="p">,</span><span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nv">right</span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Empty</span><span class="p">)</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="p">()</span>
<span class="kd">fun</span><span class="w"> </span><span class="o"><</span><span class="n">T</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">Comparable</span><span class="err"><</span><span class="n">T</span><span class="o">></span><span class="err">> </span><span class="nf">contains</span><span class="p">(</span><span class="n">node</span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="p">,</span><span class="w"> </span><span class="n">data</span><span class="p">:</span><span class="w"> </span><span class="n">T</span><span class="p">):</span><span class="w"> </span><span class="kt">Boolean</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">when</span><span class="w"> </span><span class="p">(</span><span class="n">node</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">Empty</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="kc">false</span>
<span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">Node</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="k">when</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">node</span><span class="p">.</span><span class="na">data</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">contains</span><span class="p">(</span><span class="n">node</span><span class="p">.</span><span class="na">left</span><span class="p">,</span><span class="w"> </span><span class="n">data</span><span class="p">)</span>
<span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="n">node</span><span class="p">.</span><span class="na">data</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">contains</span><span class="p">(</span><span class="n">node</span><span class="p">.</span><span class="na">right</span><span class="p">,</span><span class="w"> </span><span class="n">data</span><span class="p">)</span>
<span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>This is a reasonable attempt. Before we dive into the problems, let's analyze
what we just did. After <code>fun</code> we write <code><T : Comparable<T>></code>, which makes this
function generic. The addition states that the type parameter <code>T</code> can be
substituted by any type that implements <code>Comparable<T></code> (which we need to be
able to use <code><</code> and <code>></code>). <code>Node<T></code> simply defines a type parameter <code>T</code> that
can be substituted with <em>any</em> type. It is a bit inconvenient that we could put
non-comparable types in a <code>Node</code> object, but we'll see that it sorts itself out
when we create the <code>Tree</code> class in part 3. For now, just ignore that detail.</p>
<p>Now, the above code won't compile, for multiple reasons. The first problem is
that we can't ask at runtime if <code>node is Node<T></code> (the compiler will say <em>Cannot
check for erased type: Node<T></em>), becuase information about generics is erased
at runtime. We could succesfully match against a wildcard type parameter
(meaning <em>any</em> type) with <code>is Node<*></code>, but then we run into the real
showstopper: we don't know whether <code>data</code> and <code>node.data</code> are actually
comparable, as they might not have the same type. With the current class
hierarchy, there is <em>no</em> reasonable way around this. An <code>ANode</code> is not
parameterized, and therefore the dynamic type of an <code>ANode</code> can be any <code>Node</code>
type (e.g. <code>Node<Int></code>, <code>Node<String></code> etc) or <code>Empty</code>. We have to put the type
parameter <code>T</code> higher up in the inheritance chain. </p>
<h3>Inheriting from a generic class</h3>
<p>Since <code>ANode</code> is the only class higher up in the inheritance chain (apart from
<code>Any</code>), this is where we need to put our type parameter. For <code>ANode</code> and <code>Node</code>,
it is straightforward.</p>
<div class="highlight"><pre><span></span><code><span class="kd">sealed</span><span class="w"> </span><span class="kd">class</span><span class="w"> </span><span class="nc">ANode</span><span class="o"><</span><span class="n">T</span><span class="o">></span>
<span class="kd">data</span><span class="w"> </span><span class="kd">class</span><span class="w"> </span><span class="nc">Node</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="kd">val</span><span class="w"> </span><span class="nv">data</span><span class="p">:</span><span class="w"> </span><span class="n">T</span><span class="p">,</span><span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nv">left</span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Empty</span><span class="p">,</span><span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nv">right</span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Empty</span><span class="p">)</span><span class="w"> </span><span class="p">:</span><span class="w"> </span>
<span class="w"> </span><span class="n">ANode</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">()</span>
</code></pre></div>
<p>Note that <code>Node<T></code> is inheriting from <code>ANode<T></code>. We cannot (and wouldn't want
to, anyway) leave it as <code>ANode</code>, because unlike Java, Kotlin does not support
raw types. Now, since we cannot inherit from <code>ANode</code>, but must specify the type
parameter with a concrete type, what do we put there for <code>Empty</code>? In the case of
<code>Node<T></code>, we simply inherit from <code>ANode<T></code>, because <code>T</code> is declared as a
parameter to <code>Node<T></code> and is therefore concrete for for <code>ANode</code>. We can't
however just do something like</p>
<div class="highlight"><pre><span></span><code><span class="kd">object</span><span class="w"> </span><span class="nc">Empty</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">()</span>
</code></pre></div>
<p>because <code>T</code> is not declared in that scope. We also can't give <code>Empty</code> a type
parameter, because <code>Empty</code> is a singleton object, and type parameters simply
don't work with singletons (it wouldn't make much sense, if you stop to think
about it for a while). What we actually want to put as the type parameter, is
nothing. Literally. We wan't <code>Nothing</code>, a concrete type in Kotlin which is a
subtype of every non-nullable type. </p>
<div class="highlight"><pre><span></span><code><span class="kd">object</span><span class="w"> </span><span class="nc">Empty</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="o"><</span><span class="n">Nothing</span><span class="o">></span><span class="p">()</span>
</code></pre></div>
<p>Note that we don't want to put <code>Any</code> (which is the supertype of every
non-nullable type), because the we wouldn't be able to assign <code>Empty</code> to any
concrete type of <code>ANode<T></code>. Now you may be angrily shouting that you
<em>still</em> can't assign <code>Empty</code> to <em>any</em> type of <code>ANode<T></code>. Unfortunately,
<code>ANode<T></code> (for any substition of <code>T</code>) is <em>invariant</em>. Let's fix that.</p>
<h2>Generics are invariant by default, but Kotlin can stretch the rules</h2>
<p>Any generc class is invariant by default. What does this mean? In short, it
means that a generic type (e.g. <code>ANode<Int></code>) is not a supertype, nor subtype,
of any other type. Formally, it means that if we have two types <code>A</code> and <code>B</code>
such that <code>B</code> is a subtype of <code>A</code>, <code>ANode<B></code> is <em>not</em> a subtype of <code>ANode<A></code>.
Take for example <code>Empty</code>, which subtypes <code>ANode<Nothing></code>. It is not a subtype
of <code>ANode<Int></code>, even though <code>Nothing</code> is a subtype of <code>Int</code>. Inconvenient, we
want <code>Empty</code> to be a subtype of <em>any</em> concrete <code>ANode</code>. We can achieve this by
using the <code>out</code> modifier, and declaring <code>Anode<out T></code>. Formally, we make
<code>ANode</code> <em>covariant</em> on the type parameter <code>T</code>. We can only do this if every use
of <code>T</code> is in an <em>out</em> position (i.e. return values). Note that this
restriction applies only to the body of <code>ANode</code>, the <code>T</code> in <code>Node<T></code> is <em>not</em>
the same type parameter as in <code>ANode<out T></code>. If you found all of that
confusing (I sure did the first time I read about it), you can read more about
variance in the
<a href="https://kotlinlang.org/docs/reference/generics.html">Kotlin docs on generics</a>.
Here is what the working class hierarchy looks like:</p>
<div class="highlight"><pre><span></span><code><span class="kd">sealed</span><span class="w"> </span><span class="kd">class</span><span class="w"> </span><span class="nc">ANode</span><span class="o"><</span><span class="k">out</span><span class="w"> </span><span class="n">T</span><span class="o">></span>
<span class="kd">object</span><span class="w"> </span><span class="nc">Empty</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="o"><</span><span class="n">Nothing</span><span class="o">></span><span class="p">()</span>
<span class="kd">data</span><span class="w"> </span><span class="kd">class</span><span class="w"> </span><span class="nc">Node</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="nv">data</span><span class="p">:</span><span class="w"> </span><span class="n">T</span><span class="p">,</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nv">left</span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Empty</span><span class="p">,</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nv">right</span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Empty</span><span class="p">)</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">()</span>
</code></pre></div>
<p><code>contains</code> now looks like this:</p>
<div class="highlight"><pre><span></span><code><span class="c1">// check if data is contained in node</span>
<span class="kd">fun</span><span class="w"> </span><span class="o"><</span><span class="n">T</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">Comparable</span><span class="err"><</span><span class="n">T</span><span class="o">></span><span class="err">> </span><span class="nf">contains</span><span class="p">(</span><span class="n">node</span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">,</span><span class="w"> </span><span class="n">data</span><span class="p">:</span><span class="w"> </span><span class="n">T</span><span class="p">):</span><span class="w"> </span><span class="kt">Boolean</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">when</span><span class="w"> </span><span class="p">(</span><span class="n">node</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">Empty</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="kc">false</span>
<span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">Node</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="k">when</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">node</span><span class="p">.</span><span class="na">data</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">contains</span><span class="p">(</span><span class="n">node</span><span class="p">.</span><span class="na">left</span><span class="p">,</span><span class="w"> </span><span class="n">data</span><span class="p">)</span>
<span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="n">node</span><span class="p">.</span><span class="na">data</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">contains</span><span class="p">(</span><span class="n">node</span><span class="p">.</span><span class="na">right</span><span class="p">,</span><span class="w"> </span><span class="n">data</span><span class="p">)</span>
<span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>Now, you may be asking yourself why in the world we can do the <code>is Node<T></code>
check <em>now</em>, while we could not before? Well, we're not actually checking at
runtime whether <code>node</code> is <code>Node<T></code>, because the compiler knows any variable
with the static type <code>ANode<T></code> is either <code>Empty</code>, or <code>Node<T></code>. So, for
example, <code>ANode<Int></code> must have dynamic type <code>Empty</code> or <code>Node<Int></code>, there are
no other possibilities. As the compiler knows this, we can in fact skip the
<code><T></code> and just write <code>is Node</code>. That's all we need of our node classes, so we
can move on to implement the <code>Tree</code> class in part 3!</p>
<h2>Final code listing</h2>
<p>This is the final state of the code that we'll be using for part 3.</p>
<div class="highlight"><pre><span></span><code><span class="kd">sealed</span><span class="w"> </span><span class="kd">class</span><span class="w"> </span><span class="nc">ANode</span><span class="o"><</span><span class="k">out</span><span class="w"> </span><span class="n">T</span><span class="o">></span>
<span class="kd">object</span><span class="w"> </span><span class="nc">Empty</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="o"><</span><span class="n">Nothing</span><span class="o">></span><span class="p">()</span>
<span class="kd">data</span><span class="w"> </span><span class="kd">class</span><span class="w"> </span><span class="nc">Node</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="nv">data</span><span class="p">:</span><span class="w"> </span><span class="n">T</span><span class="p">,</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nv">left</span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Empty</span><span class="p">,</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nv">right</span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Empty</span><span class="p">)</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">()</span>
<span class="c1">// check if data is contained in node</span>
<span class="kd">fun</span><span class="w"> </span><span class="o"><</span><span class="n">T</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">Comparable</span><span class="err"><</span><span class="n">T</span><span class="o">></span><span class="err">> </span><span class="nf">contains</span><span class="p">(</span><span class="n">node</span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">,</span><span class="w"> </span><span class="n">data</span><span class="p">:</span><span class="w"> </span><span class="n">T</span><span class="p">):</span><span class="w"> </span><span class="kt">Boolean</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">when</span><span class="w"> </span><span class="p">(</span><span class="n">node</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">Empty</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="kc">false</span>
<span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">Node</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="k">when</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">node</span><span class="p">.</span><span class="na">data</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">contains</span><span class="p">(</span><span class="n">node</span><span class="p">.</span><span class="na">left</span><span class="p">,</span><span class="w"> </span><span class="n">data</span><span class="p">)</span>
<span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="n">node</span><span class="p">.</span><span class="na">data</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">contains</span><span class="p">(</span><span class="n">node</span><span class="p">.</span><span class="na">right</span><span class="p">,</span><span class="w"> </span><span class="n">data</span><span class="p">)</span>
<span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
<span class="kd">fun</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="n">args</span><span class="p">:</span><span class="w"> </span><span class="n">Array</span><span class="o"><</span><span class="kt">String</span><span class="o">></span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">// create the tree</span>
<span class="w"> </span><span class="c1">// 6</span>
<span class="w"> </span><span class="c1">// / \</span>
<span class="w"> </span><span class="c1">// 3 9</span>
<span class="w"> </span><span class="c1">// \</span>
<span class="w"> </span><span class="c1">// 4</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="nv">root</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Node</span><span class="p">(</span><span class="m">6</span><span class="p">,</span>
<span class="w"> </span><span class="n">left</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Node</span><span class="p">(</span><span class="m">3</span><span class="p">,</span>
<span class="w"> </span><span class="n">right</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Node</span><span class="p">(</span><span class="m">4</span><span class="p">)),</span>
<span class="w"> </span><span class="n">right</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Node</span><span class="p">(</span><span class="m">9</span><span class="p">))</span>
<span class="w"> </span><span class="n">println</span><span class="p">(</span><span class="s">"Search for elements in the tree"</span><span class="p">)</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="n">data</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">listOf</span><span class="p">(</span><span class="m">6</span><span class="p">,</span><span class="w"> </span><span class="m">3</span><span class="p">,</span><span class="w"> </span><span class="m">4</span><span class="p">,</span><span class="w"> </span><span class="m">9</span><span class="p">))</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">println</span><span class="p">(</span><span class="n">contains</span><span class="p">(</span><span class="n">root</span><span class="p">,</span><span class="w"> </span><span class="n">data</span><span class="p">))</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">println</span><span class="p">(</span><span class="s">"Search for elements not in the tree"</span><span class="p">)</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="n">data</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">listOf</span><span class="p">(</span><span class="m">10</span><span class="p">,</span><span class="w"> </span><span class="m">2</span><span class="p">,</span><span class="w"> </span><span class="m">1</span><span class="p">,</span><span class="w"> </span><span class="o">-</span><span class="m">12</span><span class="p">))</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">println</span><span class="p">(</span><span class="n">contains</span><span class="p">(</span><span class="n">root</span><span class="p">,</span><span class="w"> </span><span class="n">data</span><span class="p">))</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>A binary search tree in Kotlin pt. 1: Representing a node2018-10-28T15:13:08+00:002018-10-30T08:27:09+00:00Simon Larséntag:slar.se,2018-10-28:/a-binary-search-tree-in-kotlin-pt-1-representing-a-node.html<p>In my journey to become a somewhat competent Kotlin developer, I've decided to
implement a few of the basic data structures that I've picked up during my
three years of computer science studies. First up, we have a generic binary
tree. This is an interesting case, because it lets us …</p><p>In my journey to become a somewhat competent Kotlin developer, I've decided to
implement a few of the basic data structures that I've picked up during my
three years of computer science studies. First up, we have a generic binary
tree. This is an interesting case, because it lets us both delve into generics
in Kotlin, and some aspects of inheritance that differ from inheritance in
Java (in a good way!). As I want to cover some topics in depth, this will be
a three-part series with the following content:</p>
<ol>
<li>In the first part, we'll develop a basic class hierarchy for representing
nodes. It covers <strong>object</strong> types, <strong>data classes</strong> and <strong>sealed classes</strong>,
as well as some other related topics. The node is however restricted to only
carry <code>Int</code> data.</li>
<li>In the second part, we'll expand upon the class hierarchy from part 1 to
create a generic node class that can hold any type of data.</li>
<li>Finally, we'll use the results of part 2 to develop a rudimentary binary tree
in (what is in my opinion) idiomatic Kotlin.</li>
</ol>
<p>If you already know about object types, data classes and sealed classes, I
recommend that you skip directly to part 2. If you are already comfortable with
generics, including generic inheritance, you may skip directly to part 3.</p>
<h2>Series index</h2>
<ol>
<li>Representing a node (this part!)</li>
<li><a href="https://slar.se/a-binary-search-tree-in-kotlin-pt-2-generic-node.html">Generic node</a></li>
<li>Generic BST with insert, contains and traversal (coming soon!)</li>
</ol>
<h2>Goals and intended audience</h2>
<p>I write articles mostly for myself, and as such, this article series is intended
for developers with some experience with Java looking to get into Kotlin. Let's
get at it then, shall we?</p>
<h2>Representing a node: A Java-like attempt</h2>
<p>As I see it, a tree node can be one of two things: existent, or non-existent.
In other words, it can be a node or an empty node. As Kotlin is, thankfully,
quite adverse to using <code>null</code>, I will refrain from doing so as well. So what we
want is an abstract node class <code>ANode</code> and sub-classes <code>Node</code> and <code>Empty</code>. Let's
give it a first try in a pretty Java-like manner, and then improve upon it with
some neat Kotlin language constructs.</p>
<div class="highlight"><pre><span></span><code><span class="kd">abstract</span><span class="w"> </span><span class="kd">class</span><span class="w"> </span><span class="nc">ANode</span>
<span class="kd">class</span><span class="w"> </span><span class="nc">Empty</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="p">()</span>
<span class="kd">class</span><span class="w"> </span><span class="nc">Node</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="nv">data</span><span class="p">:</span><span class="w"> </span><span class="kt">Int</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nv">left</span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nv">right</span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span>
<span class="w"> </span><span class="k">constructor</span><span class="p">(</span><span class="n">data</span><span class="p">:</span><span class="w"> </span><span class="kt">Int</span><span class="p">)</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="k">super</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="na">data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">data</span><span class="p">;</span>
<span class="w"> </span><span class="n">right</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Empty</span><span class="p">()</span>
<span class="w"> </span><span class="n">left</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Empty</span><span class="p">()</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>If you've had any experience with any remotely Java-looking language, you can
probably guess what's going on here. There's the abstract <code>ANode</code> class, the
<code>Empty</code> class representing the absence of a node and the <code>Node</code> class
representing an actual node. Note also that we have not delved into generics
yet, this is a node that can only hold <code>Int</code> data. That's fine for now, we'll
expand upon this implementation with generics in part 2. When we later implement
the binary tree, we will often want to distinguish between a <code>Node</code> and <code>Empty</code>.
One such case is when we search the tree for a given value, to see if it is
contained in the tree. This operation can be succinctly expressed using
recursion, but let us leave that for part 2. For now, let's just check the first
node (the <em>root</em>), without exploring its children.</p>
<div class="highlight"><pre><span></span><code><span class="c1">// check if data is contained in node</span>
<span class="kd">fun</span><span class="w"> </span><span class="nf">contains</span><span class="p">(</span><span class="n">node</span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="p">,</span><span class="w"> </span><span class="n">data</span><span class="p">:</span><span class="w"> </span><span class="kt">Int</span><span class="p">):</span><span class="w"> </span><span class="kt">Boolean</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">when</span><span class="w"> </span><span class="p">(</span><span class="n">node</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">Empty</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="kc">false</span>
<span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">Node</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">node</span><span class="p">.</span><span class="na">data</span><span class="w"> </span><span class="c1">// note implicit cast</span>
<span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="k">throw</span><span class="w"> </span><span class="n">IllegalArgumentException</span><span class="p">(</span><span class="s">"node argument was neither Empty nor Node!"</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div>
<p>This is of course a pretty stupid function at this point, but we'll make it much
more worthwhile in part 2. Note the
<a href="https://kotlinlang.org/docs/reference/basic-syntax.html">expression body</a>
used here, in combination with a
<a href="https://kotlinlang.org/docs/reference/control-flow.html"><code>when</code> expression</a>.
If you are unfamiliar with those concepts, follow the links and read up on them,
they will be crucial when implementing the tree algorithms in parts 2 and 3.
Also note the
<a href="https://kotlinlang.org/docs/reference/typecasts.html">implicit cast</a> occurring
on the second line of the function. Since we used <code>is Node</code> to match <code>node</code>, the
compiler can infer that <code>node</code> is in fact a <code>Node</code> object, and we can safely
dereference it with <code>node.data</code>! Finally, note that the <code>else</code> case is needed as
the compiler does not know that there are only two subclasses of <code>ANode</code> (even
though we currently do, in this very small project). We'll see how to resolve
that shortly. Let's try this function out:</p>
<div class="highlight"><pre><span></span><code><span class="o">>>></span><span class="w"> </span><span class="n">contains</span><span class="p">(</span><span class="n">Empty</span><span class="p">(),</span><span class="w"> </span><span class="m">2</span><span class="p">)</span>
<span class="kc">false</span>
<span class="o">>>></span><span class="w"> </span><span class="n">contains</span><span class="p">(</span><span class="n">Node</span><span class="p">(</span><span class="m">2</span><span class="p">),</span><span class="w"> </span><span class="m">2</span><span class="p">)</span>
<span class="kc">true</span>
<span class="o">>>></span><span class="w"> </span><span class="n">contains</span><span class="p">(</span><span class="n">Node</span><span class="p">(</span><span class="m">2</span><span class="p">),</span><span class="w"> </span><span class="m">3</span><span class="p">)</span>
<span class="kc">false</span>
</code></pre></div>
<p>It seems to work just fine. We <em>could</em> leave the class hierarchy like this and
jump straight into generics. There are, however, three notable problems with the
node classes.</p>
<ol>
<li>For each empty node we need, a new instance of <code>Empty</code> is created. This is
wasteful.</li>
<li>The body of <code>Node</code> is a whole lot of code for very little functionality.</li>
<li>The compiler can't tell that <code>Node</code> and <code>Empty</code> are the only subtypes of
<code>ANode</code>, forcing us to use an <code>else</code> in the <code>when</code> expression.</li>
</ol>
<p>As it turns out, all of these problems are easy to solve in Kotlin!</p>
<h3>Problem 1 solution: Singleton objects</h3>
<p>Problem number 1 can be solved very easily, as Kotlin has language support for
the singleton pattern. We simply swap this declaration</p>
<div class="highlight"><pre><span></span><code><span class="kd">class</span><span class="w"> </span><span class="nc">Empty</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="p">()</span>
</code></pre></div>
<p>for this declaration</p>
<div class="highlight"><pre><span></span><code><span class="kd">object</span><span class="w"> </span><span class="nc">Empty</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="p">()</span>
</code></pre></div>
<p><code>Empty</code> is now a singleton object, so we can assign it without instantiating
<code>Empty</code>s all over the place. The constructor for <code>Node</code> now looks like this:</p>
<div class="highlight"><pre><span></span><code><span class="k">constructor</span><span class="p">(</span><span class="n">data</span><span class="p">:</span><span class="w"> </span><span class="kt">Int</span><span class="p">)</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="k">super</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="na">data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">data</span><span class="p">;</span>
<span class="w"> </span><span class="n">right</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Empty</span><span class="w"> </span><span class="c1">// note the lack of parentheses!</span>
<span class="w"> </span><span class="n">left</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Empty</span>
<span class="p">}</span>
</code></pre></div>
<p>One problem solved, two to go!</p>
<h3>Problem 2 solution: Primary constructors and data classes</h3>
<p>We can solve problem number 2 with Kotlin's syntax for
<a href="https://kotlinlang.org/docs/reference/classes.html">primary constructors</a>.
Instead of defining <code>Node</code> the Java way, we do it the Kotlin way:</p>
<div class="highlight"><pre><span></span><code><span class="kd">class</span><span class="w"> </span><span class="nc">Node</span><span class="p">(</span><span class="kd">val</span><span class="w"> </span><span class="nv">data</span><span class="p">:</span><span class="w"> </span><span class="kt">Int</span><span class="p">,</span><span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nv">right</span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Empty</span><span class="p">,</span><span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nv">left</span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Empty</span><span class="p">)</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="p">()</span>
</code></pre></div>
<p>This is <em>almost</em> equivalent to the previous declaration, with the exception that
<code>right</code> and <code>left</code> are assigned default values in the header such that they can
be replaced by explicit arguments when calling the constructor. Note that
<code>ANode</code> must be instantiated right there in the header as well. However, since
we know that <code>Node</code> will always be a simple container, we can do one better here
by prepending <code>data</code> to the declaration.</p>
<div class="highlight"><pre><span></span><code><span class="kd">data</span><span class="w"> </span><span class="kd">class</span><span class="w"> </span><span class="nc">Node</span><span class="p">(</span><span class="kd">val</span><span class="w"> </span><span class="nv">data</span><span class="p">:</span><span class="w"> </span><span class="kt">Int</span><span class="p">,</span><span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nv">right</span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Empty</span><span class="p">,</span><span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nv">left</span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Empty</span><span class="p">)</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="p">()</span>
</code></pre></div>
<p>This makes <code>Node</code> a
<a href="https://kotlinlang.org/docs/reference/data-classes.html">data class</a>, which
among other things come with implementations of <code>equals</code> and <code>toString</code>. A
fortunate accident here is that the <code>toString</code> of <code>Node</code> will actually let us
view the whole tree with very little effort, as <code>toString</code> will be called on
both <code>left</code> and <code>right</code>, recursively (this will be demonstrated in part 3). Do
be careful not to create a cycle, though, as this will cause a stack overflow,
endlessly calling <code>toString</code> (a tree, by definition, has no cycles, so we are
good in this case).</p>
<h3>Problem 3 solution: Sealed classes</h3>
<p>To reiterate, the problem was that the compiler can't tell that <code>Node</code> and
<code>Empty</code> are the only subtypes of <code>ANode</code>. Therefore, we needed the <code>else</code> in the
<code>when</code> expression to cover up the non-existent case of the argument to
<code>contains</code> being anything else.</p>
<div class="highlight"><pre><span></span><code><span class="c1">// check if data is contained in node</span>
<span class="kd">fun</span><span class="w"> </span><span class="nf">contains</span><span class="p">(</span><span class="n">node</span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="p">,</span><span class="w"> </span><span class="n">data</span><span class="p">:</span><span class="w"> </span><span class="kt">Int</span><span class="p">):</span><span class="w"> </span><span class="kt">Boolean</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">when</span><span class="w"> </span><span class="p">(</span><span class="n">node</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">Empty</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="kc">false</span>
<span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">Node</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">node</span><span class="p">.</span><span class="na">data</span>
<span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="k">throw</span><span class="w"> </span><span class="n">IllegalArgumentException</span><span class="p">(</span><span class="s">"node argument was neither Empty nor Node!"</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div>
<p>We can, however, <em>tell</em> the compiler that <code>Node</code> and <code>Empty</code> <em>are</em> the only
subtypes by making <code>ANode</code> a
<a href="https://kotlinlang.org/docs/reference/sealed-classes.html_"><code>sealed</code> class</a>.
Any subclass of a sealed class must be declared inside the same file, which lets
the compiler know precisely which subtypes can exist. To accomplish this, we
simply replace the <code>abstract</code> modifier with <code>sealed</code> (because <code>sealed</code> implies
<code>abstract</code>, we don't need the latter).</p>
<div class="highlight"><pre><span></span><code><span class="kd">sealed</span><span class="w"> </span><span class="kd">class</span><span class="w"> </span><span class="nc">ANode</span>
</code></pre></div>
<p>We can now drop the <code>else</code> from <code>contains</code>, because the compiler knows that a
variable with static type <code>ANode</code> is either <code>Empty</code>, or a <code>Node</code>, there are no
other possibilities.</p>
<div class="highlight"><pre><span></span><code><span class="c1">// check if data is contained in node</span>
<span class="n">fun</span><span class="w"> </span><span class="n">contains</span><span class="p">(</span><span class="n">node</span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="p">,</span><span class="w"> </span><span class="n">data</span><span class="p">:</span><span class="w"> </span><span class="n">Int</span><span class="p">):</span><span class="w"> </span><span class="n">Boolean</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="n">when</span><span class="w"> </span><span class="p">(</span><span class="n">node</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">is</span><span class="w"> </span><span class="n">Empty</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">false</span>
<span class="w"> </span><span class="n">is</span><span class="w"> </span><span class="n">Node</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">node</span><span class="p">.</span><span class="n">data</span>
<span class="p">}</span>
</code></pre></div>
<p>Let's give it a try</p>
<div class="highlight"><pre><span></span><code><span class="o">>>></span><span class="w"> </span><span class="n">contains</span><span class="p">(</span><span class="n">Node</span><span class="p">(</span><span class="m">2</span><span class="p">),</span><span class="w"> </span><span class="m">2</span><span class="p">)</span>
<span class="kc">true</span>
<span class="o">>>></span><span class="w"> </span><span class="n">contains</span><span class="p">(</span><span class="n">Empty</span><span class="p">,</span><span class="w"> </span><span class="m">2</span><span class="p">)</span>
<span class="kc">false</span>
</code></pre></div>
<p>Neat, now we have a good base for venturing into the fraught land of generics in
<a href="https://slar.se/a-binary-search-tree-in-kotlin-pt-2-generic-node.html">part 2</a>.</p>
<h2>Final code listing</h2>
<p>The final version of the code, that we'll use in part 2, can be found below.
I've also included a main function such that you can run the code in your
preferred way, right off the bat!</p>
<div class="highlight"><pre><span></span><code><span class="kd">sealed</span><span class="w"> </span><span class="kd">class</span><span class="w"> </span><span class="nc">ANode</span>
<span class="kd">object</span><span class="w"> </span><span class="nc">Empty</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="p">()</span>
<span class="kd">data</span><span class="w"> </span><span class="kd">class</span><span class="w"> </span><span class="nc">Node</span><span class="p">(</span><span class="kd">val</span><span class="w"> </span><span class="nv">data</span><span class="p">:</span><span class="w"> </span><span class="kt">Int</span><span class="p">,</span><span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nv">left</span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Empty</span><span class="p">,</span><span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nv">right</span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Empty</span><span class="p">)</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="p">()</span>
<span class="c1">// check if data is contained in node</span>
<span class="kd">fun</span><span class="w"> </span><span class="nf">contains</span><span class="p">(</span><span class="n">node</span><span class="p">:</span><span class="w"> </span><span class="n">ANode</span><span class="p">,</span><span class="w"> </span><span class="n">data</span><span class="p">:</span><span class="w"> </span><span class="kt">Int</span><span class="p">):</span><span class="w"> </span><span class="kt">Boolean</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">when</span><span class="w"> </span><span class="p">(</span><span class="n">node</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">Empty</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="kc">false</span>
<span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">Node</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">node</span><span class="p">.</span><span class="na">data</span><span class="w"> </span><span class="c1">// note implicit cast</span>
<span class="p">}</span>
<span class="kd">fun</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="n">args</span><span class="p">:</span><span class="w"> </span><span class="n">Array</span><span class="o"><</span><span class="kt">String</span><span class="o">></span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">println</span><span class="p">(</span><span class="n">contains</span><span class="p">(</span><span class="n">Empty</span><span class="p">,</span><span class="w"> </span><span class="m">2</span><span class="p">))</span>
<span class="w"> </span><span class="n">println</span><span class="p">(</span><span class="n">contains</span><span class="p">(</span><span class="n">Node</span><span class="p">(</span><span class="m">2</span><span class="p">),</span><span class="w"> </span><span class="m">2</span><span class="p">))</span>
<span class="w"> </span><span class="n">println</span><span class="p">(</span><span class="n">contains</span><span class="p">(</span><span class="n">Node</span><span class="p">(</span><span class="m">2</span><span class="p">),</span><span class="w"> </span><span class="m">3</span><span class="p">))</span>
<span class="p">}</span>
</code></pre></div>Creating a standalone (runnable) Kotlin .jar file with IntelliJ and Gradle2018-10-17T20:15:46+00:002018-10-29T21:02:18+00:00Simon Larséntag:slar.se,2018-10-17:/creating-a-standalone-runnable-kotlin-jar-file-with-intellij-and-gradle.html<p>I've recently started dabbling in some Kotlin, and have found it a very pleasant
experience. One of the first things I wanted to do was to create a standalone
<code>.jar</code> file, including the Kotlin runtime and any other dependencies. This,
as it turns out, was a bit tricky. In this …</p><p>I've recently started dabbling in some Kotlin, and have found it a very pleasant
experience. One of the first things I wanted to do was to create a standalone
<code>.jar</code> file, including the Kotlin runtime and any other dependencies. This,
as it turns out, was a bit tricky. In this short article, I will walk you
through creating a small command line application using the
awesome <a href="https://ajalt.github.io/clikt/"><code>clikt</code> library</a>, and then packaging
it into a standalone <code>.jar</code>.</p>
<h2>Setting up</h2>
<p>Start out with creating a new project by going to <code>File -> New -> Project</code>, select
Gradle in the leftmost menu bar (i.e. <em>not</em> Kotlin), and then tick the Kotlin
box in the <code>Additional Libraries and Frameworks</code> menu. Then just fill in any
GroupId, ArtifactId and Version (I will use <code>slarse</code>, <code>app</code> and <code>0.1</code> for these
fields, respectively). Then just click <code>Next</code> with the defaults until the
project is created.</p>
<h3>Initial Gradle configuration</h3>
<p>In the project root, you should now have a file called <code>build.gradle</code>, which
looks something like this:</p>
<div class="highlight"><pre><span></span><code><span class="err">plugi</span><span class="kc">ns</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">'org.je</span><span class="kc">t</span><span class="err">brai</span><span class="kc">ns</span><span class="err">.ko</span><span class="kc">tl</span><span class="err">i</span><span class="kc">n</span><span class="err">.jvm'</span><span class="w"> </span><span class="err">versio</span><span class="kc">n</span><span class="w"> </span><span class="err">'</span><span class="mf">1.2.51</span><span class="err">'</span>
<span class="p">}</span>
<span class="err">group</span><span class="w"> </span><span class="err">'slarse'</span>
<span class="err">versio</span><span class="kc">n</span><span class="w"> </span><span class="err">'</span><span class="mf">0.1</span><span class="err">'</span>
<span class="err">reposi</span><span class="kc">t</span><span class="err">ories</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">mave</span><span class="kc">n</span><span class="err">Ce</span><span class="kc">ntral</span><span class="err">()</span>
<span class="p">}</span>
<span class="err">depe</span><span class="kc">n</span><span class="err">de</span><span class="kc">n</span><span class="err">cies</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">compile</span><span class="w"> </span><span class="s2">"org.jetbrains.kotlin:kotlin-stdlib-jdk8"</span>
<span class="p">}</span>
<span class="err">compileKo</span><span class="kc">tl</span><span class="err">i</span><span class="kc">n</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">ko</span><span class="kc">tl</span><span class="err">i</span><span class="kc">n</span><span class="err">Op</span><span class="kc">t</span><span class="err">io</span><span class="kc">ns</span><span class="err">.jvmTarge</span><span class="kc">t</span><span class="w"> </span><span class="err">=</span><span class="w"> </span><span class="s2">"1.8"</span>
<span class="p">}</span>
<span class="err">compileTes</span><span class="kc">t</span><span class="err">Ko</span><span class="kc">tl</span><span class="err">i</span><span class="kc">n</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">ko</span><span class="kc">tl</span><span class="err">i</span><span class="kc">n</span><span class="err">Op</span><span class="kc">t</span><span class="err">io</span><span class="kc">ns</span><span class="err">.jvmTarge</span><span class="kc">t</span><span class="w"> </span><span class="err">=</span><span class="w"> </span><span class="s2">"1.8"</span>
<span class="p">}</span>
</code></pre></div>
<p>Before we can compile a project with <code>clikt</code>, we need to add it as a dependency.
We can do that by adding <code>compile "com.github.ajalt:clikt:1.5.0"</code> in the
dependencies section. It should now look like this:</p>
<div class="highlight"><pre><span></span><code><span class="err">depe</span><span class="kc">n</span><span class="err">de</span><span class="kc">n</span><span class="err">cies</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">compile</span><span class="w"> </span><span class="s2">"org.jetbrains.kotlin:kotlin-stdlib-jdk8"</span>
<span class="w"> </span><span class="err">compile</span><span class="w"> </span><span class="s2">"com.github.ajalt:clikt:1.5.0"</span>
<span class="p">}</span>
</code></pre></div>
<p>Then hit the little refresh symbol in the bottom left corner
(should say <code>Refresh Gradle Project</code> when you hover your mouse over it) to
download the new dependency. And that's it for now!
We'll get back to the <code>gradle.build</code> file once we
want to configure our <code>jar</code> task, but let's create the app first!</p>
<h2>Creating the application</h2>
<p>Let's make this easy: we'll just use the sample application available from the
<a href="https://ajalt.github.io/clikt/"><code>clikt</code> documentation</a>. It looks like this:</p>
<div class="highlight"><pre><span></span><code><span class="kd">class</span><span class="w"> </span><span class="nc">Hello</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">CliktCommand</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="nv">count</span><span class="p">:</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="n">option</span><span class="p">(</span><span class="n">help</span><span class="o">=</span><span class="s">"Number of greetings"</span><span class="p">).</span><span class="na">int</span><span class="p">().</span><span class="na">default</span><span class="p">(</span><span class="m">1</span><span class="p">)</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="nv">name</span><span class="p">:</span><span class="w"> </span><span class="kt">String</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="n">option</span><span class="p">(</span><span class="n">help</span><span class="o">=</span><span class="s">"The person to greet"</span><span class="p">).</span><span class="na">prompt</span><span class="p">(</span><span class="s">"Your name"</span><span class="p">)</span>
<span class="w"> </span><span class="kd">override</span><span class="w"> </span><span class="kd">fun</span><span class="w"> </span><span class="nf">run</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="n">i</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">1.</span><span class="p">.</span><span class="na">count</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">echo</span><span class="p">(</span><span class="s">"Hello </span><span class="si">$</span><span class="n">name</span><span class="s">!"</span><span class="p">)</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
<span class="kd">fun</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="n">args</span><span class="p">:</span><span class="w"> </span><span class="n">Array</span><span class="o"><</span><span class="kt">String</span><span class="o">></span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Hello</span><span class="p">().</span><span class="na">main</span><span class="p">(</span><span class="n">args</span><span class="p">)</span>
</code></pre></div>
<p>Create a Kotlin file called <code>main.kt</code> at <code>src/main/kotlin/main.kt</code> and paste
the above code into it. Note that we are using the default package here
(i.e. not defining a package) for the sake of simplicity.</p>
<p>For this to compile, we will need to add the following imports at the top:</p>
<div class="highlight"><pre><span></span><code><span class="k">import</span><span class="w"> </span><span class="nn">com.github.ajalt.clikt.core.CliktCommand</span>
<span class="k">import</span><span class="w"> </span><span class="nn">com.github.ajalt.clikt.parameters.options.default</span>
<span class="k">import</span><span class="w"> </span><span class="nn">com.github.ajalt.clikt.parameters.options.option</span>
<span class="k">import</span><span class="w"> </span><span class="nn">com.github.ajalt.clikt.parameters.options.prompt</span>
<span class="k">import</span><span class="w"> </span><span class="nn">com.github.ajalt.clikt.parameters.types.int</span>
</code></pre></div>
<p>And that's it for the application, you should now be able to run it as usual.
When running it, there should appear a prompt in the terminal saying <code>Your name:</code>.
With that out of the way, the only thing left to do is to package our
fantastic application into a standalone <code>.jar</code> file.</p>
<h2>Packaging the application into a standalone <code>.jar</code> file</h2>
<p>This is actually not very difficult, but you need to know what to do. We need
to create a so-called "fat" jar, which includes both the Kotlin runtime and the
<code>clikt</code> library. We also need to specify the name of our main class.</p>
<div class="highlight"><pre><span></span><code><span class="err">jar</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">ma</span><span class="kc">n</span><span class="err">i</span><span class="kc">fest</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">a</span><span class="kc">ttr</span><span class="err">ibu</span><span class="kc">tes</span><span class="w"> </span><span class="err">'Mai</span><span class="kc">n</span><span class="mi">-</span><span class="err">Class'</span><span class="p">:</span><span class="w"> </span><span class="err">'Mai</span><span class="kc">n</span><span class="err">K</span><span class="kc">t</span><span class="err">'</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="kc">fr</span><span class="err">om</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">co</span><span class="kc">nf</span><span class="err">igura</span><span class="kc">t</span><span class="err">io</span><span class="kc">ns</span><span class="err">.compile.collec</span><span class="kc">t</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">i</span><span class="kc">t</span><span class="err">.isDirec</span><span class="kc">t</span><span class="err">ory()</span><span class="w"> </span><span class="err">?</span><span class="w"> </span><span class="err">i</span><span class="kc">t</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="err">zipTree(i</span><span class="kc">t</span><span class="err">)</span><span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>Note that the class file generated by Kotlin for a file called <code>something.kt</code>
will be <code>SomethingKt.class</code>, which is why our main class is called <code>MainKt</code>.
With that in mind, the <code>manifest</code> section is self-explanatory: we specify
the main class. The <code>from</code> section collects all compile dependencies
(that we specified in the <code>dependencies</code> section) and package them with the
<code>.jar</code> file. The little piece of logic in the lambda is to properly add
directories and <code>.jar</code> files, respectively (directories are just added,
<code>.jar</code> files are unzipped and added).</p>
<blockquote>
<p><strong>Important:</strong> The main class file must be specified with its fully qualified
name. For example, if I were to define <code>main.kt</code> in the package <code>se.slarse</code>,
then I would need to put <code>se.slarse.MainKt</code> instead of just <code>MainKt</code> in the
manifest.</p>
</blockquote>
<p>Anyway, that's really all we need to do.
It should now be possible to run the <code>jar</code> Gradle task to produce a <code>.jar</code> file
in <code>build/libs/<ArtifactId>-<Version></code> (so in my case it is at
<code>build/libs/app-0.1.jar</code>). And that's it, hope it helped someone!</p>
<h1>Full source code and <code>build.gradle</code></h1>
<p>Here are both of the files we wrote in this tutorial, in their entirety.</p>
<div class="highlight"><pre><span></span><code><span class="c1">// main.kt</span>
<span class="k">import</span><span class="w"> </span><span class="nn">com.github.ajalt.clikt.core.CliktCommand</span>
<span class="k">import</span><span class="w"> </span><span class="nn">com.github.ajalt.clikt.parameters.options.default</span>
<span class="k">import</span><span class="w"> </span><span class="nn">com.github.ajalt.clikt.parameters.options.option</span>
<span class="k">import</span><span class="w"> </span><span class="nn">com.github.ajalt.clikt.parameters.options.prompt</span>
<span class="k">import</span><span class="w"> </span><span class="nn">com.github.ajalt.clikt.parameters.types.int</span>
<span class="kd">class</span><span class="w"> </span><span class="nc">Hello</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">CliktCommand</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="nv">count</span><span class="p">:</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="n">option</span><span class="p">(</span><span class="n">help</span><span class="o">=</span><span class="s">"Number of greetings"</span><span class="p">).</span><span class="na">int</span><span class="p">().</span><span class="na">default</span><span class="p">(</span><span class="m">1</span><span class="p">)</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="nv">name</span><span class="p">:</span><span class="w"> </span><span class="kt">String</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="n">option</span><span class="p">(</span><span class="n">help</span><span class="o">=</span><span class="s">"The person to greet"</span><span class="p">).</span><span class="na">prompt</span><span class="p">(</span><span class="s">"Your name"</span><span class="p">)</span>
<span class="w"> </span><span class="kd">override</span><span class="w"> </span><span class="kd">fun</span><span class="w"> </span><span class="nf">run</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="n">i</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">1.</span><span class="p">.</span><span class="na">count</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">echo</span><span class="p">(</span><span class="s">"Hello </span><span class="si">$</span><span class="n">name</span><span class="s">!"</span><span class="p">)</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
<span class="kd">fun</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="n">args</span><span class="p">:</span><span class="w"> </span><span class="n">Array</span><span class="o"><</span><span class="kt">String</span><span class="o">></span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Hello</span><span class="p">().</span><span class="na">main</span><span class="p">(</span><span class="n">args</span><span class="p">)</span>
</code></pre></div>
<hr>
<div class="highlight"><pre><span></span><code><span class="c1">// build.gradle</span>
<span class="err">plugi</span><span class="kc">ns</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">id</span><span class="w"> </span><span class="err">'org.je</span><span class="kc">t</span><span class="err">brai</span><span class="kc">ns</span><span class="err">.ko</span><span class="kc">tl</span><span class="err">i</span><span class="kc">n</span><span class="err">.jvm'</span><span class="w"> </span><span class="err">versio</span><span class="kc">n</span><span class="w"> </span><span class="err">'</span><span class="mf">1.2.51</span><span class="err">'</span>
<span class="p">}</span>
<span class="err">group</span><span class="w"> </span><span class="err">'se.slarse'</span>
<span class="err">versio</span><span class="kc">n</span><span class="w"> </span><span class="err">'</span><span class="mf">0.1</span><span class="err">'</span>
<span class="err">reposi</span><span class="kc">t</span><span class="err">ories</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">mave</span><span class="kc">n</span><span class="err">Ce</span><span class="kc">ntral</span><span class="err">()</span>
<span class="p">}</span>
<span class="err">depe</span><span class="kc">n</span><span class="err">de</span><span class="kc">n</span><span class="err">cies</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">compile</span><span class="w"> </span><span class="s2">"org.jetbrains.kotlin:kotlin-stdlib-jdk8"</span>
<span class="w"> </span><span class="err">compile</span><span class="w"> </span><span class="s2">"com.github.ajalt:clikt:1.5.0"</span>
<span class="p">}</span>
<span class="err">compileKo</span><span class="kc">tl</span><span class="err">i</span><span class="kc">n</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">ko</span><span class="kc">tl</span><span class="err">i</span><span class="kc">n</span><span class="err">Op</span><span class="kc">t</span><span class="err">io</span><span class="kc">ns</span><span class="err">.jvmTarge</span><span class="kc">t</span><span class="w"> </span><span class="err">=</span><span class="w"> </span><span class="s2">"1.8"</span>
<span class="p">}</span>
<span class="err">compileTes</span><span class="kc">t</span><span class="err">Ko</span><span class="kc">tl</span><span class="err">i</span><span class="kc">n</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">ko</span><span class="kc">tl</span><span class="err">i</span><span class="kc">n</span><span class="err">Op</span><span class="kc">t</span><span class="err">io</span><span class="kc">ns</span><span class="err">.jvmTarge</span><span class="kc">t</span><span class="w"> </span><span class="err">=</span><span class="w"> </span><span class="s2">"1.8"</span>
<span class="p">}</span>
<span class="err">jar</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">ma</span><span class="kc">n</span><span class="err">i</span><span class="kc">fest</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">a</span><span class="kc">ttr</span><span class="err">ibu</span><span class="kc">tes</span><span class="w"> </span><span class="err">'Mai</span><span class="kc">n</span><span class="mi">-</span><span class="err">Class'</span><span class="p">:</span><span class="w"> </span><span class="err">'Mai</span><span class="kc">n</span><span class="err">K</span><span class="kc">t</span><span class="err">'</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="kc">fr</span><span class="err">om</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">co</span><span class="kc">nf</span><span class="err">igura</span><span class="kc">t</span><span class="err">io</span><span class="kc">ns</span><span class="err">.compile.collec</span><span class="kc">t</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">i</span><span class="kc">t</span><span class="err">.isDirec</span><span class="kc">t</span><span class="err">ory()</span><span class="w"> </span><span class="err">?</span><span class="w"> </span><span class="err">i</span><span class="kc">t</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="err">zipTree(i</span><span class="kc">t</span><span class="err">)</span><span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>Awesome Python Podcasts2018-07-03T07:34:33+00:002018-10-29T21:03:07+00:00Simon Larséntag:slar.se,2018-07-03:/awesome-python-podcasts.html<p>Whenever I find myself occupied with some monotonous task, I
very much enjoy listening to podcasts. As programming is my
number one passion, and Python is my favorite language, I tend
to listen to podcasts that relate to them. In this post, I'll
give a brief overview of my three …</p><p>Whenever I find myself occupied with some monotonous task, I
very much enjoy listening to podcasts. As programming is my
number one passion, and Python is my favorite language, I tend
to listen to podcasts that relate to them. In this post, I'll
give a brief overview of my three favorite podcasts, and just
why I enjoy them as much as I do!</p>
<h3>Python Bytes</h3>
<p><a href="https://pythonbytes.fm/"><em>Python Bytes</em></a> was the first podcast I ever
listened to. It's a really neat show that comes out on a weekly basis,
and focuses on delivering news and headlines in the Python community.
The best part about the show is that they highlight awesome Python
packages and tools that I would not have heard about otherwise. The
episodes are fairly short, usually around the 20 minute mark, so they
fit in even on a short commute. The episodes have, as far as I can recall,
been published every week without fail for almost two years now, which
is really nice. The hosts (Brian Okken and Micheal Kennedy) have great
chemistry, and the show is recorded with decent enough equipment and
well edited. All in all, I really enjoy the show and highly recommend it!</p>
<h3>Talk Python To Me</h3>
<p><a href="https://talkpython.fm/"><em>Talk Python To Me</em></a> is Micheal Kennedy's (from
Python Bytes) own show. In each episode, Micheal invites someone (sometimes
multiple people at once) from
the Python community to come talk about what they do. The episodes
are fairly lengthy and often reach for the 1 hour mark, but they are
also mostly entertaining throughout. As with Python Bytes, the episodes
are well edited, meaning that awkward pauses and the like shine with their
absence. Talk Python To Me is probably my favorite podcast right now and
I can't recommend it enough.</p>
<h3>Podcast.__init__</h3>
<p><a href="https://www.podcastinit.com/">_Podcast.__init___</a> is very similar to
Talk Python To Me, seeing as both shows revolve around inviting prominent
personalities to talk about their work. I haven't listened to all that many
of the episodes, but I really enjoyed the first 7-8 episodes. Initially,
the show had two hosts, but in the later episodes one of the original hosts
is conspicuously abscent, which to me was to the detriment of the show. It's
still a good show, mind you, but I enjoyed the early episodes more than the
few late ones that I've listened to. The length of the episodes seem hover
around the 1 hour mark, +/- some 20 minutes. I will probably revisit this
post once I've listened to a few more of the episodes, but as it stands I
recommend listening to the show from the beginning.</p>
<h3>Did I miss something?</h3>
<p>Those were my 3 top picks for Python podcasts. If you feel like I've missed
some great podcast(s), feel free to drop a comment!</p>What the self? Python's self demystified!2018-05-01T16:04:01+00:002018-05-01T16:04:01+00:00Simon Larséntag:slar.se,2018-05-01:/what-the-self-pythons-self-demystified.html<p>Any Python programmer will sooner or later want to (or have to) write a class.
With classes come <code>self</code>, the <em>seemingly</em> (do note the emphasis there)
magical keyword that you just have to write out as the first parameter to every
method. To really understand classes, you need to understand …</p><p>Any Python programmer will sooner or later want to (or have to) write a class.
With classes come <code>self</code>, the <em>seemingly</em> (do note the emphasis there)
magical keyword that you just have to write out as the first parameter to every
method. To really understand classes, you need to understand what <code>self</code>
actually is: neither magical, nor a keyword. Let's demystify this integral part
of Python classes!</p>
<h2><code>self</code> is not a keyword</h2>
<p>This is pretty easy to prove. Just open a Python interpreter and import the
<code>keyword</code> module.</p>
<div class="highlight"><pre><span></span><code><span class="o">>>></span> <span class="n">keyword</span><span class="o">.</span><span class="n">iskeyword</span><span class="p">(</span><span class="s2">"for"</span><span class="p">)</span>
<span class="kc">True</span> <span class="c1"># aha, makes sense</span>
<span class="o">>>></span> <span class="n">keyword</span><span class="o">.</span><span class="n">iskeyword</span><span class="p">(</span><span class="s2">"else"</span><span class="p">)</span>
<span class="kc">True</span> <span class="c1"># seems to be working</span>
<span class="o">>>></span> <span class="n">keyword</span><span class="o">.</span><span class="n">iskeyword</span><span class="p">(</span><span class="s2">"self"</span><span class="p">)</span>
<span class="kc">False</span> <span class="c1"># proof!</span>
</code></pre></div>
<p>Alternatively, one can always consult the
<a href="https://docs.python.org/3/reference/lexical_analysis.html#keywords">list of keywords in the Python docs</a>.
Since <code>self</code> is not a keyword, it has no special significance in the language
itself. We can also verify that it's not some funky builtin by simply typing
it out in the interpreter.</p>
<div class="highlight"><pre><span></span><code><span class="o">>>></span> <span class="bp">self</span>
<span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span>
<span class="n">File</span> <span class="s2">"<stdin>"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1</span><span class="p">,</span> <span class="ow">in</span> <span class="o"><</span><span class="n">module</span><span class="o">></span>
<span class="ne">NameError</span><span class="p">:</span> <span class="n">name</span> <span class="s1">'self'</span> <span class="ow">is</span> <span class="ow">not</span> <span class="n">defined</span>
<span class="n">name</span> <span class="s1">'self'</span> <span class="ow">is</span> <span class="ow">not</span> <span class="n">defined</span>
</code></pre></div>
<p>This can <a href="https://docs.python.org/3.6/library/constants.html">verified here</a>).
So, what in the world is <code>self</code>? Actually, it's just a variable name.</p>
<blockquote>
<p><strong>Trivia:</strong> If you try <code>keyword.iskeyword("True")</code> and
<code>keyword.iskeyword("False")</code> in both Python2 and Python3, you will find that
both are keywords in Python3, but not in Python2 (in 2, <code>True</code> and <code>False</code>
are just builtin constants). In fact, <code>True</code> and <code>False</code> are not even
write-protected in Python2, leading to shenanigans such as <code>True, False =
False, True</code> being possible. In Python3, the keyword status of <code>True</code> and
<code>False</code> make such an assignment a syntax error.</p>
</blockquote>
<h2><code>self</code> is just a variable name</h2>
<p>Consider the following code snippet of a (pretty useless) class that just
stores two values (that are just assumed to be addable with each other),
and defines a method that returns the sum of the values.</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">Tuple</span><span class="p">:</span>
<span class="w"> </span><span class="sd">"""A class for storing two values."""</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">first</span><span class="p">,</span> <span class="n">second</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">first</span> <span class="o">=</span> <span class="n">first</span>
<span class="bp">self</span><span class="o">.</span><span class="n">second</span> <span class="o">=</span> <span class="n">second</span>
<span class="k">def</span> <span class="nf">sum</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Return the sum of 'first` and 'second'."""</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">first</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">second</span>
</code></pre></div>
<p>Okay, so the class is terrible, but that really doesn't matter for the
purposes of this article. Now we see <code>self</code> in action for the first time.
From the code, it's purpose is quite clear: it refers to the object instance
on which the method is called. Usage looks something like this:</p>
<div class="highlight"><pre><span></span><code><span class="o">>>></span> <span class="n">t</span> <span class="o">=</span> <span class="n">Tuple</span><span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">first</span>
<span class="mi">4</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">second</span>
<span class="mi">3</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">sum</span><span class="p">()</span> <span class="c1"># `self` in `sum` refers to `t`</span>
<span class="mi">7</span>
</code></pre></div>
<p>So, <code>self</code> is just a reference to the object on which the method is called (in
this case, <code>t</code>). This will probably become more apparent when reading
<a href="#two-ways-to-call-methods">Two ways to call methods</a> further down, but just
suspend your disbelief for moment and assume it is so.
But, considering the <code>self</code> is just a variable to which <code>t</code> is assigned, what
happens if we replace <code>self</code> with, say, <code>donkey</code>?</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">Tuple</span><span class="p">:</span>
<span class="w"> </span><span class="sd">"""A class for storing two values."""</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="n">donkey</span><span class="p">,</span> <span class="n">first</span><span class="p">,</span> <span class="n">second</span><span class="p">):</span>
<span class="n">donkey</span><span class="o">.</span><span class="n">first</span> <span class="o">=</span> <span class="n">first</span>
<span class="n">donkey</span><span class="o">.</span><span class="n">second</span> <span class="o">=</span> <span class="n">second</span>
<span class="k">def</span> <span class="nf">sum</span><span class="p">(</span><span class="n">donkey</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Return the sum of 'first' and 'second'."""</span>
<span class="k">return</span> <span class="n">donkey</span><span class="o">.</span><span class="n">first</span> <span class="o">+</span> <span class="n">donkey</span><span class="o">.</span><span class="n">second</span>
</code></pre></div>
<p>In fact, this will work exactly the same as when the first parameter was called
<code>self</code>, and usage is unchanged:</p>
<div class="highlight"><pre><span></span><code><span class="o">>>></span> <span class="n">t</span> <span class="o">=</span> <span class="n">Tuple</span><span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">first</span>
<span class="mi">4</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">second</span>
<span class="mi">3</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">sum</span><span class="p">()</span>
<span class="mi">7</span>
</code></pre></div>
<p>To summarize, the first parameter in a method is simply a variable that refers to
the instance on which the method was called. Naming it <code>self</code> is
<a href="https://docs.python.org/3.6/howto/descriptor.html#functions-and-methods">just a convention</a>
, and we could name it anything. Note also that there is no technical
need for consistency across methods, we could name the first parameter to the
<code>__init__</code> method <code>donkey</code>, and the first (only) parameter to the <code>sum</code> method
<code>shrek</code>, and it'd still work.</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">Tuple</span><span class="p">:</span>
<span class="w"> </span><span class="sd">"""A class for storing two values."""</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="n">donkey</span><span class="p">,</span> <span class="n">first</span><span class="p">,</span> <span class="n">second</span><span class="p">):</span>
<span class="n">donkey</span><span class="o">.</span><span class="n">first</span> <span class="o">=</span> <span class="n">first</span>
<span class="n">donkey</span><span class="o">.</span><span class="n">second</span> <span class="o">=</span> <span class="n">second</span>
<span class="k">def</span> <span class="nf">sum</span><span class="p">(</span><span class="n">shrek</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Return the sum of 'first' and 'second'."""</span>
<span class="k">return</span> <span class="n">shrek</span><span class="o">.</span><span class="n">first</span> <span class="o">+</span> <span class="n">shrek</span><span class="o">.</span><span class="n">second</span>
</code></pre></div>
<blockquote>
<p><strong>IMPORTANT:</strong> <em>Always</em> name the first parameter to a method <code>self</code>. The
convention exists for a reason: it makes your code more readable.</p>
</blockquote>
<p>We could leave it at this, and hopefully walk away with a slightly better
understanding of why we put <code>self</code> everywhere in methods. But I think diving
just a little bit deeper into where the first argument to methods actually
comes from will prove fruitful.</p>
<h2>Two ways of calling methods</h2>
<p>Methods are defined on the class itself, and not on the instance. There is
actually a way to call a method directly on the class, that is equivalent to
the way we usually call methods on the instance.</p>
<div class="highlight"><pre><span></span><code><span class="o">>>></span> <span class="n">t</span> <span class="o">=</span> <span class="n">Tuple</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">6</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">sum</span><span class="p">()</span> <span class="c1"># regular method call</span>
<span class="mi">11</span>
<span class="o">>>></span> <span class="n">Tuple</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="c1"># calling the method on the class itself, and passing</span>
<span class="mi">11</span> <span class="c1"># `t` as the `self` argument</span>
</code></pre></div>
<p>The second way of calling <code>sum</code> is explicit about where <code>self</code> comes from: it's
passed in as the first argument. If we think of the first, "regular" way of
calling methods as shorthand for the second, it's suddenly entirely clear what
the first argument actually is (the instance itself). Calling a method that has
more parameters than just <code>self</code> works as expected: simply pass in the
additional arguments. To be super clear, with the following method added to
<code>Tuple</code></p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">sum_mod</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">mod</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Return the sum of the members modulo 'mod'."""</span>
<span class="k">return</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">first</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">second</span><span class="p">)</span> <span class="o">%</span> <span class="n">mod</span>
</code></pre></div>
<p>the following two method calls are equivalent</p>
<div class="highlight"><pre><span></span><code><span class="o">>>></span> <span class="n">t</span> <span class="o">=</span> <span class="n">Tuple</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">7</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">sum_mod</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
<span class="mi">2</span>
<span class="o">>>></span> <span class="n">Tuple</span><span class="o">.</span><span class="n">sum_mod</span><span class="p">(</span><span class="n">t</span><span class="p">,</span> <span class="mi">5</span><span class="p">)</span>
<span class="mi">2</span>
</code></pre></div>
<p>And that pretty much concludes what I wanted to cover in this article!</p>
<h2>Summary</h2>
<p>In the beginning, someone (probably Guido) said <em>let there be <code>self</code></em>. And there was.
The first parameter to a method is since then, <em>by convention</em>, called <code>self</code>, and
refers to the object on which the method was called.
Calling a method on some object <code>t</code> (e.g. <code>t.sum()</code>) can be viewed as shortand for
calling the method on its class, and passing in a reference to <code>t</code> as the first
argument (e.g. <code>Tuple.sum(t)</code>). If you are interested in learning about the dark magic
going on behind the scenes, you can read up on the official documentation for the
<a href="https://docs.python.org/3.6/howto/descriptor.html">descriptor protocol</a>, and
more specifically the
<a href="https://docs.python.org/3.6/howto/descriptor.html#functions-and-methods">Functions and Methods</a>
part of it. It is however somewhat advanced, and I don't find it essential to
understanding the semantics of method calls in Python. I hope you have found
this article enlightening, stay tuned for more Python in the coming week!</p>Properties as Pythonic setters2018-04-05T18:26:10+00:002018-04-05T18:26:10+00:00Simon Larséntag:slar.se,2018-04-05:/properties-as-pythonic-setters.html<p>This is the second part in a two part series on Python properties. In
<a href="https://slar.se/properties-as-pythonic-getters.html">Part 1</a> (which readers will be assumed to
have at least skimmed through), we saw how a property can be used to create a
read-only attribute that can be accessed like any data attribute (i.e …</p><p>This is the second part in a two part series on Python properties. In
<a href="https://slar.se/properties-as-pythonic-getters.html">Part 1</a> (which readers will be assumed to
have at least skimmed through), we saw how a property can be used to create a
read-only attribute that can be accessed like any data attribute (i.e with
<code>obj.attr</code>), but raises an <code>AttributeError</code> when written to. Now, we will look
at how to expand the property to also allow us to write to <code>count</code> like it's a
normal data attribute (i.e. with <code>t.count = 42</code>), while also doing input
validation.</p>
<h3>A property as a Pythonic setter</h3>
<p>Using the <code>Ticker</code> class version from the final listing in
<a href="https://slar.se/properties-as-pythonic-getters.html">Part 1</a>, we are unable to set the <code>count</code>
attribute to <em>any</em> value.</p>
<div class="highlight"><pre><span></span><code>>>><span class="w"> </span><span class="nv">t</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>Ticker<span class="o">(</span><span class="m">24</span><span class="o">)</span><span class="w"> </span><span class="c1"># valid range for count is thus [0, 23]</span>
>>><span class="w"> </span>t.count<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">11</span><span class="w"> </span><span class="c1"># this is well within that range</span>
Traceback<span class="w"> </span><span class="o">(</span>most<span class="w"> </span>recent<span class="w"> </span>call<span class="w"> </span>last<span class="o">)</span>:
<span class="w"> </span>File<span class="w"> </span><span class="s2">"<stdin>"</span>,<span class="w"> </span>line<span class="w"> </span><span class="m">1</span>,<span class="w"> </span><span class="k">in</span><span class="w"> </span><module>
AttributeError:<span class="w"> </span>can<span class="s1">'t set attribute</span>
<span class="s1">can'</span>t<span class="w"> </span><span class="nb">set</span><span class="w"> </span>attribute
>>><span class="w"> </span><span class="k">for</span><span class="w"> </span>_<span class="w"> </span><span class="k">in</span><span class="w"> </span>range<span class="o">(</span><span class="m">11</span><span class="o">)</span>:<span class="w"> </span><span class="c1"># doing it the hard way ...</span>
...<span class="w"> </span>t.tick<span class="o">()</span>
>>><span class="w"> </span>t.count
<span class="m">11</span>
</code></pre></div>
<p>This presents something of a usability issue, as the only way to set the
<code>Ticker</code>'s internal count to a specific value (using the public API) is by
calling <code>tick()</code> an appropriate amount of times. If we were to use the Ticker
as, say, a clock, we'd definitely want to be able to set <code>count</code> to a value
within the range <code>[0, _end)</code> by simple assignment. Fortunately, there is a
simple way to expand a property with a setter method using the <code>@<name>.setter</code>
decorator, where <code><name></code> is replaced with the name of the property. For the
<code>count</code> property of the <code>Ticker</code> class, it looks like this:</p>
<div class="highlight"><pre><span></span><code><span class="nd">@count</span><span class="o">.</span><span class="n">setter</span>
<span class="k">def</span> <span class="nf">count</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">val</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Set the internal count to val."""</span>
<span class="k">if</span> <span class="n">val</span> <span class="o"><</span> <span class="mi">0</span> <span class="ow">or</span> <span class="n">val</span> <span class="o">>=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_end</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">val</span><span class="si">}</span><span class="s2"> is out of range for attribute count."</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_count</span> <span class="o">=</span> <span class="n">val</span>
</code></pre></div>
<blockquote>
<p><strong>Note:</strong> A string literal preceeded with an <code>f</code> is an <em>f-string</em>. This is a
Python 3.6 feature. For backwards compatability, you could change to using
<code>string.format</code> like this:
<code>"{} is out of range for attribute count.".format(val)</code></p>
</blockquote>
<p>The code should be fairly self-explanatory. The setter takes a value <code>val</code> as
an argument. If <code>val</code> is outside of the allowed range <code>[0, _end)</code>, a
<code>ValueError</code> is raised. Otherwise, <code>_count</code> is set to <code>val</code>. The error message
could be more informative, but I did not want to obscure the important parts
with a lot of text. We have thus defeated the aforementioned usability issue,
and usage now looks like this:</p>
<div class="highlight"><pre><span></span><code><span class="o">>>></span> <span class="n">t</span> <span class="o">=</span> <span class="n">Ticker</span><span class="p">(</span><span class="mi">24</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">count</span>
<span class="mi">0</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">tick</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">count</span>
<span class="mi">1</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">count</span> <span class="o">=</span> <span class="mi">11</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">count</span>
<span class="mi">11</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">tick</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">count</span>
<span class="mi">12</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">count</span> <span class="o">=</span> <span class="mi">24</span>
<span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span>
<span class="n">File</span> <span class="s2">"<stdin>"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1</span><span class="p">,</span> <span class="ow">in</span> <span class="o"><</span><span class="n">module</span><span class="o">></span>
<span class="n">File</span> <span class="s2">"<stdin>"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">23</span><span class="p">,</span> <span class="ow">in</span> <span class="n">count</span>
<span class="ne">ValueError</span><span class="p">:</span> <span class="mi">24</span> <span class="ow">is</span> <span class="n">out</span> <span class="n">of</span> <span class="nb">range</span> <span class="k">for</span> <span class="n">attribute</span> <span class="n">count</span><span class="o">.</span>
<span class="mi">24</span> <span class="ow">is</span> <span class="n">out</span> <span class="n">of</span> <span class="nb">range</span> <span class="k">for</span> <span class="n">attribute</span> <span class="n">count</span><span class="o">.</span>
</code></pre></div>
<p>Seems to work just the way we want it to!</p>
<h3>Ticker full listing (with getter/setter property)</h3>
<p>Here is the full listing of the <code>Ticker</code> class.</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">Ticker</span><span class="p">:</span>
<span class="w"> </span><span class="sd">"""A Ticker ticks from 0 to an upper limit, and then starts over."""</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">end</span><span class="p">:</span> <span class="nb">int</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Create a Ticker that starts over at end"""</span>
<span class="k">if</span> <span class="n">end</span> <span class="o"><=</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">"end must be greater than 0!"</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_end</span> <span class="o">=</span> <span class="n">end</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_count</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">def</span> <span class="nf">tick</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Increment the internal count by 1."""</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_count</span> <span class="o">=</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_count</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span> <span class="o">%</span> <span class="bp">self</span><span class="o">.</span><span class="n">_end</span>
<span class="nd">@property</span>
<span class="k">def</span> <span class="nf">count</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Return the current count."""</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_count</span>
<span class="nd">@count</span><span class="o">.</span><span class="n">setter</span>
<span class="k">def</span> <span class="nf">count</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">val</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Set the internal count to val."""</span>
<span class="k">if</span> <span class="n">val</span> <span class="o"><</span> <span class="mi">0</span> <span class="ow">or</span> <span class="n">val</span> <span class="o">>=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_end</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">val</span><span class="si">}</span><span class="s2"> is out of range for attribute count."</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_count</span> <span class="o">=</span> <span class="n">val</span>
</code></pre></div>Properties as Pythonic getters2018-04-05T18:25:55+00:002018-04-05T18:25:55+00:00Simon Larséntag:slar.se,2018-04-05:/properties-as-pythonic-getters.html<p>If you come from either Java or C++, you've probably written your fair
share of getter and setter (also called accessor and mutator) methods.
It is common for programmers that transition from such a language to Python
to carry over this practice. In many cases in Python, we simply forego …</p><p>If you come from either Java or C++, you've probably written your fair
share of getter and setter (also called accessor and mutator) methods.
It is common for programmers that transition from such a language to Python
to carry over this practice. In many cases in Python, we simply forego the
abstraction and access the attributes directly. Sometimes, however, getters
and setters are useful for providing write-protection and input validation.
In this two-part series, we are going to explore how to make Pythonic setters
and getters using one of my favorite Python features: <strong>properties</strong>. </p>
<h3>Part 1 (this part): Properties as Pythonic getters</h3>
<p>In this first part, we take a look at how to use a property
to implement a read-only data attribute that can be accessed just like any other
data attribute (e.g. like <code>obj.attr</code>). Writing to it will, however, result in an
<code>AttributeError</code>. This is useful for preventing users
from accidentally changing the internal state of an object in an unintended
way, while still providing a uniform API. For example, we might want a
way to access the root element of a binary tree, but without risking to alter
its container.</p>
<h3><a href="https://slar.se/properties-as-pythonic-setters.html">Part 2: Properties as Pythonic setters</a></h3>
<p>In the second part, we'll have a look at how we can use properties to also
implement a setter method, with input validation, that can be utulized just
like any plain ol' data attribute (e.g. like <code>obj.attr = 42</code>). This is useful
when the attribute has some legal set of values.</p>
<h2>The Ticker class</h2>
<p>For the purpose of learning properties, we will develop a fairly useless class
called <code>Ticker</code>. All it does is tick from <code>0</code> to some boundary, and then restart
from <code>0</code>. Two <code>Ticker</code> instances could, for example, represent a rudimentary
clock with hour and minute counts. The first version of <code>Ticker</code> is outlined
below.</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">Ticker</span><span class="p">:</span>
<span class="w"> </span><span class="sd">"""A Ticker ticks from 0 to an upper limit, and then starts over."""</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">end</span><span class="p">:</span> <span class="nb">int</span><span class="p">):</span> <span class="c1"># ': int' is an optional type hint</span>
<span class="w"> </span><span class="sd">"""Create a Ticker that starts over at end"""</span>
<span class="k">if</span> <span class="n">end</span> <span class="o"><=</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">"end must be greater than 0!"</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_end</span> <span class="o">=</span> <span class="n">end</span>
<span class="bp">self</span><span class="o">.</span><span class="n">count</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">def</span> <span class="nf">tick</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Increment the internal count by 1."""</span>
<span class="bp">self</span><span class="o">.</span><span class="n">count</span> <span class="o">=</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">count</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span> <span class="o">%</span> <span class="bp">self</span><span class="o">.</span><span class="n">_end</span>
</code></pre></div>
<p>We can use this class something like this:</p>
<div class="highlight"><pre><span></span><code><span class="o">>>></span> <span class="n">t</span> <span class="o">=</span> <span class="n">Ticker</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">count</span>
<span class="mi">0</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">tick</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">tick</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">count</span>
<span class="mi">2</span>
<span class="o">>>></span> <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">3</span><span class="p">):</span>
<span class="o">...</span> <span class="n">t</span><span class="o">.</span><span class="n">tick</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">count</span>
<span class="mi">0</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">count</span> <span class="o">=</span> <span class="mi">42</span> <span class="c1"># uh oh...</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">count</span>
<span class="mi">42</span> <span class="c1"># this is an illegal state</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">tick</span><span class="p">()</span> <span class="c1"># back to a legal state in the next tick</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">count</span>
<span class="mi">3</span>
</code></pre></div>
<p>As long as the <code>count</code> variable is only read from, there are no issues with this
design. Unfortunately, directly assigning to <code>count</code> may put the <code>Ticker</code> in an
illegal state, i.e. such that <code>count</code> is outside of its expected range of
<code>[0, _end)</code>. This isn't so much an issue for the <code>Ticker</code> itself, as it is
returned to a legal state on the next tick. Other functionality
depending on the <code>Ticker</code> to keep within the <code>[0, _end)</code> range could however be
in for a nasty surprise, meaning that there is a serious usability issue here.</p>
<p>Thus to the crux:</p>
<blockquote>
<p>How do we protect the <code>count</code> variable from being put in an illegal state,
while still allowing access to it?</p>
</blockquote>
<h2>Solving the problem</h2>
<p>First of all, we should make the <code>count</code> variable private (which in Python
equates to prepending an underscore). The issue that remains to be resolved is
how to expose <code>_count</code> in the public API of the class.</p>
<h3>Solution 1: A Java-style getter</h3>
<p>A Java or C++ programmer might instinctevly think of a traditional getter method.</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">get_count</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Return the current count."""</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_count</span>
</code></pre></div>
<p>This solution has two issues: it breaks the api, and it makes us think about
<code>count</code> as something more complicated than the mere data attribute that it
is. It would be much preferable if we could access <code>_count</code> just like we
accessed it before it was made private (i.e. with <code>t.count</code>), but at the same
time provide write protection (such that <code>t.count = 42</code> raises an error).
Enter the <em>property</em>.</p>
<h3>Solution 2: Using a property as a read-only data attribute</h3>
<p>Implementing the same functionality as <code>get_count()</code> with a property is dead
simple.</p>
<div class="highlight"><pre><span></span><code><span class="nd">@property</span>
<span class="k">def</span> <span class="nf">count</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Return the current count."""</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_count</span>
</code></pre></div>
<p>We use the <code>@property</code> decorator to say that the <code>count</code> method is a property.
This will let us invoke the <code>count</code> method without providing the parens, so
it will look like we are just accessing a data attribute named <code>count</code>.
Usage now looks like below:</p>
<div class="highlight"><pre><span></span><code><span class="o">>>></span> <span class="n">t</span> <span class="o">=</span> <span class="n">Ticker</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">count</span>
<span class="mi">0</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">tick</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">tick</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">count</span>
<span class="mi">2</span>
<span class="o">>>></span> <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">3</span><span class="p">):</span>
<span class="o">...</span> <span class="n">t</span><span class="o">.</span><span class="n">tick</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">count</span>
<span class="mi">0</span>
<span class="o">>>></span> <span class="n">t</span><span class="o">.</span><span class="n">count</span> <span class="o">=</span> <span class="mi">42</span>
<span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span>
<span class="n">File</span> <span class="s2">"<stdin>"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1</span><span class="p">,</span> <span class="ow">in</span> <span class="o"><</span><span class="n">module</span><span class="o">></span>
<span class="ne">AttributeError</span><span class="p">:</span> <span class="n">can</span><span class="s1">'t set attribute</span>
<span class="n">can</span><span class="s1">'t set attribute</span>
</code></pre></div>
<p>Excellent! We have the exact same API as when <code>count</code> was a public attribute,
but without the risk of accidental overwriting. This is precisely what we wanted,
and a Pythonic way of dealing with the issue of providing read access to fragile
state variables.</p>
<h3>Ticker full listing</h3>
<p>It always annoys me when I get to the conclusion of some tutorial, and the end
result is just assumed to be obvious. Therefore, here is the full listing of
<code>Ticker</code> with a property as a getter.</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">Ticker</span><span class="p">:</span>
<span class="w"> </span><span class="sd">"""A Ticker ticks from 0 to an upper limit, and then starts over."""</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">end</span><span class="p">:</span> <span class="nb">int</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Create a Ticker that starts over at end"""</span>
<span class="k">if</span> <span class="n">end</span> <span class="o"><=</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">"end must be greater than 0!"</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_end</span> <span class="o">=</span> <span class="n">end</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_count</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">def</span> <span class="nf">tick</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Increment the internal count by 1."""</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_count</span> <span class="o">=</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_count</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span> <span class="o">%</span> <span class="bp">self</span><span class="o">.</span><span class="n">_end</span>
<span class="nd">@property</span>
<span class="k">def</span> <span class="nf">count</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Return the current count."""</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_count</span>
</code></pre></div>
<p>Now is about the time to move on to
<a href="https://slar.se/properties-as-pythonic-setters.html">Part 2</a>, in which we expand on the <code>count</code>
property to allow us to set the internal count, but only within the range <code>[0,
_end)</code>!</p>