PUNKTO

PUNKTO | Notetaking Is Life Changing

Wed, 03 Dec 2025 00:00:00 PST

Notetaking Is Life Changing

The first section of this blogpost is on the topic of maintaining a personal knowledgebase in general, and the second section is about my personal setup for information retention and organization. You can skip ahead to my technical details by clicking here.

Write It Down

It is imperative that you write it down. The importance of writing it down cannot be overstated.

The thoughts that you have in your mind right now are only in your mind right now. You might not need them now. You might need them later. They might not be in your mind later, when you need them. Your mind is an ephemeral rolling storage device. Once it loses power all the information contained therein is gone, so writing things down is the only way for you to continue propogating information after your death. Once enough time passes (even if you don't die), the overwhelming majority of the thoughts in your mind will be replaced with new thoughts. Only those that are foundational or reinforced through some type of spaced repetition are going to last a long time. If you write down your thoughts, you can keep them after you forget them.

You know that thought that you had in the shower? That thought was a seed. When you posted that thought to social media or in the group chat with the boys, you ate the seed. Maybe it was a tasty seed. Maybe it felt good to eat. Maybe other people saw the seed and said "oh that's a good looking seed, you're so cool" and you got gratification from a lot of likes 👍 or upvotes ⬆ or retoots 🔃 or (You)s 🐸 and now you've received your reward for finding that seed and it's gone. You ate it. It's digested in your guts and now it's useless. It's become shit. Instead of doing that, you could save the seed. Put it in a packet. Keep the packets in a seed vault. When you have enough seeds in a packet, you can plant and water them and tend to them as they grow into big trees that bear lots of fruit. Those fruits have more seeds in them. Those fruits are tastier than the seeds alone. You can share the fruits. They're more filling and nutrient-rich. If you write it down somewhere and hold onto it for a while, you get a tastier snack with more nutrients and more impact on the world. Delay your gratification by writing it down and sitting on it until it's worth more than a couple of bytes.

Do you remember all the stuff that you learned in high school? Maybe you're young enough that you do. I'm 10+ years post-graduation and I don't remember any of it. Most of it was probably useless, but I did learn it at one point. I sat down and spent the time and energy to learn about the holocaust and the allgeories behind Animal Farm a half dozen times each. Where's the proof of work? Where's the evidence, even just for myself, that I did that? Where's the record in my own words and terms? It's gone. If I had written it down somewhere that lasted, it would still be here. In the one year of community college that I attended before dropping out, I took a course on the history of the Western world. The notes for that class got written down digitally. I still have them. Do I need to reference them and know what Herodotus thought about the Persian empire? No, that information will never be useful to me. But I have it for my own personal reference and enjoyment.

Do you remember the exact steps that you took on that complicated server install? What about your thought process behind the Docker container that you made? If it was very recent, you probably do. Do you remember the ones that you did 5 years ago? I'm willing to bet not. You don't have to try to remember it if you just write it down. I can't speak to your brain, but I know that mine is really bad at remembering things. I can free up energy and space and effort by giving up on remembering things. Instead I just write it down and move on. I can focus my brainpower on the task at hand, I don't need to waste any trying to remember details of the email server migration that I did last weekend.

Maybe your name is Plato and you think that your writing will out-mode your memories. Maybe it will, but your next memories will also out-mode your current memories. Your current memories are going to be out-moded anyways, so you might as well write them down. Also Plato had a lot less ideas and information to keep track of than you do. The information landscape was simpler back then. You could fit it all in your monkey brain and it wasn't generally portable to other people. Now there's too much to keep track of and what you do figure out can be useful for others.

If you think that you understand something, you might be right. A good way to tell if you actually understand it or not is to try to explain it to someone else. In the asbsence of another person, you can at least put your attempted understanding into words by writing it down. I guarantee that this will force you to flesh out your concepts. You'll find holes in your reasoning, gaps that you missed, and you'll reveal what you really think to yourself. Your own thoughts are, surprisingly often, a mystery to yourself. You can reveal them by writing them down. You can think so much faster than you can do anything else. You can't act as fast as you can think. You can't read, speak, or write as fast either. Of all of these, writing is the slowest. When you write it down, you force yourself to slow your thinking way down to the speed that you can write. This forces you to spend more time on your thoughts, giving them more body, greater clarity, and more cohesion.

Teams and organizations maintain knowledgebases and wikis in part for collaboration, in part to replace people when they are gone, and in part to force people to think through things. You are on a team with your past self, and it makes sense for you to collaborate with that person like you would any team mate, by writing down the things that you figure out so that they don't have to figure it out from scratch. Having a robust knowledgebase is like standing on the shoulders of giants, but you can be your own giant, and the shoulder that you stand on can be your own. In order to do this though, you have to write it down.

Some Systems That Others Use

These aren't new revelations. Other people have had these ideas about recording information before. A quick search for "personal knowledgebase software" turns up a never-ending plethora of tools and apps. Seriously there are so many. Some of them surely are good.

The application Obsidian has been quite popular lately among Internet Computer Dudes, and for good reason. It's an open source app that keeps all of your documents under your control and stores them in markdown, the current lingua franca of hand-written text. I have friends who use Obsidian with great success and they are quite happy with it. That's fine. If Obsidian works for you and you like it, then Obsidian seems like a great choice. I personally can't stand writing huge amounts of text outside of my ${EDITOR} because I inevitably wind up hitting Ctrl+W to delete a word and that closes the window in all other applications. I also don't like Markdown. Specifically the difference between bold and italic text in Markdown betrays the concept that bold is just italic but more so. Italic text in Markdown is surrounded by a single asterisk. Bold text is surrounded by two asterisks. Bold and italic text is surrounded by three asterisks. I don't think that this makes sense, because I see bold and italic text as two separate things, not different degrees of the same thing. This is my main gripe with Markdown and it's why anything that uses Markdown is a non-starter for me.

There are a bazillion other similar applications. Applications like Joplin, Wiki.js, BookStack, LogSeq (with their friendly CLA that will definitely never come back to bite you), and an ever-growing library of others all compete for the same market segment. They are all fighting over the same piece of noosphere turf. They also all offer first-class mobile phone experiences (as does Obsidian), which is personally antithetical to my own use-case. I record my whole entire life in my notes, and I don't want it exposed on a device so riddled with backdoors like a mobile telephone. Admittedly, I'm currently safe from state-level threats, but who knows how long that will remain so. I'm confident that my LUKS disks will be just as safe in 10 years as they were 10 years ago.

There's also apps like AnyType which are great if your primary goal for software is to look cute. I've personally watched AnyType delete text seconds after it was typed by the user. This is probably the worst possible bug for any note-taking application to have, so I'll keep avoiding it forever. AnyType has its proprietary cousin, Notion, which is a locked-in walled garden that has recently started to shoe-horn AI junk.

The incomprehensible sea of choices in note-taking applications and the extreme amount of overhead required to use some of them (especially the self-hosted ones) has lead a lot of people to just ditch the whole thing and go for whatever default "notes" app they have available. This is really sensible and fits the bill for most people. You can spend a lot of time and energy learning and using the tool. You can easily spend more time and energy on the tool than it saves you. With something basic, you don't run into that issue.

The logical next step to the default notes app is to use one big text file, or "OBTF". A lot of people have had a lot of success with this technique. They simply put every piece of information that they care about into (as the name implies) one big text file and call it a day. There is some software written to make this more comfortable as well. If this is appealing to you, then by all means go ahead with that. My personal knowledgebase is (at the time of writing) 7.2 MiB across 1000+ text files. Putting all of that into one big text file for me would be too bulky.

There is a very attractive and friendly application called Linwood Butterfly which acts as an infinite canvas for you to take hand-written notes. This is really cool but I personally have no use for it. If I was taking lots of notes on a tablet or something similar, I would be all about this. But as it is I just use paper for hand-written things and the rest gets typed up, not scribbled down.

Pen and paper, in fact, is an all time classic #1 top hit blockbuster tool for information storage. Personal knowledgebases are not some newfangled tech bro innovation, they've been around for a long time, just with different names. The oldest and probably most well-known name for this type of thing when done on paper is the Commonplace Book. These were remarkably popular with commoners and the elite alike throughout most of history.

The most well-known commonplace book belonged to Marcus Aurelius, the Roman emperor from 161 to 180 AD. His personal commonplace book became what we know as "Meditations", a foundational work for modern stoicism. They were his personal writings that were never intended for mass publication, just his own notes. Various influential Greek thinkers long before him held similar practices, as did some later figures such as Thomas Jefferson, Napolean, and even Bill Gates.

The big shift between commonplace books and modern personal knowledgebases is the curation of the information that gets put inside of them. Pen and paper used to be a relatively expensive luxury good, so it didn't make sense to use them to record meaningless or trivial information. Many old commonplace book proponents advocated for using them to store wisdom, not facts. Personal knowledgebases, on the other hand, are commonly used to hold both. I think that this is fine. People need to keep track of more information today then they did in 161 AD and both paper and disk space have gotten remarkably cheap. We are no longer bound by the limitations of two millenia (or two centuries) ago.

Given the amount and structure of the information stored in the personal knowledgebases of most users, it is potentially more apt to compare them to Zettelkatsen. This is a practice of recording small pieces of information on individual notecards, then tagging and storing them in some structured manner.

Due to the size of zettelkatsen notecards and the inability to restructure organization and ordering in commonplace books, digital personal knowledgebases are wholey different beasts. They allow for a manner of information retreival and organization that simply wasn't practical before the digitization of the world. They are a relatively new superpower that you can tap into to improve your efficacy in various areas of life. This is primarily how I have used them in mine.

How I Write It Down

Some of my notes start on paper and some of them start digitally. Whether the notes start digitally or on paper is decided in large part by what is nearest to my hands when I need to record some piece of information. Paper notes usually get digitized, but not always.

Paper Notes

I carry around a small form-factor notebook in my wallet. The notebook is the same size as the popular "Field Notes" brand (3.5" x 5.5"), remarkably similar to the size of a passport. I buy off-brand in bulk because they are cheaper that way. My wallet, a Tintris brand "Journal Cover" is specifically advertised as being able to comfortably hold a Field Notes notebook, and it does a fine job. I'm not affiliated with them at all, it's just a statement of fact that that's the wallet I use. I have one pen that I just get refill carts for (again in bulk to save money), and I use this to record most anything that I want to keep when I'm out and about. I also use it when I'm home but don't want to gaze into the glowing rectangle.

Some people prefer to use the notes app on their phone. That's fine, but my phone is a little satanic rectangle and I am dealt psychic damage whenever I look at it. If I pull out my phone to take some note, I am provided with various information about unread emails, unread spam SMS, and unread messages on various messaging apps. I don't like seeing that. So instead I use pen and paper. Some people prefer to use their PKB app on their phone. That's fine, but as I mentioned above, I don't trust my phone to keep all of that information safe.

For longer form stuff, such as notes taken during meetings (where it would be unprofessional to bring a laptop) or notes taken during conference talks, I have some wire-bound full-size notebooks that I absolutely adore. They're the Cambridge Mead "Quad Writing Pad". They occasionally get discontinued and then brought back, so I buy them in bulk whenever they are available. I use these to take long-form notes instead of typing them sometimes because I write a lot slower than I type, which forces me to process and compress the information with my brain. This usually helps me to actually gain a better understanding of the information that I'm writing down, and digitizing them later helps with retention due to spaced repetition. It does have a higher time investment, which is the only real downside.

Paper notes get digitized if it is relevant for them to be. Not all of the paper notes should be digitized, so not all of them are.

Digital Notes

I mentioned a lot of different options for digital PKBs above. I don't use any of them, and instead brewed my own setup which seems to get slightly more complicated every year. I'm sure that in 3 years there will be more to it than I write here, but what follows is a dsecription of my setup as of December 2025.

You can download the whole tarball to play with this yourself, click here.

I consolidated everything across my config into one tarball for this post, so if something doesn't work quite as described please let me know and I'll adjust the tarball for others. Everything here assumes that your dokuwiki files are stored in ~/Projects/dokuwiki/[pages|media]/ as mentioned below. That's where it is on my system.

On my machine all of the nvim config is in init.lua, but in the tarball I moved them to .config/nvim/lua/dokuwiki_customization.lua. That way it doesn't conflict with your existing init.lua file and you can just add this line to yours to activate everything:

require("dokuwiki_customization")

Here's a quick and easy key for the customization that this brings to the table (no pun intended):

Mode	Keypress	Function
Insert	Alt+b	bold syntax around the cursor
Visual	Alt+b	bold syntax around the selection
Insert	Alt+i	italic syntax around the cursor
Visual	Alt+i	italic syntax around the selection
Insert	Alt+u	underlined syntax around the cursor
Visual	Alt+u	underlined syntax around the selection
Insert	Alt+t	monospaced syntax around the cursor
Visual	Alt+t	monospaced syntax around the selection
Insert	Alt+l	hyperlink syntax around the cursor
Visual	Alt+l	hyperlink syntax around the selection
Insert	Alt+N	Header number N (1-5) syntax around the cursor
Visual	Alt+N	Header number N (1-5) syntax around the selection
Normal	Enter	Follow dokuwiki link under the cursor or open the web link your web browser
Normal	BackSpace	Go back one page
Normal	\o	Open media referenced under the cursor with xdg-open
Normal	\w	Open current page in localhost DokuWiki instance
Normal	\r	Start audio recording
Normal	\s	Save audio recording
Normal	\\	Collapse or expand all folds
Bash Shell	transcribe	Transcribe audio recordings
Bash Shell	wikify [URL]	Import an external web article into the wiki (requires rdrview in your ${PATH})
Bash Shell	meeting [NAME]	Create a new meeting minutes file with the current date using the designated template
Bash Shell	quotes	Open the quotes file

My requirements for the system are what shaped my choices in technology. This is by no means the "correct" way to do it for everyone, but it is the correct way to do it for me and it might be the correct way to do it for you if you and I are similar people. I need a setup that:

Allows me to use my text editor, NeoVim.
Allows me to collaborate with other people if need be.
Can be easily rendered into rich text instead of the plaintext format that I write it in.
Is extensible to accomodate my changing needs and desires.

DokuWiki

DokuWiki is the foundation that everything in my PKB (or wiki) is based on. I like it because the markup syntax is nice to look at, it stores everything as text files, and the setup of the web application is remarkably easy. DokuWiki also allows me to collaborate with other people who don't live on the terminal. By spinning up a DW instance with a subset of the files from my PKB, I can share these in a sensible way with normal people, and I can accept changes from them.

Additionally, DokuWiki has a gitbacked plugin. This allows for using a remote git repo as the store for DW's text files. This is great because then you get version management and easy syncronization across multiple systems. I am able to sync my PKB from my laptop to my desktop and a webserver while maintaining the ability to change the files from all three without much forethought. Since I can have multiple git repos with different included files living in the same directory on my computer, I am also able to split off subsets of my wiki for collaboration with distinct teams while keeping everything cleanly integrated into one filesystem tree from my perspective.

Setting up DokuWiki isn't absolutely necessary for any of the rest of the customization below, but it is remarkably easy. On Ubuntu (as root):

apt install php php-xml php-fpm nginx
mkdir /var/www/dokuwiki/
cd /var/www/dokuwiki/
wget https://download.dokuwiki.org/src/dokuwiki/dokuwiki-stable.tgz
tar xvz --strip-components=1 -f dokuwiki-stable.tgz

And then put this into /etc/nginx/sites-enabled/default:

server {
        listen 127.0.0.1:80;
        server_name localhost;

        client_body_buffer_size 128k;
        client_max_body_size 4M;

        index doku.php;
        root /var/www/dokuwiki;

        location '~ ^/data/' {
                internal  ;
        }

        location / {
                try_files $uri $uri/ @dokuwiki;
        }

        location @dokuwiki {
                rewrite ^/(.*) /doku.php?id=$1&$args last;
                rewrite ^/_detail/(.*) /lib/exe/detail.php?media=$1 last;
                rewrite ^/_export/([^/]+)/(.*) /doku.php?do=export_$1&id=$2 last;
                rewrite ^/_media/(.*) /lib/exe/fetch.php?media=$1 last;
        }

        location ~ /(conf/|bin\/inc\/install.php) {
                deny all;
        }

        location ~ \.php$ {
                fastcgi_param REDIRECT_STATUS 200;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                fastcgi_pass unix:/run/php/php-fpm.sock;
                include fastcgi_params;
                try_files $uri $uri/ /doku.php;
        }

        location ~ ^/lib.*\.(js|css|gif|png|ico|jpg|jpeg)$ {
                expires 365d;
        }
}

Then just hit the URL in your browser and continue the setup. If you're doing this on a remote server, configuring the gitbacked plugin is pretty easy so I won't get into it here. I do this on my local machine so that I can easily see what it will look like in the web view as soon as I save a file, so I symlink /var/www/dokuwiki/data/[media|pages]/ to directories in my git tree.

NeoVim

I use NeoVim as my text editor. I used to use regular Vim but I switched away when Bram Moolenaar died. I thought that with his death, Vim would stop seeing development work done and it would stagnate. This was before I had formed ideas on free software immortality. At this point NeoVim is just what I use and my fingers type nvim whenever I want to edit something. I've passed the point of no return.

Since nvim is such an extensible text editor, it is able to integrate quite nicely into the DW tree structure. It also allows me to do things that would be quite laborious otherwise, which we'll get into in a bit. I have wound up with a lot of completely custom functionality written into my nvim environment, and I will warn you that some of it was vibecoded. Sorry, my brain is too small.

Syntax Highlighting

When working with a fancy markup language like DW's, I like to have syntax highlighting in my terminal. For this I use a modified version of Florian Preinstorfer's vim-dokuwiki. My version can be downloaded in the tarball above.

My changes allow for folding (with za) of html, php, code, and file blocks. This is useful since I wind up pasting in sometimes long terminal output in code blocks and being able to hide them is useful.

I need this syntax highlighting to apply whenever I open a DW file, which defaults to having the .dw file extension, but I personally prefer the .txt file extension. So for this I have this snippet set in my nvim's configuration:

-- Source Dokuwiki syntax file when a Dokuwiki file is loaded
vim.api.nvim_create_autocmd({"BufRead","BufNewFile"}, {
  pattern = {'*.txt', '*.dw', '*.dokuwiki'},
  command = "set filetype=dokuwiki"
})

Syntax Automation

DokuWiki's syntax is nice, but it sure does take a lot of typing to make a top level header. It's a whopping 12 equals signs! I have this section in my nvim config which makes doing most syntax a lot easier:

vim.cmd('vmap  "zdi**z**')
vim.cmd('imap  ****1hi')
vim.cmd('vmap  "zdi//z//')
vim.cmd('imap  ////1hi')
vim.cmd('vmap  "zdi__z__')
vim.cmd('imap  ____1hi')
vim.cmd('vmap  "zdi\'\'z\'\'')
vim.cmd('imap  \'\'\'\'1hi')
vim.cmd('vmap  "zdi[[z]]')
vim.cmd('imap  [[]]2ha')
vim.cmd('vmap  "zdi====== z ======')
vim.cmd('imap  ======  ======6hi')
vim.cmd('vmap  "zdi===== z =====')
vim.cmd('imap  =====  =====5hi')
vim.cmd('vmap  "zdi==== z ====')
vim.cmd('imap  ====  ====4hi')
vim.cmd('vmap  "zdi=== z ===')
vim.cmd('imap  ===  ===3hi')
vim.cmd('vmap  "zdi== z ==')
vim.cmd('imap  ==  ==2hi')

With this, Alt+b makes somehing bold, Alt+i makes something italic, Alt+u makes something underlined, Alt+t makes something monospaced, and Alt+l makes something into a link. Additionally, Alt+ any number 1-5 creates a header of that depth. This makes writing way easier, and it works in both visual mode (with the target text highlighted) and in insert mode (it just wraps your cursor in the syntax).

Media Opening

This is where the vibe coding starts.

DokuWiki pages are technically intended to be viewed in a web browser, where they can have embedded images and audio. I thought this was nice and I wanted to be able to open these files directly from my editor without having to go anywhere else, so I first set the keymap of \o to run a lua script:

vim.keymap.set("n", "o", require("openblock").open_under_cursor, { desc = "Open file or directory from {{:...}} block", noremap = true, silent = true})

And the openblock.lua is included in the above-linked tarball.

This is intended to be done in normal mode when the cursor is on top of a referenced file. It simply opens the target file with xdg-open, so whatever you have configured as your default application for that filetype should just spring up.

Link Following

When I am editing a file in my DW tree and I have a DW link to another file, I would like to be able to go straight to that file without having to type out the whole :e path/to/file.txt thing. Instead I just want to go to it and hit [Enter] on my keyboard and go there. I accomplish this with a vimscript (again vibecoded) that gets sourced from the nvim config:

vim.cmd("source ~/.config/nvim/custom/followlink.vim")

The full vimscript is (you guessed it) in the above-linked tarball. It just follows the link (from the CWD) under the cursor when you press [Enter] on your keyboard.

Browser Opening

Since I have a DokuWiki instance running on my local machine, I wanted an easy way to open the current page in a browser to see what it looks like there. The vimscript gets imported from the nvim config and the whole script is available in the tarball. It was vibecoded. Perhaps you can see a pattern.

vim.cmd("source ~/.config/nvim/custom/browser_open.vim")

From any page, simply hit \w and your browser should open the page that you are editing.

Audio Recorder

I have found it useful at times to have an easy audio recording functionality built into my text editor so that I can put live audio into my wiki. Same pattern as above:

vim.cmd("source ~/.config/nvim/custom/audio_recorder.vim")

Except there are two scripts for this one. I didn't actually vibecode these, I wrote them myself :) In fact nothing from here on out was vibecoded.

You start an audio recording with \r and save it with \s. Upon saving it adds a section below the embedded audio containing a special tag that the next script will look for.

Transcription

One cool thing about the last 3 years is that neural networks have had a glowup and now you can do automatic audio trasncription and it works pretty good. I have a beefy GPU in my desktop that can do this transcription easily, but I don't want it done on my laptop, which is weak by comparison.

For this I use an external script (although a small amount of vibe coding would surely get it integrated perfectly into the editor, I don't mind having it separate). I put this into my ${PATH} and just call it when I want to transcribe the audio recordings that have been generated. I just run this command on my desktop.

The command is transcribe and the script is in the above tarball (wow shocker).

Online Article Imports

One really cool feature that I saw on Notion and wanted to implement in my own wiki was the ability to easily import articles from the web into my own knowledgebase for posterity, archival, or reference. I implemented this by writing a wrapper script around rdrview, a standalone application that serves the same functionality as the "reader view" button in any common web browser. Specifically rdrview uses the same thing as Firefox.

Installing it is pretty easy, just do:

sudo apt install libxml2-dev libseccomp-dev libcurl4-openssl-dev
wget https://github.com/eafer/rdrview/archive/refs/tags/v0.1.5.tar.gz
cd rdrview-0.1.5/
tar xvzf v0.1.5.tar.gz
make
make install

It will put it in /usr/local/ by default. If you want it somewhere else, like to set it up with Environment Modules, then you have to modify the Makefile.

sed -i 's,/usr/local,/modules/sw/rdrview/0.1.5,' Makefile

Once you have rdrview installed, the bash script included in the above tarball should let you just run wikify [URL] and import a web article.

If you instead run wikify [URL] x, then it won't open the article in neovim after creating it.

Meeting Minutes

The Robert's Rules say you gotta have minutes for a meeting. I take meeting minutes of all the meetings that I go to and catalogue them in my wiki. It's helpful to have templates because some meetings follow regular structures that repeat every time. In order to make this easier on myself, I place my template files in meeting_minutes/${ORG}/${TYPE}-template.txt. Then when I want to create a new meeting from that template, I just run the command meeting ${TYPE} and the script in the above tarball figures out the rest.

It creates a new file named meeting_minutse/${ORG}/${DATE}-${TYPE}.txt with the starting contents as the corresponding template file and opens it in nvim. The date format is specifically chosen to be sorted by ls and show up in the correct order, so I sadly couldn't do the American date format for it, even though I generally prefer the American date format.

"That's Going In Quotes"

Spending formative years in VoIP rooms on the Internet has taught me that when someone says something funny or poignant or particularly characteristic, you should write it down in a "quotes file" so that you can refer to it later. My recorded quotes live in my wiki in the notes/quotes.txt file, and I open the file simply by running the command quotes. This is really fast and easy and lets me write things down before I forget them.

Reinventing The Wheel

The astute among the readers may note that I've basically recreated the feature set of VimWiki but with slightly more system integration. This is true. I used VimWiki at one point, and I quite enjoyed it. However some stuff happened that resulted in me losing my VimWiki setup, so I rebuilt from the ground up with DokuWiki instead until I wound up with what you see here.

The extra astute among the readers may note that I'm using NeoVim as an extensible text editor and I could have saved a lot of steps if I just did this all in Emacs. It even has Org-mode! Unfortunately, I can't figure out how to close nvim, so I can't try any other text editors.

PUNKTO | SeaGL 2025 Absolutely Rocked

Tue, 11 Nov 2025 00:00:00 PST

SeaGL 2025 Absolutely Rocked

I attended SeaGL, the Seattle GNU/Linux conference, this year. It ran on Nov 7th and 8th of 2025. This blogpost is a retrospective / review of this year's conference. I've been to SeaGL many times before, and I've even contributed some volunteer labor to the event in the past. This year's conference was the best SeaGL that I've ever been to, and quite likely the best FOSS-focused conference that I've ever been to. This SeaGL has planted more seeds in my mind for future conference talks and blogposts and fun projects than any other conference I've been to.

I had a great time and I look forward to next year's SeaGL. I hope that the conference is able to garner some sponsors for next year's event so that they can continue operating into the future. As I understand it, they are extremely broke and while the software might be Free, UW is expensive.

My Talk

I spoke at SeaGL this year. This was my third time speaking at SeaGL and my sixth time speaking at conferences overall. This presentation was a bit of a "homecoming" for me, since SeaGL was the first (and second) conference that I ever spoke at. I spoke on the subject of software (im)mortality with a talk titled What Is Free May Never Die. A recording is available here.

The core idea of my presentation was that Free software doesn't die. When the forces of society conspire to kill software, they are only able to kill proprietary software. They can't kill Free software. I explain this idea much more fully in the recording, so I recommend watching that. If you don't have time to watch a 20 minute video, you can also just read the whole script here and look at the slides here. I tried to write it so that the slides weren't 100% necessary to get the point, and I think that I did a good job on this front.

Goals Of My Presentation

The core idea popped into my head while I was in the shower one day. In our popular culture it is common to take these types of ideas from the shower and plaster them on social media for instant gratification. I see these kinds of ideas as seeds, and posting them online is a form of eating the seeds. Instead of eating the seeds directly, I like to filter them into seed packets in my seed vault (text files in my personal wiki). Once I have enough seeds in a packet, I sow them, water them, tend to them, and harvest the fruit. This presentation was the fruit. By doing it this way, I'm able to eat the fruit (way more satisfying and tasty than the seed) and create more seeds. The main goal of this talk was to get the core idea out of my brain and plant seeds in the brains of others, where they might water them and grow them into fruit. I think that the framework that I presented in my talk might be practically useful in helping to motivate other people to write and maintain Free software. It might even be helpful in driving people to use Free software, but I'm not as sure about that. Only time will tell if I was successful in this primary goal.

I also wanted to give this non-technical talk to expand my capabilities and repertoire in speaking. I had previously only spoken about technical (or historical) topics, which are relatively easy to speak about. For those types of talks, you're just standing in the front of a room repeating things that are verifiably true. People can disagree with your presentation style and your stated values and all sorts of things, but the skeleton of your talk (the raw facts) are raw facts no matter how you slice them. This was a more "philosophical" talk, and I had quite a difficult time writing it. I did lean heavily on historical facts for a good portion of my talk, but these were provided solely to back up the factuality of the core idea, something that can't be arrived at by purely technical operations. In this sense my talk was successful because I did a fairly good job on it.

Missing The Mark

My talk wasn't perfect (no presentation is). There were things that I could have done better.

Comparing the script that I wrote to the actual words that I said in the recording, I missed a couple of spots that I thought would be impactful. More rehearsals (I did 15) would help with this.
People didn't know instinctively when the "time for applause" was because I didn't build to it properly. I needed to build the anticipation towards climax better. I did a really good job of this in my LibrePlanet 2024 talk.
I went over time. I knew that this would happen because I had a 20 minute slot and my practice runs kept coming out to slighty more than 20 minutes. I even cleared this up with the conference organizers beforehand and was told that it wouldn't be a problem for me to go over slightly (I anticipated 45 seconds to 2 minutes). However, I forgot to mention this to my room proctor, who proceeded to rush me off stage as soon as my time was up. This was a shortfall on my part: it was my responsibility to make this clear before I started the presentation.

The Feedback

The reception on this presentation was the best that I have ever had for a conference talk. People came up to me throughout the conference and during both after parties to say how much they liked it. The feedback that I can recall now as:

It was like a YouTube video essay (I think that this was intended as a compliment but it could have been an underhanded diss)
It was like a long Ted talk
Your timing was impeccable, you knew your slides so well
I felt moved and touched
I tuned in to only part of your talk and was blown away
I tuned into your talk from another simultaneous talk and listened only to the audio, then recognized your voice from across the room at a loud after-party and came to thank you
This makes me feel bad about spending my professional life writing proprietary software

One of the pieces of software that I list as "dormant, waiting for someone to come pick it up" in my talk was BitKeeper. Toby Betts, the guy who talks about Temple OS, mentioned that he is actually the guy who is carrying the torch on BitKeeper. He compiled it and uses it personally because he doesn't like Git. This beautifully underlines my core point.

Some people also raised questions about the core idea (which is what I wanted).

What are the implications of AI on this?

My gut reaction was that AI doesn't have much to do about whether or not software can or does die. I personally view AI as a benefit to Free software. Proprietary software developers have a profit motive to use AI, and it produces a lower quality product (slop, vibe code), which is rough around the edges and has weird bugs. This helps to lower proprietary software's quality of work to what Free software users are accustomed to from Free software. Levelling the playing field like this works in our benefit.

I figured that there is some impact on software's longevity by being scooped up by an LLM. If you don't care about getting rich or getting recognition, then LLMs help to propogate your ideas. By copying your work and repeating it to other people, the pattern of symbols produced by your brain are further advanced into the future. This is a huge benefit if your primary goal is that your work become widely used (and immortalized).

The person who raised this question, however, works at Microsoft on an open-source project called Amplifier. He mentioned that their current private development work (to be publicly available Soon™) is able to produce high-quality and complex software based on a description of what the software should accomplish. I even talked with him about being able to do it all local and offline so that there is no reliance on the big dogs of "AI". He said that that was his current project. If this really is possible and becomes generally available, then it will drastically lower the bar for the general public to write their own software for their own personal needs. The ability to do this will change the landscape, as people won't need to purchase proprietary software in order to accomplish their goals in the future, they'll just describe it. It's possible that in the future, the vast majority of software that people use is personal software. If this happens, then I suspect the intrinsic human nature to share and want to be shared will take hold and we'll see a proliferation of Free software. Perhaps that's wishful thinking.

What if nobody has a copy?

One person asked this question about Free software's longevity. If Alice writes a piece of Free software and distributes it to Bob and Eve, and then all three of them die and their disks are encrypted and nobody else can ever get a copy, then yes the Free software is effectively dead, not dormant. This underlines how important it is to share your software with the general public if you want it to last.

The Followup

I think that there is more to be explored here, and I intend to if I ever speak on this topic in the future. Some of these expanded topics are:

What if the government makes Free software illegal?
Does the amount of software really matter? It seems like the amount of software that people use is more important.
Proprietary software being taken away form the users is a bigger problem than I initially highlighted. This is a real thing that is happening more and more often. Software hasn't been in popular use for long enough for the general population to have seen this happen enough and care.
What type of software is most like a handaxe/calculus?

Other Talks

I saw several other talks during the conference this year. There were some that I wanted to see but didn't get to either because I was engaged in discussions with people or because there were multiple talks that I wanted to see going on at the same time. Here are some thoughts on a select small number of the talks that I saw.

Fediverse Keynote

The opening talk of the entire conference was about the Fediverse, given by Evan Prodromou. Evan previously wrote a book specifically about programming for ActivityPub. I found his talk intriguing and he raised some interesting points.

The thing that stuck with me the most from Evan's talk was when he described the history of the blogosphere and went on to say that modern post-social-media attempts to return to such a setup are impotent nostalgia. He said that the past seems better but that it wasn't, and he said that it seems like we can return to it but we can't. For obvious reasons, this led me to cast a critical gaze at my own blog. I don't have any social media accounts, and I maintain this website as my primary public-facing persona. Am I suffering from nostalgia, trying in futility to return to a past that was worse (and I didn't even directly experience)?

I've looked at the possibility of making these blogposts available over ActivityPub in the past, but I've run into some interesting issues with doing so. First and foremost, this is a static website generated using an SSG that I cobbled together. Since ActivityPub is by its very nature a participatory technology, I would have to completely rewrite my website and learn new tools to do so. The real enticing factor that even makes me look in this direction is that syndicating this content on ActivityPub might make it more accessible for some people. I already have an RSS feed for these posts, but having them show up in a social-media-like interface that people are familiar with might help them get read. If more people read my blog then that counts as a "win" for me, and the Venn Diagram of people who use the fediverse and who use feed readers is not a circle.

There is an additional problem posed by the formatting of my blog. For the most part these posts are a lot of text in HTML with an image here or there. However there are also posts that have a huge number of images in image galleries. I don't know of any ActivityPub implementations that can support the way that I write these blogs.

Thankfully, I had the opportunity to talk with Evan about my specific blog and implementing it on the Fediverse at Friday's SeaGL after-party. He assured me that huge amounts of HTML text with images interspersed is not a problem at all for the fediverse and is in fact well-supported. The only issue for my blog is the fancy image galleries. One way around this limitation would be to have each gallery with text be a separate post. I think that this is a reasonable method to structure things. An alternative would be to use one of the many available RSS -> Fediverse bridges that exist on the public Internet.

Maybe I will eventually syndicate this blog's content on the fediverse. If I do, I'll wind up hosting my own server, since I can't ever seem to find a stable one hosted by others with a ToS that I can honestly agree to. It's the same reason that I run my own blog instead of using something like bearblog and my own Matrix server instead of using the default matrix.org one. Content policies are a slippery slope and I've seen them abused too many times to trust them with my free expression of thoughts. A smart man learns from his mistakes, but a wise man learns from the mistakes of others.

Local Offline AI

Adam Monsen gave a talk about running AI models on local hardware and I found it highly educational. It was an effective quick rundown of the local model landscape. I expect to replicate some of his successes going forward on my own hardware and I may even post about them on this blog.

James's Presentation

James Sundquist gave a presentation titled Physical Theatre, made using open source tooling. I've known James since LFNW 2025 and I've followed his podcast online since then as well. I'm not sure what James's talk had to do with Free software. He mentioned an open source script-writing application early in the presentation and then later made an off-handed comment that one of his props was open hardware. Other than that, his presentation was a theatre performance, not a tech talk.

It rocked.

I loved it. My girlfriend with me loved it. Everyone in the room loved it. It was a hilarious, engaging, wide-ranging tour of fun. James performed portions of his original stage works, all of which were originally written for groups but had been adapted for performance by one man. You don't notice that there is only one guy on stage. James had the energy and explosion of a whole crew. I will happily see any performance that he gives over any intriguing technical topic. I hope that we are never scheduled to speak at the same time.

If you ever have the opportunity to see James perform at a conference or in-person under the name Living Cartoon Company, I strongly encourage you to do so. He doesn't disappoint.

Keysigning

In the days leading up to SeaGL, discussions in the Matrix chatrooms led some people to want to engage in a GPG keysigning party. We planned it all out ahead of time, and it went pretty well. There were a few hiccups here and there, but we managed to get everything sorted out.

I produced a page on my website to prove that I am the same guy in person (people could compare my picture to my face in real life), printed out copies of my key, and even made digital copies on cheapo disposable laser-engraved flash drives. Other people brought their keys on paper (printed or hand-written) or pointed others to their websites or other keyservers to find their keys. We verified each other through a number of means, ranging from attested proofs like mine to government-issued ID, to recognizable people.

One thing that we all did after signing each other's keys was encrypt the signatures and send them back to each other. This was an effective way to prove that people did in fact have control of the corresponding private key.

The keysigning party was a lot of fun and I got my GPG key signed by five people (with hopefully a sixth to come Soon™). I signed the keys of:

Morgan from mcmillian.dev
Trevor from johnsonfarms.us
Ted Matsumura from the LPI
Adam Monsen, the aforementioned speaker and SeaGL cofounder
Matt from comp.uter.science
Edward Ly the software engineer and DDR chart artist

I expect to write a more detailed writeup of keysigning parties in the future, and I may even speak on the topic at conferences going forward. It was a really good time.

Element

I also engaged in Matrix keysigning with a few people. Matrix encrypts messages E2E and so runs into the same "is this really that person's key?" issue that GPG runs into. With Element X, the modern mobile client, some of us were able to able to verify each others' keys. There was a little bit of tomfoolery in that you have to click a surprising number of times to find the button on someone's profile to verify them, and it takes an unreasonable amount of time to load. Additionally, one person using legacy Element mobile (not the X variety) was unable to participate in this signing since that version expects a QR code instead of emoji comparison.

Element's keysigning was clunky and slow and unintuitive. I hope that it improves in the future. Admittedly GPG keysigning was clunkier and slower and less intuitive, but at least people expected that and were prepared for it. Chat applications have trained people to expect things instantly, and that expectation was not met with the key verification.

SeaGL had a large number of social events outside of the main conference this year and they all rocked. Everyone was friendly and sociable and I had a great time at everything. What follows are some notes from a few of these events (though not all).

After-parties

The Friday after-party was at Ada's Technical Books in Capital Hill. They have a rentable bar attached to the bookstore which SeaGL rented out. Here I talked with a handful of people from the conference on a wide range of topics. This is where an attendee recognized my voice from across the room and where I talked with Evan about ActivityPub for this blog. At one point early in the party, I saw two dudes sitting against the wall alone, not talking to anyone or engaging with anything. I struck up a conversation with them and I'm glad I did. I got to meet new people and make new friends.

One of these people was Edward (aforementioned keysigning party attendee). He's a really cool dude and I look forward to talking with him and meeting him at future conferences. I also met a guy named Wade who was from Florida, up at UW for an space exploration meeting. He stumbled upon SeaGL and just thought that the people seemed friendly. Only at the after-party did he find out that it's actually a real conference. Hearing Wade's life story was amazing, and I'm glad that I struck up conversation with him. I hope that he had a great rest of his time in Seattle, and I'm sad that I didn't get his contact information to keep in touch.

I also bought a book at Ada's during the Friday after-party. The book is Game Boy Coding Adventure. Long-time readers of this blog who read the Personal History section of my STK blogpost will see this as harkening back to some early childhood memories. I'm excited to work through this book, and I may wind up creating something worth posting on this blog in the future. I bought this book at the after-party instead of during the conference because I didn't want to carry it around all day. I'm happy to patronize the sponsors of SeaGL, I just hope that it shows up in their numbers as a SeaGL-related sale.

The Saturday after-party was at Big Time Brewery in the U-District. This party likewise resulted in some fun and engaging conversations with other conference-goers. We got to compare all three form-factors of Framework Laptops (12, 13, and 16), I met a bread delivery driver who is working on his own open source game inventory management project, and I had some onion rings that were actually made correctly (most onion rings are undercooked and result in a poor eating experience). Big Time is always a nice place, and their food is surprisingly good for a bar.

TeaGL

There is a lot of overlap between people who are extremely particular about their computers and people who are extremely particular about their hot beverages. As a result there is a sizeable population of SeaGL participants who love to drink tea. So there is an event that happens during SeaGL called TeaGL. It is run by Robin, a self-described "tea maid" from somewhere in the desert lands. She makes a wonderful tea to-order in the Chinese tea ceremony style, and I love it.

This year featured the return of the TeaGL tea swap. Attendees could sign up beforehand to be partnered with a random TeaGL Buddy with whom they would trade teas during the conference. Me and my gf got assigned each other, and we received the emails while sitting on the couch right next to each other. We laughed about it a bit and then I emailed Linnea, the officiant for the TeaGL tea swap, about the issue. Linnea re-partenred us with new people (herself and Ammar, who I later learned was voluntold to be my Buddy).

Ammar put down that he usually drinks bagged tea and that he likes hibiscus. I went to the local health foods shop and bought loose leaf jasmine and hibiscus leaves, then mixed them together in a glass jar. Since this was loose leaf and he mentioned usually drinking bagged, I didn't know if he had an infuser so I also got him an infuser. I thought that I had done pretty good, Ammar did in fact not have an infuser (his had broken) and it turns out that he loves the tea :)

However, Ammar won TeaGL 2025. He knocked it out of the park, into the next park over, and resulted in 4 home runs in each park. To start, he 3D-printed (in a plant-safe material) a container for holding tea. The symbol on the front is the Chinese symbol for tea.

Inside is a herbal tea mixture of mint, thyme, and basil that he grew himself hydroponically.

I am amazed. Flabbergasted. Shocked. I feel like a spoiled boy on Christmas morning. This stuff is delicious. It's not "technically" tea, but who cares? It rocks, it clears out my sinuses and it is tasty. I feel like when I won the grand prize for the LFNW raffle one year.

The GF Report

My girlfriend accompanies me to all sorts of nerd conferences, she's been to five of them now, and this SeaGL was her favorite as well. I'll let her speak in her own words:

I thought everyone was really nice and I liked James's performance. I really liked the focus on moving people away from tech monopolies. A lot of it felt very newbie-welcoming. More so than any other conference I've been to. It felt like someone who didn't know anything could have these conversations about stuff easier. It felt more accessible than other conferences as someone who isn't an expert. Since I've been to a few in the area now, I've made friends, and it's easier to have these conversations with friends.

Findings

SeaGL is a great conference to attend. This is regardless of whether or not you're an expert or brand new to the scene. It would be a wonderful conference to sponsor. If I had a tech company, I would want to sponsor SeaGL to get my name in front of people at all stages of their career or adventure in open source.

Coverage From Around The Net

Ted Matsumura wrote a post about SeaGL as well, which you can find here (archive).

Trevor wrote up a blogpost about his cool badge that he wore to the PGP party. You can find it here (archive).

Adam's notes on his local offline AI talk can be found here (archive).

PUNKTO | Hosting An STK Server

Wed, 27 Aug 2025 00:00:00 PST

Hosting An STK Server

This blogpost details a bit of my personal history with the game, some recent (at the time of writing) news in the game's development, and how you can go about setting up your own dedicated STK server.

When people ask about free software videogames, SuperTuxKart is one of the most frequently cited examples of a fully funcitonal game that you can play. Before Steam came to Linux and popularized playing proprietary videogames on the platform, STK was one of the most polished games that you could play without a proprietary operating system or the (then anemic) WINE.

Personal History

When I was in elementary school, my uncle (who was in middle school at the time) blew my mind by showing me that he could play Gameboy games on his computer with something called an "emulator". He explained that an emulator is when your computer pretends to be a Gameboy so that it can play Gameboy games. If this was sorcery, what I saw next was black magic. He showed me some basic texture replacement and he made Professor Oak at the beginning of Pokemon Blue say my name instead of his normal "Hello there! Welcome to the world of POKEMON!". He showed me so much cool stuff that he could do on his computer and at the end he said "but if you really want to do hardcore stuff, you need Linux". We then had to eat dinner so he couldn't tell me more.

In the Summer of 2011, my mother insisted that I take a course at Portland's Saturday Academy because she didn't want me sitting around playing videogames and doing nothing for the entirety of Summer break(just 6 out of 7 days each week). I looked through their course catalog and saw one course that said "Linux" in the title. The name stuck out because it's not something that you hear very often when you're just a regular kid, but I remembered it from the conversation that I had had with my uncle all those years ago. Armed with a O'Reilly Linux Pocket Guide from my mom and a hope that I might get to modify Pokemon Blue, I walked into class. We were taught the basics of building a computer, installing Linux, using Linux, and to finish it all off we downloaded and played SuperTuxKart. This content was spread out over the whole summer, and I left the class with the foundational knowledge and skills that would someday become my career and main hobby.

My impression of STK from this first interaction was that it was clunky but functional.

Despite having received this book a whopping 14 years ago and becoming a professional full-time tax-paying bill-having Linux Sysadmin, I still have not read more than the first 3 pages of this book.

Almost exactly 8 years later, in the Summer of 2019, I was on a motorcycle camping trip on the Mongolian steppe with some dudes from the Internet. The camping trip was a week long, and I don't remember the exact details of how , but we were able to charge our phones. Additionally, there was surprisingly good 4G connectivity nearly everywhere we went, even when you could scarcely see a building in the distance. Everyone had Android and one guy also had an iPhone as a "travel phone". One of the last days of the trip, we couldn't find wood to build a fire so we didn't want to sit outside and get eaten up by mosquitos, secluding us to our respective tents. It was also a rare occasion where we had poor Internet connectivity. I wanted to play SuperTuxKart with everyone but our Internet connection was too poor for everyone to be able to download it quickly. F-Droid has a built-in feature to share installed apps over a local network connection, but for whatever reason we couldn't get it to work. All we wanted to do was shared the software and raced karts together. After several failed downloads each, we were all able to (slowly) get the game on our phones and even play a little bit, using the iPhone as a WLAN router. But by the time the last guy had it, some of us had already started going to sleep.

Despite the hiccups, it was a blast. There's something surreal about playing SuperTuxKart on your phone with strangers from the Internet while laying in a tent in the middle of nowhere in a foreign country. I can't fully describe with langauge what it feels like, but it is amazing. This memory stayed with me and changed my perception of both camping and of SuperTuxKart. No more was it a "clunky but functional" game that I didn't really want to play. It had transformed into "a great party game" in my mind, and even "a core part of the camping experience".

Fast forward another couple of years and I was glamping in an RV at Michigan International Speedway to see NASCAR with some friends from the Internet. I knew the gameplan before I packed my bag, before I even bought my ticket or booked my flight. I wanted to play STK with the guys while camping in an RV. I pre-downloaded the current version of STK for every platform and did all the work beforehand to set up my laptop as a WiFi hotspot. I wrote instructions on installation, printed and laminated some steps to get people bootstrapped, and headed off to Michigan.

This time there were no hiccups. Everybody could participate. We didn't need Internet, I had brought everything that we could possibly need. Everyone had a great time. The game was still clunky and floaty, but if you're having fun with your friends, the clunkiness is a source of comedy, not a source of frustration. The game is still frustratingly unpolished if you're playing the Story Mode or against bots, but against other humans with the goal of having a good time it is some of the best fun you can have while camping after the fire has gone out for the night.

I wanted to keep playing after the camping was over. I wanted to play over the Internet when I got home. There are plenty of public STK servers available, but my Internet at home is ADSL and my ping outside of my geographical region is abysmal. I couldn't reasonably do this from my house because I'm double-NAT'd and on ADSL. There's no way that my network can take even 8 people connecting from the outside. Once I had my own server hardware colocated in a nearby datacenter, I had no more excuse, and it was time to run my own dedicated STK server.

How To Run The Server

There's not an "easy" way to run a dedicated STK server.

The STK team has put out some documentation that is helpful in running the server, but it is missing some crucial information. There are some other guides which are all also missing some pieces, and there is a Docker container that hasn't been updated in 3 years. The good news here is that I'm a professional and I'm used to figuring things out with bad or missing docs. This section of the blog is meant to be a complete guide for everything that you need to host your STK server. It is exactly what I have done to host mine.

Prerequisites

I'm doing all of this work on Ubuntu 24.04. If you're doing it on something else, you may have to adjust these instructions slightly.

We're going to compile a custom version of STK so we need to do some initial setup. I recommend compiling instead of using a pre-packaged version for a couple of reasons:

The pipeline that you create to build and deploy the software will be an educational experience
You'll be able to build and deploy git versions of the server faster than new packaged versions are made available
By compiling the server-only version of the binary, there is less functionality and therefore less attack surface

Apt Packages

First, install some apt prereqs that will be needed to do the compilation and get the rest of what we need.

apt install build-essential cmake libbluetooth-dev \
libcurl4-openssl-dev libenet-dev libfreetype-dev libfribidi-dev \
libgl1-mesa-dev libglew-dev libjpeg-dev libogg-dev libopenal-dev \
libpng-dev libssl-dev libvorbis-dev libxrandr-dev libx11-dev \
nettle-dev pkg-config zlib1g-dev

Source Code

Then we need to download the latest stable release tarball from GitHub and the latest assets snapshot from Sourceforge. You will be able to compile STK and even launch it with just the tarball from GitHub, but it will immediately crash (and the compilation will complain) if you don't have the assets. Yes, even for server-only mode. I don't think that this should be necessary but that's just the way that it is.

On my system, this gave me a 1.4.tar.gz and a supertuxkart-code-r18614-stk-assets.zip. If you're doing this in the future, then you'll probably wind up with different version numbers in your filenames. That's okay.

User & Dir

We'll additionally want to make a user to run the service and a directory to house everything.

useradd -d /opt/supertuxkart/ -c "SuperTuxKart Server" -m -s /usr/bin/nologin supertuxkart
mv 1.4.tar.gz supertuxkart-code-r18614-stk-assets.zip /opt/supertuxkart/

Account

As a last prerequisite, you'll need an STK online account. You can make one on this page. Keep track of your username and password.

Compilation

Unzip the zip file and untar the tar file. This will yield two new directories, named supertuxkart-code-r18614-stk-assets and stk-code-1.4, respectively. Move the assets to the appropriate place underneath the stk-code where the compilation is going to look for them, and make a new directory to build the software from.

mkdir stk-code-1.4/stk-assets/ stk-code-1.4/build/
mv supertuxkart-code-r18614-stk-assets/* stk-code-1.4/stk-assets/
cd stk-code-1.4/buid/

Then you just run the compilation:

cmake .. -DCMAKE_INSTALL_PREFIX=/opt/supertuxkart/ -DSERVER_ONLY=ON
make install -j $(nproc --all)

Then make sure that everything is owned by the supertuxkart user that we made earlier:

chown -R supertuxkart:supertuxkart /opt/supertuxkart/

Server Setup

Once you have the software successfully compiled, you need to set up the server to run automatically. The first step in this is to make sure that you are signed into your STK Online account that you made earlier. You can do that from the commandline by running:

sudo -u supertuxkart /opt/supertuxkart/bin/supertuxkart --init-user --login=YOUR_USERNAME --password=YOUR_PASSWORD

This will create the file /opt/supertuxkart/.config/supertuxkart/config-0.10/players.xml which contains your username and auth token. It doesn't include your password which is nice. Admitedly, I have no idea how to invalidate an auth token. I don't think that this functionality is exposed to server operators, probably instead it is exclusive to the STK devs who run the STK online service.

Next you need to modify the config file. A default one is provided here. This is what mine looks like:

Make your own and put it in /opt/supertuxkart/something.xml. If you copy mine, you'll want to change some values:

server-name Unless you want to advertise my blog and confuse people with a billion identically-named servers in the server list, you should set this to whatever you want your server to be named.
server-port You will need to change this if that port is already in use on your server or if you're running multiple different dedicated STK servers on the same box (for example to have different rulesets).
server-max-players Is the maximum number of players that can be on the server. Crank this up if you have a lot of friends.

You'll also want to put something in /opt/supertuxkart/motd.txt. This is the message of the day that will be shown to all players who connect.

Your firewall needs to allow incoming connections to the STK daemon, so go ahead and open that up. I'm using firewalld. Adjust this if you're using ufw or raw nftables.

firewall-cmd --add-port=2759/udp --permanent
firewall-cmd --add-port=2757/udp --permanent
firewall-cmd --add-port=2759/udp
firewall-cmd --add-port=2757/udp

Lastly, you'll want to make a SystemD service to run the game at all times. Put this content in /etc/systemd/system/stk.service:

[Unit]
Description=Supertuxkart Server
After=network.target

[Service]
ExecStart=/opt/supertuxkart/bin/supertuxkart --server-config=/opt/supertuxkart/YOUR_CONFIG_FILE.xml --wan-server
WorkingDirectory=/opt/supertuxkart/
Restart=always
User=supertuxkart

[Install]
WantedBy=multi-user.target

Then reload the daemon and start the service:

systemctl daemon-reload
systemctl start stk.service

At this point your server should be publicly available. You can open up the current STK version on any platform, connect, and race with friends and strangers from around the world.

Experience Running The Server

The "Bad"

The bad isn't really that bad if you're not a big baby.

Occasionally, there will be an issue with the main STK online server. It will go down and as a result your STK daemon will fail to connect and then just sit there not functioning and not being advertised to other users. To resolve this, just do a:

systemctl restart stk.service

The admin console is disabled in the XML above because a SystemD service has no stdin with which to use it. In some places online, the STK devs recommend launching the application in a screen or tmux session. However, if you do this, then the process uses a whole entire CPU to wait for admin input.

The alternative to the admin console, if for example you need to ban a specific IP address or username, is to interact directly with the Sqlite3 database. This is obviously a non-ideal solution.

You would only need the admin console to kick or ban players. However, I have not experienced even one person who I would not want to play STK with in the time that I've been hosting a public server. The most "abuse" that could happen is plaintext in the game chat. Oh no, free speech!

Maybe someone has or could make a client that cheats at SuperTuxKart, but I think that the motivation for doing that is probably pretty low, given the extremely small playerbase. You would get way more ROI with hacked clients making people mad on a popular game like Counter Strike, Fortnight, or Over Watch.

The above config promotes the first-connected person on the server to be able to change gamemodes and difficulties. This is super useful because when strangers use your server without you there, they can play however they want. However, there is no ability to tell the server "whenever the user myusername connects, promote him to the boss position". I suppose that this theoretically opens up the possibility of someone connecting, becoming the admin, and then going AFK forever. Remember to just have fun and not take things too seriously.

The Good

If you look at the public server list, you'll see the punkto.org STK server listed. You may notice that nobody is online. In fact, if you scroll through the server list, you may notice that nobody is racing on many of the public servers.

However, a peculiar thing happens when you start playing on any public server. Other people will intermitently (usually in the evening in the Western hemisphere) look at the public server list and filter by number of active players. They'll see someone playing and join on in. You eventually wind up with a snowball effect where everyone is joining in on the server(s) with the highest playercount because everyone wants to play with the most people.

This has happened to me a few times on my own server. Sometimes I get on and run a few races and nobody else gets on. Sometimes I get on and then the usercount snowballs all the way up to the limit. When there's other people, there is fun to be had.

A river stops being a river when you put it in a bucket. A river's only a river when it flows. I could show you screenshots or describe with extreme detail what it's like to be playing a videogame with strangers on the Internet for free, but all the descriptions in the world don't do it justice. Fun is only fun when it's experienced. If you played CS or TF2 back in the day, it's similar to that. There's dudes with crazy high skill levels. There's dudes who just want to have a laugh and drive backwards for the whole race. There's dudes who talk mad shit on the Internet.

Playing STK online reminds me of the "good old days" when the Internet was the wild west and people didn't take things so seriously. I love it and I'll never stop.

Recent Split

Recently, there has been some silly drama in the STK development team and a portion of it has split off from the rest, forming STK Evolution. If you like, you can read a whole long blogpost about it here.

The STK Evolution people seem dedicated to shipping v1.5 soon, providing a better STK Online server that doesn't drop dedicated servers all the time, and continuing to work on STK in the longterm. They're also in control of the github.

As soon as 1.5 is available, from whoever ships it first, the punkto.org STK server will have it available. I look forward to being among the first dedicated servers on the new STK Online.

Happy racing.

10/20/2025 Update

Today 1.5 was released after nearly 3 years of waiting. The punkto.org dedicated server was updated 2 hours after the release was issued! If you are following the above instructions, the service will fail to start. This is because you now need to manually move the data files from the stk-assets directory to the data directory. Like so:

root@server# cd /opt/supertuxkart/stk-assets/
root@server stk-assets# mv karts library models music sfx textures tracks /opt/supertuxkart/share/supertuxkart/data/

Obviously replace /prefix with wherever you have your STK files stored.

Additionally, it appears that the STK developers settled their differences. Work will continue under the original domain, and the next version of STK (after 1.5) will be called SuperTuxKart-Evolution, replacing the previous projected version 2.0.

When 2.0 releases, it is my hope that punkto.org is among the first dedicated servers to host it.

PUNKTO | Language Controls Thought

Fri, 15 Aug 2025 00:00:00 PST

Language Controls Thought

The words that you use influence the way that you think about things, which in turn influence the language that you use. This creates a feedback loop that you can intentionally break out of if you choose to do so.

If the above sentence rings true for you then you might want to just skip this whole post. It's a lot of paragraphs to say 3 words.

This Is An Abnormal Post

This is a pretty "philosophical" post within which I talk about abstract and fuzzy things like "emotion" and "ownership". I don't normally share lines of thought of this variety with others, and I much less publish them publicly. While I do spend a lot of time thinking about and writing about this type of thing, I frequently don't even refer to my "philosophical" writings after I have put them down, be it in ink or in bits. It is valuable to assess why I usually don't do this and then address why I am doing it this time.

I don't think that the term "philosophical" is quite right for this but I can't think of a better word so I'll just put it in quotes.

I find the writing process to be beneficial to the thinking process. The thinking flows into the writing, which flows back into the thinking. This creates a feedback loop which causes both things (the writing and the thinking) to change. I am of the opinion that this change is usually for the better, although sometimes things can get caught in a negative feedback loop. You can think way faster than you can write, and by forcing yourself to write down what you are thinking, you functionally force yourself to think slower. Thinking slower is usually thinking better. Writing is even slower than typing and so forces you to slown down your thinking even more, which (in my experience) leads to less jumping around in thought. Additionally, when you read a sentence, you assess it from an outside perspective. Writing down your thoughts is an effective way to see your thoughts from an outsider's perspective. For these reasons (and because I just like writing lots of words), I write down my "philosophical" thoughts and opinions.

Maybe the first draft of everything should be done in pen. Plus a pen and paper can't run out of battery on a long flight.

That being said, I don't normally refer to my philosophical writings much after I have written them down. When I do, I usually find myself thinking one of two things:

This is so obvious that it didn't need to be written down.
This is stupid.

And for this reason I don't normally share my writings with friends or family except when they are requested, and I very rarely publish them on the open Internet. The last time that I did was in the post on this blog about imposter syndrome, and I still sometimes think that I shouldn't have published that. This post was specifically requested by a close friend of mine. During a car drive, we had a long conversation about the topic of emotional ownership and speech influencing thought. He asked me if I had some blog that he doesn't read (you're looking at it), and requested that I please elucidate my thoughts and post it so that he can read it. I typically would just send it directly to the requestor, but this is a lot of words to get lost in a chatlog and he did specically request that I send it to him in blog format.

Speech Influences Thought

The way that you say things betrays the way that you think about things. On a facial level, this is the whole point of language as an invention. It goes deeper, however. The vocabulary that you use reveals a great deal of information about your personal history and your presuppositions. The easiest example for this is regional accents. If someone uses the term "wicked" to mean cool, you might be able to assume that they are from America's Northeast. If they use the word "jojos" to refer to potato wedges, they may well be from the Pacific Northwest (or apparently the upper Midwest). People who say "sodapop" instead of "soda" or "pop" are probably from or influenced by Portland, Oregon. These of course are not always true, but they illustrate an example effectively. If you say "you made me sad" when you feel sadness, you are also parading the additional information that you hold the other person responsible for the emotion in your mind. The point here is that word choice is "leaky" in a secrecy-sense.

Not only does this disclose ideas to other people, it also reinforces the ideas in your own mind. Your conception of reality is just one of many possible interpretations. Reality is a chaotic mess that you and I impose order onto. You take in information from your senses and build up an idea of how the world is. Hearing other people express the same idea reinforces your idea and makes you feel more comfortable with its "trueness". This is the basis for how filter bubbles exist and doesn't need to be rehashed here. When you speak your idea out loud, you also hear yourself say it much in the same way that you read the sentences that you write. This hearing also reinforces your idea.

If you find that some thought is damaging do you, changing the way that you think about it may be able to stop its damage. You may see this and think "well I just need to stop the damaging thought from happening", but in many cases this is unrealistic. You will find yourself damaged by other humans and by the world as a whole. If you rely on others to stop your thoughts, then you will find yourself disappointed. This is because other people can be unreliable and difficult to change. To do so will require a lot of work. It will also require a lot of work to change yourself. However, the work done by the outside world would be a charity from the rest of the world to you. This charity is not owed and shouldn't be expected. The work done inside yourself is an investment that pays immediate dividends.

Saying "you made me sad" not only discloses to the other party that you feel that they are responsible for your sadness, it also reinforces this sentiment in yourself. You feel the emotion of sadness and assign it as an effect that has been caused by another person. By making them responsible for making you sad, you also make them responsible for making you happy. This functionally takes the ability to make yourself happy out of your own hands, forcing you into a state of dependence or reliance. If you depend on someone else to make you happy, then you are giving control of yourself over to that other person.

An effective way to change how you think about something is to change the way that you talk about it. By using different words to frame the reality that you are experiencing, you change the way that you experience it.

Some Lucid Examples

The above is a bit whishy-washy and abstract. Some actionable examples are in order.

Debby vs Negative Self-Talk

A friend once mentioned that he would sometimes feel depressed about his situation in life. He saw a manner in which he could immediately improve things, yet did not take action. In response to this, he described what he called a "stupid HR bitch" (hereafter Debby) in his mind that would talk down to him about this. She would call him stupid and lazy, say that he would never ammount to or achieve anything, and generally degrade him. He would occasionally reflect on this turn of events and call Debby by her name, attributing the thoughts to her.

This manner of framing places blame on Debby for her bullying behavior. It places responsibility on her to change. The problem with this is immediately apparent: Debby doesn't exist. She is an imaginary mental construct living entirely within the realm of thought. Instead of an imaginary friend, my buddy had created an imaginary enemy.

This manner of framing the situation was hindering any of his efforts to remedy it. He needed to change the way that he thought about the situation, but he was reinforcing the way that he thought about the situation every time that he thought about or spoke about the situation. This pattern of thought was keeping him at the bottom of the hole and preventing him from building the ladder to get out. Changing the vocabulary used to frame the situation can help in building the ladder.

I suggest changing the framing from "Debby" to "Negative Self-Talk". This has the advantage of placing the onus to change on the thinker instead of the thought. Debby is a seemingly external, unaccountable, and uncontrolable force who is beholden to no one. Negative self-talk is something that you do personally and I think that the naming accomplishes several helpful things.

First and foremost, it identifies the pattern as negative. It is possible to think "maybe Debby is right" or "maybe Debby isn't so bad", but something named "Negative..." will always be negative and undesireable. Using choice words with pre-determined "goodness" in this manner is often done to shut down opposing viewpoints in politics and debate, so its effectiveness is well-proven. Take (for example) the "pro-choice" vs "pro-life" argument in abortion. One side frames their opponents as hating choice and freedom, therefore loving slavery. The other side frames their opponents as hating life, therefore loving death (or killing). This tactic gets the job done in framing the way that people think about the issue.

Secondly, it identifies the involved parties more accurately than "Debby". It places the origin of the activity squarely on the responsible individual, the self. Debby is an "external" party who is unaccountable. But in reality the negative thoughts origintate in your mind and live entirely in your mind. The only active party is yourself. Identifying the active parties, who are capable of change (and who it is realistic to expect change from), is a helpful step in effecting the change.

Lastly, it identifies what the actual behavior is. It's just talk. In real life (outside the confines of the mind), Debby may have actual power. She may be able to demote you or fire you. She could make your life more difficult. She has more than just talk. But by calling the talk what it is (just talk), some influence is removed from it. Sticks and stones may break my bones (and Debby may fire me), but words will never hurt me.

In this manner, changing the way that you frame a pattern of thought can help you to change it.

There Is A Child Kicking My Seat

The usefulness of this technique doesn't stop at purely ephemeral and mental concepts. It can help you during physical inconvenience and discomfort.

As I type this, I am sitting in an airplane on a flight across the country. I don't particularly want to be on this flight. I am returning home to:

an overprovisioned workload in my job
an overprovisioned workload at home
people who are upset with me personally
a wildfire that is filling the air with smoke and may be burning down my house at this very moment
a prevailing culture that despises "my type" of people

And I am leaving good friends who I rarely see, low prices on most things, and agreeable weather.

I am in a middle seat, the plane is hitting turbulence and the view out the window is pure white clouds to the horizon. My flight is completely full, with not a single open seat. There is a family with several infants spread across 3 rows, surrounding me. I don't really want to be here.

The infants are screaming at the top of their lungs (really hitting that screech point where you wonder how they can get that high) for extended periods of time and are kicking my seat. There isn't anywhere that I could move, since all seats are occupied and nobody in their right mind would trade with me. It is extremely easy to fall into the framing of "these kids are ruining my time" or even just "these kids are annoying me". Letting these negative emotions ferment and fester would only serve to negatively impact my mental state, since there is no action that I can take to remedy the situation. Instead, I can change the way that I think about it.

I don't really want to be here, I would rather be where I started from. The kids don't really want to be here either. That's why they're kicking and screaming. Children also tend to have exceedingly accurate insights into things. They are able to see a more raw version of the world, unclouded by past expereince, and often find truths that fly right over the head of most adults. Having a child agree (unprompted) with your assessment of some situation is quite satisfying. These kids agree with me. They aren't annoying me, they are validating me.

Emotional Ownership

Things that you can exert immediate control over are (in a functional / egoism sense) your property. They are not the property of someone else because someone else cannot control them. The watch on my wrist is mine and mine alone. If the current function on the display needs to change, I am the one who changes it. Nobody else can change my watch because I wouldn't let them, they would be unable to. This is the case for all physical property. It is also the case for emotions.

By saying that someone or something else "made you" feel a certain way, you place ownership and responsibility for that feeling on an external actor. If this feeling is negative, it implies a responsibility to the effecting party to remedy the negativity. If it is positive, it implies a desire for the effecting party to continue in the future.

Emotions start in your body (mostly in your mind I think) and they exist entirely within your body and then they eventually die without having left your body. They are thoughts. They are the most "your property" that something can possibly be. As such, nobody else is capable of thrusting an emotion into you. Your emotions arise as an internal reaction to external stimulus. Consider the example of:

You made me feel sad when you said that you hate me

versus:

I felt sad when you said that you hated me

The former places the responsibility for an internal phenomenon (sadness) on an external actor (you). The latter places this internal phenomonon (sadness) on the internal actor (I) and still places the actual action (said that you hated me) on the external actor (you). This manner of framing is not only technically correct, it puts you in control of your own emotions because they are something that you do, not something that is done to you.

By framing the way that I think about emotions in this way, I have been able to overcome a huge number of counter-productive and time-consuming thoughts. Instead of festering on some person who did me wrong and how upset they made me, I can instead recognize that the way I feel about the situation is entirely dependent on how I frame it. I can take ownership over my own emotional state and spend less time being miserable. Being miserable sucks and I generally prefer to avoid it if at all possible. Negative emotions such as hatred and disdain will turn your heart inside out and make you evil, so being able to take control over them is unimaginably useful.

PUNKTO | Sakura Con 2025

Fri, 09 May 2025 00:00:00 PST

Sakura Con 2025

These images were originally published on a now-defunct Telegram channel and this blogpost has been backdated to be at the correct time in history. I've taken this opportunity to better select the images to share for telling the story and conveying the experience, and I've written much more explanation than I originally did when it was a late-night post to a Telegram channel after a long day.

In May of 2025, I went to Sakura Con in Seattle with my girlfriend. This was my second year attending and her first. To attend the conference, you have to have a badge and (if you're an adult) a wristband. You can purchase your badge up to a year in advance, but the earliest that you can get it is the night before the conference kicks off, Thursday night. So, after my workday ended on Thursday, we moseyed on over to the Seattle Convention Center and stood in line.

Line World

The line was a trainwreck. By my watch, we stood in line for 3 hours and 30 minutes (and 7 seconds). There was some funky line-juggling going on and some people who got in line 2 hours after us were getting their badges 30 minutes before us. I'm happy for them: I don't want anyone to stand in line for that long, but seeing this happen did breed some resentment in the people around me who had waited for longer. By the time we got our wristbands and badges at the front of the line, the line had grown even longer than it was when we entered it. I did some rough math using paces as a unit of measurement, and by my estimate the line had grown by another hour.

My understanding from talking with other attendees for the duration of the conference is that these badge-line issues did not let up, and similar experiences were had by all until the final day of the conference, when there were relatively few people still needing to get their badges. The badge is printed off as-needed when you get to the front of the line, so it's not like they had to dig around in boxes to find stuff. I think that the line issue could be alleviated in the future with a few changes:

More lanes: There was one long line that fed into multiple lanes, and each lane had one staff member who was handing out badges. While there were well over a dozen lanes at the end, only a small handful were in use. More staff (or volunteers) is needed to make this happen.
Eliminate ID checking: IDs are checked against membership to ensure that you're not stealing someone else's badge and to check if you are a child or adult.
- Membership verification could be simplified by emailing attendees a unique QR code that gets scanned. This could even be integrated into the badge-printing computer to speed up the process of looking people up.
- Age verification could be faster too. By only checking people who look like they're below 35, a huge portion of attendees could avoid this step. This works for liqour stores. It should work for an anime convention.
Stop yapping: By my observations, staff (or volunteers) spent a lot of time talking to the people who they were badging in. Sometimes as long as 10 minutes per person (although the average that I timed was 2 minutes per person). This needless chitchat holds up the line. Get the people their badge and get them on their way. If there is any issue, send them to a separate Ticket Resolution desk. That's what concerts do and it works great.
Really stop yapping: The desk attendee recites some basic rules to every person that they hand a badge to. This takes time. If you need to give a list of information to a huge group of people, don't tell them individually. Instead, do like how airport security does: put up huge signs in the most common languages.

I do wonder what kind of self-service options could be available, like how you can print out your boarding pass at the airport without talking to a clerk.

What To Do @ The Con

The next day, Sakura Con properly started. At this conference, there are a few different groups of things to do:

Dance / Party
Play videogames
Buy stuff
Attend panels
Look at cosplay

Within the last few years, I've surpassed the point where partying is a lot of fun and I have begun to metamorphasize permaturely into someone with an early bedtime who doesn't like loud places. Perhaps by the end of my 30's I will emerge from this cocoon as a crotchety old man. As a result, I didn't attend the crazy rave that goes until the wee hours of the morning.

We did check out the other main attractions, however.

Some images of the overall vibe inside the conference.

Come Spend Your Money

The vendor hall and artist alley are the two main places at Sakura Con where you can find yourself parted from your precious dollars and cents in exchange for a wide variety of cute and cool things. Despite these rooms being absolutely massive, they were packed with people. In many cases the aisles were impassable and you had to either go around or force your way through. Additionally, the artist alley (where the smaller businesses sell their wares) had a delayed opening because The Cheesecake Factory next door caught on fire and filled the whole area with smoke.

Art, both man-made and AI generated, was available for purchase in as many forms as you can image. There were post cards, stickers, posters, tapestries, patches, enamel pins, and so on to infinity. All sporting designs both original and copied from any popular culture franchise. There was obviously a strong Japan-leaning tendency among what you could find, though it wasn't exclusive. Some francises were decidedly American, and some stuff only had a passing "oh I guess cuteness as a concept is Japanese".

If 2 dimensional art isn't your jam, you can also find the largest assortment that I have personally evrer seen (admittedly I don't go looking) of action figures. From popular anime to video games to gundam to...honestly I don't know what some of this is. If it goes into a plexiglass case or a mason jar on an image board, you can buy it at Sakura Con.

You'll find plenty of stuffed animals and other soft things if you are the most cozy-core person that you know. All of it can be had for much cheaper online on sites like AliExpress or Temu. But if you find the Squishmallow section of your supermarket lacking and you are allergic to waiting 2 weeks for shipping, Sakura Con has you covered.

If all the walking around and looking at stuff works up your appetite, you can find some snacks or a nice refreshment, some of which is clearly Japan-inspired or Japan-sourced (pictured below). There's a food court that serves standard American fare and I recall one vendor booth selling locally-made meat sticks. These weren't even pretending to be on-theme, but they were pretty good as far as meat sticks go.

You could also find various (toy, unsharpened) video game weapons, one booth that actually sold real sharpened blades, video game accessories, and DnD supplies. I don't think that DnD has anything to do with Japan, but the overlap of the populations is pretty sizeable.

And if you get tired of all of the lifeless junk completely boxing you in from every angle, slowly eating away at your wallet and your ability to enjoy things that aren't part of a Brand(TM), you can find an island of life and youth by purchasing a real live kitten.

Looking back at these pictures, I am astonished by the fact that everything seems to contain plastic.

Polyester and spandex in the clothing.
Vinyl for the stickers.
If you buy any printed or drawn artwork on paper, it gets put into a plastic sleeve.
The figurines are not just assembled with injection-molded plastic pieces. They're held together with plastic glue and painted with acrylic.
The plushes are made of synthetic plastic material and stuffed with poly-fil (made of plastic). Cotton or wool could be used instead but it isn't. It's just plastic.
All the foodstuff comes packaged in plastic, sometimes in multiple layers of plastic in fact.
Swords with plastic handles.
Dice made of plastic.
The giant metal shield might be plastic-free, but it probably has some plastic hardware on the backside or the paint is likely again acrylic.
Even the cat is full of microplastics.

Conference Talks

A conference isn't much of a conference without some talks to go to, and Sakura Con (even though the Con is short for Convention) is no exception. However, in the spirit of Comic Con, Sakura Con calls them "panels". I go to technical conferences to network, talk shop, and see (and give) talks, so I was excited to see talks at Sakura Con. My experience the year prior taught me to be careful about what to attend here, as the bar for entry isn't particularly high. This year was much better and I mostly just went to talks that I actually enjoyed. You can find a full schedule of the available talks (and more) here.

Early on Saturday morning we found our way to the Japanese Harp talk. I had neither seen nor heard Japanese Harp before, and it is an intriguing looking and sounding instrument. Being culturally isolated form the West, Japan long ago developed its own unique sound. In some ways, this sound can only be properly represented with these unique instruments. The speaker's presentation style was a bit dulll, but what really shocked me was that they encouraged members of the audience to come up and play on these extremely expensive and difficult-to-obtain instruments. Everyone who did showed the appropriate amount of care and respect. I'm not sure why that surprised me, but it did. When all was said and done, the harps were wrapped up in giant cloths (I had expected hard carrying cases) as if foreshadowing what we would experience later.

When I took the picture of the cloth-wrapped harp, we didn't know what the wrapping technique was called. But we soon learned all about the Japanese art of furoshiki. A woman from the Japanese Consulate gave a panel on gift-wrapping and bag-making using just a single piece of cloth. That's a bit of a reductive explanation, she showed us hands-on how to use single pieces of square fabric for a wide assortment of tasks that I never would have imagined using them for. If this sounds interesting to you, I cannot encourage enough checking out her YouTube video which is nearly beat-for-beat identical to the hands-on instruction at the panel.

Since learning about furoshiki, we have used them to augment or supplant backpacks, shopping bags, packing cubes, and more around the house. I am overjoyed that I went to this panel, it had a measurable positive impact on my life. Immediately after Sakura Con was over, we went to a Daiso and found the same furoshikis that we used in the class. If you're looking, they're labeled as "bandana" instead. I would love to someday own real Japanese furoshiki cloths, but they're a bit expensive from overseas. I prefer carrying these Daiso bandanas instead of canvas bags to the grocery store because they fold up smaller and they are a lot of fun.

In a similar hands-on vein to the furoshiki panel, we found ourselves at an origami panel. Here we learned a bit about the history of origami, the etymology of the term, and got some hands-on practice folding paper. In the panel, they taught the whole class how to fold this dragon and alien, walking through step by step slowly. For the stragglers (like myself) who couldn't figure some parts out even after seeing it directly in front of me, the panel team even had people walking around the room to give hands-on help to the students. The whole event was really well-done.

We had such a good time folding origami at the origami panel that we intentionally spent all of the rest of our unscheduled time for the remainder of the conference at the origami zone. There was a whole section of a room dedicated to origami whenever there were people there who wanted to fold. We made friends with the people from PAPER (Puget Area Paperfolding Enthusiasts Roundtable) and they showed us techniques for folding various other things, including:

Boxes with lids and organizer inserts
Platonic solids
A cute flower
A golden snitch

The last of which (apparently) had no instructions recorded anywhere online, it has just been passed around from one folder to another and somehow made its way to us. It was a lot of fun. The helpers pointed us towards some origami books and I am happy to now count origami among my hobbies that I greatly enjoy.

Part of the origami exhibit was showing off some of the things that PAPER members have folded. These include quite elaborate original designs and beautiful executions of known patterns.

High on the fumes of hobbies and tools that you can pick up from the Japanese dollar store (Daiso), we found our way to a panel specifically dedicated to Daiso crafts. Here we were regaled with the infinitude of ways that you can spend an endless amount of time making stuff from supplies that you can get for relatively cheap. As a form of recreation and entertainment, this makes some sense, but I wasn't nearly as enchanted here as I was by the origami and furoshiki panels.

There is an item on the Sakura Con schedule that I saw last year and didn't get to attend. It repeats every single day in the same room. I am talking about the Tea Ceremony panel. I fancy myself a bit of a tea snob, and appreciating fine teas is a bit of an "us thing" for me and my gf, so I was excited to be able to attend this year. I only got a few pictures before the woman acting as the narrator asked the audience to please not take pictures or video of the performance itself. While I was hoping for tea, instead we were shown the etiquette and mannerisms common in Japanese tea ceremonies, along with the meaning behind them.

This was educational, but I left feeling slightly disappointed that there was no tea. If you see this panel on the schedule and just want some tea, you can safely skip it.

The final memorable talk that we attended was one about ROM hacking to translate a Japanese videogame. I'm used to seeing technical talks at technical conferences, but this was not something that I had at all expected to see at Sakura Con. Fundamentally, this talk was about passion driving a community to reverse engineer an entire Nintendo DS videogame in order to make it accessible to English-speakers.

The technical breadth of this talk was stupendous, an absolute tour de force on the speaker's part. It was interactive with the audience as well, with the crowd collectively choosing which topics to cover. The different aspects of the game were reverse-engineered independently of each other, and some of the most memorable aspects were:

How the hexacdemical code for a chess puzzle minigame was laid out like a chess board if you rendered it 8x8
Just how gosh darn complicated the various layers of the maps in the game were
The plethora of leftover assets from development and marketing that were shipped with the final ROM but not used in the game itself. The speaker introduced us to the website The Cutting Room Floor, which serves as a catalog of this type of thing.

If this topic interests you, I implore you to check out the talk of the same name but with slightly different content that the speaker gave at a previous conference. Or you can find his website here.

The speaker politely asked attendees not to post pictures of his face on the Internet, so I have obscured his likeness using a Green Pepper, which I think he would appreciate since he was also using a Framework 13 running Linux (I am typing this on my Framework 13 running Linux).

Every Girl's Crazy 'Bout A Sharp Dressed Man

Next (and last) on the list of attractions that started this blogpost is cosplay. Sakura Con (alongside other similar conventions) is known far and wide for being the place where people can dress up as their favorite characters or just generally wear whatever cool unorthodox clothing that they normally don't. This is a big draw for these events for me. I love to see the spins and the original twists, and of course I love to see characters that I recognize from the sreen walking around in real life.

Every cosplay-friendly event is bound to have your regular cast of well-known faces. Some of these (Gengar and Deku Link) are enormous, and I looked like a child standing next to them. I'm short which doesn't help but they are still enormous.

Among the well-known faces, there is one who reigns supreme at an anime convention. Hatsune Miku, the anime girl that even your grandparents can recognize, the #1 mover of anime merchandise in the world, the blue-haired vocaloid, was everywhere that you looked. Being the most popular, there were plenty of bad Miku cosplays. There were plenty of unimaginative Miku cosplays. What I really loved were the unique Mikus that I saw. Here I have photos of my two favorite Mikus from the event. First is a Chainsaw Man / Miku hybrid with leeks for weapons. And second that we have is a rare Bard Miku. I spoke with her a bit and she said that lots of people don't recognize her as a Miku at first, but I spotted it instantly. She asked me to hold my hands out in a cup (as if to hold water), then told me that there was a leak in it while giving me a hand-drawn leek sticker. These Mikus took the cake.

You'll also find group cosplays, where a whole group of friends will dress up together to match a theme. I think that this helps to identify groups of people and makes it harder to meet new friends, but it sure does look cool.

I'm not sure if these two were dressed up as established characters or if these were original costumes, but I thought that both looked so damn cool. I found them playing against each other in a Tetris tournament in the gaming zone and asked to take their pictures. As an aside, the girl absolutely obliterated the guy in Tetris. He didn't even stand a chance.

Not all cosplay is an established brand, however. Not everyone wants to dress up as a videogame or an anime. Some people just want to dress up really cool and don't often get an excuse to do it. I asked some of these people why they chose their theme costume and their answers were that they just really like strawberries/moths/mushrooms and think that they are cool. These costumes were completely hand-made and well done.

The fella with the sign, as far as I'm aware, just rides the escalators and walks around the conference to mess with people. I saw him last year as well.

Seeing cosplay is the largest draw to Sakura Con for me, but you don't even need to be a paying attendee (and wait in the super long badge line) to see cosplay. Just hanging out on the sidewalk outside the convention center is enough to see some of the coolest costumes that you've ever seen. I don't know what some of these are, but they look great.

The Rest

That's about it for the Con from me. I had a decent time, though I probably won't be going back again next year. It's a bit pricey and it takes a lot of time. There's a lot of standing around and waiting and walking. The goods for sale are too aggressively priced, and I feel like I've seen the talks that I care to see from this style of convention for the rest of my life. What remains in this blogpost are some additional pictures from the weekend.

Food

If you are in search of food in Seattle, you will inevitably find yourself a vendor on the street hawking hot dogs for what seem to outsiders like unimaginable prices. These prices are Washington standard, only slightly inflated due to being so close to major events. And these aren't any typical hot dogs, these are Seattle Dogs. A toasted bun slathered in cream cheese with a hot dog and the whole thing topped in grilled onions. This is a Seattle staple, and you can find them most anywhere you go. At Sakura Con I got a Seattle Dog from DeeZ Dogz, and I am not lying or overexaggerating when I say that it was one of the best Seattle Dogs I've had. I went back for a second one the following day.

For dinner one night, I took my girlfriend to a nice little establishment downtown named The Palace Kitchen. In a city widely known for not really having much good food, this restaurant is considered one of the "gems" of the city, and they didn't disappoint. We asked to split an entree since we were both working with shrunken stomachs at the time, and they very nicely plated both halves separately. I'm not sure why that stands out so much to me, but it does.

The last place worth mentioning where we got food was Tai Tung (usually called "Chop Suey"), Seattle's oldest continuously operating Chinese restaurant. It is extremely hyped on review sites and in the press. It is a juggernaut in the local culture. I found it wholy unremarkable. This was disappointing, but oh well.

Photography

I am by no means a "good photographer" or an "artist", but I like to run around with my little point and shoot. Sometimes I even point it and things and shoot. Sometimes I even like the pictures that I take. The patio at the convention center gave me a good vantage point to get some architecture shots of nearby buildings. Some of these I took on the route between the venue and our lodging.

That image of the old Bell building is among my computer wallpapers now.

PUNKTO | The DOS Rule

Tue, 04 Apr 2025 00:00:00 PST

The DOS Rule

This blogpost is why I get invited to all the parties.

Background

DOS is a card game released by Mattel in 2018 as a "sequel" to the popular card game UNO. Similarities between the two games are obviously apparent. In 2022, Mattel released a 2nd Edition of the game DOS with some updated rules and a slightly different set of cards. I own the 1st Edition of the game and play it somewhat regularly with my girlfriend. The 1-on-1 matches are a good deal of fun and move at a good pace for a slow weeknight. We also will occasionally bring the card game to other functions to play with larger groups of people (up to 4). The official rules say that the game caps out at 4 players, and I think that this is largely a result of the fact that there are only 112 cards and the hands can get quite large. I think that the game would work well with up to 8 players if you just combined two decks, but then shuffling would become even more difficult than it already is.

The game's rules are pretty straightforward and don't leave much room for interpretation. For the most part they are quite explicit on the order of play and the various different rules. However, this is not true for The DOS Rule.

The DOS Rule and Confusion

In my 1st Edition copy of the game, with instructions copyrighted in 2022 (slightly different than the above-linked 2017 copyright version), there is a section with the heading DOS RULE, and it reads as follows:

If you ever have exactly two cards in your hand, you must shout out "DOS!" (meaning "two"). If you forget to do so and another player calls out "DOS!" before you, you must immediately draw two cards as a penalty. If this happens during your turn, do not add the penalty cards to your hand until the end of your turn.

This rule leads to a lot of confusion for me.

What does it mean for a card to be "in your hand"
When precisely do you have to call out DOS?
At what point can someone else call it out for you?

What Does "In Your Hand" Mean?

The rule says that you must call it out "If you ever have exactly two cards in your hand". In other words, once you reach a state of having two cards in your hand, you obtain an obligation to shout out "DOS". But when do you reach that state? Take this example:

You have three cards in your right hand
Using your left hand, you remove one of the cards from your right hand in order to play it

At this moment, you have two cards "in your hand", while having three cards "in your hands". You could concievably return the third card to your right hand and select a different one instead. For this reason, I think that it is reasonable to say that you have three in "your hand" still. The "hand" here isn't a literal "hand", but is instead a logical grouping of cards. The game could be played by someone with no hands and this rule should still be enforceable on them for the purposes of disability accessibility. Mattel makes the cards accessible to colorblind individuals by printing shapes that correspond with the colors, so I don't it is reasonable that they would require all players to have hands attached to their body. You may gripe that this rule only serves to punish players and that the handless people already have it hard enough, but I used to have an aunt who had no hands and she was able to live her life just fine. When you spend your whole life without hands, you get a lot better at doing things with your feet. If memory serves, she could even play the piano. It is more reasonable for "your hand" to refer to the group of cards under your immediate control.

This begs the question: When does a card leave your immediate control?

Perhaps you are playing with the "touch-move rule", named after the chess rule of the same name. This rule says that once you touch a piece, you must move it, and that once you release a piece, you cannot change where you moved it to. A parallel rule could be (and sometimes is, as a house rule) drawn in the game of DOS. Here, once you begin moving a card toward the piles, you must play that card. Once you begin to move the card toward the piles, it is effectively no longer under your immediate control, no longer in your hand, and could knock you down to two cards, triggering the DOS rule. You may think that you are "saved" because you could play the card on any of the available piles, but it doesn't actually matter which pile you play it on because it could still knock you down to two cards. However, the touch-move rule is not an official rule in DOS and so I will ignore this potential.

In a game of DOS where you can [hum-ho|flip flop|waffle|be indecisive|move back and forth] and still retain a card in your immediate control (aka a standard game of DOS), I believe that reasonable judgement and common sense dictate that the card is no longer in your hand once it is both:

Touching a valid pile
Not touching your body

These two requirements work together to cover all of the edge cases that I can think of. If you drop a card on accident, whether you were in the process of playing it or not, it doesn't leave your "hand". This is unless, of course, you actually drop it on a valid pile, which is indistinguishable from "splashing the pot".

When Do You Call DOS?

Now that we have established more detailed rules for when you have two cards in your hand, we can turn our attention to the next important question: when do you have to say DOS? The rules say "If you ever have exactly two cards in your hand, which seems pretty immediate to me. You gain the obligation to call out the word "DOS" at the exact same instant that you have exactly two cards in your hand.

Ideally, in card games and in life, you would take care of your obligations as soon as they are assigned to you. This ideal is rarely met in life for most people (including myself, look at how often this blog gets updated), but meeting it in card games is much more achievable. It seems reasonable to me that most people would call "DOS" as soon as they realize that they have only two cards. This assumption is reflected in real-world observations.

When Can Someone DOS You?

The moment that you enter a state of having 2 cards in your hand, you are not racing the entire rest of the table to shout out DOS. This would be result in irreconcileable arguments about who started to say the word first, who finished saying the word first, and who heard whom at what time. You would also have an issue where one person (me) would spend the whole entire game saying the word "DOS" over and over again nonstop. There is no penalty in the rules for calling this out incorrectly, so there is no cost with calling it out. Technically, if you were consistent and fast enough, you could beat everyone else to the punch on their own DOS, effectively making it impossible for you to lose the game. Some people might consider this tactic "cheating", but there's nothing actually against it in the rules, so it would be a valid play (and ruin the game) if the competition to say "DOS" began at the moment that you obtained the obligation. These problems would be completely impassable with multiple people who are prone to arguing.

Thankfully, the race for all other players to call out "DOS" is gated by the active player forgetting to do so. You only have to worry about someone else DOS'ing you "If you forget to do so". There are two starting pistols, not just one, and the starting player gets a head start:

The active player's starting pistol fires at the moment that they have exactly two cards.
All other players' starting pistol fires once the active player has forgotten to call out "DOS".

However, this raises a new and more complicated question that will take several more paragraphs to fully flesh out. How can you tell when someone has forgotten to do something?

It isn't reasonable to interpret that someone has forgotten to call out DOS if they have only just entered the state of needing to. It also isn't reasonable to interpret that they have forgotten if it is in their current "stack" of upcoming actions and they just haven't reached it yet. Often in DOS, you reach the state of having exactly two cards at some point during your turn, where you already committed to taking other actions, such as:

playing additional cards
moving piles of cards into the discard pile
reshuffling the deck
rearranging your hand
replenishing discarded piles
gloating

Take for example a list of chores. If you assign someone the tasks of doing the laundry, doing the dishes, and sweeping, it would be completely unreasonable to say that they have "forgotten" to sweep if they are still doing the dishes. It would be equally unreasonable for them to be accused of forgetfulness if the dishes are undone and they are sweeping. Choosing the order of non-depending tasks is one of the fundamental rights of man, a truth which is self-evident. They have not even necessarily forgotten to do one of the obligatory tasks if they are sitting on the couch with tasks undone - people need breaks and can't work nonstop for their whole lives. This is where the metaphor begins to fall apart.

If someone is engaged in game-based action besides saying "DOS", they may not have forgotten. They may just not have gotten around to it. However, if they enter a state of rest after finishing some action(s), it would be reasonable to expect that they have forgotten to say the magic word. Why does this double standard exist between chores and cardgames? Because cardgames are competetive, and in turn-based games you are expected to get a move on and complete all your actions so that the next player can take their turn. If you are resting during your turn, you are holding up the game for everyone else at the table (obviously not necessarily a literal table, see above discussion around "hand"). It is naturally expected that players will remain active for the duration of their turn and only rest when they believe that they have completed all of their actions. Maybe this would be different in a card game with long-lived turns. I'm not sure that such a card game exists, but if a card game had individual turns that lasted hours or days, people could reasonably be expected to rest during their turn - this is not the case with DOS, however. DOS turns almost universally take less than a minute.

I think that it is obvious to expect that someone has forgotten to say the word if they end their turn without saying it, starting the competition between all other players to call them out on this fact. However, this is a difficult position to back up when considering rules from the other games in this series.

DOS 2nd Edition

The 2nd Edition rules are much clearer and obviate the need for some of the above speculation. These newer rules read:

The moment you have 2 or fewer cards in your hand, you must yell “DOS!” to alert the other players you are about to win. However, if someone catches you and calls out “DOS!” before you (and before the next player begins their turn), then you must draw 2 cards!

I don't own the 2nd Edition version of the game. I own the 1st Edition, which is a fundamentally different game. The 2nd Edition has a distinct (and incompatible) ruleset, a different deck of cards, distinct packaging, and the Mattel website (archive) lists "DOS Second Edition" as "like UNO and DOS card games".

UNO

The game DOS is billed as a "sequel" to UNO. Their (similar) UNO Rule reads as follows:

When you play our next-to-last card, you must yell "UNO" (meaning "one") to indicate that you have only one card left. If you don't yell "UNO" and you are caught before the next player begins their turn, you must draw two cards.

"Forget" And The Turn Cutoff

Neither the previous nor subsequent games in the trilogy mention "forgetting". It appears to me that in both of those games, the two starting pistols fire at the same exact time. Interestingly, they both explicitly cancel out your obligation to say DOS or UNO at the beginning of the next player's turn. If nobody noticed that you only had one/two card(s) before the next player begins, you got away with the crime scott-free. If we take the above rationale around when an obligation is forgotten, this leaves only a small portion of time after you end your turn but before the next player begins their turn when you could be called out. This greatly increases the challenge of calling someone else's obligation, and may even make it impossible.

When does the next player's turn begin? Does it begin the instant that your turn is over? Does it begin as soon as they start to physically take their first action? Agruably, the first action is to assess the cards in your hand, something that doesn't have an easy physical "tell". Is there any in-between time where it is nobody's turn? It is preposterous to assume that a turn-based card game could have a portion of time where it is nobody's turn. Turns are a fundamental part of the game and to have a section of time "outside" of turns would be to have a section of time "outside" of the game itself.

I think that this is very telling to the assumption that you could only call "DOS" after someone has finished their turn. The same rationale used to acheive that conclusion in DOS 1st Edition could be used in UNO and in DOS 2nd Edition. If the logic behind the firing of the second pistol is in fact sound, it would completely cancel the efficacy of the rule itself and render the whole thing useless. You would never get to actually call "UNO" or "DOS" since the next player's turn begins the moment that your turn ends.

In other words, the second starting pistol must fire at some point during the active player's turn.

High Priority Obligations

The only explanation that I can think of that reconciles this is that the obligation to call out "DOS" (or "UNO") is a high priority obligation, one that trumps all other obligations and plans, rocketing itself up to the top of the list. This is further supported by the fact that this rule has its own header section. It isn't included at the end of some other paragraph or as a bullet point, the game developers thought it so important of an obligation that it should be separately outlined with a bold heading.

With this interpretation, the active player's starting pistol fires as soon as they have exactly two cards in their hand, and everyone else's starting pistol fires as soon as the active player does anything other than say "DOS". This interpretation is the only way that I can think of to mesh together the fact that the active player must forget and the historical understanding that the second starting pistol must fire during the active player's turn. The only way that the historical, future, and present rules make sense is if the second pistol fires the moment that the active player begins some other action.

"Next Player"

The "next player" cutoff isn't present in the 1st Edition of DOS. The game technically provides no cutoff, meaning that you could be called out on forgetting to say "DOS" several turns ago. I feel that the spirit of the history of the rule, however, still forces us to take the High Priority Obligation explanation. Additionally, I don't think that the "next player" cutoff is well-enough defined for DOS 1st Edition. In either of the other games, the "next player" is obvious. However, in DOS 2nd Edition, the "next player" could refer to either:

The next player to have a turn after the currently active player.
The next player to have a turn after the player who enters a state of having exactly two cards in their hand.

DOS is designed in such a way that you may enter a state of having exactly two cards in your hand on someone else's turn. The 1st Edition's rule says:

If this happens during your turn

The very nature of this word "If" means that these events could happen outside of your turn. This is clear to anyone who has played DOS (1st Edition) and has in fact happened to me on several occasions. It happens like this:

You have one card in your hand
Another player scores a Double Color Match
You are forced to draw one more card, giving you two and obligating you to say "DOS"

In this position, when does the second starting pistol fire? Does it wait until the player with two cards begins their next turn, taking another action? Or could it fire immediately at the moment that any player takes a new action? We are left speculating on this point, but what seems the most obvious (and fun) to me is that the second starting pistol would fire as soon as any player takes a new action.

Intermediary DOS

In the 1st Edition of DOS, you can play two cards at the same time on one pile. This means that you could go from having 3 cards to having 1 card in a single action, in a single play. Two cards leave your "hand" at effectively the same moment. Do you have an intermediary instant of having only 2 cards in your hand, or do you go straight from 3 to 1? Zeno would have an anyeurysm at the discontinuity of the latter assumption. Going from one point to another without passing through all points in between is unfathomable to everyone except for sci-fi authors.

However, you must consider that the actions in card games are not perfectly continuous and are instead broken up into blocks (I've been calling them "actions" earlier in this post). At the beginning of the action, you have 3 cards. At the end of the action, you have 1 card. How you got from the beginning to the end and what happened in between has no bearing on these states. Additionally, if the two cards exit "your hand" at the same instant, there is no actual point where only one has left your hand.

I don't think that an "intermediary DOS" is well-established, but I still call it out every time anyways just in case it exists.

Conclusions

A card exits your hand the moment that it is not touching your hand and is touching a (legal) pile. A card enters your hand the moment that it is touching your hand and is not touching the draw pile. If you ever have exactly 2 cards in your hand, you immediately obtain a high priority obligation to shout out "DOS". If the next action by any player is anything other than you shouting "DOS", then all other players obtain an opportunity to call you on your laziness/forgetfulness. There is no cutoff for other players calling you on this, even if you no longer have exactly 2 cards in your hand. If they do, then you must draw two cards. If it is your turn, add these two cards to your hand at the end of your turn.

I wish that Mattel had been clearer when writing their rules.

Maximum Penalty

You add the penalty cards to your hand at the end of your turn. The rules are very explicit about that. The penalty cards are not in your "hand" until you are finished with all of the other actions from your turn. If you earlier were in a state of having exactly 2 cards in your hand and nobody called you until you later had 3 cards in your hand during your next turn, you would have to finish playing your turn and then take the penalty cards. If you played 1 card, dropping you back down to 2, on your turn, you would trigger a second occurence of having only 2 cards, given that the penalty cards haven't entered your hand yet. You could be penalized with up to 4 penalty cards on one turn.

PUNKTO | 4 Days In Playa Del Carmen

Tue, 11 Mar 2025 00:00:00 PST

4 Days In Playa Del Carmen

These images were originally published on a now-defunct Telegram channel and this blogpost has been backdated to be at the correct time in history. I've taken this opportunity to better select the images to share for telling the story and conveying the experience, and I've written much more explanation than I originally did when it was a late-night post to a Telegram channel after a long day.

In March of 2025, I travelled to Playa Del Carmen, Quintana Roo, Mexico with my family for a short vacation. The trip lasted from March 6th to 10th, just before the Spring Break crowd took the world's vacation destinations by storm. Even though the trip was short, we managed to cram a whole lot into the long weekend. Being my first trip South of the border, this was a great opportunity to put my Spanish learning to the test.

Our group after landing in Cancún. I'm holding the camera.

Starting In Portland

Our journey didn't start out in Seattle, which would have made more sense for just myself and the girlfriend. Instead we started in Portland since my brother and his wife live down there and the logistics for them getting to Seattle were a nightmare. We stayed in Portland the night before the flight since our flight was at the buttcrack of dawn.

For dinner the night before the flight, we went to the Cartlandia location on 82nd Ave. The food that we got was stupendous. Look at the sheer length of that Bahn Mi. It was as tasty as it was long. And for a good price too (coming from the perspective of Washington prices). This seems like a good point to give a little bit of history on the Portland food scene. The basic flow of events, as best as I understand them, is along these lines:

Long ago, Portland was a mostly unremarkable city from the gastronomic perspective
In 1963, a fella named Horst Mager immigrated from Germany and opened a restaurant in NE Portland.
Horst was unimpressed by the culinary scene in Portland and so opened a culinary institute to teach people how to make food actually good.
The graduates of Horst's institute transformed the city's food scene by collectively raising the bar and raising the population's expectations.
Lower quality establishments no longer met the public's high expecations and either adapted or died.
Portland punched above its weight class in the "foodie" circles for many years.
The 2008 housing bubble popped and drove many people into destitution.
The Portland city council lowered the restrictions and streamlined the process of opening mobile food carts.
The native food cart population experienced an explosion in numbers.

As a result of the above series of events, Portland is notable for being one of the best cities for street food on planet Earth. Additionally, a much higher proportion of the food available in Portland is really good compared to other cities. You can find food trucks in nearly every empty space in the city, and you can be fairly confident that what you get will be good. This time, Portland did not disappoint (not a surprise).

PDX

We got to the airport at about 1:30am. Our flight left at 5-something. You might find yourself wondering "why did you get to the airport so incredibly early". If you're not early, you're late. Much to my shock when we arrived, they had remodelled the airport since the last time that I passed through.

The remodel looks amazing. There's these beautiful wood facades everywhere. There's art installations everywhere. It's a place that's actually nice to be instead of a purgatory of hell that airports usually wind up being. I had plenty of time to explore thanks to the fact that we got the airport so early, so I saw most of the airport.

Even though they changed the carpet, you still have to take this picture to let your mom know that you made it.

While walking around the empty airport, I was able to take what I feel are some intriguing shots. These spaces are designed to hold and move a lot of people. But here they are empty. Like a giant just before waking up. My favorite picture here is the drinks fridge. It was at a convenience stand that didn't open for a few more hours.

In my wandering, I came across an installation of objects from Guillermo del Toro's Pinocchio. This film was shot in Portland, so the props didn't have far to go. I hadn't (and still haven't) seen the film, but it was intriguing to see how detailed and expressive the models actually are in real life. I normally expect things to be phoned in, with simple faces in real life that are "enhanced" in CGI.

I also found a statue of former Oregon governor Vic Atiyeh. Worth noting here is that the former governor's statue is holding a boarding pass from Portland to Tokyo, and this boarding pass is sporting a large SSSS on the front. This "quad" signifies that the holder has been "selected" for Secondary Security Screening Selection. Maybe this occasionally happens randomly, but it happens every time you fly if you're on the "Selectee List", one notch down from the No-Fly list. Actually they're the same list, it's just whether or not a box is ticked that determines if you are allowed on airplanes or not. Having this on your boarding pass makes your time at airports quite uncomfortable: everything takes longer, you are scrutinized under a microscope, and your freedom of movement is limited. Thousands of dollars in man-hours are wasted every time a selectee has a flight. I was at first astounded to see this on the boarding pass of a statue of a government official, but it could well be a nod to Vic's Middle Eastern heritage and the "random selection" of anyone profiled as a threat. Related reading.

Next stop, Cancún (actually Salt Lake City but you get the idea)

Lodging

After getting on the airplane in Portland, we had a layover in Salt Lake City (really weird vibes at that airport), and then took off for Cancún. We made it through customs without any fuss and got into a rideshare that we had reserved online to take us the short road trip from the Cancún airport to Playa Del Carmen. We checked into our lodging and they gave us these wrist bands to "get discounts" (easily identify us as tourists) and told us that if we didn't return them we would be heavily fined. We were not in fact heavily fined when we didn't return them.

The condo that we had rented was pretty great. I don't have many pictures of the inside, but it was pretty swanky. Three floors, three bedrooms, three bathrooms, a jacuzzi on the roof, and a patio on the second floor. Tropical foliage all over the place and a shared swimming pool for all the guests at the complex. The bottom floor shower was quite large and had some nice tile-work. The water was not potable.

The art hanging throughout the condo certainly had a theme. You can really tell that it was all painted by the same artist with a...taste. Everyone is caked up, even the child in the first painting. Even the dog. You just know that that bird is packing a dump truck beneath those feathers.

Artistry

It's not just inside the hotel. In fact, everywhere that you look in Playa Del Carmen, you will find artwork. Coming from a culture that takes itself too seriously to just have cool art on the walls, it was so refreshing to see a display of such talent everywhere that I went.

There was some low quality or gang-related grafitti, but the vast majority of it was actually well done. Sometimes it was related to a business, but quite often it was just art for its own sake. The purple stripes on the image with the woman aiming a rifle with a scope says "pinta o muere" (paint or die). I adore the pattern work in the parrot's feathers.

Not forced into just putting paint on walls for art, the local population hangs papel picado down entire avenues. I'm not sure why there was a Christmas installation in March, but I can't talk much since I leave stockings up and listen to Christmas music year-round. I'm a holly-jolly guy, and I loved the yuletide area that existed for no reason. We saw live performances of dance (I'm not sure which type) in the street for diners, plenty of statues and art installations using recycled materials.

Speaking of recycled materials, I found this chair made of old ties. Here we would "recycle" these tires: melt them down or shred them up, repurpose them as a new source material, and otherwise put a bunch of carbon into the atmosphere to reuse them. Instead of doing that, a box cutter and some rivets could turn the tires into a chair that can actually be used immediately. I wish that more recycled goods had the soul of their materials' past lives. This chair was quite comfortable.

There really was artwork every where that I looked in Mexico. Even little things like a gate had been done up with some flair. In all of the first-world countries that I've been to, there hasn't been nearly this level of art on display. Walking around America has no SOVL like this.

Food

There was a point in my life where I wanted to travel to see sights and have experiences. Now I want to travel for food. It is (now) the first thing that I consider when planning a trip. Everything else comes second. Playa Del Carmen is not particularly known for its culinary scene (unlike Portland, as discussed earlier), and rightly so. It wasn't that great. Most of the food was remarkable for how unremarkable it was. I do not recommend visiting PDC for with gastro-vations (gastronomic motivations, aka "hunger").

Our first dinner in the country was just me and my gf. We went to a location named Amate 38 because it is on 38th avenue. At the time I thought that "amate" meant "he/she loves you", but I've continued to improve my Spanish and I am now pretty sure that it either means "love yourself" or refers to a type of artwork made on traditional bark paper. The decor and artistry here was really nice, you can tell that they thought things through and executed their vision well. The best part of the meal were these handmade fresh tortillas. The rest of the food was pretty bland. We had gone here becasue Amate 38 is widely lauded as one of the best locations in Playa Del Carmen, so our expectations were pretty high and we left feeling a bit disappointed.

Birria is a staple Mexican food in Washington. You can get it at many Mexican restaurants and food trucks. Making it is pretty easy, it just takes a lot of time. Various meats are stewed together for many hours, the softened meat is popular as a filling (in addition to cheese) in quesadillas, which is usually dipped in the soup broth. Sometimes the stew is just consumed by itself. It's quite simple, and difficult to mess up. The birria at this little shack was okay. Most of the actual flavor came from the little cups of sauce that were on the table. It wasn't fantastic, but it wasn't bad. This will be a recurring trend with the meals that we had in PDC.

At the birrieria, it didn't seem like the employees spoke any English. I spoke just a tiny bit of Spanish at the time, enough to get by at restaurants. It was quite confidence-building to be able to navigate the interaction here. Admittedly, the types of conversations that you have at a restaurant that only serves birria are pretty limited. You could have a robot generate the whole conversation and it would be pretty accurate.

This was the first time that I got to have Sidral Mundet, a sparkling apple juice drink popular in Quintana Roo. It quickly became my favorite commonly-available drink in Mexico.

Our breakfast one morning was at a restaurant called Lara y Luca. The venue was lushly adorned with plants, and the food was plated with an artist's touch. It was okay. No major problems with the food, it just looked better than it tasted. The juices were astoundingly fresh, however. They were really nice. Non-potable tap water throughout most of Mexico leads to many restaurants serving aguas frescas (fresh waters). They don't disappoint. Finding juices like this in the US is a rare treat, but they seem to be commonplace (at least where we went) in Mexico.

When we swam in the cenotes (more on that later), we worked up quite an appetite. The buffet was about to close, but the men running it kept it open and gave us a discount from the posted price. Here you can see some plates of the various buffet food mixing together. Among the options was cochinita pibil, which I called "chibli pibli", prompting a laugh from our waiter. This is a traditional (pre-Hispanic) dish native to the Yucatán peninsula, and it was actually quite good. I'm not sure how much was the food's quality and how much was my extreme hunger from swimming for hours, but the meal here was my favorite on the entire trip.

At one point, we walked into a tucked-away restaurant with an exceedingly eclectic menu and astronomical prices. The waiter told us that he couldn't seat us without a reservation, since all open tables for the day had been spoken-for. I inquired about the menu and he said that it was because the it was a kosher restaurant catering to Jewish diners. I asked him where in the area he would recommend for food, and he pointed us toward Zublime, a short walk away.

The food here was actually pretty good. The plating was stupendous. I can still taste the creamy sauce that the shrimp is in. If you find yourself on the island of Cozumel, I would recommend this place. However, it isn't really Mexican food. In the last picture there, that small grey ball is actually ice cream. The schnitzel was really nicely breaded as well.

Speaking of eating non-Mexican food in Mexico, we went to a French bakery joint one morning for breakfast. The food was not remarkable. It tasted like Shari's (RIP Shari's).

There was a seafood restaurant next to our hotel where our wristbands got us a discount. I had been wary to check this place out since our hotel was right on 5th ave, the main drag through town. I reasoned that anywhere so close to a massive tourist area would certainly be low quality since they would have spent all their money on the location and they would be catering to drunks. I'm used to the US where the best restaurants are the sketchiest and the restaurants closest to the attractions are the worst. I was wrong, though. This place was fair. The soup was packed with flavor, the chips were fresh, and the tacos were really nice.

I do wish that we had more time. More time to explore the culinary scene. I didn't get to try tacos árabes or go to this literal hole-in-the-wall joint that smelled amazing.

Not all food is prepared meals in restaurants. Our actual first order of businesss after leaving the airport, before getting to our condo, was getting snacks. The watermelon salsaghetti skwinkles went hard. I wish I had some skwinkles right now to be completely honest.

Look at this gigantic melon!

Beat The Heat

Mexico was hot. It was 80+ degrees and humid. We had come from the cold PNW and wound up sweating our butts off. We needed to find ways to escape the sun, at least during the hottest parts of the day.

Cenotes are underground rivers and water systems that are all interconnected. The entire Yucatán peninsula is on top of one gigantic cenote network. We went swimming in it. Not only did this water-cool us (by swimming) but it got us out of the sun (by being underground). It was stupendous. I didn't bring my camera with during the guided tour because water is wet and my camera is dry and I prefer to keep it that way. There were humongous ledges that you could jump off of into the water, and I jumped off every single one of them (sometimes multiple times). I have no evidence to back this up.

Again, swimming in water water-cools you and so keeps you from succuming to the heat. I had never gone snorkeling before. It was a blast. I highly recommend going snorkeling if you get a chance. I again didn't bring my camera because water is wet and electronics don't like being wet, but the guide had an underwater Go Pro which got some cool shots. I got to swim deeper than I ever had before, down to a ship wreck. There was a point where I was swimming around on my stomach and a whole school of fish passed right under and around me. I got to see coral up close for the first time, and I even saw a barracuda as big as my arm. It was an awesome experience.

A word to the wise: if you're going to be swimming in direct sunlight for a long time, wear a long sleeve shirt. Three of the people in our party got badly sunburnt. For me it was the worst sunburn of my life, quite agonizing and really put a damper on the rest of the trip. In fact it put a damper on the next two weeks as sharp bolts of pain would snake across my upper back at random. A slight breeze left me writhing in agony. I wish I had worn a shirt. You can't put sunscreen on in a protected reef, and it washes off quickly anyway, and sunscreen gives you alzheimers.

The food in the first pic above was actually used on the tour boat to make guacamole lunch for the whole party. The guac was fantastic. Everything was fresh and (presumably) local.

The coral reef snorkeling tour actually departed from Cozumel, a small island across the water from Playa Del Carmen. In order to get there, we had to take a ferry. The views from the water were great. I was struck by how nice the ferries looked. Coming from Washington state, where the ferries are government-operated and they look government operated (run-down, dilapidated, kept running out of reluctance), the ferries from PDC to Cozumel looked like they were from a Sci-Fi movie or a tech startup. Admittedly these ferries are much smaller, only carrying people and not cars, but they looked really nice.

On the ferry ride back, we hit some extremely rough water and everyone had to deboard the boat one at a time, jumping from boat to land, from a ramp extending out of the top deck. That was fun but I don't have any pictures of it.

One of the things that we could see on Cozumel from the water was a giant pyramid-like structure. It's immense and you feel like you're on your way to Giza as the boat draws nearer. Could it be made of stone or glass and steel? What architectural and culture treasures lie in store here? When you get up close, you find out that it's a hotel with a gigantic thatched roof.

We made a friend on Cozumel as well

One day we were walking around aimlessly and some members of our party were not coping with the heat very well. So we found a nearby art gallery that had A/C and popped on in. I must admit that most of the art went over my head. I don't really understand the message conveyed by wood growing out of gravel in plastic pipes. The plane in sand rocks, I really liked that. The jaguar was cool but I couldn't take a proper picture of it. The jaguar's installation was massive, taking up a whole corner of the place. The animal was laying on the ground, killed as you can see, and on every nearby wall were CCTV cameras pointed directly at it. I couldn't take one picture that fully captured it. The tile work on the car near the entrance was really nice, it was done so meticulously with such great attention to detail.

Guayabera Journey

I wasn't just in Mexico to have a good time. I had a mission.

Close your eyes and imagine a "Mexican Shirt". Not a polo. Okay try imagining a "Cuban shirt". Great now open your eyes. The shirt that you were just picturing in your head is called a guayabera. It wasn't invented in Cuba, it's just that Castro liked them so they're heavily associated with him. I wanted to find some of these shirts for a couple of reasons. Firstly, they look awesome. In fact people in third-world countries frequently have unmatchable swag. From Mongolian Deels to West African Dashiki to the Arabian Bisht, the world's poor have impeccable fashion sense. I own a deel, but I can never wear it because people whine that I'm doing cultural appropriation by wearing clothes that don't belong to my heritage. 100% of the Mongols who have talked to me while I'm wearing my deel have ppreciated it, but Seattleites complain. Since I have Mexican heritage, I get a free pass to wear guayaberas, which are incredibly stylish. Secondly, they have lots of pockets (sometimes as many as 4).

We wandered around and I saw a few for sale (among other torso clothing) for exhorbinant prices (~$100 USD per shirt). I spoke (in Spanish, to practice) with a man who wasn't working at any of the tourist traps about where I could find a reasonably priced one, and he pointed me to a store on Google Maps named GUAYABERAS ADELA. The next day, instead of trying to stay out of the heat, I decided to go directly into the heat and walk 1.5 miles in 85 degree weather and 85 percent humidity, straight to Guayaberas Adela. When I got there, I could find no such business. Instead I found a garage with a little old lady sitting on a plastic chair.

Her English was broken, and my Spanish was way worse. I inquired to her if she knew where I could find the store, and she informed me that it was her store, but that she had retired several years ago after selling Guayaberas all her life because it grew tiresome in her old age. I asked if she could point me towards somewhere and she looked up the map on my phone to point out the correct location. Despite the langauge barrier, this elderly woman went out of her way to make sure that I knew exactly which door of my destination building to go into. It was heartwarming.

Along this long walk outside of the immediate tourist vicinity, we certainly got some strange looks. At no point did I feel threatened, I could just tell that the locals though that the gringos were lost. It didn't help that I was constantly taking pictures of things that most people probably don't care about.

Another half mile down the road, I came upon my target, Mercado Público Municipal. This is a large building with many different vendor stalls where sellers hawk various wares. Most of it was clothing, but there was some home goods as well. This is where I found a guayabera seller. There was again a language barrier. My Spanish skills were just enough to squeak by, but the clerk went out of his way to make sure that we understood each other. From him I purchased 4 plain white guayaberas with 4 pockets each, 1 long sleeve dark blue guayabera with 4 pockets, and 1 short sleeve teal guayabera with red embroidery and 0 pockets. The total bill came out to about $120 USD, much more agreeable than the $100 per each that I would have paid on the main tourist drag.

I love these shirts, and you can even see me wearing one in my FOSSY 2025 talk. I wear them often and nobody can talk crap about cultural appropriation because they're my culture.

It was about food time and on the way out of the mercado we stopped to chat with some small elderly women about dining suggestions. After a bit of (deserved) giggling at my poor Spanish skills and my inability to roll my R's, they realized how far we would have to walk in the heat to get where we were going. Instead of just telling us to be careful out there, these women hailed the next bus, got us on board, told us how much to pay, and told the bus driver where we were going so that he could stop at the correct place. Their kindness and generosity touched my heart.

What's PDC Like?

I haven't directly touched much on the actual experience of being in Playa Del Carmen. It's a huge tourist destination. The main drag through town, 5th avenue, is a 24/7 party. The 2014 dance music playlist never stops playing, the lights never stop flashing, and the sellers never stop selling. It has the same kind of "party for drunks" vibe that you get in places like Vegas, except even more-so. I've never seen so many sex references in public signage. I've never seen so many sex shops. The number of pharmacies selling things like ozempic, steroids, antibiotics, and viagra openly blew my mind. There's copyright infringing (hey Nintendo, C&Deez nuts) shirts being sold at discount prices (for American standards), and the kind of egregiously "funny" graphic tees that derive most of their humor from capitalization and kerning choices.

There were sketchy massage parlors with scantily-clad "masseuses" aggressively coming after any single male passersby. There were all manner of injection-molded trinkets and nonsense things and stuff that just said the words "Playa Del Carmen" on the side. There were random dudes running around going up to groups asking "hey do you wanna get high?" and then sprinting away to the next group upon rejection.

I've never been so accosted by sellers. I told one man that I didn't have time to stop at his store. He said that that's okay and that I could have some of his time. You expect to see SpiderMan harassing you for pocket change in exchange for a photo at Time's Square in New York City, but here we experienced it in Mexico. These people have developed a sixth sense for detecting a "mark". They can spot who they can effectively encourage to go into their store and spend some money, and they are quite good at this. They repeatedly marked my step-dad. It repeatedly worked.

I was saddened that the main drag of Playa Del Carmen (a city in Mexico) didn't seem to have much Mexican identity. It gave up most of whatever it had to absorb the party identity. But unlike other tourist locations around the world, which have some unique spin or draw to their attractions, PDC's party identity is based solely on party. Like a ball of meat with no skeleton to give it structure or a bowl of fake plastic fruit at the dinner table. Everybody in the tourist center spoke English, there was little opportunity for a language barrier to be noticed. The closest that I got inside the tourist center was in an ice cream shop. I was trying to order vanilla ice cream and the guy at the counter could not understand what I was saying until I pronounced it "bye-knee-juh" the way that a Comprehensible Input video on YouTube did.

Infra & Structure

I like to take pictures of the world around me. Infrastructure is one of my favorite subjects.

I wasn't consciously aware of how well-hidden and minimized the electrical infrastructure in the USA (or other first-world countries I've been to) is. Many places in PDC display this kind of rats-nest approach to high voltage wiring and metering. It's a bit difficult to see due to the fact that water is invisible, but in the 5th picture above there is a steady stream of water running onto the electrical meter box. There were areas where the electrical wiring was sagging so low that you had to duck to walk down the sidewalk. There were (I think, I didn't have my multimeter) exposed lines at human-level. Perhaps on a long enough timescale this area will give birth to a new type of human who has evolved electrical immunity. Or just caution.

Similar levels of polish and OSHA-compliance were to be seen in some of the construction. This crane(?) was hobbled together with sticks that were presumably sourced from the worksite. This kind of carbon efficiency makes American construction companies shiver. You may notice that the man painting the beam of the market, standing on a scaffolding, only has one leg. I have never seen a construction worker with a prosthetic limb before. I always assumed that if construciton workers lost a limb they just retired and lived off of worker's comp. Perhaps no such system exists in Mexico.

I similarly like taking pictures of buildings and open spaces. I've been told that I have a particular style when it comes to buildings. I tend to show the sky. I just try to take pictures that I think look cool. It was easy to get shots of empty spaces in the wee hours of the morning. I was waking up at about 5am every day since my body was on West coast time and PDC is 3 hours ahead. At this point in my photography hobby, I hadn't figured out that you can turn the exposure dial, so the dark shots don't look as good as they could have if I were to retake them again now.

That Abarrotes La Reforma building is apparently a convenience store, though I don't think that it was open for business when we were passing by. I think that building looks particularly distinguished.

Vehicles

Vehicles are similarly fun to take pictures of. Certainly one of the classic photography subject matters.

Being a poor country, Mexico utilizes vehicles for people's primary transport that we don't or can't in the US. I hadn't seen a tuktuk on this continent before, and it was refreshing to be greeted by these cute little three wheeled vehicles once again. Mopeds are also widely used as people's main transportation device. You can't get away with that in the USA, insurance companies won't allow you to daily a moped. You have to have a car if you want to also have a bike.

Speaking of economical, I need to talk about the colectivos. In America we call these "shuttles" and they usually take us to and from airports and hotels. Quite often they are free. In Mexico, these same types of vans are used as mainstream public transport (in addition to actual busses). The doors don't latch, so as they drive around the doors swing open and shut (unless someone holds it shut). Don't fall out or get your fingers crushed. If you do, that's on you. We didn't know about these when we first arrived, so when we were walking down the street (on the guayaberas mission) and a white van pulled up and the door slid open to let someone out, we were a little concerned for our safety. No need to fear though, this is just regular public transport. The price was agreeable as well, $20 MXN (about $1) per ride. The only issue was that the maps were not published anywhere online or in print. The local population just figures out the bus routes given enough time.

The artistry that I mention earlier also extended to people's vehicles. It feels like you can't go a furlong without finding someone's art project or painting on a functional surface in this place. That's refreshing, especially when compared to the boring corporate surfaces in the PNW.

The opposite of an economical ride, I'm used to seeing Garda armored trucks up here. They have a distinct silhouette and style. This was not that. This gives much more "killdozer" vibes. I imagine that the armoring and precautions on a Mexican armored vehicle are much higher than on an American one, given cartel violence.

Lastly, a parking mechanism that we saw quite a lot of. I believe it is a form of anti-theft. A strange little buggy on the island of Cozumel. And a VW bus that simply won't quit.

Words In The World

I have a folder on my computer named "Words In The World" where I save all the pictures of words that I have seen written in public that make me go "hmmm". Here are the additions to that folder from PDC. I love that Kentucky Fried Chicken becomes Pollo Kentoky in Mexico. This is especially entertaining because Kentucky is re-spelled to be pronounced correctly, but it keeps the K's, a letter only used in loan-words. I had heard of "bistec" being an mishearing of the English "beef steak", but I hadn't actually witnessed it until this point. I got a new term for buttcheeks. I have no idea what a "Color Change" is, and I refuse to look it up because that would spoil the fun.

Closing Remarks

Everyone that I interacted with directly was friendly and kind. Coming from an area renowned for its unfriendly and cold people, this was a very welcome change of pace. Not just the sales people in the tourist-heavy areas, either. I was riding a bus and a woman recognized that I was not from around there, so she (being bilingual) offered to communicate with the driver on my behalf. Everyone helped me through my questionable Spanish skills. At 3am a convenience store worker dragged out extra crates from the back to let me look at cigarette flavors (it's for a friend). When I'm in America (specifically Washington) and I ask for help finding something, I get a "google it", a shrug, a silent finger pointing in a direction, or simply ignored.

I enjoyed Mexico, and I will be returning. I won't be going back to the party city of Playa Del Carmen.

PUNKTO | CAnada, CAlifornia, and a CAve

Sat, 01 Mar 2025 00:00:00 PST

CAnada, CAlifornia, and a CAve

These images were originally published on a now-defunct Telegram channel and this blogpost has been backdated to be at the correct time in history. I've taken this opportunity to better select the images to share for telling the story and conveying the experience, and I've written much more explanation than I originally did when it was a late-night post to a Telegram channel after a long day.

In February of 2025 my girlfriend and I had a simple plan. We wanted to go to CAnada for a nice romantic weekend for Valentine's day (really an excuse to eat chicken tenders), and then a week later hit CAlifornia to see a John Mulaney show. Towards the end of the month we also explored a CAve in Yakima, WA. This wasn't part of the original plan but I took pictures that I want to share so it goes in the blogpost and we get a third CA. Overall we had a great time everywhere and we took some cool pictures and ate some tasty food.

CAnada

Canadian Food

The plan for the romantic getaway to Canada starts in Ireland. When we were in Dublin, we had a spice bag for the first time from an award-winning Chinese takeout restuarant. A spice bag is a Chinese food from ireland the same way that the fortune cookie is a Chinese food from the USA. It is prepared by stir frying chicken tenders and french fries in five-spice with onions and bell peppers, then tossing that into a paper bag and putting curry sauce into a little single-use cup. It is eaten by tearing the bag open, pouring the curry sauce on top of the food, and going to town with a fork. It is absolutely delicious, probably the best food that we had in Dublin.

When we got home, we looked high and low for somewhere that made a spice bag. After a few hours of searching online, we couldn't find a single restaurant in Washington, Oregon, or California that had it on the menu. Feeling something like the opposite of glee, we hung our heads and moved on with our lives. That is, until my gf found an Irish pub in British Columbia that had a "spice tray" on the menu. With border crossing, it was a drive that we could make in 1 day if we wanted to. So we did. Our first stop in Canada, before we even checked into our hotel, was at Donnellan's Irish Pub in Vancouver, BC.

We both ordered the spice bag. It was 100% legit (based on our memory of one late-night spice bag months prior) and delicious. They hit all the points that make a spice bag a spice bag (to my understanding). My only gripe was that the portion size was quite small, but that's probably because this was listed as an appetizer instead of an entree. They also had "Oklahoma style smash burgers" which was one of the better burgers I'd had in recent memory. Someone was talking recently about incorporating Canada into the USA. Their burger skills are high enough that maybe that could work. We shouldn't let them be a state though, maybe just a territory or something like Puerto Rico. Their journey to proper statehood should require more than just making good burgers (an admittedly American trait), but also things like winning the Super Bowl and landing on the moon.

This was really the main reason that we went up to Canada, but since it was close to Valentine's anyways we just made a whole weekend out of it.

We went to a place called Neverland Tea while we were in Vancouver. This is a "tea salon" and has earned a long mention here as well as my continued patronage every time that I go to Canada in the future. The tea here was fantastic. I'm a bit of a tea snob. This is exceedingly high praise. If you enjoy tea, this establishment receives the highest recommendation that I can give for a tea salon. I rate it 12 out of 10. The only other place that I know where I can reliably get tea this good is my own kitchen.

We had high tea at Fortnum & Mason's in London. The Queen of England started the high tea tradition at Fortnum & Mason's. They are widely regarded and praised as some of the best high tea that you can get in England, a country known for its love of tea. Neverland outdid Fortum & Mason's by leagues in the quality and preparation of both the tea and the food. All of this at 1/4 the cost.

The whole place was decorated in such a tasteful manner for Valentine's day. It is typically quite easy to go overboard with PINK and wind up being gaudy, but Neverland acheived an elegance that signaled thought and restraint. Their china was gorgeous and impeccable. For Valentine's day the salon had a special menu, which we opted for out of a selection of three. You need a reservation to get in here, and when we arrived the table that had been set aside for us had already been decorated to match the menu (and season). I'm used to arriving somewhere with a reservation and being seated at any open table. The waitress at Neverland knew exactly where we were to sit the moment that I told her my name. This intentionality in decor and table reservation betrays the thought that they put into everything.

The food was delightful. I'm running out of positive adjectives. Nearly everything on the tower of plates was somehow pink, and everything was delicious. The only complaint that either of us had was that the little triangular sandwiches on the bottom level weren't to our taste, though that is certainly just a matter of personal preference.

And just look at how beautifully everything is arranged. How elegant and subtle the decorations on the platters are. The way that the quiche is nestled between the finger sandwich and the roll. I loved it.

The most important thing at a tea salon (or anywhere that serves high tea) is the tea itself. Neverland has a sizeable selection, and you can see a full list of their flavors here. They serve blacks, greens, whites, oolongs, herbals, and even rooibos. I got a plain black and my gf got a scented black. Both were delightful. But most importantly they were prepared correctly. This is an embarassingly rare bit of good news and deserves some explanation.

China (and many Asian cultures) has a special tea ceremony method of tea preparation which I am not talking about here. That is its own thing outside the realm of Western tea. As far as Western tea traditions go, many people follow in the footsteps of the British and completely ruin their tea. A lot of people think that they don't like tea when in fact they just don't know how to make tea, and they would probably enjoy it if they tasted some that was correctly prepared.

The beverage that we call "tea" is made by taking dried tea leaves and placing them in hot water, a process called "steeping". The method of drying and processing determines which type of tea (green/black/white/oolong/puer) is created. These different types of tea (which are all made of leaves from tea plants) have different properties and flavor profiles. Additionally, there are herbal and rooibos teas which are technically not teas since they do not use leaves from the tea plant. Herbal "teas" are whatever dried herbs you want to steep and rooibos is a specific type of herbal tea using a specific African plant.

Since these are all different things, they behave differently when you expose them to hot water. The terpenes and tannins in each type of tea are released at different temperatures and at different steep-times. Many people (such as the British) place their tea (which in many cases is not tea) into tea bags in boiling water and leave them there for the entire time that it takes to consume the drink. This results in bitter tea because the tannins are over-extracted. Any tea can become bitter if you steep it at too high of a temp or for too long, but teas don't have to be bitter. Oblivious to this, most people (such as the British) add milk and sugar to their tea to offset this unpleasant bitter flavor, masking the flavor of the tea. This is an abomination.

Different types of teas deserve to have their appropriate temperaturees and times observed. Herbal tea is the most forgiving because it's not actually tea. Of the actual teas, black tea can take the harshest treatment and is generally not too bad if you use boiling water, but ideally it should be steeped at 200-210 F, not 212. The other teas are more delicate and will become bitter if you expose them to the same treatment that black tea can withstand. If you want to actually enjoy the tea flavor of your tea and you don't just want sweet milk, I implore you to follow the guidelines on the tea package.

Speaking of tea packages, the teabags that you can find at the store are typically terrible. Open one up and find that they do not appear to contain any tea leaves, but instead a dusty powder made of who-knows-what. Since you can't actually see the leaves, you have no indication that what you are looking at is leaves. It could well be twigs, dirt, bugs, or any manner of other nonsense additives. Given the profit motive of companies and the lack of care from the average consumer, I find it pretty likely that this is the case. Tea should be purchased loose leaf and it should actually be leaves.

Lastly, tea should not be steeped in a bag or a small infuser ball. The tea leaves need to fully unfurl in order to allow the water to permeate them and extract all of the delicious flavors, but keeping them locked up in a tiny enclosure does not allow them to do this. Instead it traps your tea leaves and prevents them from realizing their full potential. Those tea leaves spent a lot of energy growing and came a long distance to get into your cup. They deserve better than cramped quarters. For this reason I generally use an infuser basket, which allows my leaves to swim around a bit.

Fortnum & Mason's followed the British tradition of ruining perfectly good tea leaves. Neverland Tea Salon in Vancouver, BC got all of these points correct! The tea was all loose leaf. It was steeped at specific temperatures for specific amounts of time. See those little white pucks in front of the spheres? Those are timers to ensure that the tea leaves infuse the water for the correct amount of time. The tea is steeped loosely in these giant spheres, giving them plenty of space to unfurl and swim around. In fact they were even stirred to encourage this swimming behavior. When the timer went off, the barista (not sure if that is the right term for a tea salon) simply had to press on a lever, which allowed the tea to instantly flow out the bottom and into a cup. The bottom of the sphere's pedestal was fitted with a strainer to avoid getting particulate into the finished cup (which would continue steeping and make the product bitter).

I was absolutely floored when I saw this and I am still flabbergasted that it was done so well. Bravo, Neverland.

One night in Canada, we needed a meal (as you tend to do at night) but it was getting quite late and most everything was closed. What was open was mostly bars, which we weren't vibing with. After some searching around online, we found Donair & Sub House in Burnaby (a town near Vancouver). In the name and on the menu it implies that they have subs. I don't see a single thing that I would call a "sub" on their Google Maps pictures. What I can vouch for is their poutine. We got poutine here with lamb and tzatziki on top. It was reasonably priced, a huge portion, and delicious. It was so good that we went back a second time before leaving town. The staff were kind and the food was banging.

Overall the food in Vancouver was delightful. It is a huge draw for us to return. Vancouver's food rivals Portland's, and both blow Seattle out of the water.

The Fun

Believe it or not, there is more to do for fun than just eat food in Canada.

When I booked the hotel, I opted for a room with a jacuzzi. I did this because jacuzzis rock and it felt somewhat romantic for a Valentine's day weekend. By pure serendipity, we found ourselves at a Canadian supermarket and purchased some bubble bath solution. I've had bubble baths in regular bathtubs before and haven't felt like they were anything particularly special since I was a child. Oh cool there's some bubbles in the bath. They mostly get in the way. We put bubbles in the jacuzzi. When we turned on the jets, the bubbles took over. They didn't get in the way, they became the way. I was making mountains, skyscrapers, and suits of armor out of the bubbles. The bubbles overflowed the bathtub and got all over the place. I thought perhaps the world might be lost in a sea of bubbles. If you ever get the chance to take a bubble bath in a jacuzzi, I implore you to give it a shot.

the glass with some purple liquid at the bottom was fruit juice, not wine

While in Canada, we visited a comedy club that was putting on a show. We sat in the front row. My gf didn't want to sit in the front row because she didn't want to get picked on. I assured her that we would not get picked on. We got picked on. The comedians found out that we were Americans and made a few jokes about how we were most likely armed and how we were "some of the good ones" by being from Seattle. I think there may have been a few other politically-charged American jokes that flew over my head. Overall I had a great time and my gf wasn't miserable.

At one point we found ourselves in a bookstore. I thought to myself "I'll find an easy Spanish book to help me learn Spanish". I typed away at their little self-service kiosk and found the above pictured book listed as "available". I moseyed on over to General Fiction and couldn't find it, so I flagged down a clerk to give me a hand. Between the three of us, we checked 4 different sections (including the back room) before finally finding the book exactly where it was supposed to be. The three of us had independently completely overlooked it multiple times. The search took about 20 minutes, and at checkout I realized that the book was actually in English the whole time.

Here are some photos from the Metropolis mall in Vancouver. We only got a few shots but the interior was wonderfully designed to evoke its namesake. I think it did a pretty good job. The bathroom pictured is not from Metropolis, but it is somewhere in Vancouver. I find the chair in the corner facing the toilet to be puzzling. I don't think that it is there to serve as an ottoman. It appears to be a cuck chair in the bathroom. Perhaps Canadian culture was heavily influenced by Germany.

Some of the fun that we had was actually waiting in line at the border. The way up North was pretty uneventful and regular, but coming back down to the USA we had a ~3 hour long line. This would typically be abysmal, but we had a secret trick up our sleeves: Uno No Mercy. This is a special version of Uno with slightly changed rules to cause people to draw more cards and some new +6, +8, and +10 cards (among others that cause additional drawing too). One hand of No Mercy took most of our time at the border line.

We also got to see the mind-bending strange sight of Non-Sign II just on the US side of the border. It is the first piece of artwork that travelers see when coming into the US through this entrypoint, and its discontinuity leaves you pondering if you've lost your mind from boredom or if it was actually made like that (both can be true).

CAlifornia

The week after we got back from Canada, we made our way down to California. The main purpose of this trip was to see a John Mulaney show. However, cameras were not permitted at the show, with information to that effect in the ticket booking page, on signage at the venue, and verbally by the opener. This rule was flouted by the attendees, but I did not even bring my camera or take a single picture the whole time. Mulaney is a favorite comedian for me and my gf, and seeing him live was a treat. It seems like he is working on fleshing out new content, and a lot of what he ran through on stage was similar to his SNL opening and other recorded performances that he has done of late. Also, he had caught a cold but performed through it anyways. Sadly, he seemed a bit miserable while on stage. I hope he's feeling better.

Leaving WA

Before leaving for California we went to Snoqualmie Pass for pancakes at the recommendation of a friend. They were okay, pretty much just standard diner fare. You get the same pancakes somewhere like Shari's or IHOP. There was a LOT of snow hanging off of the roof and piled up all over the place. When we got to Cali it was 70 degrees and sunny, with no snow to be found. Elevation and latitude are wild.

We got to Seatac quite early because if you're not early you're late. Since I have a fancy Capital One card, we were able to get into the lounge in the S gates, and wow was it delightful. There were comfortable chairs, soft music, power outlets, and endless free food and drinks. I had never been to the lounge before and it was a nice experience. I'm glad to be able to visit places like this. However, C1 is changing their terms and I won't be able to bring guests into the lounge starting in 2026. The lounge is a significantly nicer place to hang out than the terminal where you're lucky to find an uncomfortable seat and it's loud and all the food is overpriced.

Lodging

Flights and show tickets and a rental car were expensive and I had just spent a bunch of money on a hotel in Canada. To keep costs low, I booked a cheap hotel in what I later learned was a sketchy area of town. It was quite scuffed and there were some interesting design decisions. There were these vines growing all over the place. The driveway was an intersection. And there were these speakers that were just playing classical music 24/7. Definitely not the worst place that we have ever stayed, but not somewhere that we would want to stay again.

When we got into our hotel, I smelled a familiar odor. It smelled faintly of maple syrup and ice, like the country that I had just come from. I followed my nose to track down the source of the odor and found it emenating from the desk in the room. It was held together with robertson #2 screws, an alternative to phillips that is popular and common in Canada. I found the smoking gun when a bilingual "Made in Canada" sticker was also on the underside of the desk. No Canadian can hide from me.

The hotel was old and run down. It hadn't been renovated in a long time. The aspect of this that I found most entertaining was the juxtaposition of a GIGAMAX ethernet port feeling the need to specify that it is not a "dial-up" connection and the unassuming power outlet with integrated USB-PD on a USB-C port.

Californian Food

Man's gotta eat

When in Rome, do as the Romans do. When in California, eat avocado toast because you can't afford a home anyways. The venue was pretty cute and decorated in a millenial fashion. The food was decent and if you're feeling frisky there's a mega church right next door.

We had dinner at a ramen restaurant in Redlands. I have since been informed that Redlands in "not a great area", though it seemed clean enough to me. When we walked in, we were the only customers in the restaurant, which gave me a slight pause of concern. Usually if a place is good, it will have people in it. My anxiety was quickly alleviated however by the Yu-Gi-Oh was playing on the TV behind the cash register (muted) and the rattlecan artwork covered the walls featuring a...gang of dogs(?) and other equally surreal paintings. As the decor would insinuate, this bowl of ramen (I got the special) was the best that I had ever had. If you find yourself near Redlands, CA, you should for sure check this place out.

Another night for dinner we found ourselves at a combination Indonesian restaurant / grocery store called Java Bistro in a town named "Rancho Cucamonga". I had thought that the name of this town was just an exclamation like "Jumping Jehosephats" until I saw it on a sign in real life. I had never had Indonesian food before and I had never been at a restaurant that was also a grocery store. My GF got a noodle soup quite similar to ramen but with a pickled egg and cream cheese rangoons. I got a bunch of stuff wrapped up together in a banana leaf. Both of our dishes were actually really nice, though the meatball soup was skipable.

Long Beach

Aquarium

We went to California for a comedy show but we had more than just one day in the state. I took this opportunity to link up with a couple of old friends that I hadn't seen in many years. One of these good friends treated us to the Long Beach aquarium. This really became the highlight of the California trip and I remember it way better than the comedy show.

Taking pictures of fish in an aquarium is actually difficult becuase you're photographing dark things through an inch of dirty glass. As a result, all of these photos are blurry, poorly lit, noisy, and with a lot of chromatic aberration. Consider it part of the aesthetic.

Intellectually, I know that coral, starfish, and sea anenome are all technically considered "animals", but they are all serving major "plant" vibes. They move so slow and without any volition that they might as well be plants. None of them have brains and they can't fight predators. I feel like venus fly traps are closer to animals than these imposters who share a kingdom with us. I like the long stringy starfish. They look like they taste spicy.

The acquarium also had birds, which are definitely for sure animals. If starfish are technically animals then there should be a separate family of organisms called "actual animals" that can look around and make sounds. Birds are some of these "actual animals". These birds were (known for their work in the air) at an acquarium (known for water) because they are birds with a particular affinity for water. I assume that there were no ducks present because they aren't considered "novel" by most of the North American population. They should get some ducks. Some seagulls would also be fitting, since they have "sea" in the name and that's pretty wet.

Alright enough messing around. Acquariums are for fish. Everything else is tangential or just a way to boost ticket sales. Close your eyes and think of a water creature. You thought of a fish, don't lie. The Long Beach aquarium had a wide assortment of fish. They had colored fish, plain fish, fish with eyebrows, fish with fangs, long fish, vertical fish, and even forgetful fish. My favorite of these fish is the one in picture #4 above, who looks like he's popping up to say hi.

Almost but not quite fish, the aquarium also had giant swimming blankets, seals who had escaped the club, a disturbingly girthy eel, and one of those creatures who will teach us how to become immortal if we can just stop them from going extinct for a little while longer. Axlotls rock, and I had never seen one before.

Jellyfish are named fish but they're just barely more animal-like than a starfish. The only reason that jellyfish get ranked up in the animal spectrum is because they move on a timescale that you can notice if you just look at them for less than a bazillion hours. These jpegs are oriented in the direction that they were shot on my camera. The jellyfish were actually swimming sideways or upside down. I wonder what causes them to swim in different directions.

Octopi are very animals. Not only do they move around quickly (really super quickly in fact) and engage in combat, they're crazy smart. Octopi disassemble stuff and mess with people for fun. This octopus was trying to show me its butthole. I know that the suction cups of an octopus are supposed to be extremely painful, but they look quite soft.

I've been to aquariums that have water insects before, but I'd enver seen crabs that were this pokey. It looks really cool that they have so many horns and I hope that they're not just decorative but also used for useful purposes. Picture #3 above looks like a fake paper mache or plastic model of a crab, but I promise it was real. I saw it move around. Very animal.

Frogs I guess technically count as water animals, but only as much as birds do. Some of these frogs are poisonous which is crazy if you think about it. Just touch some slime and you get to die an agonizing death.

This is actually a 3D model of a shark with a USA flag texture applied to it. There is not an America Shark in nature.

The Rest Of Long Beach

After leaving the aquarium, I walked around the rest of the immediate vicinity with my gf, my friend, and a couple of his kids. I took some pictures that I think are intriguing, though it's hard not to be intrigued by such captivating architecture. The last three pictures just so happen to have similar coloration to this very website, and they are also my favorite images from the bunch, likely in large part due to their colors.

My buddy told me that this area is where Formula One races happen from time to time, which threw me for a loop. Sometimes I forget that things like that are actually real and not just in videogames and movies. It's like seeing Big Ben for the first time. Yeah you've seen pictures and you might know that it's real in your brain, but you haven't actually been there and thought "yeah here it is".

Image #4 has a steak knife in a potted plant. Californians aren't allowed to have guns so they have to scatter melee weapons around the place just in case they find themselves in a self-defense situation.

There were a number of street vendors in the Long Beach area around the aquarium. It seemed like a sizeable tourist destination. The grey-market copyright-infringement booths fusing popular culture and poncho aesthetics seemed so odd to me. It didn't feel like real life. Southern California is a strange place.

Look at how cool this dude is. He's pushing a stroller while also rollerblading around town in a tank top with sunglasses on. I can only hope to someday reach such levels of cool.

A CAve

The final CA for this blogpost was a cave that we explored. Specifically it was Grafitti Cave in Yakima, WA. An outing like this typically isn't the kind of thing that I put in a travelogue, but I took some pics that I relaly like and I can put whatever I want on my blog since it's my blog. Also the name is better with a third CA.

Grafitti cave is in a nice area of highway 97 with plenty of parking, and the cave itself is easy to get to. It's named grafitti cave for two main reasons:

There is a lot of grafitti
It is a cave

I understand that there is a certain "code of honor" among grafitti artists to not cover up something unless you are going to do better. At places like this though, it is easy to see that this rule is disregarded. I suspect that this is a result of locations like this being encourage for newbies, who do not have the respect for the art of others quite fully developed yet.

Here are a few of my favorite pieces from this exploration. MRSA is a sizeable name in the Seattle grafitti scene, so seeing them out here wasn't exactly a surprise but it was cool. I appreciate the color fading on Sonic's hair and the broken heart in the cracked pavement is very "wow this is deep" but I like it anyways.

Aside from the grafitti and the cave, this area of Washington is quite scenic and is worth looking at in its own right. I often have difficutly successfully communicating the scale of scenery and distance in photographs that I take, but I think that I did a good job of it in image #4 above. Perhaps the trick is to have a human form as reference, since our brains instinctively know how big a human is. In the final picture above I found a DOTCOM tag. DOTCOM was one of the most prolific Seattle artists who was busted when the police brought down MSP.

I also got to get some cool shots of my gf in the cave. I feel that these are "artsy". The final one is actually a trick of the light. I wanted it to look like her flash light was reflecting off the rocks and lighting her up. In reality, that is the last remaining rays of sunlight filtering through and illuminating the front of her body.

I promise that I really legitimately did not see this sign until we were on our way out. There are at least three entry points onto the trail that leads to the entrance of the grafitti cave, but only one had this sign. I'm sowwy mister government I promise I won't do it again :^)

PUNKTO | A Long Journey

Thu, 16 May 2024 00:00:00 PST

A Long Journey

This Blogpost Is Different Than The Others

Most of the blogposts on my website have been technical writeups. This post is not a technical writeup, it is more of a travelogue interspersed with self-reflection. Between 4/26/24 and 5/13/24, I travelled across the US and back, from one coast to the other. Summing up the miles (7662), I travelled nearly 1/3 the circumference of the Earth. These miles were driven in a 2022 Toyota Prius by

Myself
My girlfriend
OpenPilot

OpenPilot did the vast majority of the driving, with one of us manually taking over for only the trickiest driving through construction and inner-cities. It really is a wonderful piece of software.

I am writing this post in part for posteriety and in part to force myself to fully flesh out some of the related ideas that are bouncing around in my head.

What Would DRIVE Me To Do That?

The reason that I went on such a voyage is because I had speaking engagements at two different conferences, each just a week apart. On April 28th I spoke at LinuxFest Northwest in Bellingham, WA. On May 5th I spoke at LibrePlanet in Boston, MA. These two conferences are on opposite ends of the continent. The two presentations that I was giving had a plethora of differences:

subject matter
duration
target audience
venue
goal

Most people would not have done what I did. Most people would have flown from one venue to the other. Most people would not have signed up to speak at two conferences so close. Most people would have used the same talk and slide deck for both conferences. What lead me to do this?

I drove because I didn't want to fly. This desire is lead by a few different factors:

I do not like flying. The actual act of the airplane going up and down isn't so bad. I struggle with the security theater.
Both my girlfriend and I wanted to see some sights on the way across the country that we would not otherwise normally be able to see.
It is somewhat fun to go on a road trip.
I am capable of going on such a road trip, financially and professionally. Smoke 'em if you got 'em.

I signed up for conferences so close together mostly on accident. I had originally submitted a talk for LFNW 2023. My talk was accepted, but the conference was unfortunately cancelled last-minute. To get my free-software conference "fix", I proceeded to submit a different talk for for LibrePlanet 2024, which was accepted. When LFNW 2024 was announced, I was invited to re-submit my presentation, but that was after I had been accepted for LibrePlanet. I didn't check the dates carefully enough and only noticed just how close they were when the conferences were drawing near.

I submitted different talks for the two conferences because the conferences had different CFPs and my submission for LFNW wasn't appropriate for LP. LibrePlanet is a Software Freedom conference, with the primary focus being on the politics and culture surrounding the base philosophy. LinuxFest is a Linux confernece, with the primary focus being on any tangential subject. LFNW focuses more heavily on technical matters than does LP. Additionally, I wanted to have a greater variety of talks so that my Trophy Case would look more impressive. I used my LFNW presentation to talk about the work that I get paid for on a daily basis and I used my LP presentation to talk about some of the work that I've done here on this blog.

Bellingham

Every time that there has been a LFNW, it has been in Bellingham, WA at Bellingham Technical College. Bellingham is a wondeful little city and every time that I go there I am struck by how nice it is. I would like someday to take a prolonged vacation in Bellingham so that I can fully immerse myself in the area. When LFNW 2023 was scheduled for the Fall of 2023 and my talk was accepted there, I did book an AirBNB for the a whole week, but with the change in conference date and obligations, I wasn't able to spend as long as I would like in the city. Nevertheless, I did still have a good time.

After checking in to our hotel, we made our way down to the Fairhaven district of Bellingham. Fairhaven was one of the original 1903 settlements that consolidated together into the city of Bellingham, and it is a very noticably separate district. It feels like a separate small town, a short drive away from the main downtown Bellingham. Today it is mostly a tourist destination, and the streets of Fiarhaven are littered with quaint shops and eateries.

I got a cone of ice cream in Fairhaven at a location called ACME Valley Ice Cream. The ice cream here was unique when compared to any other ice cream that I have had throughout my life. It was chewy. They advertise it as "full strength ice cream" and claim not to inject any air during the manufacturing process. It was a real treat and will sit apart from all other ice cream in my memory going forward.

Fairhaven is also home to Village Books, a bookstore spanning 3 floors and including a cafe. It is also attached to a restaurant. Aside from these details, there is not much that sets this bookstore apart from most other bookstores in the world. They have a large piece of blown-glass art hanging up in a gigantic window and the normal assortment of gifts and trinkets that you are likely to find in stores that ostensibly specialize in books. I have been to this bookstore on previous trips to Bellingham and my family has taken to calling it the "Harry Potter Book Store" due to its winding staircases.

Other locations unrelated to LFNW that I went in Bellingham were:

A Mexican grill in Fairhaven that I did not care for. I had once been there 7 years ago with a friend, but the food was of poor quality and preparation.
A sushi joint attached to a movie theater. They were about to close and so the selection was quite limited, but they had a giant Buddha statue in the center of the restaurant made of pure brass.
A restaurant specizlizing in waffles. They had both sweet and savory options available, and I went with savory as it was quite late in the day. They include a small plastic dinosaur with each order and have dinosaurs decorating the entirety of the restaurant. The interior decoration is definitely catered towards the college crowd.

All of the other activities that I did in Bellingham were directly related to LFNW.

LFNW

LFNW always starts off with a game night on the Friday night just before the Fest begins on Saturday (on 'Fore-Fest Friday). This game night definitely did not disappoint, the organizers did the most elaborate and involved game night that I have ever seen at LFNW. In the past, a swarm of open-source nerds have gathered in a large room with free pizza and stacks of board games to engage in cut-throat rounds of Ticket To Ride and Settlers of Catan. This year, however, the game night party spanned three locations.

It started at a bar. Finding the LFNW section was a bit difficult due to the entire road being blocked out for a massive party, but once I found the LFNW section at the back of the bar, I had the opportunity to "talk shop" with some other attendees and reminisce about how nice it was to finally have such an event in person again. After the attendees were sufficiently drunk, the party moved a few blocks away to a barcade so that they could drink some more. This place was full of pinball machines, which was great because I love pinball (although my skill level does not match my enthusiasm level). After a few rounds of bouncing a steel ball around a cabinet, the party expanded next-door to a board and card game shop. In here there was the typical plethora of baord games to be played by anyone who could find a partner or team. Simultaneously there was a game of LFNW trivia in the style of Jeopardy being played by four teams. Each team had a button at their table to push when they knew the answer. These buttons were wired to the GPIO ports of a Raspberry Pi for the purposes of knowing who was first. The RPI was running some custom-written terminal-only software to figure out which team had pushed the button first, and it was plugged in to a large TV. The failures that you can imagine with this setup occured and provided some nice comic relief. I did not participate in this, however, and instead learned to play Stratego with my girlfriend while our table-neighbors played a Star Wars themed chess set where they had to decide which pieces filled which roles because none matched and there were no instructions.

The fest-proper began the next morning at BTC, as it always has.

This was the first full LFNW back in person since the pandemic. They had originally scheduled it for Fall of 2023 but had to cancel because the building that LFNW has historically used was closed for renovations. The organizers put on a mini-fest in Fall of 2023, but I did not attend because I had SeaGL obligations instead. During this proper LFNW 2024, the original building was still closed for renovations and so the main expo hall was instead hosted inside of the building that has previously been the cafeteria. The cafeteria is notably smaller than the old expo room, but this turned out to be fine, since there were far fewer vendors in the hall than there have historically been. I could definitely feel that this was the first full fest back, as many old portions of the conference were missing.

The signage was poor, which made finding the correct buildings and the location for the JB-provided lunch quite difficult without asking someone for help. It is good to drive community interaction, but a map or better signage would have saved quite a bit of confusion for myself and others. Thankfully, the LFNW audience is relatively high-trust and collaborative, and so once the knowledge was made available to one person on where everything was, it quickly became disseminated throughout the attendee population.

There were no badges this year. In previous years, LFNW has provided badges that have been extremely helpful, including a printed schedule and map as well as information about the different vendors. The badges have previously also helped to signify different people as attendees, vendors, speakers, or staff. Finally, the badges have been wondeful souveneirs that last well after the end of the conference. One year LFNW even had plantable badges which were made of recycled paper and had embedded within local wildflower seeds. The lack of badges was sad and I hope that they can return next year.

Attendance to this LFNW was lower than any previous LFNW that I have been to. This is of course understandable, it was the first full fest back in-person. People need to remember that conferences exist and are good fun things to go to. I hope that in the coming years, LFNW is able to repopulate back to its former glory and be a bustling center of software freedom in the PNW.

The vendors this year, as I mentioned before, were fewer than in previous years. Some old vendors who have been consistent across the years did still show up, notably Jupter Broadcasting, PostgreSQL, the FSF, and Candela Technologies. There was also a new vendor that I had never heard of before, Gadfly AI. Upon seeing the name, I was immediately skeptical that they would be worthwhile to engage with, since anyone who puts AI in their company name is likely trying to grift their customers. However, they were distributing some zines which advocated for giving Richard Stallman a voice. RMS has been discriminated against recently by neurotypicals who do not understand how to communicate with neurodivergent people. As a result, he has become an uncouth name to mention and an unpopular person to support. I hope to see more of Gadfly AI at future conferences and I wish them success and plenty in their grifts. I hope that they take all of the VC money that they can get their hands on and use it to further the goals of software freedom.

In addition to the Friday pre-presentation-party, LFNW has always held a party on Saturday evening. Sandwiching more conference-sanctioned drinking between the two days of the conference is a tradition that is well-received by the attendees. I went to the restaurant where I had a nice dinner with some strangers who I have seen from afar in previous years. It was a good time. Instead of following next door to the pub when the dinner venue closed, I went back to my hotel room to prepare for my talk.

My LNFW Talk

My talk at LFNW was decent. I'm happy that I gave it, as I did a good job of presenting Modules to people who had never heard of it before. The recording of my talk should serve as good introductory material for future users and administrators of the software. There were some shortcomings with my presentation, though, which I am accutely aware of.

There were some issues with my talk that were caused by the organization of the conference. These are worth mentioning, but not to cast blame on the conference organizers. My goal here is not to say "you messed up my talk!". It was my talk and the shortcomings of it are my responsibility, the blame is entirely on me. My goal here is to record my experience and to hopefully help future presenters in setting their expectations when speaking at this conference or similar conferences.

Firstly, I seemed to have a shrinking timeslot. When I originally submitted the talk for LFNW 2023, I submitted it for a 45-minute timeslot. When I submitted the same talk for 2024, the only option was for a 35 minute timeslot. On the final published schedule, I was given 30 minutes. The time difference between when my talk started and when the following talk started was 2.5 hours. This lack of consistency confused me, and when I reached out to conference organizers for clarity, I received no reply. I had no idea how much time I should or could go for, so I was forced to attempt to reduce a talk that was originally 45 minutes down into 30. As a result of this, I spoke too fast and rushed through some of the content. I didn't feel that I had the time to give the talk that I had originally planned to give.

Secondly, my talk about a special piece of software that most people haven't heard of was scheduled at the same time as a talk by an esteemed security researcher about how he (and his team) found a large CVE in MySQL. This latter talk is obviously more interesting and most of the conference-goers wanted to see that talk instead. I can't fault the conference for scheduling us at the same time - that's just the way that the schedule worked out. As a result of this, however, my talk was poorly attended. This left me nervous throughout the presentation. This is actually a shortcoming of myself, however. I should have named the talk something a little more click-baity and I should have been fine with a such a small turnout. This is all in hindsight. Perhaps someday I will give the same talk with a snazzier name.

Thirdly, I feel that the room that my talk was in was quite difficult to find. It was on the second floor of a building with poor signage on the outside and no signage on the inside. The hallway that you had to walk down to get to the stairwell was completely unlit. I think that this may have convinced some people that they were in the wrong place. It was not a welcoming sight and even I was skeptical that I was in the right building, despite having scouted the location out on the day before:

The shortcomings of the conference were things that I could have adapted to if I had been better prepared. I should have provided my own signage to show people which direction my talk was in. I should have named the talk something more enticing. I should have been calmer in the face of the unknown timeslot and a low turnout. These shortcomings of the conference are things that I could have shored up if I had done better. In 10 years, the recording will survive, and it is not effected by any of these conference issues, only by how I reacted to them.

The feedback that I received from my friends and family who have watched the presentation has been universally positive. They say that I am being too critical of myself. I believe that they are viewing my performance through rose-colored glasses and that the only way that I can improve is to know what I have done wrong. This way, I can be more cognizant of not doing those things in the future. I want to address the issues that I brought to the presentation. They vastly outnumber the issues provided by the conference, and in the long run they are the only ones that matter. A recording of my talk can be found here. I have noticed these problems that came from me:

I spoke too fast during various parts of the talk. Mostly I did this when I was going over something that the immediate audience was very familiar with. I didn't want to bore them, but moving so quickly through this content does a disservice to the recording, where the audience for the rest of time might not be as experienced.
I failed to properly enunciate my words. Even with the very nice microphone provided by LFNW for the recording, some of the things that I said are difficult to make out in the video.
I made motions with my hands. This is good because it is better than fidgeting with them or putting them in my pockets. However, my hand motions usually had absolutely nothing to do with what I was saying. At various points in the video I am just waving my arms around without any point to it. It would have been better if my hand motions helped to illustrate points.
I had no wireless clicker, which meant that I had to keep walking back and forth between the laptop and the screen to progress through slides.
My slides were fine for a large projector and screen, but for the poor projection quality provided by the projectors, some of the text inside of screenshots was too small.
My projected image had a green tint over everything. I think that this was an issue with the laptop that I was using to present (a burner laptop from Costco) and it caused the whole slideshow to look slightly off.
I did not know my slides well enough. I had only done two rehearsals of the presentation and at one point I had to take a minute to go back because I had gotten ahead of myself and lost my place.

Driving From Bellingham to Boston

After LFNW was over, it was time to hit the road. I had a long way to go, and a short time to get there, needing to travel from Bellingham, WA to Boston, MA by car in just under 5 days. To make things a little more challenging, there were sights to see and people to break bread with along the way. Mostly though, it was a lot of driving on one road (Interstate 90) from one side of the nation to the other. I-90 is the longest interstate in the USA and I drove (or was driven) over the entire thing. In order to fully appreciate the journey, you should know about the sleeping arrangements.

Catching Z's

Hotels are expensive and run the risk of bed bugs. They have draconian check-in and check-out times and their continental breakfast is universally forgettable. Additionally, it takes a good amount of time to check in and out of a hotel. We were in a rush and did not want to deal with any of this. Fortunately, I own a 2022 Toyota Prius. This is one of the best vehicles for this trip if you are under 6 feet tall. This vehicle is compatible with OpenPilot, so it mostly drives itself, greatly reducing the fatigue load on the driver(s). Additionally, with the rear seats folded down and custom mattress pads made by myself and my girlfriend, it doubles as a rolling bed (this is where the 6 foot limit comes in) that uses less than a gallon of gas each night for climate control. Normally when people hear about this, their first response is "You're living in your car?!?!" and "You don't have to do that", but it is much nicer than most hotels. It may be helpful to think of it as a capsule hotel on wheels. This allowed us to make good time across the nation and saved a good amount of cash.

But while asleep, the vehicle had to be parked somewhere. It had to be parked somewhere where nobody would care. I have form-fitted sunshades for every window and extremely dark cermaic tint, so nobody can see inside, but a glance at the windsheild (with the sunshade visible) and hearing the engine occasionally start to charge the hybrid battery is enough to inform attentive strangers as to what is going on. In some states, you are permitted by law to do this at their highway rest areas. Washington was one of these states, and on the Eastern edge of the state is where we slept the first night, since we had started driving in the mid-afternoon, just after LFNW ended. Some states, however, do not permit you do sleep in your car at rest areas, or they limit you to something like 3 hours. New York is one of these, but despite the legal code being present to prevent it, there was no signage and we found that the law was unenforced. There were many drivers of RVs, campers, and vans who were obviously car-camping. Many places online will recommend Walmart, sometimes mentioning that you should get permission from the store manager first. This is old out-dated advice. Based on my phone calls with various Walmarts, they no longer allow this. Love's truckstops, however, had no such issues with allowing overnight parking. Even in cases where there were signs saying "1hr maximum parking", the store attendants confirmed that it was not an issue. The Love's bathrooms were always clean and they even had showers, which were very welcome after a few days on the road.

Between rest areas and Love's, we always found a place to sleep and were never bothered by security guards or police. This allowed us to spend more time with the wheels rolling and less time getting into and out of hotels.

South Dakota

We made several stops on the way from WA to MA. Our first major stop was Mount Rushmore, where we saw the giant faces carved out of a mountain and learned a bit about how it was done. This is an awe inspiring monument and I highly recommend that anyone see if if they are able. We took many cheesy tourist pictures here, as we did everywhere we went, and then hit the road. After leaving the mountain, we blasted through the state of South Dakota at an extreme speed, blowing past Wall Drug, Prairie Dog Town, 1880 Town, the Minuteman Missile site, and other things that we would have liked to see. We had a mission. We wanted to get to the world's only Corn Palace in Mitchell, SD, before they closed for the day. We arrived with about half an hour to explore the palace, but to our dismay we were blocked from entry. The palace was closed to tourism. They were holding their annual circus. I was crushed. Weeks of anticipation had lead up to this, I was so excited to see the palace, to go inside and to bask in the corniness of the place. The devastation of not being able to enter left a sour taste in my mouth as we drove away. We had ignored other locations to get to the palace, and it was all for nothing.

Leaving Mitchell, our next stop was in Souix Falls, SD for dinner. Unable to find anything that was open late and looked decent, we settled on a location with low hopes. We went to The Dive, a place that looked like a dive bar and called itself a dive bar. It was also right across the street from a college. In my experience of living and working near colleges, this signifies low quality and high cost. I am happy to say that The Dive proved me wrong on all counts. The burgers that we had there were reasonably priced and delicious. All of the sauces are made in-house and were unbeaten by any bottled sauce that I have ever had. Every aspect of the meal was done very well. This was a lovely little pick-me-up after the crushing let-down that was the closure of the Corn Palace.

Chicago

Our next stop was Chicago, IL. We had high hopes because there were several things in that city that we wanted to see. Not only were there normal tourist attractions, there were internet meme tourist attractions. The schedule wouldn't permit us to visit the American Football House in Urbana, but we could stop at the 100 Gecs Tree and the Chicago Rat Hole.

Security at the tree was tough. They didn't like that we were there and were yelling at us to leave while we were trying to find the exact tree. They were calling in backup (or maybe phoning in law enforcement) while I took my one picture. We sped away and weren't able to pose with the tree, failing to recreate the famous album cover.

It was somewhat difficult to find the exact tree. Most places online just describe it as being a tree of this type at a particular office building. This office building is surrounded with trees of this type. The exact tree is located at 42°01'49.0"N 87°54'44.2"W.

On to the rat hole, we had the exact coordinates for it in Chicago and street-parked nearby. We walked in the direction of the mark on the map, but could not locate it. After making several passes over where it is supposed to be, I checked the Wikipedia page again:

The Chicago rat hole was a hole shaped like a rat...

and

On April 24, 2024, the Chicago Department of Transportation removed the sidewalk slab containing the hole...

This is where the rat hole would have been if it had not been removed exactly 7 days prior to our arrival.

This was a bit of a disappointment, so we made our way over to something that will certainly not be removed or closed. We drove over to Millenium Park to see The Bean, known officially as Cloud Gate. This is a beautiful scultpure and provides amazing shots for photographers. My girlfriend was excited because she has always wanted to take fun photos with the bean. I was excited because I like beans. The bean was closed for construction on the plaza. We were again let down by closure of tourist locations. With one stop left in Chicago, we were almost ready to head out.

Our final destination in Chicago was Pequod's Pizza. This location is rated #1 in the US for Chicago Deep Dish Pizza, and it did not disappoint. Not only were we able to get a seat with no reservation, our pizza was made quickly and was delicious. The wait staff was helpful and kind in guiding us through what to request on/in the pizza. This was the highlight of Chicago, it was the only destination in the city that did not present significant problems.

Ohio

Leaving Chicago, the next destination was quite a distance away. We were headed for Cleveland, OH. I drove through the entire state of Indiana without stopping and found myself with an old friend. This friend, who I had not seen in many years, welcomed us to his area by showing us around. We visited the USS Cod, an old WWII submarine parked on the shore of Lake Erie in Ohio. This was a real treat. The submarine is beautifully preserved and the combination of literature and audio recordings inside are educational and entertaining. The men who operate the venue surrounding the Cod were friendly, helpful, and informative. They answered all of the questions that we had and were great conversation partners. They wished us happy trails as we went a block down the road to our next destination.

We wound up next at the Rock and Roll Hall of Fame. I don't care much about the history of Rock and Roll, but this museum was a spectacular treat for my girlfriend. She adored the exhibits and activities and we spent a good while walking around and appreciating the artifacts. What I found most interesting was that many of the outfits in the museum, which seem so perfect and larger-than-life in music videos and on stage, are not spectacularly made. When you get a chance to see them up close, you can see the seams and the design decisions that are obviously there to make performing in them easier, but that make them look worse than the imagination and the movie magic lets on.

Our final stop in Ohio was the gas station from the YouTube channel Gas Station Encounters. I have been watching the videos of this channel for years, and to finally be in the very location that I have seen on-screen so many times was surreal. If you don't know what it is, it would seem like just a regular gas station. But having seen it so many times beforehand made it special. The man working behind the counter even took our picture together in front of their golden Play Button. This was a real treat. I have been in many gas stations and have been disappointed by their wares, but this Marathon Gas location had an astounding selection of items and none of them were expired.

Boston

After leaving Ohio, our next stop was Boston. We woke up on Friday morning with many miles still left to go and a hotel to check into in just a few hours. Our hotel was The Verb, located right next door to Fenway Park, and we got there on Friday evening. This hotel was recommended by the FSF for the conference, and it was one the only hotel near the venue that actually had vacancy when I went to book it. This hotel was like a miniature version of the Rock and Roll Hall of Fame, with memorabilia everywhere and record players in each room. From the outside and the common areas, this hotel is great. Once inside the room, however, there were various things that did not work or were poorly designed. This wasn't so bad because we didn't spend much time in the hotel room except to sleep.

The night before the conference we did a little bit of driving around and sight-seeing, since we would not have much more of a chance later on. I got to experience being tail-gated while driving a reasonable speed through a 1-lane college alley with cars parked on either side in the dark on a Friday night with drunk college kids everywhere. After this trip, I understand why the drivers of this state are called "Massholes".

LibrePlanet

LibrePlanet started the next morning. LP had badges, and mine even identified me as a speaker. That was quite exciting. As far as attendance and vendors, LP was actually smaller than LFNW, which was surprising to me. I had expected it to be much larger. The conference was also organized a bit differently. There were two rooms for in-person talks, but if you wanted to watch anything from the third track, you had to tune in digitally. All the talks of the third track were presented remotely. This made it easier to pick which talks to attend, since at any moment there were only two in-person talks to go to. This conference had great signage and I never was lost or confused. In many ways, it was a better conference than LFNW.

LFNW did beat LP in festivities, however. I think that this may be due to a difference in drinking and party culture on the East vs West coast. The main activity from LP was a historic walk through Boston at the end of Saturday. This walk was lead by the FSF and took participants through many historic locations with significance from the revolutionary war and Boston's early history. During this walk, we were given trivia questions about the FSF with the promise of a prize for whoever got the most questions correct. Additionally, the FSF equipped some of the attendees with protest signs reading "FREE YOUR COMPUTER". I held one of these during the walk and it sparked some conversations about software freedom with strangers who were passing by on the street. I even managed to start a chant with a drunk man in the backseat of a cab.

During Saturday at the conference and during the walk thereafter, I am happy to say that I made a friend. An elderly blind man named Dave needed some help finding his way between sections of the conference, so my girlfriend and I lended him a hand. Or more appropriately, we lended him an arm. Dave's preferred method of guidance was to take someone's arm and follow them that way. This worked great, and he even joined us for the walk. It was very good that Dave joined us for the walk, too. As a local Bostonian, he was able to, without even seeing where we were, tell us all the history and significance of the points that we were passing on our journey. Without this, we would have seen various places but we would not have fully been able to appreciate them, as the FSF guide did not provide much educational information during the route.

When we got to our final destination on the walk, the historical Quincy Market, I popped off to the restroom. When I came back, all of the restaurants were closing and the FSF leadership had disappeared. Perhaps I missed something while I was peeing, but there were a few other attendees at the market who were just as confused as I was. This stung a bit, but I did manage to get a large quantity of Chinese food for $7 as the restaurant closed. Eventually everyone was kicked out by security and we rode the train back to the venue to drive back to the hotel. I am glad to have gone on this walk and to have made a friend, but I feel that the execution on the FSF's part could have been better. I hope that they continue the tradition in coming years with improvements. This walk has the potential to become a lovely experience to tourists and non-locals.

Getting back from the walk very late, we had just enough time to eat and fall asleep before waking up the next morning halfway through the opening keynote. We made our way to the second day of LP, but I didn't see many talks that day because I was busy preparing for my own presentation.

I saw some wondeful people at LP. I can't write about anyone without mentioning Richard Stallman, who hardly needs introduction. This was my first time meeting him in person, and he is just as pedantic and rude as the Internet makes him out to be. A lot of criticism against him comes from neurotypicals who are unaware of how to interact with neurodivergent people. I got to ask him a question during his Q&A and I even got him to sign a book for me. This will always be a cherished memory. I also saw someone who looked extremely similar to Frederick Brennan, but I wasn't 100% certain that it was him and I thought it would be impolite to ask, although he does now live with family on the East Coast, so it could well have been. I also saw Ian Kelling, the FSF sysadmin. He is originally from the West Coast and we have met before. It was wonderful and surprising to see his friendly face again, this far away from home.

My LP Talk

My talk at LibrePlanet was leagues better than my talk at LinuxFest. The issues with the venue from LFNW were not present at LP. The FSF provided laptops to do presentations on, which removed any room for issues with presenter equipment. I wasn't scheduled at the same time as a blockbuster, and they even put me in the big room on the massive stage, the same stage that the keynotes were given from. My only regret with the location is not specific to my talk. The elevator in the building where I was speaking was out, which made it impossible for some people (RMS, Hotwheels, and another person in a wheel chair at least).

I also improved on many of my mistakes from LFNW. I rehearsed twice as many times for my LP presentation as I did for my LFNW presentation. The title was more enticing. The subject matter was more interesting. I spoke slower and more clearly. I knew my slides like the back of my hand. However, I made some new mistakes. 35 minutes into my talk, I misread the time and thought that I was going over. I had a 45 minute talk and had planned to use the whole 45 minutes. Thinking that I was 45 minutes in to the presentation when I was actually 35 minutes into the presentation, I assumed that I must have been going to slow and sped up to compensate. As a result, I missed out on making a few points and saying a few things. Nevertheless, the presentation ended well (and on schedule) to applause from the crowd. My other mistake was with my hands. Instead of waving them around meaninglessly, I spent most of the talk fidgeting with my badge lanyard. I'm not sure which hand choice is worse, but the optimal hand choice will always be to use them to illustrate points.

Unfortuantely, immediately after my talk, the room monitor came up to tell me that there had been an issue with the recording. A 3 minute segment in the middle had been lost due to a disk error. This was fine, he said, as I was to go to the other room of the conference and re-record that segment. The three clips could be spliced together seamlessly in post-production. However, when I made it to the other room, I learned the sad truth. Only the last 15 minutes of my talk (incl questions) had been recorded. The whole front portion of my presentation was lost to time. This was devastating, but thankfully my girlfriend and my good friend in the audience were each independently recording the presentation. We shuffled these files around and eventually got everything turned over to the FSF. Hopefully they will be able to splice together something workable from all the segments.

If you have recordings of any talks from this most recent LibrePlanet, please send them to the FSF so that they can make the recordings available. I still have my slides and at least 1 full recording of the talk. Although the audio is poor, I believe that I can make something workable out of this recording if the FSF is unable to. I await their published version or additional word from them.

Homecoming

Leaving Boston, we did not travel straight home. Instead, we went North a ways to spend time with some old friends who were living in New England. They prepared for us delicious fresh lobster, let us hang out with their pets, and showed us around the area. We ate good food and saw beautiful sights. I got to see an absolutely massive library and eat something called "breakfast pizza", a genre of Italian food that I didn't previously know existed. Saying our goodbyes, we set off to return home via a slightly different route than we took to get there.

Niagara Falls

Our first stop on the way back was Niagara Falls. These falls are a specacular thing to see. Photos and videos don't do it justice. To truly experience it for yourself you have to stand there. We didn't get on a boat to get close to the falls due to time constraints, but we did get to go down to the Cave of the Winds and touch a neighboring waterfall. One thing that did strike me while standing there was that the best view of America's greatest waterfall is actually from the Canadian side of the river. I feel that there is some poignant metaphor here and I hope to be able to tease it out eventually.

In the town of Niagara Falls, we stopped at what looked like a run-down submarine sandwich shop, but the sandwiches were anything but run-down. Viola's Submarine Shop does amazing sandwiches quickly for an affordable price. If you are in the area, look past the exterior and pop in for a sub. It is well worth it. The sandwiches are made fresh and the filling is grilled right in front of you. The people behind the counter are friendly, although they are Italian and so it may be difficult to tell at first.

Columbus, OH

Columbus, OH was our next interim destination. We went to meet up with one of my girlfriend's old friends who had moved to the area. We went around the city a bit and got some ramen for dinner, then we found our way to a book store with 32 rooms. Libraries and book stores, as you maybe can tell, have a certain draw to me and my girlfriend. We walked around this one for a couple of hours, looking at books and catching up with an old friend. I even bought a couple of books, which is rare for me. Usually I just take pictures of books to download later on libgen.

That night, before going to bed, we went to the Field of Corn in Dublin. There were multiple purposes for this stop:

It's cool
My friend from Ohio, who goes on about corn all the time, has never been. I got to beat him to it
To take sp00ky pictures at night

The next day we were still in Columbus and we were not going to let such an opportunity go to waste. I had never been to a Waffle House before, and had always wanted to go. Since there was one quite nearby where we had slept that night, we went for breakfast. The fame of the chain did not leave me disappointed. I wasn't expecting the best waffle that I had ever had in my life, but I was quite surprised by how good the food was, given that it is a 24/7 fast diner chain known for being run-down. After the Waffle House, we went to Columbus's historic North Market. This was a bit of a disappointment. It was just overpriced food vendors and one shop that sold AliExpress-sourced stickers and postcards. If you are in Columbus, the North Market is very skip-able, you won't be missing out on much.

Next, we went to the Wright Brothers Memorial in nearby Dayton. This placed us on top of a hill overlooking some historic locations in the development of flight. Although the location of the first flight was far away in Kitty Hawk, this location included a beautiful park and wonderful views. The building for the Interpretive Center, however, was unfortunately closed for the season. Departing this memorial, we made our way to our last stop in Ohio, the Honda Heritage Center. This was a real treat, since they are only open for 5 hours on Thursdays, and we just happened to be in the area during those 5 hours on a Thursday. I'm more of a Toyota fanboy myself, but I can certainly appreciate the engineering and cultural impact that Honda has had. They make cool cars and motorcycles and it is hard to live a life in the USA that has not been effected by Honda in one way or another. The Heritage Center was magnificent. The old vehicles preserved as museum pieces were all in immaculate condition and there were not glass walls preventing me from taking good pictures of them.

Columbus, IN

After Columbus, OH, we travelled to Columbus, IN. Or as I have recently started calling it, Columbus 2.0. Columbus 2.0 is a small town in the middle of nowhere in Indiana and it has some of the most beautiful architecture in the country. There are other places with nicely designed buildings, but they are not as densely nice. They are also much more populated than Columbus 2.0. Columbus, IN is where the headquarters of Cummins is. This the same Cummins that builds the high-reliability high-power motors that go into a great many trucks. The founder of Cummins is from Columbus, IN, and liked architecture so much that to this day the company still sponsors new architecture developments in the city. The man who designed the public library in this small town also designed The Louvre. We walked the streets of the city on a self-guided tour via their app for a couple of hours and appreciated a great many historic and beautiful buildings.

Columbus, IN is also home to a soda fountain museum. Soda fountains have fallen out of favor in modern America, but they were once a large part of public life. Being able to see so many soda fountain machines and read a bit about their history was fantastic and brought a smile to my face. I do wish that artisinal soda would become a bigger deal in America again. The PNW focus on microbrews leaves me (sober) out of the loop at many parties and events.

Columbus 2.0 also brought me a small surprise. A friendly man on the sidewalk, going in the opposite direction of us, handed me his business card, told me that he does stereo photogrophy, and then walked away. On the back of his business card was this YouTube link. I implore you, if you are reading this blog post, to open up the linked video and watch it from start to finish. This man is an absolute genius, although his presentation is a little bland. Previously responsible for making sure that nuclear bombs don't blow up on accident, he is now responsible for making sure that Cummins engines don't blow up on accident.

With beautiful architecture, interesting museums, and eccentric geniuses, Columbus 2.0 has much to offer. I wish that we could have spent more time there, but the schedule did not allow for it. We had to be off to our next destination, the largest truck stop in the world.

Iowa 80

Iowa 80, on Interstate 80 in Iowa, is the world's largest truckstop. It is home to a dentist, a chiropractor, a hairdresser, all of the trucking accessories that a big-rig driver could dream of, a music store, and every kind of trinket and doodad that you have ever seen at a gas station in your life and wondered "why would someone buy this?". It is truly an amazing feat of American consumerism mixed with our tendency to build massive infrastructure in the middle of nowhere.

Much more interesting than this truckstop, however, is the Iowa 80 trucking museum right next door. This is a museum that offers free entry and sports an aircraft hanger full of vintage and beautifully-restored trucks. In this building you can get the entire history of trucking from the humble beginnings of local deliveries to the modern day as the defacto primary method for moving goods over long distances. You will see the journey that consolidated many different historical trucking companies under shared umbrellas and you will get to see some beautiful pieces of artwork in the form of truck bodies. This place is a sight to behold and well worth a stop if you are passing through.

South Dakota, round 2

We got cockblocked by South Dakota the first time, but it would not happen again. Passing through the second time on the same road, but travelling in the other direction, we hit all the spots that we wanted to hit the first time. First off, we went to the Corn Palace and there was not a circus going on, so we got to stop in and appreicate the detailed history of corn palaces and the beautiful murals made entirely of corn. This lived up to the hype for sure. Up next was the Minuteman Missile Silo. While we didn't get to see any actual missiles, there was much educational material on nuclear armaments going from their inception to the present day. An important historic site for our nation, this was a cool stop. Leaving the missile site we went 0.5 miles down the road to a little place called Badlands Ranch Store, where we got up close and personal with prairie dogs and took many photographs. We did not touch them, however, because I had been informed that they carry the black death. I have now been informed that they don't actually do that and that touching them would have been fine.

Our last stop in South Dakota was the famous Wall Drug. If you are ever considering going, skip it. This is a tourist trap to end all tourist traps. There is nothing interesting there. The food is all bad and the entire joint is one big overpriced gift shop with dilapidated plastic attractions.

While we didn't stop for the natural ancient caves or the 1880 Town because we were in a bit of a rush to get home, these weren't huge draws for us in the first place.

Closing Thoughts

After leaving Wall Drug, we didn't stop except to sleep and eat until we made it all the way back home. I was getting homesick towards the end of the trip and it was with a sigh of relief that I finally got back to my own bed. I think that this duration of trip is the limit for what I can comfortably do in my car, and in the future if I ever present at LP I will seek to fly.

I am approaching a time of great changes in my life. At work, my IT director is retiring and we are replacing him. I hope that the new director will be just as wonderful to work for as my current one, but only time will tell for sure. It is possible that I will be on the job hunt sooner rather than later.

The reality that I have grown accustomed to over the last 3 years is being shaken. On Monday I have a very important appointment, and if it goes poorly then I may as well be dead. If it goes well, then we are so back. If this is my last post on this blog, thanks for tuning in. Put this on the Internet Archive or something before the VPS expires.

If I make it to the other side of springtime, I have much that I want to do. There are many seeds knocking around in my brain, waiting to grow in to new conference talks, new blogposts, new software projects, and new opportunities.

PUNKTO | Imposter Syndrome

Wed, 20 Dec 2023 00:00:00 PST

Imposter Syndrome

I have been full-time a Linux user for about 10 years now, and a professional Linux user for most of that time. I say "professional Linux user" because I feel it is more descriptive then the long list of different job titles that I've held where the responsibilities basically boil down to being really good at the commandline. One thing that I have noticed is that a lot of people who wind up in this position develop a sort of imposter syndrome. I'm not entirely excluded from this list - I have felt like I was just pretending to know what I was doing. However, I don't really feel that way anymore and I think it might be valuable to look back at a decade of experience and re-assess the things that changed my mindset from then to now.

Everyone is pretending (tech)

One job that I worked had me helping on a big project to replace the webapp that ran basically the entire company. It had all the bells and whistles and did everything:

Customers interfaced with this tool
Employees used this tool as critical parts of their workflow
This tool handled compliance with various government bodies and regulations
Payroll was processed by this tool
This tool allowed us to interface with other companies in the same industry
This tool acted as a central print server for some reason

The tool in question was an off-the-shelf offering from some external company that sold it to basically everyone in the industry. It required a lot of infrastructure. Most of the servers that I worked to build and maintain at the time were primarily used to support this one tool. We had IT people in three different teams supporting different parts of this tool. When the time came to replace it because the software was reaching end-of-life, we didn't have the manpower, knowledge, or bandwidth to do it ourselves. Sensibly, we hired the company who designed and sold us the software to help us with the migration.

Given that these outsiders were going to be signing into our production systems that handled all sorts of vital information, management asked if there was any way that we could keep an eye on them. Having just finished reading Sudo Mastery by Michael W Lucas, I suggested that we could record their login sessions with sudoreplay and a few setting changes. I made a couple of changes to the sudoers file, the ~/.bashrc for the profile that the developers would log in as, and I wrote a friendly little script that would allow me to view their sessions based on various different sort criteria. When they started to log in and do their work, every keystroke and every character printed to the terminal was faithfully recorded. Looking back at my work now, it was very janky. The bashrc would first check if it was running inside of sudo, and if not it would start a new shell with sudo. Sudo was configured to allow the user to do this without asking for a password. All of the replay files got stored on an NFS (No Fucking Security) share that was accessible on all of the servers. However, for all its faults, my script did one very important thing: it got the job done. This is always the most important thing in computers. Code quality, infrastructure reliability, and user-friendliness are all important, but nothing trumps functionality. Functionality is the gold standard. If some tool or setup doesn't do the job that it needs to do, then it is useless. If it does the job while being poorly designed, it is better than not doing the job while being well-designed. I was only pretending to know what I was doing, and I did a job that I would now consider sub-par. However, I got the job done, and that was the most important part.

What I saw in the replays shocked me. The vendor employees, the people who worked for the same company that developed the software in the first place, the people whose primary job is to get rented out for these exact migrations, had no idea what they were doing. I watched them display the lowest level of general Linux knowledge and the most copy-pasting that I had ever seen in my career. They would run the same command multiple times, changing nothing, and see it continue to spit out the same error messages. They would paste (non-idempotent) commands from some document out of order and then do it again in the right order. I watched them fail to utilize even the most basic flags for grep and tail. I even watched them use passwords that belonged to other companies on our systems. This last one raised a lot of anger when I brought it up in a meeting - what if they were putting our passwords into other people's systems as well? However, they did one thing right: they got the job done. My employer paid their employer nearly 6-figures.

It doesn't matter if you are only pretending to know what you are doing. If you are just pretending to know what you are doing, then you are just like everyone else. What matters is if you can get it done.

Everyone is pretending (non-tech)

I am privileged to have a lot of non-IT friends and family, and this has given me the opportunity to see this same pattern play out in non-tech fields.

When working on cars with some friends who are professional or hobbyist mechanics, I have seen them pull out their phones and hunt down a YouTube video while underneath a car. In the thick of it, covered in grease, with the problem (literally and figuratively) over his head, I watched one of my friends pull up a video to figure out how to get to some particular part that needed to be replaced. This has happened many times. I have done it while working on my car. You have probably done it while working on your car. At the end of the day, the car runs again and drives away. The job got done.

I knew a general contractor. This guy made his living by working on house projects for others. They would pay him no small sum of money to renovate their bathrooms or put a gate in a fence. He had the expertise to do any job that came his way - he had built multiple houses from the foundation to the finish work in his life. A new client wanted some fresh flooring in a spare bedroom, and the specific type of flooring was something that the contractor had never seen before, it had been invented and made commonplace since the last time he did a floor. So what did he do? He looked up a YouTube video and he got the job done.

Knowing what you're going to do is impossible

You cannot look into the future and see what you are going to do tomorrow. You cannot look into the future and see what you are going to do in 10 minutes. You can have an expectation, sure, but you cannot know for certain. If there is a hammer in your one hand, a nail in the other, and you are standing in front of two boards of wood that need to be nailed together, you can guess as to how exactly you are going to get it done, but you can't really know for sure until you've done it. You can know what you did, but you can never know what you are going to do. All you can have before you do something is a guess, however educated it may be.

When I need to deploy some new server software, I have no idea how complicated it will be. Probably it will just be a package installation and some changes to a configuration file. Nginx is a good example of this. However, it could require custom-compilation, various services as dependencies, extensive modification of multiple configuration files and changing settings that can only be changed graphically or through a web interface. LibreNMS is a good example of this type of setup. When I start out, I cannot know how to do something. All I can do is say how it was done after I have finished. For every new task there are unknown unknowns.

Everyone is wrong

When I was in job training to become a security guard, the instructor was going over what you should do if you have to caution-tape a crime scene so that it doesn't get disturbed while you wait for police to arrive. He said that you should take "how big you think the scene should be and double it". Being a smartass 18 year old, I asked if we should double the area or double the perimeter, to which he replied "both". This is impossible. Take a square with the perimeter of 4 feet and double its perimeter to 8 feet. Now instead of having 1 foot on each side, it has 2. However, the area of the square has changed from 1sqft to 4sqft. Doubling the perimeter quadruples the area with squares. Circles work the same. Take a circle with a circumference of 4 feet and double it to 8 feet. The circle's area in this example grows from 1.27 sqft to 5.09 sqft, again a quadrupling of area for a doubling of perimeter. The instructor in this training didn't know the geometric details, he just wanted to tell people that they should go bigger than they expect because you can always make it smaller afterwrads, but a contaminated crime scene can't easily be uncontaminated. While his statement may have been wrong, it got the job done.

I was having some trouble with Ubuntu's new autobuilder, Subiquity. I actually wrote a whole blogpost about it here, but long before that blogpost I was having issues with the partitioning setup. The partitioning needs to be different for BIOS vs UEFI and it needs to have a bunch of special things set that aren't documented anywhere. I wanted to deploy different systems with different partitioning schemes and different amounts of SWAP space, but the Subiquity auto-partitioner was just too confusing. Even when I manually partitioned a system and then recovered the Subiquity-generated autobuild file, it was so opaque and specific to that server that I couldn't get my desired partitioning scheme to work on other servers via Subiquity. My solution to this was to go with the absolute default behavior of the Subiquity auto-builder, which is to create a 50GiB root partition and leave the rest unprovisioned. Since I had to do this to dozens of servers, I wrote a SaltStack state to do the rest of the provisioning. Salt doesn't include this functionality by default, so I had to make it myself. This required a lot of custom Python for acquiring information about the disk and doing math to calculate how big different partitions should be in order to meet my requirements. This was terrible, janky, bad code. I disliked this when I was doing it and I regret it now. After a few months of using this, I hunkered down and spent the 3 days that I needed to fully understand Subiquity's auto-partitioner through a mixture of trial&error and documentation. I was wrong to do it in my original manner and I wasted a bunch of time. However, I did in fact get the job done. For a few months, my custom solution was used to build out a large number of servers.

Everybody is human and makes mistakes. You need these failures in order to succeed. This is unintuitive. You don't make a good meal out of bad ingredients. You don't build a good philosophy out of bad presuppositions. You don't make an awesome castle out of broken Legos. You can't polish a turd. But failure is a prerequisite for success. Failure gives you knowledge and perspective. That knowledge and perspective gives you success. Success is when you get the job done.

This is the conclusion part

Of the ~5 people who might read my blog, I don't think that any of them often experience imposter syndrome. Maybe someone will find this through a web search in 10 years. Maybe it will help them.

PUNKTO | GPL Praxis: How To Get OpenProject 13 Enterprise Edition (For Free)

Sun, 17 Sep 2023 00:00:00 PST

GPL Praxis: How To Get OpenProject 13 Enterprise Edition (For Free)

CLICK HERE TO GO TO INSTRUCTIONS

OpenProject is a Libre project management application. The primary interface for the application is through the web browser, but a terminal client is under active development. OpenProject (OP) is sometimes useful for managing projects. Recently a new major version, version 13, was realeased. This new version of the software includes various new features such as time tracking and iCalendar integration.

Previously, I have been disappointed by the fact that some features are locked behind an enterprise license. In this post I will explore reverse engineering OP and removing these restrictions without paying. There are a few motivations:

OpenProject is licensed under the GPL, which gives me the right to make any changes that I wish to the software.
- If you don't use your rights, you will lose them.
- Exercising your freedoms under the GPL is the only way to fully appreciate them.
The basic enterprise license costs $7.25/mo per user with a minimum of 5 users, and I'm not paying $435/yr for an installation of which I am the only user.
I want Kanban boards in my web browser for free on PM software that I host myself.
Yar har fiddle dee dee.

"Reverse engineering" may not be the most accurate term here since I am not decompling anything. However, I am delving into a completely alien codebase in a language that I do not know to try to understand enough to make changes.

Installing OpenProject

I opted to install OP via the officially supported method on Ubuntu 22.04 of just using their repos. The official installation walks through setting up the repository and key and installing the software and then passes off to another section on the initial configuration. I want this as automated as I can get it, so instead of using the installer in interactive mode, I run it in non-interactive mode. This requires that I already have a postgresql database up and running. This isn't a tutorial on PSQL setup so I'm gonna breeze past this. You need:

A Postgres user (probably named openproject)
A Postgres database (probably named openproject) owned by the aforementioned user

You also probably want to front OP with Nginx or some other reverse proxy. This will keep you from exposing OP directly to the network and you can manage your SSL certs in one less place. My Nginx config looks like this:

server {
    listen 443 ssl;
    ssl_certificate YOUR_SSL_CERT;
    ssl_certificate_key YOUR_SSL_KEY;
    server_name YOUR_SERVER_FQDN;
    client_max_body_size 100M;
    client_body_buffer_size 128k;
    root /opt/openproject/public;
    location ~ / {
        proxy_pass_request_headers on;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Host $host:$server_port;
        proxy_set_header X-Forwarded-Server $host:$server_port;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://127.0.0.1:6000;
    }
}

This will of course return a 502 because OP is not yet set up. To set OP up, you can put a file at /etc/openproject/installer.dat with the contents:

openproject/edition default
postgres/autoinstall reuse
postgres/db_host 127.0.0.1
postgres/db_port 5432
postgres/db_username openproject
postgres/db_password DATABASE_PASSWORD_HERE
postgres/db_name openproject
server/autoinstall skip
smtp/autoinstall skip
memcached/autoinstall install
server/hostname YOUR_SERVER_FQDN
server/ssl no
openproject/admin_email example@example.com

And then run the command sudo openproject configure. This will set up OP for you and then it should be listening in the web browser. The default creds are admin/admin and you will be prompted to change this password on your first login. If you have accomplished this then you have met all of the prerequisites for getting OP Enterprise Edition for free.

Tracking Down The Upgrade Functionality

In the webapp, I got a huge annoying banner upon login that I didn't have an enterprise license. This looked like a promising place to start. However, clicking on the "Upgrade Now" button brought me to an external site, which was not what I'm looking for (yet). I instead clicked on the user button in the top right corner ("OA" for "Openproject Admin" by default) and go to Administration and then Enterprise Edition, where I found a page where I would put my "Enterprise edition support token" if I had one.

At first I thought "maybe I can just go to the path /admin/enterprise in the nginx root to find something", but sadly the file /opt/openproject/public/admin/enterprise does not exist (and neither does its parent directory). Looking at the nginx config, the requests are being passed to a process running on port 6000. You can inspect what is actually running on port 6000 like so:

root@openproject:/opt/openproject/public# netstat -tulpane | grep :6000
tcp        0      0 127.0.0.1:6000          0.0.0.0:*               LISTEN      997        121934     30374/puma 6.3.0 (t
tcp        0      0 127.0.0.1:34586         127.0.0.1:6000          TIME_WAIT   0          0          -
tcp        0      0 127.0.0.1:33480         127.0.0.1:6000          TIME_WAIT   0          0          -
tcp        0      0 127.0.0.1:46618         127.0.0.1:6000          TIME_WAIT   0          0          -

I have no clue what "puma" is but it is the application that is doing the serving, so let's look at that:

root@openproject:/opt/openproject/public# ps -ef | grep puma
openpro+   30374   30318  0 18:12 ?        00:00:11 puma 6.3.0 (tcp://127.0.0.1:6000) [openproject]
openpro+   30397   30374  0 18:12 ?        00:00:05 puma: cluster worker 0: 30374 [openproject]
openpro+   30401   30374  0 18:12 ?        00:00:04 puma: cluster worker 1: 30374 [openproject]
root@openproject:/opt/openproject/public# systemctl status | grep puma -B 2
             ├─openproject-web-1.service
             │ ├─30318 /bin/bash -e ./packaging/scripts/web
             │ ├─30374 puma 6.3.0 (tcp://127.0.0.1:6000) [openproject]
             │ ├─30397 puma: cluster worker 0: 30374 [openproject]
             │ └─30401 puma: cluster worker 1: 30374 [openproject]

It looks like the process is being run from the openproject-web-1 systemd service. Looking at the contents of /etc/systemd/system/openproject-web-1.service to see where the actual starting point for the web application is yeilds:

root@openproject:/opt/openproject/public# cat /etc/systemd/system/openproject-web-1.service
[Unit]
StopWhenUnneeded=true
Requires=openproject-web.service
After=openproject-web.service

[Service]
Environment=APP_PROCESS_INDEX=1
ExecStart=/usr/bin/openproject run web
Restart=always
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=%n

[Install]
WantedBy=openproject-web.service

This command in ExecStart is the main OP executable, and thankfully it isn't even a binary, it is just a really long Bash script generated by some tool called "pkgr". In this script I found what happens when using the run subcommand:

while : ; do
  case "$1" in
    run)
    {...}
    exec sh -c "cd $(_p ${APP_HOME}) && $runnable"
    {...}

That ${APP_HOME} variable is set in /etc/default/openproject to be /opt/openproject. I could have come to this same conclusion by looking at the output of dpkg-query -L openproject and sifting through the files there, but this way was more fun. Looking at some of the other variable setting in this script, I eventually figured out that the binary that gets run by the above exec command is /opt/openproject/vendor/pkgr/processes/web, which itself just launches /opt/openproject/scripts/web, which in turn launches our puma process from earlier:

#!/bin/bash -e

HOST="${HOST:=127.0.0.1}"
PORT="${PORT:=8080}"

bundle exec rails server -u puma -b $HOST -p $PORT

Everything from here on is Ruby and I was well out of my depth. Ignorance, however, is not a sufficient excuse to not do something. Now that I had found the OP application files, I just needed to find the code that manages the enterprise license. Running the command grep -ril enterprrise . in the /opt/openproject/ directory will show everywhere that that string shows up in the codebase, but there are a lot of results. Of the 843 files containing this string, a lot can be completely ignored.

Files in ./modules/. This directory contains OP-specific Ruby code for web features. All of the modules highlighted by the grep search are either completely or partially locked behind the Enterprise paywall. These modules are:
- boards for Kanban boards
- auth_plugins for different authentication methods
- team_planner for the team planner
- grids for laying out pages
- ldap_groups for accessing group membership from LDAP
- openid_connect for OpenID integration with Google Workspace or Azure AD
- bim retools OP specifically for use in construction
- dashboards is the part that manages per-project dashboards
- two_factor_authentication for 2FA
Files in ./db/migrate/ which I assume is used for database migration when OP gets updated. When you upgrade OP, you should get to bring your license with you, and the code here provides for that.
Files in ./lookbook/docs/ which dictate which colors to use for various things, such as warnings that you should buy an enterprise license.
./lib_static/open_project/configuration/helpers.rb, among other things provides a link for users to go sign up for a 14 day free trial of OP Enterprise on a managed installation "in the cloud".
Some files in ./.heroku/ for tools which have their own enterprise licenses unrelated to OP.
Files in ./frontend/ and ./public/assets/frontend/ which contain HTML and JavaScript and TypeScript and graphics and all that good stuff.
Temp files in ./tmp/cache/.
Some documentation about how to use the OP API in ./docs/.
Some files in ./config/ which are mostly just locales so that OP can translate "PLEASE GIVE US MONEY" into any language.
Files in ./spec/ which are used for testing the code.
Some of (but not all of) the files in ./lib/, ./vendor/bundle/ruby/3.2.0/gems/, and ./app/ that are obviously related to the modules which depend, in whole or in part, on an enterprise license.
Some .erb files, which are not related to the epic rap battles of history, but instead are Ruby templates for HTML pages.

All of these aren't worth looking at beacuse they are not where the code to import and validate an enterprise token is coming from. Of the original 843 files, this leaves only 9 to actually look through:

./lib/constraints/enterprise.rb
./lib/open_project/enterprise.rb
./vendor/bundle/ruby/3.2.0/gems/openproject-token-3.0.1/lib/open_project/token.rb
./app/helpers/enterprise_helper.rb
./app/helpers/enterprise_trial_helper.rb
./app/services/authorization/enterprise_service.rb
./app/contracts/concerns/requires_enterprise_guard.rb
./app/models/enterprise_token.rb
./app/models/token/enterprise_trial_key.rb

The very first file here just creates a function called "matches" in the "Enterprise" module. The next file, ./lib/open_project/enterprises.rb appears to set some methods that can be used for enterprise tokens . The "token" method checks the EnterpriseToken.current.presence, which is defined in the ./app/models/enterprise_token.rb file. The "current" method of that class calls the RequestStore module and then returns (I think) the content of the key that it is querying, in this case the EE (enterprise edition) token. Grepping for RequestStore.fetch shows a lot of places in the OP code that this is used, so to avoid breaking other functionality, I don't think that this is the best place to make modifications. The ./app/helpers/enterprise_helper.rb, enterprise_trial_helper.rb, and ./app/models/token/enterprise_trial_key.rb files are all used for (I think) creating the metadata for the trial license.

That only leaves three files left. One is the Ruby gem file token.rb which takes in the token data that OpenProject (the company) gives you when you pay them money, validates it, and sets it up in your database. The other two files, which are in the ./app/ directory, look like the place where restrictions and enforcements of what is and is not allowed without an EE token are defined and enforced. This leaves two options with how to move forward:

Disable restrictions so that OP doesn't stop you from doing anything without a token
Modify token validation so that OP doesn't check if your token is legit

Changing The Constraints

The first place that I tried to get around this licensing issue was in the ./app/services/authorization/enterprise_service.rb file. That has some particularly interesting snippets:

  GUARDED_ACTIONS = %i(
    attribute_help_texts
    baseline_comparison
    board_view
    conditional_highlighting
    custom_actions
    custom_fields_in_projects_list
    date_alerts
    define_custom_style
    edit_attribute_groups
    grid_widget_wp_graph
    ldap_groups
    openid_providers
    placeholder_users
    readonly_work_packages
    team_planner_view
    two_factor_authentication
    work_package_query_relation_columns
  ).freeze

and

  # Return a true ServiceResult if the token contains this particular action.
  def call(action)
    allowed =
      if token.nil? || token.token_object.nil? || token.expired?
        false
      else
        process(action)
      end

    result(allowed)
  end

Initially, this first section looks like a list of actions that OP won't let you do unless you have an EE token. Those are in fact all of the different places where pieces of the application are guarded behind a paywall. The syntax is something that I hadn't seen before, the %i().freeze formatting being completely alien to me. With some looking around online, it appears that the "freeze" part of this is used to make the variable immutable, which means that it can't be changed after that point in the code. The parentheses give us easy string interpoltaion without needing to worry about quotes. The "%i" part at the beginning converts ths contents of the parentheses into an array of "symbols", which are just immutable strings. So I tried to replace this list of guarded actions like so:

  GUARDED_ACTIONS = %i().freeze

And then restarted openproject (systemctl restart openproject). Doing this did not immeidately allow me to use Kanban boards in the browser interface, which was a bummer. Maybe the second section of code, which appears to check if a token is present or expired, was my real target. This second section of code defines a method named call which takes in an action name, sets a boolean variable named allowed based on whether or not a token is present/absent/expired, and then runs result with that boolean. This final result function goes on to use something else from another module (ServiceResult), but I suspected that the check for token validity was all that I needed to bypass. I tried changing:

    allowed =
      if token.nil? || token.token_object.nil? || token.expired?
        false
      else
        process(action)
      end

To:

    allowed = true

Sadly, this still did not allow me to access enterprise features without a license. At this point I didn't even know for sure if OP was getting to my modified code or not, since my dissection of what this file is for was pure speculation. To test, this, I figured that I would honor the time-tested tradition of debugging with print statements. One quick "how to hello world in Ruby" later, I added a puts "we dem boyz" to the part of the function where I forcefully set allowed to true. Restarting openproject and reloading a kanban page did in fact print many copies of this string to the systemd journal for openproject-web-1.service. However, the restrictions were still in place.

"Maybe I need to look a level deeper at the ServiceResult call" was my next thought. This class is defined in ./app/services/service_result.rb, and it is really long and complicated, but it does fortunately have two lines at the top that look like they might be very impactful:

  SUCCESS = true
  FAILURE = false

So I changed the false for failure to a true. Sadly, I still got a "please buy OP EE" message when I wanted to just move rectangles around three columns with my mouse. Some functions, however, I was able to use, such as the team planner and site themeing. Looking instead at one of the actual restrictions in the code, I see a chain of calls to various other classes and methods just to check if some action is allowed:

./app/controllers/boards/boards_controller.rb

    def restricted_board_type?
      !EnterpriseToken.allows_to?(:board_view) && board_grid_params[:attribute] != 'basic'
    end

./app/models/enterprise_token.rb

class EnterpriseToken < ApplicationRecord
  class << self
    def current
      RequestStore.fetch(:current_ee_token) do
        set_current_token
      end
    end

    def table_exists?
      connection.data_source_exists? table_name
    end

    def allows_to?(action)
      Authorization::EnterpriseService.new(current).call(action).result
    end

    def active?
      current && !current.expired?
    end

    def show_banners?
      OpenProject::Configuration.ee_manager_visible? && !active?
    end

    def set_current_token
      token = EnterpriseToken.order(Arel.sql('created_at DESC')).first

      if token&.token_object
        token
      end
    end
  end

"Maybe I can force the changes in enterprise_token.rb", I thought. I set the functions table_exists, allows_to, and active to just be true, I set show_banners to false because I don't want to see the banners in the web application, and I didn't modify the set_current_token because it looked too complicated. This actually allowed me to get the Kanban boards working in the web browser. This is all well and good but modifications spread across several different files with unknown implications elsewhere is not really what I was going for when I set out on this path. The real best solution would be a clean solution. After some tinkering to see how much I could possibly reduce my modifications, the actual modifications that matter are:

./app/models/enterprise_token.rb

    def table_exists?
      #connection.data_source_exists? table_name
      true
    end

    def allows_to?(action)
      #Authorization::EnterpriseService.new(current).call(action).result
      true
    end

    def active?
      #current && !current.expired?
      true
    end

    def show_banners?
      #OpenProject::Configuration.ee_manager_visible? && !active?
      false
    end

Making just these changes allows me to access the themeing in the administration view and the Kanban boards. Technically, this meets my original desires for getting EE features without paying. However, this isn't a great solution. Here I am replacing a bunch of different lines of code (after writing this blogpost I discovered that I was not the first person to think of this). However, I worried that there could be soemthing that I was missing since it seemed like OP checked in multiple places for token validity. What if I could trick OP into thinking that I had a perfectly valid key? Then I could let all of the key validity checks keep going without worry, since OP would think that the key is totally valid.

Spoofing A Key

When you attempt to import a token into OpenProject on the /admin/enterprise page, the software looks at the token and makes sure that it is legit, then it grants you the EE features. This is done in the file ./vendor/bundle/ruby/3.2.0/gems/openproject-token-3.0.1/lib/open_project/token.rb, provided by this Ruby gem. When you attempt to add a new license, the software checks that it is in fact a license file, checks that it is from the OP Foundation, and sets various values about user count and expiration. To assist in debugging this script, I got a free trial version of the license, which lasts for 2 weeks. This is plenty of time to figure out how to remove such silly checks. In order to get a trial license, you can just go to this page. Keep in mind that you need a verified email address.

get the useful information out of this token, I added a piece to the code that writes the data from my temporary trial license to a file. I added this just after the final JSON is acquired:

** ./vendor/bundle/ruby/3.2.0/gems/openproject-token-3.0.1/lib/open_project/token.rb**

out_file = File.new("/tmp/op_json.txt", "w")
out_file.puts(json)
out_file.close

Then I imported the token. This wrote the file mentioned above, which had plaintext JSON data:

{"version":"2.0","subscriber":"Firstname Lastname","mail":"someemail@domain.tld","company":"Company Name","domain":"openproject.domain.tld","issued_at":"2023-09-31","starts_at":"2023-09-31","expires_at":"2023-12-52","reprieve_days":14,"restrictions":{"active_user_count":10}}

And that looks like something that I can edit. Modifying this JSON and replacing the code with a static declaration of my very legitimate license:

** ./vendor/bundle/ruby/3.2.0/gems/openproject-token-3.0.1/lib/open_project/token.rb**

        #data = Armor.decode(data)
        #json = extractor.read(data)
    json = '{"version": "1.9","subscriber": "Richard Stallman","mail": "rms@gnu.org","issued_at": "1970-01-01","starts_at": "1970-01-01","expires_at": "2970-01-01","reprieve_days": 14,"restrictions": {"active_user_count": 1000}}'
    attributes = JSON.parse(json)

The different options here all have reasoning behind them:

version is set to any number below 2.0 because if it is set too high then OP will do domain name validation which will prevent this from working across all OP installations.
subscriber and mail are set to RMS because I am sharing the software and I want to be free (hackers)
The dates are set to fun dates
active_user_count is set to 1000 because there's no way that anybody is using OP for more than 1000 users.

After making this change and restarting OP, I went to the page where you can enter a new enterprise key. Entering the letter a and clicking the "Save" button did now show the fake license and unlocked all enterprise features. This was an improvement to the previous solution because it required editing less lines of source code. However, this still required modifying the actual source code files of OP, which could be updated to require different workarounds in the future, increasing the amount of work that I have to do when OP 14 comes out if they completely rework the token validation code. At this point I had stars in my eyes and a dream in my head. Is there a way to do accomplish this task without editing any source code files from OP? Would it be possible to make my own completely valid token from scratch? The only way to find out would be to dive into the actual token extraction process.

The Enterprise Token

The actual token license key looks like this:

-----BEGIN OPENPROJECT-EE TOKEN-----
owGbwMvMwMX4qcPly8HAJ8aMa9YlCSTl5KfH56YWFyemp+qVVJSksvz3icwvVUgs
SlUoSy2qVCjOS03MrlSw1uTi8sgvTwWK6Sh4KiTmKuTllyik52fmpSuU5CsUZ+SX
K1QC9eVWKiQml5Qm5ihkp1YqlGQklgBVpwOVphXl5yr4B7jpcXF5KmTn5ZcrKoSm
Z4DMAnH0FDxL1IsVskqLgerVIYangA3PLFHILFZIzMnR4+pkcmFhYORi0BNTZNFf
ZJn8p9bcaI1gsgLMQ6xMINcLyBTl55c4FJTmZZfk6+UXpTNwcQrAlIh18f+PeiRj
01x4Qk5dwCam5+7VpVmhp3mc5y8q7Huh9mJC3osz9+48MptwNCv66f1d800SOWr2
CO5hnbYw+sq/mXvyO6YuuvegYU9XTO68gsO7nqlc47zWFRckO3HHPNM1LD4LvRyf
WP8XeCOYXXGpQbxhqZUos/UG8eTk19tfJ4doqDQmKPouPBfR+Vb4msY873m/Gvq/
Bs1c4p7oGqVbleudW9at8Fj38dZdkXJzXARfbkz9sje6wjbx4NP6AH/5+QJ111dk
eqz+WvhVkEeGU//Tw0vLH87lz76sZdEurSRkL3gwN3uywume3VpGZm8yckMONX7n
4+Yystu894P2haTFmju175gv/TNz+v3Pa/znuXx8Xb3UqyO2viQ3WPiiyuo16oZa
OrdNcm2muq64Z515KDx3wQLj8w+YwqIZc/iTgu5urKmZWJBoHqx6ND/CZOLat2u4
Mhe933lyc7bR/PW7K79cTXI8zb/A6PqmHVZq3q233AXfcjQrfGFe/E726+mXBoov
zNpDvx088jm4L3fGifnfTrbnGFwyTPNI3qiZ8bdT/1rWFP9tCW9PSHKcO/nnaeG7
lpq78ao8L9zWOKqt+FNdvjDwo5Ni6uYty5548IZFrqpK2TdHmEfLYDXvjPvf3hSF
ZOptcG5yTzU8XJ660fDNu5pNWxpSlr8K2btgWudqtp8LVhs/EfhiZrfut+HmvX7H
/pYfmmuwdaE9AA==
=e1w4
-----END OPENPROJECT-EE TOKEN-----

Which looks a lot like a PGP ASCII-armored message. Changing the header and footer of the file from OPENPROJECT-EE TOKEN to PGP MESSAGE did not allow me to gpg --deamor ee-token.asc, however. Luckily, I know base64 when I see it. Taking out the middle part and putting it into a base64 decoder, I got some JSON output with plaintext (JSON) keys and (JSON) values that are again base64 encoded:

{
    "data": "SSdtIG5vdCBzaG93aW5nIHlvdSBsb2wK\n",
    "key": "dGhlIGluZHVzdHJpYWwgY29uc2VxdWVuY2VzIGFuZCBpdHMgZXRjCkxpc3RlbiBtYW4gZG8geW91IGV2ZXIgZ2V0IHRpcmVkIG9mIHNlZWluZyB0aGUgc2FtZSBpZGVhcyBvbmxpbmUgYWxsIHRoZSB0aW1lPyBNYW5raW5kIHdhc24ndCBtZWFudCB0byBoYXZlIHRoZSBzYW1lIHRob3VnaHRzIGV2ZXJ5IGRheSBmb3JldmVyLiBJIGRvbid0IHRoaW5rIHRoYXQgY2FuIGJlIG9yZ2FuaWMgb3IgaGVhbHRoeSBmb3IgdXMuIE1heWJlIGl0IHdvdWxkIGJlIGEgZ29vZCBpZGVhIHRvIGN1dCBvZmYgZXh0ZXJuYWwgc291cmNlcyBvZiB0aG91Z2h0LiBJIGRvbid0IGtub3csIEknbSBqdXN0IGEgZHVkZSB3aXRoIGEga2V5Ym9hcmQu=\n"
    "iv": "c2hvcnR5==\n"
}

But when I plugged those values into a base64 decoder, I got binary gibberish. Judging by the contents of vendor/bundle/ruby/3.2.0/gems/openproject-token-3.0.1/lib/open_project/token/extractor.rb, this is because these values are either encrypted or binary representations of the data. The "data" section is the actual stuff that I want, the "key" section is the decryption key for that data, and the initialization vector is the "aes_iv":

./vendor/bundle/ruby/3.2.0/gems/openproject-token-3.0.1/lib/open_project/token/extractor.rb

        encrypted_data  = Base64.decode64(encryption_data["data"])
        encrypted_key   = Base64.decode64(encryption_data["key"])
        aes_iv          = Base64.decode64(encryption_data["iv"])

And that AES encrypted key is itself actually encrypted with an RSA key. AES is symmetric and RSA is asymmetric. The RSA public key is needed to get the AES key out. The openproject-token gem actually just calls out to the openssl library, using the verify_recover function via the backwards compatibility of the deprecated "public_decrypt" function.

./vendor/bundle/ruby/3.2.0/gems/openproject-token-3.0.1/lib/open_project/token/extractor.rb

        begin
          # Decrypt the AES key using asymmetric RSA encryption.
          aes_key = self.key.public_decrypt(encrypted_key)
        rescue OpenSSL::PKey::RSAError
          raise DecryptionError, "AES encryption key could not be decrypted."
        end

        # Decrypt the data using symmetric AES encryption.
        cipher = OpenSSL::Cipher::AES128.new(:CBC)
        cipher.decrypt

./vendor/ruby-3.2.1/lib/ruby/3.2.0/openssl/pkey.rb

    def public_decrypt(string, padding = PKCS1_PADDING)
      n or raise OpenSSL::PKey::RSAError, "incomplete RSA"
      begin
        verify_recover(nil, string, {
          "rsa_padding_mode" => translate_padding_mode(padding),
        })
      rescue OpenSSL::PKey::PKeyError
        raise OpenSSL::PKey::RSAError, $!.message
      end
    end

At this point I had to figure out where the RSA public/private keypair comes from. The extractor.rb file gets its RSA key from the token.rb file, which gets it from:

** ./vendor/bundle/ruby/3.2.0/gems/openproject-token-3.0.1/lib/open_project/token.rb**

      def key=(key)
        if key && !key.is_a?(OpenSSL::PKey::RSA)
          raise ArgumentError, "Key is missing."
        end

        @key = key
        @extractor = Extractor.new(self.key)
      end

But I had no idea where the "key" here came from. I assumed probably that it was passed to the Token object when it is defined, so if I could find the location in the OP code where that happens, maybe I could find the key. By doing a recursive grep for [^a-z]Token and looking through the results similar to how I found the original relevant files for the enterprise licnse itself, I was able to track down that the RSA key was pulled in from the disk:

./config/initializers/ee_token.rb

begin
  data = File.read(Rails.root.join(".openproject-token.pub"))
  key = OpenSSL::PKey::RSA.new(data)
  OpenProject::Token.key = key
rescue StandardError
  warn "WARNING: Missing .openproject-token.pub key"
end

It can be a bit difficult to fully wrap your head around what all is going on here, so I wrote a script that imitates OP's functionality to go from an "OPENPROJECT-EE TOKEN" to actual usable license JSON:

#!/usr/bin/env ruby

require "base64"
require "openssl"
require "json"

token_file = 'REAL.token'
pub_key = 'REAL.pub'

# Read in Token text
ee_token = File.read token_file

# Remove header and footer
header = "-----BEGIN OPENPROJECT-EE TOKEN-----"
footer = "-----END OPENPROJECT-EE TOKEN-----"
match = ee_token.match /#{header}\r?\n(.+?)\r?\n#{footer}/m

# Base64 decode string
json_data = Base64.decode64(match[1].chomp)

# Parse JSON
encryption_data = JSON.parse(json_data)

# Base64-decode RSA-encrypted AES key, AES IV, and AES-encrypted License
encrypted_data = Base64.decode64(encryption_data['data'])
encrypted_key = Base64.decode64(encryption_data['key'])
aes_iv = Base64.decode64(encryption_data['iv'])

# RSA decrypt AES key
rsa_public_key = OpenSSL::PKey::RSA.new File.read pub_key
aes_key = rsa_public_key.public_decrypt(encrypted_key)

# AES decrypt the license
cipher = OpenSSL::Cipher::AES128.new(:CBC)
cipher.decrypt
cipher.key = aes_key
cipher.iv = aes_iv
data = cipher.update(encrypted_data) + cipher.final

# Parse License JSON
license = JSON.parse(data)
puts license

Step by step, this script exactly imitates what OP does:

Read in the token data sent via email from the OP Foundation
Strip off the header and footer
Convert the base64 text from the token to JSON
Use the OP Foundation's public RSA key to decrypt the AES key
Use the decrypted AES key and IV to decrypt the license text
Parse the license text to JSON

Writing this script was very helpful in the process of fully understanding what OP is doing at each step, and it helped me learn a little bit of Ruby. This script requires two files in the current working directory:

REAL.token, a plaintext copy of the OP enterprise token that they send over email.
REAL.pub, which is an RSA key distributed with the OP package. It can be found at /opt/openproject/.openproject-token.pub

Since asymmetrical encryption is used to secure the AES key, I needed the private key in order to create and sign my own EE tokens. I assume that this private key is kept safe on the OP Foundation's servers and cannot be accessed by regular people, as that would allow for exactly what I am trying to do here. Probably this file is named openproject-token.key and the contents look something like this:

I looked (not very hard) but couldn't locate an accidentally-public private key with the regular search engines. This means that a universal key for all OP installations is not possible (until some point when such a key can be found). But if I replace the public key file with my own public key, then I should be able to generate my own license from it.

Recreating The Token

The structure of the token is like so:

This seems pretty complicated at first glance. Certainly it is more complicated than it needs to be. This could be as simple as

RSA encrypt the license JSON
Base64-encode that encrypted version
Slap a header and footer on

I have no idea why this extra step of AES encryption is used, execept possibly as a deterrance for people reverse engineering the token.

To start, I needed to create my own RSA public/private keypair, since the OP Foundation was not interested in sharing their private copy (I'm just assuming here, since I didn't ask). This can be done pretty easily on Linux with the commands:

openssl genrsa -out private.rsa 4096
openssl rsa -in private.rsa -pubout -out public.rsa

Which creates two files in the CWD, private.rsa for the private key and public.rsa for the public key. The public.rsa key needs to go in the OP installation directory. On Ubuntu server, the location that it needs to go is /opt/openproject/.openproject-token.pub. I replaced the file that was there with this newly generated one. After doing this, the next steps (in order) are:

AES-encrypt the license
RSA-encrypt the AES-encryption key
Base64-encoded the AES-encrypted license, RSA-encrypted AES key, and the AES IV
Store those encoded versions in JSON
Base64-encode that JSON
Slap a header and footer on that encoded JSON

I tried a few ways to do this directly on the shell, but I ran into issues when trying to Base64-encode the AES IV. This was becasue the IV is a hexadecimal number and when I cat $iv | base64, that Base64-encodes the ASCII version of the string, not the hex version of the string. Eventually I bit the bullet and wrote a Ruby script inspired by OP itself to do the heavy lifting:

#!/usr/bin/env ruby

require 'base64'
require 'openssl'
require 'json'

# License Plaintext
license_plaintext = '{"version": "1.9","subscriber": "Richard Stallman","mail": "rms@gnu.org","issued_at": "1970-01-01","starts_at": "1970-01-01","expires_at": "2970-01-01","reprieve_days": 14,"restrictions": {"active_user_count": 1000}}'

# AES encryption setup
aes = OpenSSL::Cipher::AES128.new(:CBC)
aes.encrypt
aes_key = aes.random_key
aes_iv = aes.random_iv
aes.key = aes_key
aes.iv = aes_iv

# Encrypt license with AES
aes_encrypted_license = aes.update(license_plaintext)
aes_encrypted_license << aes.final

# RSA encryption stuff
rsa_private_key = OpenSSL::PKey::RSA.new File.read 'private.rsa'

# Encrypt AES key with RSA
rsa_aes_key = rsa_private_key.private_encrypt(aes_key)

# Base64 encode RSA-encrypted AES key, AES IV, and AES-encrypted License
encoded_license = Base64.encode64(aes_encrypted_license)
encoded_key = Base64.encode64(rsa_aes_key)
encoded_iv = Base64.encode64(aes_iv)

# Make a JSON out of the above data
json_data = JSON[{
  'data': encoded_license,
  'key': encoded_key,
  'iv': encoded_iv
}]

# Base64 encode the JSON
data = Base64.encode64(json_data)

# Print with header and footer
puts '-----BEGIN OPENPROJECT-EE TOKEN-----'
puts data
puts '-----END OPENPROJECT-EE TOKEN-----'

Running this code spits out a working OP license which grants unlimited enterprise features.

How To Use This Hack

If you self-host OpenProject, you can remove those annoying banners and get Enterprise Edition for free by following this 2-step process:

curl -s https://punkto.org/no_cdn_lol/openproject/ruby/public.rsa | sudo tee /var/openproject/.openproject-token.pub && sudo systemctl restart openproject
- The exact destination location may have some variation on your installation. This filepath should be correct for any installations that resulted from following the official packaged installation instructions.
Go to the page on your installation to enter an enterprise license. This is the domain.tld/admin/enterprise page by default and paste in this key:

-----BEGIN OPENPROJECT-EE TOKEN-----
eyJkYXRhIjoieTN3MlpRek9aemJ0UGlhT0I4RE1qRys3UGhta2FVR1BkLzhh
RkcyZFE4SEs0Njh5VEhjeE9tTnRSRTV3XG5qY2srSHpieWZhUU9PeFZMVFRm
bFovbDdkWitkV2pySjAzajhlMTRQQXk4RXVhSGE4UlpOT3lHc1BUbytcblh3
OHprU3k0NC9yekM5L09wRW1pQkNHbFRCK0R2M3RTNzJqWWtKTnR5VWRBMmg0
WGFiRFk2OUJaNmI4ZVxuNHdLSGplT3ZTNXcxWUFqQmpkYmZ2ZHFYYXlOZlUv
V1ZYSGF1YkhIR3p3SjUxeUxhY3U5QUFaSEFQMEVLXG5mTERnaWlubXVXWjJZ
eGJVcnlmeHR2SGU2cldvS3owaDdPa1VXQXhmeGkwL3RuUnJ4cDczdS95eGUr
YjZcbnk1bWVwOUI4dU5ZZ1NCWUl0Q28yXG4iLCJrZXkiOiJUMjR2SmxxOXFt
M0JoU09LWCtXb2ZSd2ZzWCtkUXUxZ0NWT2ZBdjRMZURTRHdzSVIvb3FnREtz
MGwvdXBcbjYxeHg1aWloY1FHbHJrcTZEeG5Tc1BlSndtZjFHeXNES0pZeVB0
ZERyeUsvZmhXQWFPNnRtU1Z6YVREQ1xueFBrdW9qR25hZXhjQ05ibDFtUXBR
QThiTDJObUJmRDIzRlNjUVE0aHJlb3NmWTRHZHBFSVRCcTlFRC9EXG5nRXZU
ZXR4RmRXVmUwVVhvT2ZXa2FvWUFzS3htZXRoR0FObXBDWEJETUhZVXJkRE11
TnZQd24yd2h4WFFcblU4VzJMdVpYRkd1aUE4NFZoR3VacTR0TForNjlhU2h3
K0V4SnJjdWJ1b0tjaklhaGxDMnNyTVV0MXNWWVxucnZ6NlNRY2NvSlh2bkhZ
Tk95WTNGSGF2WU5KOG96MWxUN0dvZGZUdFVvdEY3aC9VV3o1RzBnM2hHKzhE
XG5NUlFsSlJDdmkxL1VhM1gvd2x1cEVYNHhQUnFtVXpDWEkrN2NPM3Zsb05v
WEhRa3RYYmg4a1F5d0NEQ0NcbmVtaHduT3Jqdk9wcGR2OVdoR0dGT0Y1amtD
d0c3MnFzZTdPWC9BMjJxaThXVC8wN3hqa0NvN0R6dThZeFxuOCtmVUlkTFpY
WTJLc0I4MEw3ZTJ0eDZFR2hiVjJhUG9wbVBsdW01Z05wcDBFbHlxelNXYWpo
UkdhNGhpXG5ua2l6N2o0YmFFZW1GYWtLYXJ1aUxKME00QzZvUW5JbUM0SHRY
Mk1HRzJDakhVY1VncWpPNURZNVN0SlBcbnViM2lwYWRwdFZkSmFjVzJ5OVhL
dTFVWWptY1ErNWRtWEdnN1RDdUZVV0VOeG90SitDQnIzZjFwYWEyT1xuNG15
NFRTL2tEU3NNWnp4Vmh5WGxjRkk9XG4iLCJpdiI6InJLVU40WHNGT2p1S0Nj
aWtrNmlWQUE9PVxuIn0=
-----END OPENPROJECT-EE TOKEN-----

There you have it, a simple working method to get a never-ending OP enterprise license. This requires no modifications to the OP source code, so this should drastically lower the barrier of entry to free OP for users around the globe.

Hackers need to pay their bills, too. By donating Monero to:

89zUPdtgXvo268ZtbzE7LBTEdo2Y75V1FUSs1rETeaRudQZy8BRs4rwTQuFzWqkQ6VU14Ei5TUBGhYCY6TmbCoEvQ2YkqC9

you will be supporting this effort and contributing to future development, maintenance, and continuous improvement.

PUNKTO | Zonerender: An RSS Feed Generator For Zonelet Sites

Wed, 06 Sep 2023 00:00:00 PST

Zonerender: An RSS Feed Generator For Zonelet Sites

I have written a tool which takes in a Zonelets site and spits out an RSS feed. This is a feature that is unfortunately missing from the stock Zonelets installation, although it is mentioned in the FAQ:

Can my Zonelet have an RSS feed?

Sort of. Neocities offers a very basic RSS feed that notifies followers when your site is updated. It doesn't have a blog-style feed where you can read the contents of posts from RSS feed-readers, and I haven't implemented a way to generate that easily. You can also encourage people to make their own Neocities accounts to follow your site.

Designing Zonelets specifically for use on Neocities is fine, but requiring its dependence on a particular service is directly contrary to the philosophy espoused by the Zonelets homepage:

The modern internet has coalesced around a few big sites, run by a few big companies. Social media giants absorb the vast majority of ordinary people's creative and expressive engagement with the internet. Many, myself included, spend much more time on social media than we would like. Creating a giant, decontextualized flood of micro-thoughts may benefit the corporate entities behind Twitter or Instagram, but does it benefit real human beings? Is this what we really want?

By leaving this as the only official option to have an RSS feed, the Zonelets author is encouraging the recreation of this coalesced Internet that they apparently are railing against. Suggesting that Zonelets authors require their readers to create accounts on Neocities just to get notifications of new posts is another step away from a decentralized web, it is another step away from self-ownership. Blogs need RSS feeds - this is a standard tool that has been widely used for the exact purpose of providing notifications and serialization of blog content for longer than most Zonelets users have been alive.

Possible Solution

It is possible that the developer of Zonelets could update Zonelets to generate an RSS feed from Javascript. I'm not actually sure if that is a possible solution since I don't think that Javascript can consistently be run from every single RSS reader out there. Maybe Zonelets could be modified to include some local script which the authors can run to generate their feed.xml file, but that in itself presents some problems:

It requires current Zonelets authors to update their installation, providing no backwards compatibility.
It requires Zonelets authors to use some external program.
- If it is an interpreted language like Python, it requries the user to install and set up all of the relevant libraries.
- If it is a compiled language like C, it requires the user to blindly trust the Zonelets developer.

I think that this is not a great solution. A good solution would make RSS feeds available to all Zonelets users (past/current/future) and require no additional work or blind trust on their part.

Extant Solutions

To create an RSS feed for a website that does not have an rss feed, various web services exist. Some popular ones which allow users to take in websites and generate RSS feeds include:

Fetch RSS is notable for its ability to take various different websites that would require a subscription and turn it into an RSS feed, including YouTube channels and TikTok feeds. However, it does not support Zonelets out of the box. Fetch RSS does provide the ability to select elements from a page to automtically parse into an RSS feed, but the real kicker here is that Zonelets archive pages are rendered in JavaScript for some reason. This means that all three of these RSS generators will not work:

Furthermore, these sites want you to make an account and log in, and they all have various post count and usage limits. This is their business model and they have to pay for all that shiny web design.

My Solution

Here at Punkto, I don't have to pay for shiny web design, so I can afford to provide boutique services free of charge. I have written a tool which takes in a Zonelets archive page (such as this one) and spits out an RSS feed. I have made this service publicly-accessible.

In order to use this service, all you need to do is point your RSS reader at https://punkto.org/zonerender?thelinktothearchive. For an example you can look at https://punkto.org/zonerender?https://zonelets.net/archive.

Doing this will cause my webserver to reach out to the requested URL and generate an RSS feed from the provided archive page. It will then go out to each article in the archive and fetch the content to put in to the actual items, which will allow readers to read the content of the blogposts from inside of their RSS reader.

If you have any issues with this service or if you find a Zonelet archive which does not work, please contact postmaster (at) this domain.

Since this is a service that I am providing free of charge on a server that I pay for out of my own pocket, I have included an XMR address to donate to (89zUPdtgXvo268ZtbzE7LBTEdo2Y75V1FUSs1rETeaRudQZy8BRs4rwTQuFzWqkQ6VU14Ei5TUBGhYCY6TmbCoEvQ2YkqC9). Donations to this address will be used to pay for a better server and they will show me that people are interested in this so I will make the service more robust. This is running on a tiny VPS far away from everything, so the service will be quite slow. Please increase the default check interval in your RSS reader to keep server load low. If performance becomes an issue, I will implement caching.

If you want to help out with Zonerender but can't or won't donate money, you can help by making improvements to the main script that runs the service and sending your changes to the aforementioned email. The script can be found here.

PUNKTO | PXE Booting And Autobuilding Ubuntu 22.04

Tue, 05 Sep 2023 00:00:00 PST

PXE Booting And Autobuilding Ubuntu 22.04

At the end of this blogpost you will be able to network-boot a fresh server and wind up with an auto-built Ubuntu Server box.

I use Ubuntu Server on a lot of systems. It is my go-to, primarily because I use Xubuntu on the desktop and having the same stuff everywhere is very comfy and reduces the overhead of things that I have to remember. I only need to keep one package manager in my head, among other things that are inconsistent across distributions. This does however, subject me to Canonical's occasional insanity. Canonical has had a long history of NIH, where they just reinvent things that were already perfectly fine for absolutely no reason. When Ubuntu 20.04 released, they started to move away from Debian Installer and towards their own homemade Subiquity. Subiquity has an interactive mode which you can use to manually provision a server, but doing that hundreds of times sucks. This post will cover the whole chain of automating this. The basic flow is:

Boot server -> BIOS/UEFI -> DHCP -> PXE (tftp) -> GRUB -> Subiquity -> cloud-init (http) -> Your server gets built

I'm assuming that you already have a functional DHCP server on the network that you want to autobuild servers on. If you don't, then go somewhere else to find instructions on that.

HTTP / TFTP Server

Shared Root

If you do not already have a server on your network to serve files over HTTP and TFTP for the PXE-boot process, then you will need to create it. With any luck, this will be the last machine that you need to build manually. Build out a server in your normal manner, I am using Ubuntu 22.04 as my base for obvious reasons. Once you have the server built out with your initial configuration and can log in to it, you will need to make a directory to serve the files from. It is not the deafult behavior, but I like to make one directory to serve the same files over both TFTP and HTTP since that is easier for me to remember.

mkdir /var/pxe

Packages

Then you will need to install some packages.

apt install nginx tftpd-hpa grub-pc grub-efi-amd64-signed

The nginx package will be your webserver. If you prefer Apache or lighttpd then install one of those instead and adjust the configuration later to accommodate. The tftpd-hpa package will be your TFTP daemon. There are a plethora of TFTP daemons available in the default repositories, but I chose this one because it is in the kernel git. The "hpa" in the package name represents the initials of the author, H. Peter Anvin. The grub-pc and grub-efi-amd64-signed packages are useful for building the GRUB PXE images that we will actually pull over the network to boot future servers from bare metal.

TFTP Config

Once these packages are installed, some configurations for nginx and tfptd need to be put into place. In order to have the TFTP daemon look into the /var/pxe/ directory for files, just modify /etc/default/tftpd-hpa to look something like this:

TFTP_USERNAME="tftp"
TFTP_DIRECTORY="/var/pxe"
TFTP_ADDRESS=":69"
TFTP_OPTIONS="--secure"

Then restart the service with systemctl restart tftpd-hpa. You will also need to open firewall port 69/UDP. With firewalld, you would run:

firewall-cmd --add-port=69/udp

A quick note on TFTP: If you want to test that everything is working correctly, you will need to install a TFTP client on your local machine. I suggest the one provided by the package tftp-hpa. Since this is developed by the same person as the daemon, I am pretty certain that they will work well together. Once that is installed, you can place a file at /var/pxe/test.txt on your server and try to get it on your commandline by running

tftp server-hostname -c get /test.txt

This should print no output and return exit code 0. It should also create a file in your CWD named test.txt with the same contents as the one on the server. If instead of this expected behavior, you get the error message Transfer timed out, then that implies that something is wrong with your firewall somewhere. It may be the firewall on your local machine. Generally if your local firewall is running at all, you won't be able to use the tftp client. This is because TFTP works over UDP, and the portion of the packet that says "hey I'm related to an earlier transaction" is not set on the response from the server, so as a result your local firewall rejects the packet. You have two options:

Turn your local firewall off with sudo systemctl disable firewalld
Open a specific port on your local firewall (sudo firewall-cmd --add-port=6969/udp) and use that specific port as the "return" port for TFTP (tftp server-hostname -R 6969:6969 -c get /test.txt)

Nginx Config

Adapt this section if you are using a different HTTPD provider. Create a new file /etc/nginx/sites-enabled/pxe-booting and fill it with the content:

server {
        listen 443 ssl;
        server_name server-hostname;
        location / {
                root /var/pxe/;
        }
}

In the above snippet we start serving the /var/pxe/ directory over HTTPS. If you do not have a valid certificate for your server's DNS name, you can serve the files over HTTP, but they will travel across the wire unencrypted. As far as I know, there is no method in the Subiquity installer (which is what will be accessing these files over HTTP/S) to ignore the validity of SSL certificates. It is pretty easy to get a LetsEncrypt certificate for a wildcard these days so that's what I do in this case, but this is not a tutorial for that.

You can test that this Nignx config is working by going to https://server-hostname/test.txt in your web browser. You should be served the same test file that you created earlier for testing TFTP.

GRUB

On the same build server that you have been on so far, you will need to create the GRUB files which will actually be booted by the bare metal servers.

PXE Images

Create and enter the directory where GRUB's files will live and create the PXE images:

mkdir /var/pxe/grub
cd /var/pxe/grub
grub-mkimage -d /usr/lib/grub/i386-pc/ -O i386-pc-pxe -o /boot/grub_i386.pxe -p '/grub' pxe tftp
grub-mkimage -d /usr/lib/grub/x86_64-efi/ -O x86_64-efi -o /boot/grub_x64.pxe -p '/grub' efinet tftp

The -p '/grub' option is critical. If you want to store your GRUB files somewhere else relative to your TFTP root, then you need to change this option. If you are serving your files from /var/pxe/grub/here/are/some/more/subdirs/, then that is what you should put after the -p instead. Now you have the GRUB PXE images, but these executables will try to find more libraries when they boot, and you need to put these libraries in place.

cp -r /usr/lib/grub/i386-pc .
cp -r /usr/lib/grub/x86_64-efi .

There is a reason that we create two .pxe images instead of just one. The i386 image is for legacy BIOS machiens. The x64 image is for modern UEFI machines. On some old hardware and some VMs, you cannot boot these modern UEFI images and so you will need the old BIOS images to fall back to.

Copy libraries from /usr/lib/grub/i368-pc/ and /usr/lib/grub/x86_64-efi/ to /grub. At this point your /var/pxe/grub/ should look like this:

root@server-hostname:/var/pxe/grub# ls
grub_i368.pxe  grub_x64.pxe  i386-pc  x86_64-efi

GRUB CFG

Grub will now happily start up, which you can test by network booting a machine. However, GRUB will not be very happy that it doesn't have a config file and it will dump you into a rescue shell. You need to write a grub.cfg file in the same grub root where the PXE files live, at /var/pxe/grub/grub.cfg in this example. You can write your GRUB's configuration to your heart's desire, picking whatever options you want. This is one my config and I have found it very useful:

default=installed_os
timeout=10
timeout_style=menu
menuentry "Ubuntu 22.04 Manual Install" --id=22_manual {
  echo "Loading Kernel..."
  linux /isos/ubuntu/22.04/casper/vmlinuz ip=dhcp url=https://server-hostname/isos/ubuntu/22.04/ubuntu-22.04.3-live-server-amd64.iso root=/dev/ram0 cloud-config-url=/dev/null net.ifnames=0 biosdevname=0
  echo "Loading Ram Disk..."
  initrd /isos/ubuntu/22.04/casper/initrd
}
menuentry "Ubuntu 22.04 No SWAP" --id=22_noswap {
  echo "Loading Kernel..."
  linux /isos/ubuntu/22.04/casper/vmlinuz ip=dhcp url=https://server-hostname/isos/ubuntu/22.04/ubuntu-22.04.3-live-server-amd64.iso root=/dev/ram0 cloud-config-url=/dev/null net.ifnames=0 biosdevname=0 autoinstall ds="nocloud-net;s=https://server-hostname/isos/ubuntu/22.04/cloud-init/noswap/"
  echo "Loading Ram Disk..."
  initrd /isos/ubuntu/22.04/casper/initrd
}
menuentry 'Installed OS (on disk)' --id=installed_os{
        exit 1
}

The cool part about this grub.cfg is that it defaults to booting an installed OS. Once the Subiquity installer successfully completes, it automatically reboots. If your default boot device is set to PXE (which it probably is if UEFI is enabled), then this means your server will keep bootlooping and rebuilding itself forever. This means that you have to sit there and babysit it to intercept a reboot and change the boot order. You could run a command that returns a non-zero exit code as the last thing that Subiquity does, but that is janky and also requires that you manually intervene before you can SSH into the server. If instead we just default to booting from the disk, then you don't have anything to worry about and you can even leave your default boot option to be PXE.

You will need to change the server-hostname in the above example to your server hostname. If you are serving files over HTTP instead of HTTPS then you will need to change that too.

If you modify your DHCP to point to one of these GRUB files, then they should boot and give you the menu that you configured.

Network Prerequisites

As mentioned earlier this is not a tutorial on how to set up DHCPD. However, you do need to modify your configuration to send the options to boot an image over the network. In ISC DHCPD that will look something like:

host hostnamehere {
  hardware ethernet de:ad:be:ef:12:34;
  fixed-address 10.10.10.10;
  filename "/boot/grub_x64.pxe"
  next-server tftpservername;
  option host-name "hostnamehere";
}

In OPNSense, you set the filename in Services -> DHCPv4 -> [Network] -> Network booting. There are various fields to fill in here, but you would fill them in with the same information as the above DHCPD config, except that you will use the IP instead of DNS name of the server.

Your DHCPD might be different. RTFM.

Ubuntu

In order to network boot the Ubuntu ISO, you first need the Ubuntu ISO, which you can get from this page. Download that file to your server:

mkdir -p /var/pxe/isos/ubuntu/22.04/casper/
cd /var/pxe/ubuntu/22.04/
wget https://mirror.math.princeton.edu/pub/ubuntu-iso/22.04.3/ubuntu-22.04.3-live-server-amd64.iso

Once the download finishes, you will need to extract two critical files from the ISO: the initrd and vmlinuz. These are the inital ramdisk (the boot environment) and the kernel that will be pulled in by GRUB and will proceed to download the ISO itself and boot that so that the server can be built. Mount the ISO and extract the files:

mount ubuntu-22.04.3-live-server-amd64.iso /mnt
cp /mnt/casper/vmlinuz casper/vmlinuz
cp /mnt/casper/initrd casper/initrd
umount /mnt

This is all of the files that you actually need to network boot Ubuntu. You should now be able to PXE boot GRUB and, if you are using the example grub.cfg that I provided, you should now be able to use the entry titled Ubntu 22.04 Manual Install. This will, if everything is working, drop you into the manual Subiquity server installer, and now all you need to do is write the Subiquity autoinstall config to get the client servers autobuilding.

Subiquity

Subiquity will look for some files over HTTP(S) if you choose the "No SWAP" option above. The location that it looks in is determined by the ds=nocloud-net;s=URLHERE setting on the kernel commandline. To create the files that it needs:

mkdir -p /var/pxe/isos/ubuntu/22.04/cloud-init/noswap/
cd /var/pxe/isos/ubuntu/22.04/cloud-init/noswap/
touch meta-data user-data vendor-data

The file vendor-data is optional. I have this file on my system but it is completely empty. Subiquity will try several times to find this file if it does not exist, but if it exists and is empty, then Subiquity will only look for it the one time. As a result, Subiquity is faster to start up if this file exists and is empty.

The file meta-data should have the content instance-id: jammy-autoinstall. Update this when 24.04 comes out and you need to change the codename.

The file user-data is the real meat and potatoes of the autoinstall and it is where you will put your configuration. The documentation on some of the pieces of Subiquity are difficult to find. These resources can be very helpful when writing a configuration:

Official Autoinstall Doc (this is confusingly worded for inexperienced users)
Autoinstall Quickstart (this is not bad if you followed the previous steps in this guide)
Full Autoinstall Reference (the most useful resource here)
Curtin Reference (Subiquity has a superset of Curtin functionality and is mostly backwards compatible with Curtin, although there are some differences)

Here is an example configuration that will do what most people probably want

#cloud-config
autoinstall:
  version: 1
  apt:
    preserve_sources_list: true
  identity:
    hostname: localhost
    password: $PUT_A_HASHED_PASSWORD_HERE
    username: YOUR_ACCOUNT_NAME_HERE
  keyboard: {layout: us, variant: ''}
  locale: en_US.UTF-8
  # interface name will probably be different
  network:
    network:
      version: 2
      ethernets:
        eth0:
          critical: true
          dhcp-identifier: mac
          dhcp4: true
  refresh-installer:
    update: yes
  packages:
    - packages
    - you
    - want
    - installed
  ssh:
    allow-pw: yes
    authorized-keys:
      - "OUTPUT OF 'cat ~/.ssh/id_rsa.pub' GOES HERE"
    install-server: true
  storage:
    layout:
      name: lvm
      sizing-policy: all
  late-commands:
    - sed -ie 's/GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="net.ifnames=0 biosdevname=0"/' /target/etc/default/grub
    - sed -ie 's/ro  $/ro net.ifnames=0 biosdevname=0  /' /target/boot/grub/grub.cfg
    - rm /target/etc/hostname
    - getent hosts $(ip -o -4 address show scope global | head -n 1 | awk '{print $4}' | awk -F '/' '{print $1}') | awk '{print $2}' | awk -F '.' '{print $1}' > /target/etc/hostname # set my hostname
    - rm /target/etc/apt/apt.conf.d/99needrestart # This prompt will prevent the apt command from completing

You should fill out the identity section above and put in your SSH key. This configuration gives you old-school interface names, sets your final hostname from DNS, and sets you up with one giant root partition. If you want a more granular setup for storage, you can configure it however you like, this is a preset. The best way to figure out what the hell you need to have in your storage section for the configuration that you want is actually to run through it once by hand and then look at the /var/log/installer/autoinstall-user-data file. The section in here will be a bit jumbled up and will be specific to the exact server that you built, but with a little bit of modification you can make it more general-purpose.

If you want to set up some custom apt source to install packages from during the install, then you can do that with this apt section (example is SaltStack repo):

  apt:
    preserve_sources_list: true
    geoip: true
    sources:
      salt:
        source: deb [arch=amd64] https://repo.saltproject.io/salt/py3/ubuntu/22.04/amd64/latest/ jammy main
        key: |
          -----BEGIN PGP PUBLIC KEY BLOCK-----

          mQGNBGPazmABDAC6qc2st6/Uh/5AL325OB5+Z1XMFM2HhQNjB/VcYbLvcCx9AXsU
          eaEmNPm6OY3p5+j8omjpXPYSU7DUQ0lIutuAtwkDMROH7uH/r9IY7iu88S6w3q89
          bgbnqhu4mrSik2RNH2NqEiJkylz5rwj4F387y+UGH3aXIGryr+Lux9WxfqoRRX7J
          WCf6KOaduLSp9lF4qdpAb4/Z5yExXtQRA9HULSJZqNVhfhWInTkVPw+vUo/P9AYv
          mJVv6HRNlTb4HCnl6AZGcAYv66J7iWukavmYKxuIbdn4gBJwE0shU9SaP70dh/LT
          WqIUuGRZBVH/LCuVGzglGYDh2iiOvR7YRMKf26/9xlR0SpeU/B1g6tRu3p+7OgjA
          vJFws+bGSPed07asam3mRZ0Y9QLCXMouWhQZQpx7Or1pUl5Wljhe2W84MfW+Ph6T
          yUm/j0yRlZJ750rGfDKA5gKIlTUXr+nTvsK3nnRiHGH2zwrC1BkPG8K6MLRluU/J
          ChgZo72AOpVNq9MAEQEAAbQ5U2FsdCBQcm9qZWN0IFBhY2thZ2luZyA8c2FsdHBy
          b2plY3QtcGFja2FnaW5nQHZtd2FyZS5jb20+iQHSBBMBCAA8FiEEEIV//dP5Hq5X
          eiHWZMu8gXPXaz8FAmPazmACGwMFCwkIBwIDIgIBBhUKCQgLAgQWAgMBAh4HAheA
          AAoJEGTLvIFz12s/yf0L/jyP/LfduA4DwpjKX9Vpk26tgis9Q0I54UerpD5ibpTA
          krzZxK1yFOPddcOjo+Xqg+I8aA+0nJkf+vsfnRgcpLs2qHZkikwZbPduZwkNUHX7
          6YPSXTwyFlzhaRycwPtvBPLFjfmjjjTi/aH4V/frfxfjH/wFvH/xiaiFsYbP3aAP
          sJNTLh3im480ugQ7P54ukdte2QHKsjJ3z4tkjnu1ogc1+ZLCSZVDxfR4gLfE6GsN
          YFNd+LF7+NtAeJRuJceXIisj8mTQYg+esTF9QtWovdg7vHVPz8mmcsrG9shGr+G9
          iwwtCig+hAGtXFAuODRMur9QfPlP6FhJw0FX/36iJ2p6APZB0EGqn7LJ91EyOnWv
          iRimLLvlGFiVB9Xxw1TxnQMNj9jmB1CA4oNqlromO/AA0ryh13TpcIo5gbn6Jcdc
          fD4Rbj5k+2HhJTkQ78GpZ0q95P08XD2dlaM2QxxKQGqADJOdV2VgjB2NDXURkInq
          6pdkcaRgAKme8b+xjCcVjLkBjQRj2s5gAQwAxmgflHInM8oKQnsXezG5etLmaUsS
          EkV5jjQFCShNn9zJEF/PWJk5Df/mbODj02wyc749dSJbRlTY3LgGz1AeywOsM1oQ
          XkhfRZZqMwqvfx8IkEPjMvGIv/UI9pqqg/TY7OiYLEDahYXHJDKmlnmCBlnU96cL
          yh7a/xY3ZC20/JwbFVAFzD4biWOrAm1YPpdKbqCPclpvRP9N6nb6hxvKKmDo7MqS
          uANZMaoqhvnGazt9n435GQkYRvtqmqmOvt8I4oCzV0Y39HfbCHhhy64HSIowKYE7
          YWIujJcfoIDQqq2378T631BxLEUPaoSOV4B8gk/Jbf3KVu4LNqJive7chR8F1C2k
          eeAKpaf2CSAe7OrbAfWysHRZ060bSJzRk3COEACk/UURY+RlIwh+LQxEKb1YQueS
          YGjxIjV1X7ScyOvam5CmqOd4do9psOS7MHcQNeUbhnjm0TyGT9DF8ELoE0NSYa+J
          PvDGHo51M33s31RUO4TtJnU5xSRb2sOKzIuBABEBAAGJAbYEGAEIACAWIQQQhX/9
          0/kerld6IdZky7yBc9drPwUCY9rOYAIbDAAKCRBky7yBc9drP8ctC/9wGi01cBAW
          BPEKEnfrKdvlsaLeRxotriupDqGSWxqVxBVd+n0Xs0zPB/kuZFTkHOHpbAWkhPr+
          hP+RJemxCKMCo7kT2FXVR1OYej8Vh+aYWZ5lw6dJGtgo3Ebib2VSKdasmIOI2CY/
          03G46jv05qK3fP6phz+RaX+9hHgh1XW9kKbdkX5lM9RQSZOof3/67IN8w+euy61O
          UhNcrsDKrp0kZxw3S+b/02oP1qADXHz2BUerkCZa4RVK1pM0UfRUooOHiEdUxKKM
          DE501hwQsMH7WuvlIR8Oc2UGkEtzgukhmhpQPSsVPg54y9US+LkpztM+yq+zRu33
          gAfssli0MvSmkbcTDD22PGbgPMseyYxfw7vuwmjdqvi9Z4jdln2gyZ6sSZdgUMYW
          PGEjZDoMzsZx9Zx6SO9XCS7XgYHVc8/B2LGSxj+rpZ6lBbywH88lNnrm/SpQB74U
          4QVLffuw76FanTH6advqdWIqtlWPoAQcEkKf5CdmfT2ei2wX1QLatTs=
          =ZKPF
          -----END PGP PUBLIC KEY BLOCK-----

Please note that you cannot both set an apt source and configure LUKS encryption. If you try, it will fail. I have reported this bug to the Canonical developers but as of the writing of this blogpost, they have still not acknowledged my issue. Remedy steps can be found in the linked ticket.

Review

Hopefully the information in this blogpost can be useful to someone someday. I did a lot of searching when I first had the opportunity to implement Subiquity autobuilding on my own infrastructure and I found official documentation and comprehensive writeups difficult to locate. If you follow the instructions in this post you should be able to:

PXE -> GRUB -> initrd/linux -> ISO -> Subiquity -> Reboot -> PXE -> GRUB -> Disk OS

All automatically. If you are rebuilding hundreds or thousands of servers then this should hopefully save you a lot of work.

PUNKTO | The Surprising Depth of Turtleneck And Chain

Thu, 31 Aug 2023 00:00:00 PST

The Surprising Depth of Turtleneck And Chain

The song Turtleneck and Chain was released by the band The Lonely Island in 2011 on their album under the same name. The song features a guest appearance by Snoop Dogg. The group is made up of three friends who met in middle school: Akiva Schaffer, Andy Samberg, and Jorma Taccone. These three all attended college to acquire various degrees in creative fields, and their training pays off greatly in the track at hand. What follows is a lyrical analysis of the song, which is much deeper than most people realize.

Full song

Lyrical Analysis

[Chorus]
Turtleneck n' chain
Turtleneck n' chain
Turtleneck n' chain
(sippin' on a light beer)
[Chorus]

From the very start, the Lonely Island lets us know the ostensible topic of the song which we have been given. The title of the album, the title of the song, and indeed the chorus are all in agreement. There is a turtleneck, and there is also a chain. The repetition of this line, in three, mirrors the repetition of the phrase up to this point. With the first repetition symbolizing the album name, the second the song title, and the third a call-out to the chorus itself. In this manner, The Lonely Island aludes to recursion. If the third line is meant to call back to the chorus itself, then the line of reference will simply continue on forever, like an ouroboros, the snake which consumes its own tail.

The turtleneck is an intruiging choice of clothing for the band to choose for this song. Some may posit that it was chosen due to its overall absurdity, a sort of call to comedy. The turtleneck was 60 years old in 2011, and it had laregely fallen out of popularity for most reasons except for hiding hickeys. This unfashionable style mixed with the "cool" pose by the members of the band on the cover creates a juxtaposition that undermines expectations and creates humor. I am here to tell you that the genius of The Lonely Island does not stop here.

The turtleneck is a unique piece of clothing. Man was originally naked in the Garden of Eden, before the original sin brought him the knoweldge of good and evil, bringing with it shame for his naked body and a need for clothing. This is the purpose that most of our clothing today holds, to shield our bodies from the sight of others. The ubiquitous turtleneck, however, takes this one step farther and extends the neck, not to cover up some shameful piece of flesh (unless a hickey is present), but instead to provide an additional sense of safety. The turtleneck helps to "protect ya neck", acheiving a base human need. This safety is occasionally missing from our reality - the indifferent cruelty of the world reminds us from time to time that the false sensation of safety which we have built up for ourselves is entirely an illusion. The turtleneck represents an attempt to restore this sense of safety to one of our most sensitive body parts - the neck. This layer of protection, even though it is just a piece of fabric, is so influential to the wearer's state of mind that he may feel protected from all physical and social harm. He becomes disconnected from his phsyical body, transforming into a floating human head, in much the same manner that a Zoom call does in our modern era.

The chain in the "turtleneck and chain" combination then cannot be devoid of meaning. The lay explanation here is of course that only cool people wear chains and only uncool people wear turtlenecks. Additionally, the small size of the chain would make it seem unimportant, like a cheap immitation of a "true" cool chain-wearer. Even this surface understanding can be greatly illuminatory when viewed through the correct lens. In the cover artwork, The Lonely Island is pictured wearing white skinny jeans, blue turtlenecks, gold chains, black sunglasses, and hair that is styled in accordance with the mores of the time. Here the chain is diminutive, a visual footnote to the rest of the ensemble. In fact the only reason that it is visible at all is due to the contrast in color compared to the turtleneck, the golden yellow chain set against an almost turqoise turtleneck. However, if we take a step back and look at the outfit as a whole, we will find something truly interesting. The white skinny jeans, the black square sunglasses, and the hair styles were fashionable at the time. The turtleneck, being outdated and too thick to be fashionable, is decidedly uncool. The chain was cool but its thickness was not. Within the cool there is some uncool, and within that uncool there is cool, and within that cool there is uncool.

This marbled manner of being represents many aspects of life. Look around yourself, and you will surely find it. A thing nested within its opposite, nested within the original thing, nested within its opposite. The further down and the further up you look, the more this pattern will hold true. Even you, the reader, are an example of this. Your mind is a liquid (blood) in a solid (brain) in a liquid (cebrospinal fluid) in a solid (skull and head) in a gas (the air). The information on this page is being displayed to you in a human manner (text) in a digital manner (HTML) in a human manner (visual) in a machine manner (screen technology). You are something alive (a breathing human) sitting in something dead (a building) in something alive (society or the world) in something dead (the vastness of space). These, and indeed more, are all aluded to by the visual language present in the album cover. They are all hinted at by the turtleneck and the chain.

That is not the sole meaning of the chain, however. You would be hard-pressed to find a better example of an eternal golden braid than a gold chain. Golden in its makeup, braided in its construction, and eternal in the fact that it is a loop, a shape without end. The chain here has additional symbolism in the form of reference to this book by Douglas Hofstadter, a work with strong themes of self-reference and recursion, which we have already seen in just this opening chorus and the album artwork.

The chain, looping back on itself, is yet another piece of recursion. When you get to the end of the "turtleneck and chain" in triplicate, realizing that the three repetitions are in fact directions, instructions on how to get to the point that you are at now, you are trapped by this chain. If you do not return to the beginning, the name of the album, but instead stay where you are, the chain will continue to loop back on itself, keeping you at its tail. If you break out of this loop created by the chain, you will find yourself in the loop created by the words.

This is allegorical of the human condition. Many people feel as if they are constantly stuck in a rut, and if they could just release themselves from this loop that they are within, they would finally be free and have a good life. They proceed to work very hard to get out of the rut, and they experience the freedom that they so desired, but it does not last. Before long, the man is back in a rut, although a new rut. He has traded his problems for new problems, but they are problems all the same. No matter how he lives, he will never truly leave the land of the ruts until he leaves the land of the living.

The usage of "chain" here instead of "necklace" has additional meaning. It serves as an instruction to the audience. It is there to say "Hey, you should be chaining this together". It is a call to action, a plea from the artists to spend the time and the energy to understand what they are saying, not just on the surface but on a true level. It is a cry out in the darkness, a call of loneliness. Is there anybody out there with ears to hear? If so, let them listen, and let them chain.

What then, are we to make of the light beer? This is something unexpected, it was not mentioned in the album title, the track title, or earlier in the chorus. This is an element that has taken us by surprise and looks as if it may throw a wrench in the works of our analysis. The lay listener would say "This is just a comedic song, and a light beer is funny because light beer is not cool to drink". However, I believe that there is more to it than that. Yes, this is one of the intended meanings of the light beer, but it is not the only intended meaning.

Alcohol has numerous effects on the drinker, among which is a physical relaxation of the muscles. The Lonely Island is not bothered in the face of all of the above absurdity, they are relaxing. While aware of their current predicament (the light in light beer a clear reference to philosophical enlightenment), The Lonely Island is not troubled. They recognize the futility of resistence against the manner in which lives are lived, and they have resigned to take the path of Buddhist enlightenment. They just want to have a good time, not attempt to move an immovable object. This mentality is a clear forerunner to our society's recent rediscovery of optimistic nihilism. "My efforts are futile, so I may as well have a beer with my friends" says The Lonely Island.

My turtleneck fat and my chain so thin
N' I'm sippin' light beer, and my crew right here
Chain razor thin, light beer in my cup
And my sweater on turtle with the neck on puff

In these lines we see an obvious pattern, one which is reflected in Eastern thought as the Yin and the Yang. The fatness of the turtleneck and the thinness of the chain offset each other, they represent two opposites in direct contact, mixing together to become a coherent whole. Likewise, the (uncool) light beer juxtaposes being popular enough to have a crew that is at hand. Here we see three men who would drink light beer, but at the same time would also have a crew in attendance.

The way that this is worded implies that "turtle" is a setting for a sweater and that "on puff" is a setting for the collar of the sweater. This reflects the zeitgeist of the time. The iPhone was a new invention, only a few years old at the time that this song was released, but the cultural impact of a computing device becoming commonplace was everpresent, and The Lonely Island chose to put it also mention it in this song. Just as you would change the settings in your iPhone, you would change the settings on your sweater. This claim of man over his experience, this personalization and customization of his tools, was new and is reflected in the absurdity of changing settings on a sweater.

Some may claim that you could configure and personalize phones before the iPhone as well. While this is true, it is not true to the same extent. A wallpaper on an old dumbphone was a holographic sticker that you put over the screen. I remember that my dad's phone had a pumpkin sticker. The settings that you could change in the software were a small number of pre-programmed ringtones and enabling or disabling T9. The expansion of options when people began moving to "smart" phones seemed limitless, not even limitable by a move to the physical world. This claim of man over the minute details of his experience was new at the time, and so it is easy to overlook in modern analysis.

Huge turtle neck and my chain so skinny
Beer light all night, stand up get dizzy
Light cup, beer drink, thin so chain
Neck turtle fat, man, I won't say it again

Here we discover that this song is not just a thesis from The Lonely Island, it is an actively lived experience. Instead of being a display of ideas, they are exposing their lives in real time to us on the track. The claim of drinking beer all night and getting dizzy when you stand up is obviously something that they have lived through and experienced, and The Lonely Island goes on to show that they are experiencing it in real time as the song is being written, performed, and listened to. The listener can readily observe the effects of alcohol on the brain, the sentences become incoherent and jumbled.

Upon getting dizzy, The Lonely Island makes an observation that the cup has become light. This is because the beer is all gone, becuase it has been drank (beer drink). The song goes on to re-iterate that the chain is thin and that the turtlneck is fat, but in a similarly jumbled manner. This reiteration of opposites is itself an opposite, with the manner in which it is presented (drunken speech) flying clear in the face of the idea itself (in and among all things are their opposites).

A further aspect of drunkenness presents itself as well. The Lonely Island here claims "I won't say it again", yet goes on later in the song to restate the fatness of the sweater's neck. This is dishonesty, a bold-faced lie. What could you expect from someone who has been drinking all night, can barely stand, and has trouble forming sentences? I would not expect promises made in such a state to hold true later on, and The Lonely Island knows this. Here they issue a warning, a bold "be careful as to the sources of your information". In a world where not everyone has acheived The Lonely Island's level of enlightenment with regards to the futility of things, you must be careful who you trust and who you rely on, as they may have ulterior motives, or they may not be of clear mind.

[Chorus]
Turtleneck (check) with another on deck
In case I spill something on it
(like some light beer?)
Yes
We be braggin' 'bout whose chain be the thinnest
Turtleneck thicker than Delta Burke swimmin' in a Guinness
(now bear witness)

This verse starts with a conversation between some of the The Lonely Island members. One enumerates a checklist that he has: two turtlenecks. This may seem asburd, but the band goes on to clarify that the second turtleneck is a contingency, a Plan B in case some beer is spilled on the sweater. This portion of the lyrics does a wonderful job of embodying, instead of explaining, a philosophy. The Lonely Island is prepared for the possibility that the turtleneck may need to be replaced.

This doctrine of preparedness is not blind, however. They do not carry copies of all other items, just the most important one (turtleneck) that is most prone to damage (spilling beer). If light beer were spilled on a chain, the chain could quickly be rinsed off (especially beacuse of how thin it is). If it were spilled on anything else, the issue would not be as terrible as if it spilled on the turtleneck.

Furthermore, the band here is treating the turtlenecks as a disposable item. It serves a purpose, but does not have an honor of its own. If something were to happen to it (spilled beer or some other demise), it could be thrown off and replaced. This unattachment to objects is a clear opposite to a problem which plagues much of America, commonly called hoarding. That items, even if damaged or outmoded, may become useful at some point and so as a result should be saved is a common sentiment among the population which suffers from this affliction called hoarding. The Lonely Island, however, comes in with a cure: care not about the item, but instead about its purpose. This is a utilitarian view, even if it is in the same breath excessively consumerist.

The song goes on to describe that The Lonely Island brags about whose chain is the thinnest. This obviously makes no sense when you consider that golden chains acheive their value by how much gold they contain, how heavy and thick and long they are. Bragging about a chain's thinness is akin to bragging about how little money is in your bank account. At first glance this seems alien, but upon further consideration, such statements are commonplace. How many times have you heard people seemingly try to one-up each other with who slept less last night, who worked longer without a break, who has had a more painful and difficult childhood? This form of performative suffering, where the the pains that someone has gone through or is still going through are displayed in a manner to garner pity or sympathy is in line with bragging about how thin your gold chain is. Here The Lonely Island shows a mirror to this customary practice, displaying the absurdity of it and in so doing explaining that they do not respect individuals who engage in such behavior.

The mention of Delta Burke swimmin in a Guinness can easily give the listener a bout of confusion on par with that of trying to understand Ruby syntax. Some context should help. Delta Burke is a woman who has been, at various times in her life, a beauty queen and an actress. She has, throughout most of her career had on-and-off lost control of her weight, which led to her being a laughing stock of some media, including tabloids and Saturday Night Live. Guinness here refers to a beer (not a light beer) from Ireland. Guinness is known widely for its texture, being thick and creamy, a trait unique compared to most other brews. The reason for this thickness is the nitrogen bubbles in the beer, but that is not quite relevant to the point at hand.

Before they created this album, The Lonely Island worked as writers for SNL. SNL has been around for a long time, and back in 1998, performed a skit specifically making fun of Delta's weight (for those confused, the joke is that black men stereotypically are attracted to overweight white women). The jab at Delta's weight (thickness) in the 2011 song is not just a reference to how thick their turtlenecks are, but also a nod to their tradition, their forerunners. The Lonely Island respects those who came before them, they respect tradition. They have learned and appreciate the good works of the past. This is the only interpretation that makes sense, given that Delta was long gone from the mainstream consciousness in 2011, when the song was released. There would have been no other reason to mention her, since the trio could just as easily have chosen another woman known for her obesity. This appreciation for history does not stop there. Guinness has been brewed in Ireland (part of the UK) for over 250 years. Andy Samberg, arguably the most well-known of the trio, has heritage from this area of the world, as seen by his mother's maiden name of Morrow (this is an old English surname). This ancestral linkage to Guinness cannot be overlooked, The Lonely Island is respectfully acknowledging their heritage from both professional and genetic standpoints.

But they can't see my chain
Metal so thin it dissolve when it rain
Got more chains than the snow patrol
Everyone razor thin like an insect stripper pole

The chain cannot be seen on account of how thin it is. This is the logical conclusion of performative suffering carried forward. If the chain is so extremely thin, it cannot be seen, and the wearer cannot receive pity for it. If so little sleeping is done, the sleeper cannot function, and so cannot even find a person to tell how little they slept last night. Here The Lonely Island hammers on the point of balance in all things - even in those things that are not deserving of respect. Moderation in all things means moderation in both the good and the bad.

What shall we make of the chain dissolving in the rain? Some things melt in the rain, but gold is not one of them. This of course means that we must have yet another metaphor stacked on top of a humorous line. There are actually multiple metaphors wrapped in here. On one hand, The Lonely Island is stating that their chains, in addition to being extremely thin, are also counterfeit, and are made of a metal such as iron or steel that does (over enough time) dissolve in the rain. They are claiming that the suffering they are poking at here is not a true suffering, highlighting that it is frequently brought about by the sufferer. They are also referring to the Wicked Witch of the West from the motion picture The Wizard of Oz. This character is pure evil, following around the protagonists and attempting to kill, kidnap, and maim them. However, when confronted with water, she melts into a pile of clothing. In a way, this symbolizes the greed displayed by wearing a gold chain around one's neck. The wearer's greed takes over their life, forcing them into an existence where they are slaves, chained to the gold and gilded in chains. When the greedy person is gone, all that is left is a pile of clothing.

The "snow patrol" here is likely a group which takes to the streets when it snows to rescue stranded travellers and block off closed roads. This is not a standard naming convention, but these men grew up in California and did not know what snow was like or the names used for various groups and pieces of equipment. In this hypothetical where there is a group named the "snow patrol" who engages in such activity, they would carry a great number of chains in their totality. Four per vehicle, plus any backups in case one broke from years of patrolling the snow. It boggles the mind to think of how many pairs of snow chains such a group would have, and yet The Lonely Island claims to have more. The reason for this is evident - when their chains break or melt from being so thin and fake, they replace them. This further alludes to a couple of messages from earlier. Firstly, the motto of preparedness in the face of the world's indifference is reinforced. By carrying a large number of thin chains, The Lonely Island is ready for whatever life may have coming. Secondly, the performative suffering of attention-seeking individuals is replaced with a new form of performative suffering as soon as the current one is solved. These people, even if they break out of their rut, will simply fall back into a new one. The self-reference in this work truly boggles the mind, as we are now again back at the beginning, the initial points being made.

The next line is a very sneaky one. At first it may seem to be claiming that every single chain is razor thin, much in the same manner that a stripper pole for insects would be. However, common sense will tell you that insects do not wear clothes, and so could not be strippers. Looking closer at the line, you can see the first word is everyone, not every one. These are the official lyrics of the song, not some distortion from myself. The use of the word everyone is our key hint here. "Everyone" is an indefinite pronoun. It does NOT mean the same thing as "every one". It refers explicitly to people, to human beings. In a song about the human condition and types of people, this can be no mistake. Just one verse ago The Lonely Island was talking about obesity, and now they are talking about thinness. Again we see the opposites mixed together. Everyone around the trio is thin, much like an insect stripper, someone who strips for insects (not an insect who strips, since as mentioned earlier insects do not wear clothes). What kind of person could strip for insects? Only the very thinnest, as fat people are too large to be seen all at once by an insect. In order for the insect to appreciate the dance, the dancer must be very skinny, possibly even anorexic. The Lonely Island was surrounded by actors and actresses, people whose appareance and reputation are their bread-winning forces. Resultingly, a vast majority of them must be thin, razor thin even, if they want to continue to live the lives that they are accustomed to. The word "pole" here is a red herring. It does not refer to a long cylinder, but instead refers to opposing extremes, such as the poles on a magnet or the poles on a battery (sometimes called terminals). This word is thrown in to further reinforce the motif of the attraction and mixture of opposites.

[Chorus]
Chain so light when a breeze roll by, man, it float (man, it float)
And my beer so light you could see right through it, like a ghost
And I also wear turtle necks

The chain, occasionally a metaphor for a false and vocal form of suffering, does not truly weigh heavily on the neck of the wearer. Instead, a slight breeze can cause it to float away like a leaf in the wind. The problems displayed in the lives of the people who this metaphor is about are quick to go away if they actually need to. Of course, we know that the missing chain will quickly be replaced by another from the collection which is the envy of the snow patrol. The beer, on the other hand, is so light that it can be seen through. It is practically invisible, barely noticable. This beer, from earlier, symbolizes both enlightenment and acceptance. These traits do not boast, they cannot be seen easily. Even if described, many will be incredulous of their veracity. In this way, they are much "like a ghost".

Sweaters swallowing my chain like it's stuck in the couch
Looking like cookie monster flossin' his mouth
Find me coolin' on the corner sippin' light beer foam
You can tell by my turtle neck and chain that I bone

This portion begins with a vivid description of how a thin gold chain looks on a thick blue turtleneck. However, thinking about the words for a moment brings more clarity to the true meaning. A normal person does not floss their mouth, but the cookie monster does, because he does not have teeth, not anymore at least. Where have his teeth gone? According to Mr. Monster himself, C is also for Cavities. He lost his teeth due to his overindulgence and self-neglect. Enough suffering, even if self-inflicted, changes people. Maybe it makes them better, maybe it makes them worse. The modern cookie monster is a far cry from the original cookie monster. He has been reformed over time, yet he still wears the scars of his youth. Now he has different needs than he did in the past. Everyone carries their burdens today and has needs that arise from decisions, good or bad, that they made in the past. Here The Lonely Island is telling us that the problems you have today may seem overwhelming, but in the future they will be problems of the past. Your choice is whether the problems of today become the burdens or the memories of tomorrow. It is up to you to change your ways.

Turtleneck, light beer, and a thin ass chain, and that's it
Turtleneck, light beer, and a thin ass chain, and that's it

Simple minimalism is all that this trio needs. For three men, three items are enough. When you have your physical needs met and you have a crew, even if the crew is small, you have all that you could ever hope for. The power of friendship resounds at the end of The Lonely Island's portion of this song, and it is pure poetry.

[Snoop Dogg:]
The rabbit kicked the bucket
Dog said, "Fuck it."
Get my 2 step on
With my turtle neck on
Thin ass chain, light beer in my glass
Macking at the bitches, the ones with the ass
Girl, you know it's true
Tell me what it do
Oh, you like my style, say you like my crew
Say you wanna' be under my chest
This turtle neck is like one of my best
Say you wanna' be under my chest
This turtle neck is like one of my best
So I guess your dress, no stress, stayin' fresh is my vendetta
I'm a crook, so my look, I'm a game go-getter
The fatter, the puffier, the fluffier, the bigger the turtleneck
The more gangster it is

The above verse was performed by Snoop Dogg, and it notably lacks the metaphorical depth of those verses written by The Lonely Island. Snoop is and was a talented and popular mainstream rapper, and for him to lend (or sell) his voice to this track could say any number of things about either The Lonely Island or Snoop, or both. Was it charity on the part of Mr. Dogg? Was it a piece of performance absurdism on the part of The Lonely Island? Did they neglect to read Snoop into the true meaning behind the lyrics, leaving him floundering to string together something that sounds catchy, but in fact lacks the substance that makes the song so great? Or is there additional meaning here that I have not read in to? I leave these questions as an exercise to the reader.

[Chorus]
[Chorus]

And at the end we are back where we started, back in a mirror of the beginning. What better way to end a song such as this, with topics such as these? Can broken people become repaired, or will they simply wind back up where they started, always running in a circle, always pushing the same rock up the same hill?

Thank you for coming to my Ted talk.

PUNKTO | Fixing Twitter Syndrome On The Web

Sat, 26 Aug 2023 00:00:00 PST

Fixing Twitter Syndrome On The Web

I Paid For All The Pixels, I'm Gonna Use All The Pixels

I recently purchased a 2560x2880 monitor, which has exasperated problems that I have had with the modern web for a while now, particularly (but not exclusively) with blogs.

Behold, Julia Evans's blog. This website is extremely narrow. I have a hidpi display, and on my screen the main content of this page looks like a pole running down the middle of it, wasting all of the space on the side. I understand the desire to have some padding for visual niceness, but this is excessive. The main bar of this page already has padding inside of it. Look at that orange piece. Look at the tasteful white padding on both sides of the text. This section is the piece that was made to be looked at, but I can barely look at it. This grey part of the website leaves behind just 20% of the space on the page, which is insane. 80% of this page is wasted grey space that I don't want to see. The image here has been scaled down, which is why it is blurry.

Julia is not the only one guilty of this. This is a problem that plagues the Internet, and it is a problem that gets worse when you get a hidpi display. This isn't a problem created by my aspect ratio, and it isn't a problem created wholy by my resolution. This is a problem with how websites are designed. You will run into this same design with:

The StackOverflow family of websites
Any readthedocs page
Most WordPress blogs
Most statically-generated blogs

I am not the only one to deal with this, this effects all users of the Internet and it particularly effects users of hidpi displays no matter their aspect ratio. If someone were to visit Julia's site on a regular 4k monitor, they would run into the same frustration. In fact it looks nearly as bad on a 1080p screen. The only time that these sites take full advantage of the display is on mobile browsers, and that is because they serve different CSS on mobile browsers since people would riot if you took away the already-limited horizontal space on telephone screens.

Twitter Syndrome

This design seems to immitate Twitter, where it makes some amount of sense. On a website where everything that you would want to read is 140 characters or less, you don't want the text to start at the far left and go all the way to the right. If you had that design, everything would be less than a line. My site is designed to use as much of your display as you let it, and this is what 140 characters looks like:

This line is 140 characters in length. I think Twitter may have expanded to 280 characters recently but I don't really keep up to date there

That's the maximum. If that was how Twitter was rendered, people might actually realize that they are spending their time reading things that lack substance. Twitter and similar microblogging sites trick the user into thinking that the text on the page is important by making it big and padded on all sides and visually isolated. Instead of allowing the content of the site to create the value, they assign a value to it by shiny visual webdesign.

This design philosophy has infected sites where the content is meant to shine on its own, without this anti-user trickery. It has come to blogs and technical documentation, where you want to display as much information to the user as you can at any time. This plague, Twitter Syndrome, has spread across the web and is getting worse. As displays continue to become higher and higher resolution, sites will get narrower and narrower. Where will you draw the line? One tenth used space? One twentieth? How will you even draw a line on a website that you don't operate?

Claim: It Is Better This Way For Eyeball Fatigue

Some might claim that this system is superior to using all of the space on a page because it reduces eyeball fatigue. The claim here is that instead of moving your eyeballs all the way across the screen when you hit the end of a line, you only have to move them a little bit to start on the next line and continue reading. The proported benefits (of my strawman) are two-fold:

Reduced physical eyeball movement
Less mental pause between lines, allowing you to better keep focus.

If your eyeballs are so weak or your attention span so short that you cannot read a wide line, I implore you to get off the computer and go outside. If a wide line is too big for you to fit it into your field of vision, then that is a problem with distance to your monitor + size of the text + aspect ratio. Modern aspect ratios are another evil thrusted upon the the citizens of the world, but that is a topic for another blogpost.

This design philosophy also produces an over-reliance on the mouse scrollwheel, since you constantly have to scroll down to see more of the page. This eventually will create hand fatique which is way worse than eyeball fatigue because at least you can close your eyes, you can't detach your hand to make it stop hurting.

Regardless of all of this, the most pressing problem is that these websites look stupid. Any arguments about the benefits and impedements are secondary to your website looking like a barbershop pole is sitting in between the user and the screen. That's the real problem, that it looks like I'm sitting in solitary confinement when I'm looking at your website, and that's a little too poignant for being on the computer.

Not A Real Solution: Ctrl+Mousewheel

Some people might say "bro just hold down control and scroll up" like that is a real solution. Doing this will usually get rid of ugly horizontal bars. It will also make the text so abysmally big that it can be seen from Mars. Additionally, many sites will start to break and look like shit if you zoom in far enough. I've seen images scale disproportionately to the rest of the page, CSS elements that are meant for one small thing in the corner taking up the entire page, and some sites that allow your zoom to change the size of the text but nothing else.

Zooming in isn't a solution to the problem. Zooming in is a solution to the site being too small, not being poorly designed.

Not A Real Solution: Just Use Half The Screen

If you instead think that you can just snap the window to take up half the screen and be done with the matter, I am sad to disappoint you. The grey bars will (probably) be smaller, but they'll still be there. Additionally, this solution isn't really future-proof. As your screen gets higher resolution over time, this problem will just come right back. A half-screen window on an 8k display has the same horizontal pixels as a fullscreen window on a 4k display. The impending creep of pixel density will come and leave none spared.

Not Everyone Is Like This

Some websites take advantage of your whole screen. These websites are pretty well designed and I think that they will last a long time. The hall of fame includes:

Richard Stallman's Website
Shy Studios
The Toyota Information System
Wikipedia (before the recent change in layout)
SaltProject Documentation

A Real Solution: Modifying Other People's Websites With userContent.css

This problem is created by the website developer(s) making choices about how they want their site to be displayed. They make these decisions in CSS. Maybe there's some way to do it in Javascript but I haven't been cursed with experiencing that yet. If you open your inspector panel in any web browser and click on an element, something somewhere should be able to show you the CSS rules that are present for that element. If you look around enough, you will eventually find one that is set to width or max-width, or sometimes both. This field can be set to a font size, a number of pixels, or a percentage (there might be more options for values, that's all I've found). I think that setting this to a specific pixel count is the worst way that Twitter Syndrome rears its ugly head. When you set your max-width to a number of pixels, you are guaranteeing that you will need to adjust it again in the future when higher-resolution displays become more common. If it is set to a percentage, at least your website will look the same for the rest of eternity, even if that look is stupid.

I use Firefox, which gives me various tools to customize my user experience on the web. I'm not sure if this could be replicated in Chromium and friends, I don't use those web browsers very frequently. I can simply click on the offending CSS value and type in what I want it to be. The website will be updated in realtime in my browser so I can see the effects of my changes. Doing this allows me to set the width of different parts of the page. I can make Julia's site from earlier take up my whole screen, which makes reading her blog a lot easier.

My order of operations for a site goes like this:

Go to a website that is shaped like a skyscraper
Mouse over the section that is stupid and Right Click -> Inspect
Hover over the different HTML sections in the inspector and watch the various parts of the website get highlighted until I find the right one, then click
- You'll know that you have found the right tag to work from when it is the highest-level tag that highlights the problematic portion of the site.
Look in the CSS pane of the inspector until I find width or max-width. Sometimes it is a different key with the word padding somewhere in the name but that is less common.
Click on the value of the aforementioned CSS and change it to 100%
Enjoy the website no longer looking stupid

This solves the problem on most websites and they become usable again. Sometimes it is quite difficult to find the CSS element to modify for some part of the page. I have found this sometimes with sidebars. For this, I like to open up the Layout panel (you have to have the 3-pane inspector toggled for this) and click around in there until something looks right.

This is all well and good but your changes won't persist across page reloads, and they are only for the page that you are on. If you do this for a page and then click a link to another page on the same site, you're screwed. To remedy this, you need to make your changes persistent. There have been various browser plugins to accomplish this throughout history, but this is actually a feature that is built into vanilla Firefox, albeit poorly documented. Everything that I talk about here was tested on Firefox 117.

To save these changes, you need a chrome/userContent.css file in your Firefox profile directory. That directory is ~/.mozilla/firefox/something.profile-name/. You'll have to look and see what your profile directory is named, since it will be different for everyone. This is different from the userChrome.css file that you use to customize the browser itself. Note the difference in filenames: Content vs Chrome. If the directory and file don't exist, then go ahead and create them. You'll also need to set your about:config key toolkit.legacyUserProfileCustomizations.stylesheets to true if it is not already set.

You need to get the modified CSS out of your Firefox. You could look at it with your eyeballs and copy it down by hand, but that is error-prone and requires you to have a working knowledge of CSS, which I don't (look at my website). Fortunately Firefox has got you covered. If you open the 3-pane inspector and go to the Changes tab, there is a nice big fat Copy all changes button which will put exactly what you need into your clipboard.

Inside of your userContent.css file you will need to create the magic CSS that will overwrite the website's CSS. Use this as a template, except using your copied CSS changes from Firefox:

@-moz-document domain("readthedocs.io") {
  .wy-nav-content {
    /* max-width: 800px; */
    max-width: 100% !important;
  }
}

Note that the !important part of every value that you modify is critical. The userContent.css file will, by default, only add new values to pages and not overwrite existing values. This is unless your values are labelled as !important. When you copy the changes out of Firefox, it doesn't bring that !important tag, so you have to type it yourself.

Once you have made your changes, you can simply restart Firefox (NB: Hamburger -> History -> Restore Previous Session if your Firefox remembers things between sessions) and you will have all of your changes. Here is a copy of my userContent.css which contains snippets for some sites that I used while figuring this whole thing out. I hope this helps someone.

PUNKTO | Broken CSS Styling In Chrome, But Works Fine In Firefox

Fri, 25 Aug 2023 00:00:00 PST

Broken CSS Styling In Chrome, But Works Fine In Firefox

I built this website yesterday, and I wanted to show it off, so I sent it to a friend and I showed it to my girlfriend. They were both supportive, but to my dismay, the navbox up at the top of the page didn't render properly for anyone but me. I had done all of the design work and the site was fine moments earlier when I had walked out of my office, so why was it broken now? We tried on a few different web browsers:

iOS Safari
Android Firefox

But the nav buttons still did not render correctly. Today, I opened up the files expecting to find that I had accidentally removed some lines before stopping work, but to my surprise everything was there, the site even looked the way that it was supposed to in Firefox. When opened in Brave (a Chromium derivative) or in Android Firefox again, the list showed up as a plain unstyled list. For clarity, this is how things are supposed to look (and how they do in Desktop Firefox):

And this is how they looked in any other browser:

The list should be:

Separated by " | "
All on one line
Nicely colored

I opened up the Inspect panel in Chrome and selected the

line. I found that there was an error in the CSS:

When I hover over the surprised little triangle, he tells me "Unknown property name", but the strikethrough ends at the bottom padding. I thought that the buttons should still render, just with the wrong colors, given the section of the CSS that is striken-through with an error. Simply removing the lines from the error in Brave does not make the list render correctly.

Idea: Maybe I Can Get Brave To Render Literally Any Styled List

It is best when testing to isolate things.



  One
  Two
  Three

Works. Wrapping the

also works. However, changing the style section to:

.navbox {
  li {
    display: inline;
  }
}

Causes everything to break. Interestingly, when I instead collapse the full "path" (I'm not a webdev, that's probably the wrong term) into one line like so, it goes back to working:

.navbox li {
    display: inline;
}

And in fact, collapsing the FQCEs (Fully Qualified CSS Element, a term that I just made up) for my website results in the rendering that I expect. A snippet:

.navbox a:link {
  color: #1f9e89;
}
.navbox a:visited{
  color: #1f9e89;
}
.navbox a:hover{
  color: white;
  background-color: #1f9e89;
}
.navbox a:active{
  color: #1f9e89;
}
.navbox {
  padding-bottom: 1%;
}

Why It Didn't Work

I'm not sure where the original broken syntax came from. When I was writing the template for this site, I had a million tabs open, and I may genuinely have invented the syntax from whole cloth. By a crazy random happenstance, it is actually a real thing.

There is a new CSS feature called nesting. This feature looks like it was introduced to the CSS specification in 2018. Chromium and its children have implemented this feature in the way that it is described in the linked documents. Firefox has implemented the standard in the latest beta version (117.0 at the time of writing), which is the version that I am using. However, Mozilla's implementation is maybe flawed and accepts CSS that should not be valid (yet). Here is an example that should and does work in both browsers:



  One
  Two
  Three

Notice the & right there before the li. That is the character that should cause nesting to work. It was missing in my original code and Firefox just accepted that it should be there, while Chromium-based browsers instead silently failed. However, the CSS Working Group has discussed relaxing the syntax to no longer require the & and a Firefox developer was supportive of that goal. From the github discussion, I found this snippet useful:

This is about dropping the requirement of an explicit & in front of descendant selectors that start with an element selector. We have already resolved to make it optional in all other cases in #7834, so not sure how making it optional in all cases makes the language messier, if anything not having this exception makes it less messy. This is exactly the syntax nested rules would have in Option 4, which I see you voted you prefer over the now current (option 3) syntax. The ability to write selectors like this is the main reason people are considering option 4, so if we could get the same syntax and actually nest, it seems the best of both worlds. Also, when we polled authors, half wanted to omit the & whenever possible in a sample size of 1.6K people, which is not negligible.

And furthermore, during a CSSWG meeting, it was resolved to no longer require the ampersand going forward. People shit on Firefox for not following current standards, but here it seems that Chromium and friends are not following future standards. Firefox users are living in the future. According to CSSWG, the new stuff should be in the specification, however the spec at the time of writing just references the github issue without actually saying that you can ignore the &.

PUNKTO | I Have A Website

Thu, 24 Aug 2023 00:00:00 PST

I Have A Website

Oh, hello. THIS IS A WEBSITE.

Platform

I built out this website so that I could have a platform to express myself which does not fall under the purview of any of the public-square arbiters that have come to dominate our society. If I were to use Twitter, people without Twitter accounts would not be able to see what I post. If I were to use Facebook, people without Facebook accounts would not be able to see what I post. If I were to use LinkedIn, people without LinkedIn accounts would not be able to see what I post. LinkedIn and Facebook have been walled gardens for as long as I can remember, but Twitter's move to requiring a login to browse is recent. This presents a problem for me. I do technically have login credentials for each of these sites, but my web browser doesn't store any session data between launches, so simply reading a post on one of these sites requires that I:

Wait for Keepass to decrypt the database. This takes some time since I have the decryption time cranked up to the max
Log into my email (also in Keepass) and get a 2FA code
- Alternatively go find my phone, wait for it to turn on, decrypt it, and then wait for SMS to refresh
Finally log into the website to see the information that I wanted

This whole ordeal takes a bit of time and normally the information on social media isn't of high quality. Furthermore, social media is a psychic poison which rots your brain and your soul. If I were to use one of these platforms, I would be enticing anyone who wants to see my rambles (maybe like 5 dudes and 1 special agent) to subject themselves to the spiritual torture of social media and any security measures that they have to go through. It would be unethical for me to require of others what I am not willing to do myself.

Why Even Bother

"Why bother putting your thoughts out there" is not a terrible question. The vast majority of people are not going to care about anything that I have to write. Those who do care about something that I write won't care about the vast majority of stuff that I do write.

Maybe someone finds something useful. Maybe I get some catharsis from writing.

Technology

Static Site Generators

I made this website all on my own. In the distant past, this same URL would reach a blog that I ran which was a Jekyll site. This worked fine for me but I had to manage a Jekyll installation, which required gem and any other related nonsense. I liked that it was a static site generator and didn't rely on a beefy webserver, since it was running on the same cheap VPS that it is running on now. However, I have grown to dislike Ruby (my main gripe is that the syntax is visually unappealing). When exploring the idea of recreating a blog, I looked at a few different options for static site generation:

Hugo Hugo is much more extensible and powerful. While Jekyll is primarily focused on creating blog content, Hugo is for "websites" of any kind. Hugo is written in Go.
Pelican is another static site generator but this time written in Python.
Zola is yet another, written in Rust.

However, all four of these options presented two main problems:

Reliance on someone else's toolchain for creating and themeing my site.
I have to learn another markup language.

The learning of another markup language is really the biggest problem here. I am already intimately familiar with DokuWiki's markup language since I use it every single day both at home and at work. I use a self-hosted DokuWiki instance to keep track of everything from recipes to server deployment procedures. DokuWiki has its own markup language and I don't want to go learn MarkDown or whatever other option these static site generators require. I just want to use DokuWiki markup, since most of what will become my early blogposts are already in that format.

Rolling My Own

DW -> HTML

Pandoc can convert from DokuWiki to HTML. This means that all I have to do is figure out how to make HTML look pretty. I don't need a flashy fancy shiny site. I wanted it as plain as possible. The site should have no JS, no external sources embedded into the page, and it should look good on every screen. This is actually really easy to do in plain HTML with a little bit of CSS. I wrote a template file by hand and split it into a top.html and bottom.html, then I wrote a Bash script to handle the conversion and everything. Here's the whole script:

#!/bin/bash

# This script generates my website based off of files in this directory
DW_FILES_ROOT="txt_root/"
HTML_FILES_ROOT="html_root/"
TEMPLATE_DIR="template/"

# Sanity check cwd
if [ ! -d ${DW_FILES_ROOT} ]; then
        echo "you're in the wrong directory bro"
        exit 1
fi

# generate arrays
dw_files=( $(find ${DW_FILES_ROOT} -type f -iname '*.txt') )
html_files=( $(find ${HTML_FILES_ROOT} -type f -iname '*.html') )
dirs=( $(find ${DW_FILES_ROOT} -mindepth 1 -type d) )

# make necessary directories
for dir in ${dirs[@]}; do
        html_dir="${HTML_FILES_ROOT}$(echo ${dir} | cut -d '/' -f 2-)"
        if [ ! -d ${html_dir} ]; then
                mkdir -p ${html_dir}
                echo "Making directory ${html_dir}"
        fi
done

# generate html files
for file in ${dw_files[@]}; do
        html_dest=${HTML_FILES_ROOT}$(echo ${file} | cut -d '/' -f 2- | sed 's/\.txt/\.html/')
        if  [ ! -z "${1}" ] || [ ! -f ${html_dest} ] || [ ${file} -nt ${html_dest} ]; then
                echo "Generating ${html_dest} from ${file} and templates"
                cat ${TEMPLATE_DIR}/top.html > ${html_dest}
                pandoc -f dokuwiki -t html ${file} >> ${html_dest}
                cat ${TEMPLATE_DIR}/bottom.html >> ${html_dest}
        fi
done

# delete html files that no longer have a txt file as their parent
for file in ${html_files[@]}; do
        dw_source=${DW_FILES_ROOT}$(echo ${file} | cut -d '/' -f 2- | sed 's/\.html/\.txt/')
        if [ ! -f ${dw_source} ]; then
                rm -v ${file}
        fi
done

# push up to webserver
rsync --delete -zav ${HTML_FILES_ROOT} username@server_address:/var/www

This script requires that I be in the directory where the website articles are written. That is fine because I always am. From this point, the entire rest of the server setup was just Nginx, which I already had running on a VPS and needed only a little bit of configuration for the particular file structure that I've built out here.

Images

You see those little badges on the bottom of this page? I love them. I love websites that have them. I made all of the ones on my site by hand. Figuring out how to do that in Gimp took longer than it took to write the page-generation script. Since they are so small, I just host them directly here on the webserver. However, if I want to host larger images or video files or any other kind of sizeable download, I will need to use something else. This VPS has a small disk and it is slow and far away. I'd want heavy content to load faster. I'm not sure what the best solution to that problem is yet, but I'm sure I'll figure it out. I don't really want to rely on a fancy CDN, because that goes against the "no external requests" part of the goal of this website.

RSS

At the time of writing this post, I have no RSS feed to speak of. I would like one though, I feel like they are an essential part of a blog. I'll probably have to hand-write the actual tool that will generate this feed.

Content

I have some goals for what this site will actually host. I want a central place to put:

Technical writeups of projects that I've done or problems that I've solved in the IT space.
Writeups of non-computer projects and problems, such as specialty cooking information or woodworking.
Maybe a little bit of philosophy. This probably won't be very good and I will likely cringe at it in 5 years time.
Links to videos/slides of talks I have given, when those are available.
The hardware and software that I use, mostly for my own posterity.

9/5/23 Update

I have replaced the original Bash script with a longer and more complicated Python script. The new script is more extesnible for future modifications (I think) and it generates a valid RSS feed. This script was also the first time that I used Python classes while writing a script. I had never used them before but found them fun to work with. I think that they were a good choice of tool to use for webpages, since each webpage could get its own object. The whole new script can be found here:

#!/usr/bin/env python3

import sys
import os
import re
import subprocess
import datetime

class Webpage:
  site_name = "Ivory and Teal"
  page_txt_extension = ".txt"
  page_html_extension = ".html"
  webpagedict = {}
  templatefiles = {}

  def __init__(self, page_filename):
    self.txt_content = None
    self.html_content = None
    self.local_txt_path = None
    self.local_html_path= None
    self.title = None
    Webpage.webpagedict[page_filename] = self

  def generate(self):
    print("  GENERATING PAGE")
    with open(self.templatefiles['html_top']) as f: html_top = f.read().replace('_PAGETITLE', self.title)
    with open(self.templatefiles['html_bottom']) as f: html_bottom = f.read()
    with open(self.local_html_path, 'w') as dest_file:
      dest_file.write(html_top)
      dest_file.write('\n')
      dest_file.write(self.html_content)
      dest_file.write('\n')
      dest_file.write(html_bottom)

def find_files(path, regex):
  '''Takes in a path and a regex. Searches the path for files matching the regex, then returns a list of those files. Does not match directories'''
  matching_files = []
  for root, dirs, files in os.walk(path):
    file_regex = re.compile(regex)
    for file in files:
      file_path = root + '/' + file
      if re.match(file_regex, file_path):
        matching_files.append(file_path)
  return matching_files

def upload_everything(website_source_root):
  # This puts stdout on my terminal which is what I want
  subprocess.run(['rsync', '--delete', '-zrv', website_source_root + '/html_root/', 'cxe@punkto.org:/var/www'])

def make_xml(website_source_root):
  print("  GENERATING RSS")
  posts = []
  # Actual time of day doesn't really matter. If your RSS reader uses this as critical information to determine if there is new content, then your RSS reader sucks.
  todays_rfc_822 = datetime.date.today().strftime("%a, %d %b %Y") + " 00:00:00 PST"
  blog_post_line_regex = re.compile('^  \* ..., [0-9]{2} ... 20[0-9]{2}')
  with open(Webpage.templatefiles['xml_top']) as f: xml_top = f.read().replace('_SITENAME', Webpage.site_name).replace('_RFC822_FORMAT_DATE', todays_rfc_822)
  with open(Webpage.templatefiles['xml_middle']) as f: xml_middle = f.read()
  with open(Webpage.templatefiles['xml_bottom']) as f: xml_bottom = f.read()
  hopefully_blog_index = website_source_root + "/txt_root/blog.txt"
  if os.path.exists(hopefully_blog_index):
    # hand-typed RFC 822 compliant dates so that I can choose what shows up as the publish date in RSS feeds
    blog_index = open(hopefully_blog_index).read()
    with open(hopefully_blog_index) as blog_index:
      for line in blog_index:
        if re.match(blog_post_line_regex, line):
          # Figure out all the variable XML data
          rfc_date = ' '.join(line.split(' ')[3:7]) + " 00:00:00 PST"
          filename = re.split(':|\|', line)[1]
          link = "https://punkto.org/blog/" + filename
          content = Webpage.webpagedict[filename].html_content.replace('src="/', 'src="https://punkto.org/').replace('href="/', 'href="https://punkto.org/')
          title = Webpage.webpagedict[filename].title
          post_xml = xml_middle.replace('_TITLE', title).replace('_PAGENAME', filename).replace('_LINK', link).replace('_CONTENT', content).replace('Thu, 24 Aug 2023 00:00:00 PST', rfc_date)
          posts.append(post_xml)
  rss_feed_path = website_source_root + "/html_root/blog.xml"
  with open(rss_feed_path, 'w') as dest_file:
    dest_file.write(xml_top)
    dest_file.write('\n')
    for post in posts:
      dest_file.write(post)
    dest_file.write('\n')
    dest_file.write(xml_bottom)

def main():
  # get the where the script lives
  website_source_root = os.path.dirname(os.path.abspath(sys.argv[0]))
  Webpage.templatefiles = {
      'html_top': website_source_root + '/template/top.html',
      'html_bottom': website_source_root + '/template/bottom.html',
      'xml_top': website_source_root + '/template/top.xml',
      'xml_middle': website_source_root + '/template/middle.xml',
      'xml_bottom': website_source_root + '/template/bottom.xml'
  }
  # Don't make any non-site files that end in txt. No README.txt that is in MarkDown format. It won't come out looking very pretty.
  # If you name a file something.html and it is not an html file then don't get mad when this script messes up.
  # Filenames must be unique. No having multiple index.html pages. This is a limitation that I don't care about. If I eventually care about it then I will change it
  for file in find_files(website_source_root, '^.*/(txt|html)_root/.*\.(txt|html)$'):
    pagename = file.replace('.','/').split('/')[-2]
    # if this page doesn't already have an object, create it
    if pagename not in Webpage.webpagedict:
      Webpage(pagename)
    if file.endswith('.txt'):
      # File paths
      Webpage.webpagedict[pagename].local_txt_path = file
      Webpage.webpagedict[pagename].local_html_path = '/'.join(file.split('/')[0:-1]).replace('txt_root', 'html_root') + '/' + pagename + Webpage.page_html_extension
      # Page TXT and HTML content
      with open(file) as f: Webpage.webpagedict[pagename].txt_content = f.read()
      # Use pandoc
      Webpage.webpagedict[pagename].html_content = str(subprocess.run(['/usr/bin/pandoc', '-f', 'dokuwiki', '-t', 'html'], input=bytes(Webpage.webpagedict[pagename].txt_content, 'utf-8'), capture_output=True).stdout, encoding='utf-8')
      self_title = Webpage.webpagedict[pagename].txt_content.split('======')[1].strip()
      if self_title != Webpage.site_name:
        Webpage.webpagedict[pagename].title = Webpage.site_name + " | " + self_title
      else:
        Webpage.webpagedict[pagename].title = Webpage.site_name
    if file.endswith('.html'):
      # File paths
      Webpage.webpagedict[pagename].local_html_path = file
      Webpage.webpagedict[pagename].local_txt_path = '/'.join(file.split('/')[0:-1]).replace('html_root', 'txt_root') + '/' + pagename + Webpage.page_txt_extension
  make_rss = False
  upload = False
  hopefully_blog_index = website_source_root + "/txt_root/blog.txt"
  for name, wp in Webpage.webpagedict.items():
    # generation of html items
    print(name)
    # Delete TXT file
    if os.path.isfile(wp.local_html_path) and not os.path.isfile(wp.local_txt_path):
      print("  DELETING")
      upload = True
      os.remove(wp.local_html_path)
    # Write HTML to the file
    elif os.path.isfile(wp.local_txt_path) and not os.path.isfile(wp.local_html_path):
      make_rss = True
      upload = True
      wp.generate()
    elif os.path.getmtime(wp.local_txt_path) > os.path.getmtime(wp.local_html_path):
      make_rss = True
      upload = True
      wp.generate()
    elif os.path.getmtime(wp.templatefiles['html_top']) > os.path.getmtime(wp.local_html_path):
      upload = True
      wp.generate()
    elif os.path.getmtime(wp.templatefiles['html_bottom']) > os.path.getmtime(wp.local_html_path):
      upload = True
      wp.generate()
    elif os.path.getmtime(wp.templatefiles['xml_bottom']) > os.path.getmtime(hopefully_blog_index):
      upload = True
      make_rss = True
    elif os.path.getmtime(wp.templatefiles['xml_middle']) > os.path.getmtime(hopefully_blog_index):
      upload = True
      make_rss = True
    elif os.path.getmtime(wp.templatefiles['xml_top']) > os.path.getmtime(hopefully_blog_index):
      upload = True
      make_rss = True
  if make_rss:
    make_xml(website_source_root)
  if upload:
    upload_everything(website_source_root)

if __name__ == "__main__":
  main()