slower

Getting Started

2026-08-24T23:19:53.978016+00:00

I’ve needed to start this for a while. I think most programmers, sys admins, dbas and others work on projects large and small where we quickly find out how to make x work with y or a produce b. Then we check into source control, walk away and forget until we need to remember a detail 6 weeks later. So that’s what I plan to record here. Little notes and snippets of code, settings, whatever might have been on my mind or come across my keyboard in recent days or weeks. Often times it will have something to do with Python, SQL or clustered systems.

The first few post will be on getting Pelican setup with GitHub Pages and Gandi. After that I’ll probably jump into PySpark, HBase, Python and execution plans. Chances are if databases, Python or distributed systems are involved I’m interested in learning and writing about it. If that sounds interesting definitely keep an eye out for more post here.

A little bit about myself. I am currently a data engineer at Humana writing a lot of Python and SQL mixing software engineering practices with data science projects. Before that I worked at a few different companies focused on the Microsoft Data Stack mixing SQL Server, SSIS, SSAS and C# to build data intensive applications. There’s more information on my LinkedIn and GitHub pages.

That’s all for now. More to come soon.

Publishing with Pelican on Windows

2026-08-24T23:19:53.977999+00:00

To get things started I thought it might be a good idea to document using Pelican on Windows with Github and Gandi for blog publishing. I’ll start by configuring Pelican and Github. Once that’s working I’ll then talk about configuring Gandi so you can use a custom domain. If you’re using a different domain provider you may need to use different settings, but Github has plenty of documentation around this that I’ll provide links for. Using Pelican on Windows isn’t that much different than macOS or Linux, but you won’t find as many tutorials or be able to use the quickstart makefile.

Github Pages Setup

The first thing you should do is login to Github and then setup a Github pages repo. You can read more detailed istructions here: https://pages.github.com/ or create a repo that follows the pattern:

I followed a pattern for User Github pages. This will be important when publishing with Pelican.

https://help.github.com/articles/user-organization-and-project-pages/

Pelican Local

With that out of the way we want to move on to setting up our project on Windows. I’m using Anaconda and I will be creating a new conda environment for this project.

The main thing to pay attention to when you go through the quickstart prompts is that you won’t need or be able to use the makefile with Windows. Once you have completed the quikstart there are a couple things to pay attention to.

Your articles should be markdown documents in the content folder.
pelicanconf.py contains various settings related to you blog.
publishconf.py can be left alone because we are using ghp-import

Publishing

Go ahead and create a file under content. Something like gettingstarted.md and add some text. Once you’ve done that switch back to the terminal prompt.

Custom Domain URL

Ok, now that we have Github setup and we can see our blog pages I want to look at the steps required to use my custom domain hosted by Gandi with the Github pages. With Gandi we want to modify our A Records to allow routing to Github. Logging into your Gandi dashboard, select domains from the menu and then DNS records. On this page you should be able to edit your DNS record and add the following:

https://wiki.gandi.net/en/dns/zone/a-record

Ok finally navigate back to your Github repo and go to the settings page. Under settings scroll down until you see Github pages. You should see a textbox allowing you to enter a custom domain. Add that, and if possible I recommend checking the enforce https box below this.

Wrapping Up

With that done you should be good to go. Whenever you want to write a new article create a markdown document in the content folder and follow the same steps above for publishing. One last note if this doesn’t work immediately you might want to wait before beginning to change settings since your A record changes can take some time to replicate.

Recursive Search with Python

2026-08-24T23:19:53.977986+00:00

Recently I received from JSON like data that I needed to transform into a tabular dataset. As part of that there was a specific key that could occur as a child of different keys at different depths in the structure. Not only could the key I needed appear at different locations and depths, but when it was located it was possible that it would have N sibling occurrences I needed to retrieve at the same location. Finally for all of these there were a set of id and date keys at the top level of the structure that I was asked to include with each search key result.

I took a couple different paths on my way to solving this. One of the first things I found was the total depth was inconsistent across the structures. Not only that, but it wasn’t uncommon to find the key scattered across depths up to 5 or 6 levels deep. The function below is what I ended up using. It’s a recursive search that relies on the fact that the data is JSON like. Instead of trying to pull the parent keys out as part of the search I have a function that parses out the id and date keys passing those into this function as base. Then a search is performed on the input object checking the dictionary collections for all instances of the search key and when located appending the search keys value to the base data, which is then added to a list of results which is returned when the entire collection has been searched.

Gotchas

This needed to be Python 2 and 3 compatible so pay attention to iterating dictionary keys and values when you have this requirement. There are different ways to handle this. I used future.
The way that Python appends to list can be tricky. This bit me when I found that results contained the right number of results, but all of my results where the same and where based on the last hit. This is because I was calling append on base which was creating bindings that I mutated on each search result. Luckily Python has acopy module in the standard library to help with this scenario.

Problem Solved

The function below represents my final result. This worked well on the sample data, and eventually was used on PySpark RDDs to process hundreds of millions of structures quickly.

import copy from future.utils
import iteritems

def search ( input , rowbase , searchkey , results ):
     """ A search function to help transform nested JSON
     like objects into tabular rows. The function takes
     a JSON like object as input along with a search key
     and returns a row for each occurrence of the key in
     the object. rowbase is expected to be a list containing
     any base data you would like associated with the
     searchkey data.
     """
     if input :
         for i in input :
             # If input contains a list run it through search
             # again since it may contain dictionaries with
             # the key being searched
             if isinstance (i, list):
                 search (i, rowbase, searchkey, results)
             # If input contains a dictionary check if it
             # contains the searchkey. Also check if any of
             # the values are list or dictionaries that need
             # to be searched
             if isinstance (i, dict):
                 for k, v in iteritems (i):
                 # If the searchkey is located deepcopy
                 # rowbase to prevent changing rowbase
                 # on future hits. Create full row and
                 # append to results
                 if k == searchkey:
                     row = copy.deepcopy(rowbase)
                     row.append(i)
                     results.append(row)
                     continue
             elif isinstance(v, list):
                 search(v, rowbase, searchkey, results)
             elif isinstance(v, dict):
                 search(v, rowbase, searchkey, results)
         # Search has been exhausted return search
         # results to caller. Results will be a
         # list of list.
         return results

Next Steps

Since this works there are a couple of ideas I want to explore with it.

This seems like a good place to gain experience with Python type annotations.
Since this needs to work in a pure Python environment as well as a PySpark environment I want to do some profiling, but I’m not sure how tools like Cython or Numba will work/interact with the PySpark piece of this. That will be interesting to explore.
It would be interesting to add depth tracking and see if there are any levels where the search key never occurs so that the function could potentially skip iteritems at that level.

Docs

For more information documentation on copy and future you can check out the documentation.

I’m sure others will have different ideas and approaches to something like this. Or you might have suggestions on something that could be done to make this faster or easier to read. If you have feedback or suggestion feel free to send them my way via up via email.

DerbyPy Introduction to Python Modules and Packages

2026-08-24T23:19:53.977974+00:00

Most programming languages offer ways to organize your code into namespaces. These namespaces are logical containers that group different names and behaviors together and isolate them to that namespace. By organizing your code with namespaces it makes it easier to structure your application without naming collisions and it can make it easier for you and others to maintain your code by adding some additional organization to your project.

In Python we can use modules and packages to create namespaces that we can then reference in other modules as we build our application.

A Python module is a .py file containing Python definitions and statements. The file name is the module name with the suffix.py appended.

As with all things in Python when we import a module it is an object, and just like other objects it has dunder (double underscore) attributes that define additional data about that module. We can use that to learn more about the module before we ever start to use it.

import pprintext

print(pprintext.doc)
print(dir(pprintext))
 A module providing extensions to pretty print structures that pprint may not handle well.

 ['builtins', 'cached', 'doc', 'file', 'loader', 'name', 'package', 'spec', 'listdirectory', 'os']From the output of dir() we can see there is a function called listdirectory that is part of this module.

pprintext.listdirectory("plugins")

 plugins/
 ipynb/
 init.py
 liquid.py
 markup.py
 requirements.txt
 .git
 README.md
 ipynb.py
 LICENSE
 .gitignore
 core.py
 pycache/
 core.cpython-36.pyc
 init.cpython-36.pyc
 markup.cpython-36.pyc
 ipynb.cpython-36.pyc
 tests/
 pelican/
 pelicanconfmarkup.py
 pelicanconfliquid.py
 theme/
 templates/
 base.html
 content/
 with-meta-file.ipynb-meta
 with-liquid-tag.ipynb
 with-metacell.ipynb
 with-meta-file.ipynb
 with-liquid-tag.md

Finally, we can see where we are importing this module from with .file and we see that this is a module local to our application.

pprintext.file '/home/alex/projects/alexhagerman.github.io/pprintext.py'### Packages

For the sake of brevity and simplicity tonight we can say that a Python package is a collection of Python modules. It is a folder that contains .py file and provides a parent namespace for the modules in the folder.

Another way of saying this is:

Just like we did with our module we can call dir() on our package to see associated attributes and objects.

import pprintextension
dir(pprintextension)

 ['all',
 'builtins',
 'cached',
 'doc',
 'file',
 'loader',
 'name',
 'package',
 'path',
 'spec',
 'network',
 'pprintextension']

Additionally, we can call help which may provide more information about the package defined in init.py. You can think of init.py as a place to put initialization behavior and documentation for your package. In the way thatinit handles initializing your class init.py handles the initialization of your package during import.init.py used to be required to make a directory a package, but as of Python 3.3 thanks to pep-420 it is no longer required. More links and information are provided at the end of the notebook.

help(pprintextension)

 Help on package pprintextension:

 NAME
 pprintextension

 DESCRIPTION
 A package providing functions to pretty print structures that may have alternative renderings from the standard
 pprint package.

 PACKAGE CONTENTS
 filesystem
 network

 DATA
 all = ['filesystem']

 FILE
 /home/alex/projects/modules-and-packages-into/pprintextension/init.pyAdditionally we can import modules from packages and refer to them directly instead of using the fully qualified namespacing syntax <package>.<module>.<object>

from pprintextension import filesystem
filesystem.listhiddendirectory()

 ./
 .ipynbcheckpoints/
 .git/
 .idea/Packages go way beyond what we have covered here. As you build packages you want to consider their structure relative to the public API you’re creating. Publishing and distributing packages is a talk or series of talks on its own. For now what we have covered is how we can group modules together in a package and some basics for how to control the initialization behavior of a package.

Finishing up

Now that we know what a Python module and package is next month we will look at the import statement. As a sneak peak I'll leave you with sys.path and you can begin exploring how this relates to our own packages and modules that make up our application as well as those we might install with tools such as pip or conda.

import sys
sys.path

 ['',
 '/home/alex/miniconda3/envs/blogging/lib/python36.zip',
 '/home/alex/miniconda3/envs/blogging/lib/python3.6',
 '/home/alex/miniconda3/envs/blogging/lib/python3.6/lib-dynload',
 '/home/alex/miniconda3/envs/blogging/lib/python3.6/site-packages',
 '/home/alex/miniconda3/envs/blogging/lib/python3.6/site-packages/IPython/extensions',
 '/home/alex/.ipython']<https://docs.python.org/3/library/sys.html#sys.path>

Additional Reading

DerbyPy Intro to PySpark

2026-08-24T23:19:53.977959+00:00

This month at DerbyPy I provided a high level introduction to PySpark. For this talk I went over the Spark execution model at a high level, talked about the difference between the PySpark Dataframe and RDD api, and provided some examples of how to use both. As part of this I put together a jupyter notebook and some scripts that can be used via spark-submit along with instructions on how to run PySpark locally.

If you’re interested in the material and presentation they can be found here.

Dealing with NULL in PySpark transformations

2026-08-24T23:19:53.977945+00:00

Lately I’ve been dealing with nested data on a semi regular basis with PySpark. One of the scenarios that tends to come up a lot is to apply transformations to semi/unstructured data to generate a tabular dataset for consumption by data scientist. When processing and transforming data I’ve previously found it beneficial to make use of the RDD data structure so that I have the ability to easily apply custom transformations the same way I would if I was interacting with normal Python data structures, but with the benefit of Spark and the functionality provided by the RDD API.

With my most recent project though I decided to spend more time working with the Spark Dataframe data structure specifically for the potential performance gains from Catalyst and Tungeston. Along with this Spark offers a set of complex types for Spark Dataframe columns to make interaction with collection types a little bit easier.

Diving in I immediately used the Databricks XML library to load some data into my dataframe which had a similar shape (although different contents) to this:

from pyspark.sql import Row
from pyspark.sql.functions import explode, first, col, monotonicallyincreasingid, when, array, lit
from pyspark.sql.column import Column, tojavacolumn

df = spark.createDataFrame([
 Row(dataCells=[Row(posx=0, posy=1, posz=.5, value=1.5, shape=[Row(type='square', len=1)]),
 Row(posx=1, posy=3, posz=.5, value=4.5, shape=[]),
 Row(posx=2, posy=5, posz=.5, value=7.5, shape=[Row(type='circle', len=.5)])
 ])
])

df.printSchema()

 root
 |-- dataCells: array (nullable = true)
 | |-- element: struct (containsNull = true)
 | | |-- posx: long (nullable = true)
 | | |-- posy: long (nullable = true)
 | | |-- posz: double (nullable = true)
 | | |-- shape: array (nullable = true)
 | | | |-- element: struct (containsNull = true)
 | | | | |-- len: long (nullable = true)
 | | | | |-- type: string (nullable = true)
 | | |-- value: double (nullable = true)df.show()

 +--------------------+
 | dataCells|
 +--------------------+
 |[[0, 1, 0.5, [[1,...|
 +--------------------+

Perfect. Nothing too crazy, but I wanted to transform the nested array of structs into column representing the members of each struct type. So I started by looking at the options available to flatten my array column and I came across which appeared to do exactly what I needed. Next I needed to take the member attributes of the structs and turn those into columns. I wasn’t able to find a built in function for this, but using the select syntax available on dataframes along with the* wildcard available on structs I was able to write my own function to do this.

def flattenstructcols(df):
 flatcols = [column[0] for column in df.dtypes if 'struct' not in column[1][:6]]
 structcolumns = [column[0] for column in df.dtypes if 'struct' in column[1][:6]]

 df = df.select(flatcols +
 [col(sc + '.' + c).alias(sc + '' + c)
 for sc in structcolumns
 for c in df.select(sc + '.*').columns])

 return dfAnd with that out of the way I’m ready to go.

flatdf = df.withColumn('dataCells', explode(col('dataCells')))
flatdf = flattenstructcols(flatdf)
flatdf.show(3)

 +--------------|--------------|--------------|---------------|---------------+
 |dataCellsposx|dataCellsposy|dataCellsposz|dataCellsshape|dataCellsvalue|
 +--------------|--------------|--------------|---------------|---------------+
 | 0| 1| 0.5| [[1, square]]| 1.5|
 | 1| 3| 0.5| []| 4.5|
 | 2| 5| 0.5| [[, circle]]| 7.5|
 +--------------|--------------|--------------|---------------|---------------+
 flatdf.printSchema()

 root
 |-- dataCellsposx: long (nullable = true)
 |-- dataCellsposy: long (nullable = true)
 |-- dataCellsposz: double (nullable = true)
 |-- dataCellsshape: array (nullable = true)
 | |-- element: struct (containsNull = true)
 | | |-- len: long (nullable = true)
 | | |-- type: string (nullable = true)
 |-- dataCellsvalue: double (nullable = true)So far so good. Let’s try it again, and if all goes well we can throw this in a loop, flatten nested columns and be on our way.

flatdf = flatdf.withColumn('dataCellsshape', explode(col('dataCellsshape')))
flatdf = flattenstructcols(flatdf)
flatdf.show(3)

 +--------------|--------------|--------------|---------------|--------------------|---------------------+
 |dataCellsposx|dataCellsposy|dataCellsposz|dataCellsvalue|dataCellsshapelen|dataCellsshapetype|
 +--------------|--------------|--------------|---------------|--------------------|---------------------+
 | 0| 1| 0.5| 1.5| 1| square|
 | 2| 5| 0.5| 7.5| null| circle|
 +--------------|--------------|--------------|---------------|--------------------|---------------------+

And now we have a problem. After back tracking I found that explode is silently dropping out my row with null in it. Let's check the docs. Interestingly I didn't see anything about this. So I checked the latest docs and just so happened to noticeexplodeouter listed right below this. It turns out in 2.2.0 a set ofouter functions where added that retain null for certain operations such as explode. Unfortunately some of these are not available in PySpark until 2.3 and I didn't have the option to migrate from 2.2.x to 2.3.x.

StackOverflow to the rescue. After reviewing the PySpark tag I didn't find any solutions with accepted answers so I went ahead and wrote my own question. Thanks to that I learned a lot about PySpark/JVM interop and about some of the disparities between the JVM API and other language APIs.

Otherwise()

flatdf = df.withColumn('dataCells', explode(col('dataCells')))
flatdf = flattenstructcols(flatdf)
flatdf.withColumn('dataCellsshapetest', explode(when(col('dataCellsshape').isNotNull(), col('dataCellsshape'))
 .otherwise(array(lit(None).cast(flatdf.select(col('dataCellsshape')
 .getItem(0))
 .dtypes[0][1]))))).show()

+--------------|--------------|--------------|---------------|---------------|--------------------+
|dataCellsposx|dataCellsposy|dataCellsposz|dataCellsshape|dataCellsvalue|dataCellsshapetest|
+--------------|--------------|--------------|---------------|---------------|--------------------+
| 0| 1| 0.5| [[1, square]]| 1.5| [1, square]|
| 2| 5| 0.5| [[, circle]]| 7.5| [, circle]|
+--------------|--------------|--------------|---------------|---------------|--------------------+

Based on some responses to my question I found another question that provided a scala solution involving .otherwise and casting the nested structure with a null literal. None in Python. This seemed like the more direct solution without making use of private functionality in the library, so I opted to try implementing the scala solution in PySpark first.

But unfortunately it appears that the explode may have a precedence behind the scenes that drops the row before otherwise is evaluated. With a quickly approaching deadline I unfortunately did not have time to dig deep into why this was with other options on the table.

Into the JVM

def explodeouter(col):
 """
 Calling the explodeouter Java function from PySpark
 """
 explodeouter = sc.jvm.org.apache.spark.sql.functions.explodeouter
 return Column(explodeouter(tojavacolumn(col)))flatdfwithnull = df.withColumn('dataCells', explode(col('dataCells')))
flatdfwithnull = flattenstructcols(flatdfwithnull)
flatdfwithnull = flatdfwithnull.withColumn("dataCellsshape", explodeouter(col("dataCellsshape")))
flatdfwithnull.show()

 +--------------|--------------|--------------|---------------|---------------+
 |dataCellsposx|dataCellsposy|dataCellsposz|dataCellsshape|dataCellsvalue|
 +--------------|--------------|--------------|---------------|---------------+
 | 0| 1| 0.5| [1, square]| 1.5|
 | 1| 3| 0.5| null| 4.5|
 | 2| 5| 0.5| [, circle]| 7.5|
 +--------------|--------------|--------------|---------------|---------------+
 flatdfwithnull = flattenstructcols(flatdfwithnull)
flatdfwithnull.show()

 +--------------|--------------|--------------|---------------|--------------------|---------------------+
 |dataCellsposx|dataCellsposy|dataCellsposz|dataCellsvalue|dataCellsshapelen|dataCellsshapetype|
 +--------------|--------------|--------------|---------------|--------------------|---------------------+
 | 0| 1| 0.5| 1.5| 1| square|
 | 1| 3| 0.5| 4.5| null| null|
 | 2| 5| 0.5| 7.5| null| circle|
 +--------------|--------------|--------------|---------------|--------------------|---------------------+

While reviewing suggested solutions I found out that SparkContext has ajvm object that provides access to org.apache.* functionality. Along with this I also noticed that Databricks has an entire "private" api used with Python and Java. Part of this API istojavacolumn which makes it possible to transform a PySpark column to a Java column to match Java method signatures. Learning all of this, and knowing that the Java API already had explodeouter implemented I reviewed the Java explodeouter method to verify the type signature and built my own function in Python to call the Java function and return the column with null in place.

And it works! With that I am able to flatten out arbitrarily nested collections in PySpark dataframes while retaining nulls when using Spark 2.2.x.

Wrapping Up

A couple of things to note; if you have an array with more than one struct as a member this will fail, and if you have a deeply nested structure the growth of this transformation is typically not sustainable on a large dataset.

I have questions that I hope to continue spending time on. For instance why are rows with null dropped at all? I wonder if the operation makes a new dataframe from the column to apply the operation to and then joins it back on an index and along the way that join loses nulls. Why are functions that are lossy not identified as such? Is there always a version lag between the JVM api and the PySpark api? I'm also curious how Catalyst handles denesting operations and adding new columns from the result of exploding arrays or flattening structs.

Finally instead of adding new columns I want to try using the MapType to instead create a new column of key, value pairs that allows me to flatten out arbitrarily deep collections into a MapType so that I can use the same methodology on much deeper structures without adding a lot of columns that are mostly null.

docker-airflow

2026-08-24T23:19:53.977928+00:00

If you’ve spent time using Python for ETL processes or working with data pipelines using tools from the Apache ecosystem then you’ve probably heard about Apache Airflow. In this post I’m going to briefly write about why I’m using Airflow, show how you can get started with Airflow using docker and I will show how I customized this setup so that you can do the same. Finally at the end I’ll talk about a couple of issues I ran into getting started with Airflow and docker.

What is Apache Airflow

From the home page:

Airflow is a platform to programmatically author, schedule and monitor workflows. Programatically being a key part so that you can create and orchestrate worflows/data pipelines using the same processes and tools that let you create reliable, scaling software.

Why Airflow

I don’t plan to write much on this subject since it’s been covered in depth else where, but at work and often times when talking about Airflow the question of why Airflow versus X traditional solution where X is something like:

inevitably comes up. The primary reason I prefer a solution like Airflow to more traditional solutions is because my ETL is code. While there are numerous benefits to ETL as code my talking points are:

Your data pipes/workflows go through the same processes that helps you create better products like TDD
Your ETL development and production can be integrated with your CI/CD process
Better debugging tools
Flexibility

That’s not to say the traditional tools don’t have their place, but my experience is that any significantly complex data pipeline ends up making use of that tools script task (C# for SSIS, Java for Informatica) and now you have an amalgamation of GUI product and untested, undocumented and non versioned code in production data pipelines.

Why conda

By day I’m a data engineer helping to build platforms, applications and pipelines to enable data scientist. Because of this conda is a tool I’ve become familiar with and it let’s me work across languages, but easily integrate those various languages into my Airflow dags.

To get started with Airflow I highly recommend reading the homepage and tutorial to get an idea of the core concepts and pick up on the vocabulary used within the framework.

After that there is a great project called docker-airflow that you can get started with. This provides a quick way to get started with Airflow in an environment with sane defaults making use of Postgres and Redis.

This project provides an example dag and also allows you to load the Airflow example dags via the LOADEX environment variable. Additionally you might want to open up the Airflow dashboard and checkout the Connections tab where you can setup things such as SSH an SSH connection to reference in your dags.

docker-airflow

To get started with Airflow I highly recommend reading the homepage and tutorial to get an idea of the core concepts and pick up on the vocabulary used within the framework.

This project provides an example dag and also allows you to load the Airflow example dags via the LOADEXenvironment variable. Additionally you might want to open up the Airflow dashboard and checkout the Connections tab where you can setup things such as SSH an SSH connection to reference in your dags.

Customizing the setup

The docker-airflow project is a great start, but it makes assumptions that may not be true of your environment such as which database you plan to use, use of environment variables, etc.

If all you’re needing to tweak is the behavior of the environment or Airflow your first stop should be airflow.cfg in the /config directory. This is a centralized location for Airflow settings and is checked after any settings from the environment are loaded. If you're trying to change settings related to work pools, ssl, kerberos, etc this is probably the best place to get started.

If you’re looking to change things related to your containers such as when to restart, dependencies, etc then your going to want to checkout either the LocalExecutor or CeleryExecutor docker-compose files.

Finally you might want to make bigger changes like I did such as using a different database, base docker image etc. Doing this requires changing quite a few items. The changes I made were:

switch to miniconda for my base image to use Intel Dist Python
switch to Microsoft SQL Server for the database
switch the task queue to RabbitMQ

Most of this was driven by a desire to experiment and to learn more about tools that I use day to day. Since I work in a data engineering shop there are packages from conda-forge that I like to use driving the miniconda switch, I've used MS SQL for the last 8 years professionally and I've been working on scaling with RabbitMQ over the last year.

The switch to miniconda was a one liner in the Dockfile:

FROM continuumio/miniconda3Then to use IDP (Intel Distribution of Python) within the container I added this towards the bottom:

RUN conda config --add channels intel\
 && conda config --add channels conda-forge \
 && conda install -y -q intelpython3core=2019.1 python=3 \
 && conda clean --all \And with that I can make use of conda packages alongside traditional Python packages within my Airflow environment.

Next up I wanted to switch to MSSQL. Doing this was a matter of switching from Postgres in docker-compose and adding the MSSQL Linux drivers to the base docker-airflow Dockerfile.

docker-compose

mssql:
 image: microsoft/mssql-server-linux:latest
 environment:
 - ACCEPTEULA=Y
 - SAPASSWORD=YourStrong!Passw0rd
 ports:
 - 1433:1433
 volumes:
 - /var/opt/mssqlYou may or may not want to preserver your database volume so keep that in mind.

Setting up the MSSQL Linux drivers is fairly straight forward following the documentation from Microsoft.

Dockerfile

ENV ACCEPTEULA=Y
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - \
 && curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list | tee /etc/apt/sources.list.d/msprod.listRUN apt-get update -yqq \
 && apt-get install -yqq mssql-tools unixodbc-dev

One thing to note if you’re using a Debian based image is that Microsoft has a somewhat obscure dependency on libssl1.0.0. Without that installed you will get some obscure unixodbc error connecting to MSSQL with sql-alchemy. To remedy this add the below to your Dockerfile.

RUN echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashprofile
RUN echo "deb http://httpredir.debian.org/debian jessie main contrib non-free\
 deb-src http://httpredir.debian.org/debian jessie main contrib non-free\n
 deb http://security.debian.org/ jessie/updates main contrib non-free\
 deb-src http://security.debian.org/ jessie/updates main contrib non-free" >> /etc/apt/sources.list.d/jessie.listRUN apt update \
 && apt install libssl1.0.0

Finally setup your connection string either in airflow.cfg or an Airflow environment variable . I like to use the Airflow environment variables and pass them in from a .env file with docker-compose.

environment:
 - LOADEX=n
 - FERNETKEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
 - EXECUTOR=Celery
 - AIRFLOWCELERYBROKERURL=${CELERYRABBITBROKER}
 - AIRFLOWCORESQLALCHEMYCONN=${SQLALCHEMYCONN}
 - AIRFLOWCELERYRESULTBACKEND=${CELERYRESULTSBACKEND}And finally the last big change I implemented was the switch to RabbitMQ instead of Redis. Similar to the MSSQL switch this was just an update to the docker-compose file.

rabbitmq:
 image: rabbitmq:3-management
 hostname: rabbitmq
 environment:
 - RABBITMQERLANGCOOKIE=${RABBITMQERLANGCOOKIE}
 - RABBITMQDEFAULTUSER=${RABBITMQDEFAULTUSER}
 - RABBITMQDEFAULTPASS=${RABBITMQDEFAULTPASS}
 - RABBITMQDEFAULTVHOST=${RABBITMQDEFAULTVHOST}

And setting up the right connection string for Celery to talk with rabbitmq. Similar to the MSSQL connection string I put this in my .env file and reference it in my docker-compose file as seen above.

CELERYRABBITBROKER=amqp://user:pass@host:port/

One thing to note is anytime you are referencing the host and running with docker-compose you can reference the service id in this case rabbitmq as the host name. And with that I have a nice Airflow environment that lets me make use of the database I’m familiar with, a durable queue and packages across the Python and Data Science ecosystems via conda.

You can find these changes in my fork of the docker-airflow project. I’ve also opened a GitHub issue with the goal of creating some way to track other community variations of docker-airflow with the hope of helping others discover setups specific to their need.

Issues so far

I’ve been using the setup above for a couple weeks now with pretty good results. I’ve made use of some libraries like hdfs3 that have their latest releases in conda-forge and my familiarity with MSSQL has saved me some maintenance time. The experience hasn’t been without it’s issues. The highlights are:

Airflow packages may not be what you want. See librabbitmq and celery. It's best to manage a requirements.txt or conda.txt with your dependencies still.
Dependency management across multiple dags. In short with a standard setup you need one package version and it needs to be installed everywhere. For an interesting approach to this read We’re All Using Airflow Wrong and How to Fix It
Silent failures. Be aware of all the reasons why a worker may provide exit code 0 especially with docker. This took a minute to catch when an NFS mount stopped showing new files being available, but the exit code 0 made things seem ok. This isn’t Airflows fault, but just something to keep in mind when using Airflow in an environment with docker and remote resources.

Reaching out

Hopefully this post helps you get started with docker-airflow. If you have questions or want to share something cool that you end up doing feel free to open up an issue on Sourcehut or reach out to me alexander.hagerman@icloud.com.

Building Vim with Anaconda Python Support

2026-08-24T23:19:53.977913+00:00

This morning I was setting up a RHEL 7 box for development using my normal dot files, but when I was ready to sit down and start working on my project I noticed I got an error from You Complete Me letting me know that the version of vim that was installed wasn't compatible. After checking EPEL for a more up to date install I decided to try pulling vim from source and building it myself.

Luckily this wasn’t too hard, but I did run into a small issue related to the vim .config --with-python* flags since I'm using conda as my Python environment manager. The short story is the vim needs some information from the Python config directory to enable python and python3 support. When you use Anaconda or Minionda to manage your environments these are in slightly different locations than the normal /usr or /lib64 paths you may find in vim build documentation. Instead they will be in your conda environment lib as seen below.

Install additional build dependencies.

sudo yum install cmake gcc-c++ make ncurses-devel

Clone vim source, configure and build. Specifically pay attention to the — with-python* flags and the config directory they use in your conda environment.

git clone https://github.com/vim/vim.gitpushd ~/vim/src./configure --with-features=huge \
--enable-multibyte \
--enable-rubyinterp=yes \
--enable-pythoninterp=yes \
--with-python-config-dir=/work/alex/miniconda3/envs/py27/lib/python2.7/config \
--enable-python3interp=yes \
--with-python3-config-dir=/work/alex/miniconda3/lib/python3.6/config-3.6m-x8664-linux-gnu \
--enable-perlinterp=yes \
--enable-luainterp=yes \
--enable-cscope \
--prefix=/home/alex/.local/vim | grep -i pythonmake && make installpopdFinally if you use a custom prefix as seen above (prevents system level changes and conflicts impacting others) you probably want to add the below to you .bashrc file.

if [ -d "$HOME/.local/vim/bin/" ] ; then
 PATH="$HOME/.local/vim/bin/:$PATH"
fi

And that’s it. You should now have an up to date vim install with Python.

Vim and Rust in 2019

2026-08-24T23:19:53.977900+00:00

I’ve been using Vim as my primary editor for a few months now. Recently I wanted to revisit some project work in Rust, but I hadn’t setup any tooling in Vim for rust yet. The first couple of hits I got on Google were great resources that I’ll provide links to, but they were also over a year old, so while using them as a starting point I’m documenting my setup since some things have changed from 2017.

Tooling

Core Installs:

Rust with rustup
Racer Autocomplete:
YouCompleteMe Language Server Protocol
vim-lsp
RLS — Rust Language Server So far this has been a fairly pain free experience. As I use this (and vim) more I will likely add some updates related to packaging, compiling and debugging in Vim, but for now these are the tools that got me started. One thing to note is that I recommend installing in the order above and following the install directions (especially for the lsp) since those appear to have made some QoL changes in the last year.

Source Articles: https://kadekillary.work/post/rust-ide/ https://ddrscott.github.io/blog/2018/getting-rusty-with-vim/

Subdomain SSL with Gitlab Pages

2026-08-24T23:19:53.977884+00:00

This is out of date, I have since switched to self hosting gitea and AWS.

A few months ago I decided to migrate my Pelican site from Github to Gitlab. This was motivated largely by that fact that Gitlab has CI/CD built in by default. During this migration I also decided it was time to setup my own SSL certificate for slower.earth. Since this was new I looked around to see if there was any documentation readily available , and I found this wonderful tutorial from Fedora Magazine.

Between that and the Gitlab custom domain and ssl I was able to get up and running pretty quickly. I had accomplished my goals:

migrate to Gitlab
setup CI/CD of the Pelican site project
setup ssl

Good to go, done in an afternoon with plenty of time to work on a new post. I thought.

About a week later I was on a different computer and instead of browsing to https://slower.earth I went to https://www.slower.earth and Firefox blocked my request citing an SSL certificate error. Wondering what I had done wrong I started tracing back through what I had done and realized that I had only setup SSL certificate for my primary domain. Luckily last year lets encrypt added support for wildcard certificates to certbot. Unfortunately that has not been included in a release so there’s a couple steps that differ from the original Fedora article above.

Setup Instructions

Below are the steps to use certbot, gitlab pages and your domain management console to setup SSL for your subdomains. This assumes you are using a Debian based OS (I’m using Ubuntu 18.04) to install Certbot. If not swap out the certbot install steps for your OS and continue.

If you read the Fedora article linked above you do not need another key in .well-known. Instead for your subdomain you will validate with certbot by a DNS record setup via your Domain Management Console.

sudo aptget install certbotcertbot certonly -a manual -d *.<yourdomainhere>.<topleveldomainhere> \
--config-dir ~/letsencrypt/config --work-dir ~/letsencrypt/work \
--logs-dir ~/letsencrypt/logs \
--server <https://acme-v02.api.letsencrypt.org/directory>Follow the instructions entering your email, reviewing ToS, etc

You will then see this prompt:

Please deploy a DNS TXT record under the name
acme-challenge.slower.earth with the following value:

NAMETYPETTLVALUEacme-challengeTXT1800your code from the terminal prompt above

Once you have this setup it’s a good idea to wait a couple minutes since this record will populate via DNS and then return to your console and hit enter.

Once certbot validates the TXT record is available as part of your domain it will provide you the new location of your fullchain.pem and privkey.pem files for use with Gitlab pages.

With these files ready to go browse to your Gitlab page settings and setup your subdomains as documented here and here.

I highly recommend reading the Gitlab documentation above, but to summarize:

In your Gitlab pages project settings click add a new site
Enter the url
Add the data from your fullchain.pem and privkey.pem files generated via certbot
Copy the gitlab-pages-verfication-code= section from the Gitlab validation record box
Login to your domain management console
Setup a new TXT record for your subdomain: NAMETYPETTLVALUEWWWTXT1800gitlab-pages-verification-code=
Setup a new A record for

Gitlab

NAMETYPETTLVALUEWWWA180035.185.44.232

Return to your Gitlab Pages settings console and click the verify button.

Wrapping Up

With that you pages should show green and verified. If you browse to the different subdomains you setup then you should get through without any SSL problems.

One thing to note is that you will need to renew your certbot certificate every 90 days. This is done via the certbot renew command. I've setup an Airflow dag to take care of this since I have Airflow managing various other things for me. You can see that here

Hopefully you find the above helpful. If you run into issues I recommend:

Make sure you used the * wildcard in the domain cert setup
Setup your acme-challenge record correctly in your domain management console and left it there
Setup the right TXT and A records for Gitlab

What is ODBC Part 1 of 3

2026-08-24T23:19:53.977869+00:00

At my last job we used pyodbc to manage database interactions in a few different projects. We did this because we interacted with 5 different relational databases, and not all of them had native driver libraries for Python. In addition to this our use of pyodbc meant that we had a nice consistent database API for on-boarding, or when somebody needed to interact with a database that might be new to them for their project. Recently though I had somebody ask me what ODBC was, and to be honest I didn’t have a good answer. I’ve used ODBC libraries in multiple languages, but I hadn’t really dug into the nuts and bolts of what it was because I hadn’t needed to. I knew enough to use it, it worked well and there were bigger problems to solve. It’s a good question though. What is ODBC?

At a high level ODBC (Open Database Connectivity) is a specification for a database API creating a standard way for applications to interact with various databases via a series of translation and application layers. It is independent of any specific database, language or operating system. The specification lays out a series of functions that expose database functionality across systems. It’s an interesting, and I would say fairly successful abstraction since many programmers know how to connect, query and process data (via ODBC) in their language, but maybe they have never read sql.h or the SQLBrowseConnect function. For the full API Reference check here.

API vs Protocol

Quick side note. You may have heard about wire protocols and databases. ODBC is not a protocol; it is an API. This is important because databases tend to define their own wire protocols (some share this now with things like the Postgres wire protocol being OSS) that dictate the sequence in which events or bytes must happen for communication to work. ODBC as an API doesn’t dictate this kind of detail, instead it describes how to expose the database functionality to the programmer consistently independent of the database.

API: describes all valid functionality and interactions Protocol: defines the sequence of operations and bytes.

Why ODBC

If databases define their own protocols and have their own way of communicating why should we worry about ODBC? Turns out there are a lot of databases you can use. Factor in an explosion of languages and operating systems and suddenly you have as many developers writing low level wrappers for database drivers as you do building your actual product. Instead ODBC provides a standard for database developers to expose functionality without developers having to reinvent new bindings for each new language, database, operating system combination. You can read more here

Next Up

Now that we know ODBC is an API I want to look at the architecture of ODBC. In my next post I will cover the driver manager, ODBC drivers and the ODBC API. After that I plan on exploring ODBC from the application layer through the driver layer with Python and pyodbc looking to trace internals and see exactly how and where different layers connect.

Contact

If you have experience with ODBC internals, want to correct something I’ve written or just want to reach out feel free to follow up via email or on .

I also have a repo with the material I used for a presentation on this at the Louisville DerbyPy meetup in March of 2019.

What is ODBC Part 2 of 3

2026-08-24T23:19:53.977854+00:00

In the first article I mentioned that ODBC (Open Database Connectivity) is a specification for a database API creating a standard way for applications to interact with various databases via a series of translation and application layers. To create this standard abstraction ODBC has two components, the driver and the driver manager.

ODBC Driver

Within ODBC the driver encapsulates the functionality needed to map various functions to underlying system calls. This functionality spans calls to connect, query, disconnect and more depending on what the target data source provides. While almost all drivers provide the prior basic interactivity others many expose more advanced functionality like concurrent cursors, query translation, encryption and more. It’s worth reviewing your ODBC driver docs to see what features you might use specific to your data source. While ODBC provides a useful abstraction for connecting to data sources it’s worth using whatever additional functionality is available to make your application perform it’s best and keep your data secure on the wire.

ODBC Driver Manager

Ok so the ODBC driver encapsulates the functionality for interacting with our data source what do we need a driver manager for? First it’s not uncommon that you may want your application to interact with various different data sources of the same type. When this happens the driver manager provides the management and concept of the DSN. The DSN (data source name) contains the information required to connect to the data source (host, port, user etc for more information checkout connection strings since the driver manager can save these to a name you specify. This way you can have one driver (for instance Postgres or Elasticsearch) that can be used to connect to various different data sources from the same vendor. In addition to this the driver manager is responsible for keeping up with what drivers are available on the system and exposing that information to applications. By knowing what drivers and DSNs are available the driver manager can sit in between your application and the ODBC driver making sure the connection information and data passed back and forth is mapped to the right system and that return calls from the driver get mapped back for use by applications.

Next Up

Last up in post 3 I plan on exploring ODBC from the application layer to the driver layer with Python and pyodbc looking to trace internals and see exactly how and where different layers connect.

Contact

If you have experience with ODBC internals, want to correct something I’ve written or just want to reach out feel free to follow up via email or on .

I also have a repo with the material I used for a presentation on this at the Louisville DerbyPy meetup in March of 2019.

What is ODBC Part 3 of 3

2026-08-24T23:19:53.977827+00:00

For more information see part one and part two

Setting Up

Just like any other piece of software we can make use of debuggers to step through our application code and see what is happening with ODBC. To do this with Python you should be running a version with debug symbols included. You can do this via:

git clone git@github.com:python/cpython.git
mkdir debug
cd debug
../configure --with-pydebug
make
make testAdditionally you will want to clone pyodbc so that we can make use of symbols.

git clone git@github.com:mkleehammer/pyodbc.git
CFLAGS='-Wall -O0 -g' python setup.py build

Finally you’ll need some code and a database to interact with. If you want I have an example repo which uses docker to start Postgres and/or MSSQL. It also contains some python example code and pyodbc in the repo for debugging.

One final note, if you wish to explore code all the way into the driver manager and/or driver you will need a debug version of each. For Mac and Linux you can do this with unixodbc found here or here and specify debug with make similar to CPython above. For a debug driver build checkout Postgres psqlodbc.

Stepping through

I’m writing this on OSX, but the concepts are the same regardless of platform. On OSX you can use LLDB or GDB (I used LLDB as a learning exercise), on Linux GDB is probably your go to and on Windows you can use WinGDB or the debugger built into Visual Studio for C/C++.

From the command line start your debugger, or if using GDB/LLDB call your tool with the -f flag specifying you want to load a file and call Python with your debugger so the Python interpreter will run the file inside your debugger.

lldb -f python -- -m pdb main.py

From here you can execute the application, use normal step, thread and frame functions to inspect the stack at different steps or get additional dump file information. Some breakpoints I found interesting can be set with:

breakpoint set --file connection.cpp --line 232
breakpoint set --file connection.cpp --line 52
breakpoint set --file cursor.cpp --line 1100
breakpoint set --file getdata.cpp --line 776runIn case it is helpful you can find an lldb to gdb map [here](https://lldb.llvm.org/use/map.html).

Contact

If you have experience with ODBC internals, want to correct something I’ve written or just want to reach out feel free to follow up via email or on .

I also have a repo with the material I used for a presentation on this at the Louisville DerbyPy meetup in March of 2019.

Using Dataclasses for Configuration

2026-08-24T23:19:53.977814+00:00

Introduced in Python 3.7 dataclasses are normal Python classes with some extra features for carrying around data and state. If you find yourself writing a class that is mostly attributes it's a dataclass.

Dataclasses have some other nifty features out of the box such as default double underscore methods, type hinting, and more.

For more information checkout the docs.

Dataclasses as configuration objects

Recently I’ve had the opportunity to work on a couple of Python 3.7 projects. In each of them I was interacting with many databases and API Endpoints. Towards the beginning of one of the projects I did something like this:

elasticconfig = {"user": os.environ["ESUSER"],
 "endpoint": os.environ["ESENDPOINT"],
 ...
 }When I checked in the code I had been working on one of the reviewers commented that this pattern was normal, but since we were using 3.7 let’s use a dataclass.

import os
from dataclasses import dataclass@dataclass
class ElasticConfiguration:
 user: str = os.environ["ESUSER"]
 endpoint: str = os.environ["ESENDPOINT"]

...Makes sense, but what’s the practical benefit? Before I wasn’t defining a class and carrying around the class model that I’m not really using.

Class attribute autocomplete. I can’t tell you how many times I used to check if I had the right , casing, abbreviation etc for the key I was calling. Now it's a class attribute, no more guessing.
Hook up mypy and find some interesting errors.
Above you’ll notice I used os.environ[]. A lot of people like to use an alternative .get()pattern with dictionaries. The problem is often times a default of None gets supplied and you're dealing with Optional[T], but still acting like it's str everywhere in your code.
postinit
Dataclasses have an interesting method called postinit that gets called by init. On configuration objects this is a handy place to put any validation function/method calls you might build around attributes.
Subjectively elastic.user is faster to type, and more appealing to the eyes than elastic["user"]. So the next time you find yourself passing around configuration information remember dataclasses may be a useful and productive alternative to passing around a dictionary.

Additional Resources

Beyond the docs here are some links I found useful when learning about Python dataclasses.

Creating a Con Badge with PyPortal

2026-08-24T23:19:53.977798+00:00

Recently I've heard about multiple people working on con badges and decided to try my hand at a simple take on the idea. Since I had just recently received my PyPortal Adabox I thought I would use that as my first platform to get started.

From the product page the PyPortal is:

*An easy-to-use IoT device that allows you to create all the things for the “Internet of Things” in minutes. Make custom touch screen interface GUIs, all open-source, and Python-powered using tinyJSON / APIs to get news, stock, weather, cat photos, and more – all over Wi-Fi with the latest technologies. Create little pocket universes of joy that connect to something good. Rotate it 90 degrees, it’s a web-connected conference badge #badgelife.*Like many other CircuitPython powered devices the PyPortal has a great Explore and Learn page available that walks you through getting the right firmware installed as well as providing hardware breakdowns, code demos and FAQ.

Once I had the PyPortal up to date and had gone through a couple demos I landed on having my first badge being a simple menu systems. While many badges will contain easter eggs or ways to interact with other badges I decided to keep it simple for this first run. I wanted my badge to be able to display a couple pieces of static data and have a couple interactive options.

I landed on a Button menu that would show a couple maps, a photo of my badge, a countdown to Gen Con, and a simple D20 roller.

Along the way I made extensive use of the docs and source code that Adafruit provides.

I also found it easy to find documentation for the module I would pull in from the library modules by referencing the list of submodules on Read the Docs

Curiosities

While building my badge I ran into some interesting edges that I hope to explore further. I'm sharing these here just in case somebody else reads this and can avoid similar pitfalls or suggest a different direction.

Large buttons seem to lead to performance and OOM errors
Originally my menu had 8 buttons (one with information about Adafruit, another with information about the project), but that wasn't stable. After 3 or 4 clicks gc or something else couldn't keep up with the memory allocation and the badge would crash with a MemoryError
My schedule was also a menu of buttons originally. This let me setup a list of tuples I could manipulate in code, but when I had 5 buttons span the screen render time was visibly slow, and lead to inconsistent OOM errors.
Different fonts have different performance characteristics
Looking back this makes sense. Different glyphs will have different structures. Depending on that a glyph can place different loads on the system. I tried a few of the "performance" font from GoogleFonts, but ultimately landed on Arial Bold for a font that looked consistent, rendered quickly and didn't have a large file size.
Better ways to sleep?
My badge spends a lot of time in the main super loop polling if a button has been pressed. At this time I don't think CircuitPython supports interrupts. I hope in the future i can figure out a better was to let the device sleep, but capture an interrupt type event such as the display being touched.
PDB for CircuitPython
I spent a lot of time running snippets in the REPL. This is a nice experience to have for an embedded device, but I do miss having PDB or WebPDB to drop a breakpoint in my code, let it run and then inspect the stack, heap etc from a given point in my program. I believe MicroPython contains this functionality so I'm guessing it's possible with CircuitPython I just haven't dug in to make it happen yet.

Lessons Learned

Similar to the interesting behaviors I found above I learned a bit about developing with CircuitPython and how it can differ from my day to day Python development along the way.

Python data structure sizes
Many code bases make liberal use of dictionaries. In fact some say that Python is built on top of the dict data structure. It's incredibly useful to look items up by key, and provides some human readability over indexing into a collection with no reference beyond position. That said Dictionaries are one of the largest builtin Python objects. One of the reasons for this is something called a load factor that I won't go into now, but suffice to say as you add more objects to a dictionary and it approaches a given load factor it will automatically grow in size. Because of this in a memory constrained environment I found myself removing dictionaries or list of dicts and using more tuples and list of tuples.
Take out the garbage
Python Garbage Collection is handled via reference counting. Because of this it's important to think about when an object (especially large objects ) you are using come in scope, and when they leave scope. In an environment like CircuitPython you may also want to call gc.collect() when you leave scopes with large objects to make sure they are garbage collected before you carry on. This can help avoid some OOM errors.
Careful wih that indirection.
I found myself removing helper functions and other pieces of code that helped keep things "clean". Often times I did this because I was hitting performance of OOM errors that would go away when I put the functionality in the parent scope. Because of this I have repeated code, and code that isn't what I would expect to pass code review day to day, but it works, achieved stability and gave the performance I'm looking for on my badge.
Testing and profiling for this environment is still a challenge for me.
I would love to be able to write a test for my function and then profile that test to capture things like stack depth, object sizes, timing, etc. And since I have a test I could do this N times to see what kind of behaviors emerge. Instead right now I manually make a change and validate. Because of this I think I'm building an intuition of what is happening, but I can't verify it which leads me to assume my understanding has gaps, and potentially wrong assumptions today. Making this better can help me address the point above.

Next Steps

So with v1 of the badge prepared and ready for Gen Con 2019 I'm going to step back and work on some other items in this space. While working on the project I found out that labels don't support an orientation flag. After mentioning this in discord I opened an issue on Github with some encouragement from @ladyada. Hopefully I can spend some cycles working on that.

I also continue to think about how to write tests for CircuitPython. Since the runtime is tied to the boards it's not as simple as running the code in a CPython unittest environment. While there is a lot of overlap in the API and behavior it's not a one to one match. I think being able to test the code would lead to faster development cycles and would open the door to better profiling and understanding of my applications behavior.

Finally I plan to back up and read Making Embdedded Systems by Elicia White and visit some other embedded getting started materials. While I had a lot of ideas for this project (and I'm happy with how it turned out) I realized that since I'm not as familiar with this type of hardware environment I struggled at times to get the functionality I was looking for with the performance I needed.

Acknowledgements

Thanks to the team at Adafruit. The devices they build and the creation of CircuitPython has lead me to pick up a hobby that continues to be fun and encourages me to think in new ways about hardware and the programs I'm writing. Additionally Adafruit has a discord where many people have been incredibly patient and helpful as I learn and ask questions.

Contact

I've really enjoyed working on this project. If you want to reach out feel free to follow up via email or on.

You can find out more about the badge and source code in the repo

More Photos

Some additional photos of the portal. I've ordered a case off thingiverse , but using the Adabox case while I wait.

Connected Roomba - Possibilities

2026-08-24T23:19:53.977782+00:00

A couple years ago at PyCon I received a kit from Adafruit containing the Circuit Playground Express. After going through the Hello World examples I boxed it up I didn’t have a project ready to go. Fast forward to the winter of 2018 when I decided I would like to be able to start our Roomba away from home because of the noise it makes, and suddenly I had the project I was looking for. Digging around I found out about the Roomba Open Interface and set out to start talking to my Roomba with CircuitPython.

Will this work

After reading through the Open Interface spec I decided it should be possible for me to control the Roomba by using the Circuit Playground Express that I had waiting on the shelf. Getting the kit out and using the clips available I connected the Playground Express TX to the Roomba RX, opened a REPL and tried to wake the Roomba, but received no response.

After some more searching I found out that certain series firmware will not respond to wake commands after 5 minutes without a signal. Knowing this, and pressing the power button once to wake the Roomba, I was able to START, STOP and DOCK the Roomba with controller code running on the Playground Express.

Next steps

After spending some more time confirming command structures, documentation and behavior between CircuitPython and the Roomba Open Interface I decided to make things easier by building a package to abstract the interactions. With basic wiring and command functionality confirmed I decided it was time to start looking at making remote signalling covered in part 2.

Connected Roomba Remote - LoRa

2026-08-24T23:19:53.977765+00:00

With a basic setup working the next thing I wanted to do was make communication wireless. Thinking about my options I ruled out using WiFi pretty quick since I didn’t want to worry about discovery and router issues. I thought about Bluetooth since I could send commands from my phone to the board on the Roomba, but decided against it due to my lack of mobile programming experience and not wanting to add yet another new thing to learn. (Side note I’ve since learned about the Bluefruit app) Looking at the other Feather options I decided to make use of LoRa for my communication layer since it would be easy to use, my packets are tiny and I didn’t have to worry about software beyond CircuitPython.

The boards

With the protocol determined and sticking with CircuitPython I found a Feather with LoRa built in, and a Pi Zero Bonnet with some buttons and a small display that would make testing easier. After reading through thedocs and tutorials for both boards I began work on signalling the Roomba to start with the push of a button.

One of the first road blocks I ran into was of my own creation. The Roomba library I wrote for prototyping was too big for the Feather. The good news was OpenInterface was still useful, and there was plenty of room for the base class after removing the debug and abstraction code, so I only compiled the commands I knew I was going to use for version 1 and continued moving forward.

class OpenInterface:
 def init(self, txpin, rxpin, brcpin, baudrate=115200):
 self.board = busio.UART(txpin, rxpin, baudrate=baudrate)
 self.txpin = txpin
 self.rxpin = rxpin
 self.brcpin = brcpin
 self.brcpin.direction = digitalio.Direction.OUTPUT
 self.baudrate = baudrate
 self.stopped = True def start(self):
 if self.stopped:
 self.wakeup() for command in (b"\x80", b"\x83", b"\x87"):
 self.board.write(command) def stop(self):
 for command in (b"\x85", b"\xAD"):
 self.board.write(command)
 self.stopped = True def wakeup(self):
 for i in range(3):
 self.brcpin.value = False
 time.sleep(0.5)
 self.brcpin.value = True
 time.sleep(0.5)
 self.brcpin.value = False
 time.sleep(0.5) self.stopped = False

After stripping things down and getting the Feather to start and stop the Roomba from the REPL I turned my attention to the Pi.

With the Pi Zero providing more resources installing the OS, setting up SSH and compiling Python 3.7 took more time then getting the Circuit Python libraries working. Blinka worked like a charm and following the docs from above I had a quick script to send start and stop packets via LoRa working in no time.

while True:
 try:
 if not startbutton.value:
 msg = "Starting Roomba."
 logger.info(msg)
 rfm9x.send(bytes("1", "ascii"))
 display.fill(0)
 display.text(msg, 25, 15, 1)
 elif not stopbutton.value:
 msg = "Stopping Roomba."
 logger.info(msg)
 rfm9x.send(bytes("0", "ascii"))
 display.fill(0)
 display.text(msg, 25, 15, 1)

The display on the bonnet was a nice touch so that I could watch the Feather in a terminal while the Pi let me know immediately which button was pressed and which command I should expect the Feather to receive.

Next Steps

With the boards talking to each other and the ability to start/stop the Roomba with the press of a button the last thing to do was make this work when we are not at home. While LoRa has a pretty good range I wanted this to work for my wife and I without having to worry about where we are. In part 3 I look at making this work with SMS.

Connected Roomba - SMS

2026-08-24T23:19:53.977746+00:00

As I mentioned before one of the primary reasons for starting this project was to let my wife and I start the Roomba when we are not at home. One device that most of us take everywhere is our phone. An easy way to to send information from your phone without a custom app, stack and hassle is SMS. While it’s easy to broadcast receiving that message can take a little work.

Twilio

Luckily monitoring a number for messages is pretty much a solved problem. Twilio offers an easy way to setup number with an attached webhook for receiving and sending messages. They also have a nice Python tutorialthat had me up and running in about 10 minutes. Since I was already using the Pi Zero to send commands to the Roomba setting up a script to watch for an SMS message and pass on the new command was simple enough.

import busio
import board
import adafruitrfm9x
from digitalio import DigitalInOut
from flask import Flask, request
from twilio.twiml.messagingresponse import MessagingResponseCS = DigitalInOut(board.CE1)

RESET = DigitalInOut(board.D25)
spi = busio.SPI(board.SCK, MOSI=board.MOSI, MISO=board.MISO)
rfm9x = adafruitrfm9x.RFM9x(spi, CS, RESET, 433.0)
rfm9x.txpower = 23app = Flask(name)

@app.route("/sms", methods=["GET", "POST"])
def smsstartroomba():
  """
  When a message is received determine which
  signal to send the Roomba and reply
  to the sender.
  """** *txt = request.values.get("Body").lower() if txt == "start":
  msg = "Starting the Roomba."
  cmd = bytes("1", "ascii")
  elif txt == "halt":
  msg = "Stopping the Roomba."
  cmd = bytes("0", "ascii")
  elif txt == "dock":
  msg = "Roomba beginning to dock."
  cmd = bytes("2", "ascii")
  else:
  msg = "Unknown command. Continuing."
  cmd = None if cmd:
  rfm9x.send(cmd) resp = MessagingResponse()
  resp.message(msg) return str(resp)

if name == "main":
 app.run(debug=False)

And with that the same board I had used to test sending messages in response to button clicks can now receive SMS payloads and translate that into a command that the Feather will use to start, stop or dock the Roomba.

Next Steps

With all the pieces assembled and working the last thing to do for version 1 was setup some redundancy, restart everything and make sure it all worked as expected without my intervention.

Connected Roomba - Wrapping Up

2026-08-24T23:19:53.977727+00:00

With everything working I wanted to make sure I didn’t have to reset everything anytime an odd decode error occurs, something loses and regains power, etc. For the Feather attached to the Roomba handling this is pretty straight forward. Everything is already running in a super loop, so all I need to add is a try/exceptblock to the while loop and discard errors. Doing the same thing for the Pi was again straight forward, but since it is running Linux I needed to make sure the applications handled failures, and that the scripts restart if the board restarts, the OS bounces, etc.

Similar to the Feather code I wrapped everything in a while loop, added exception handlers, but I also added logging so that I could understand if errors are created by the OS, the code or something else:

import logging

LOGFORMAT = "%(asctime)s:%(levelname)s:%(message)s"

logging.basicConfig(
 filename="/home/pi/logs/button.log",
 level=logging.INFO,
 format=LOGFORMAT,
 datefmt="%m/%d/%Y %I:%M:%S %p",
)

logger = logging.getLogger(name)...if name == "main":
while True:
 try:
 ...
 except BaseException as e:
  logger.exception(e)
 pass

And since this is running on Linux setting up cron to handle starting the applications after reboot was one command away.

sudo crontab -e@reboot cd /home/pi/ && /home/pi/.virtualenvs/lora-pi/bin/python /home/pi/projects/roombasupervisor/buttonlistener.py 2>&1 >> /home/pi/logs/button.log
@reboot cd /home/pi/ && /home/pi/.virtualenvs/lora-pi/bin/python /home/pi/projects/roombasupervisor/smslistener.py 2>&1 >> /home/pi/logs/sms.log
@reboot sleep 10 && cd /home/pi/ && /home/pi/ngrok http 5000 2>&1 >> /home/pi/logs/ngrok.log
@reboot sleep 20 && curl http://127.0.0.1:4040/api/tunnels 2>&1 > /home/pi/logs/ngrokdetails.log

Wrapping Up

Since this was my first project interacting with an embedded system I learned quite a bit along the way. Abstractions are something that are useful, but can add bloat and load that won’t work in constrained environments. I wasn’t able to use the Roomba library I built with the Circuit Playground on the Feather that I connected to the Roomba. CircuitPython made learning and prototyping easy with a REPL and constant connection to the Open Interface. It also allowed me to focus on learning more about the boards and data interactions since I wasn’t busy rebuilding my software toolchain for a new environment. That said it has also inspired me to learn more and dig deeper into the embedded world since there are a lot of things I can’t user (interupts). There is a lot that I don’t know or understand yet, but with the help of some books and boards I am sure I will be busy expanding my understanding for the next few years.

Contact

I really enjoyed working on this project. If you want to reach out feel free to follow up via email or on .

Postgres Advisory Locks with Asyncio

2026-08-24T23:19:53.977710+00:00

Recently, here on the Cloud team at Elastic we started working on building a new service in Python 3.7. This service fetches data from a Postgres database, transforms it, and then submits that data to another service. Like many cloud-based services, ours runs in an orchestrated container environment where N instances can be running at any time. Often that’s a good thing, but our service has a few critical sections where only one instance should be able to process data. Since we are retrieving data from Postgres, we decided to go ahead and make use of advisory locks to control these critical sections. In this article I want to explain what advisory locks are, provide an implementation, and test to verify functionality.

Advisory locks

Postgres provides the ability to create locks that only have meaning within the context of your application. These are advisory locks. You use advisory locks to control an application’s ability to process data. Anytime your application is about to enter a critical path, you attempt to acquire the lock. When you acquire the lock, you can safely continue processing.

async with AdvisoryLock("goldleader", dbconfig) as connection:If it fails, then your application may retry, wait, or exit. Since this lock is external to the application, this allows for multiple instances of the application to run while providing safe critical path concurrency.

Building the lock

As part of our work, we wanted to make using advisory locks easy. To do this, we created the PostgresAdvisoryLock context manager. Since this is meant to be used with asyncio and asyncpg, we control the acquisition and release of the lock via aenter and aexit.

class AdvisoryLock:
 async def aenter(self) -> asyncpg.connection.Connection:
 self.lockedconnection = await asyncpg.connect(...)
 await self.setlock()
 if self.gotlock:
 return self.lockedconnection
 else:
 if self.lockedconnection:
 await self.lockedconnection.close()
 raise AdvisoryLockException async def aexit(self, exctype, excval, exctb):
 await self.release()Now this can be called like any other async context manager.

async with AdvisoryLock(config, "appname") as connection:
 val = await connection.fetchrow("SELECT 1")

Testing the lock

Now that the PostgresAdvisoryLock class is implemented, we need to test it. To start we verify the base functionality by acquiring the lock, running a query, and validating we can't get the lock inside the same scope. I recommend using the asynctest library to help work with asyncio inside unittest.

async def testgetresultswithlock(self):
 async with AdvisoryLock("goldleader", dbconfig) as connection:
 val = await connection.fetchrow("SELECT 1;")
 self.assertEqual(val[0], 1) async def testlockpreventssecondlock(self):
 with self.assertRaises(AdvisoryLockException):
 async with AdvisoryLock("goldleader", dbconfig) as connection:
 await connection.fetchrow("SELECT 1;")
 async with AdvisoryLock("goldleader", dbconfig) as secondconnection:
 await secondconnection.fetchrow("SELECT 1;")

Since we are going to use this to control the execution of code across many processes, we also need to verify external process behavior. To do this we use the asyncio subprocess.createsubprocessexec function to create a new process. This process attempts to get the lock our main process already has, and it should fail.

async def testadvisorylockpreventsaccessfromseparateprocess(self):
 with self.assertRaises(AdvisoryLockException):
 async with AdvisoryLock("goldleader", dbconfig) as connection:
 proc = await asyncio.subprocess.createsubprocessexec(
 sys.executable,
 "-c",
 executable,
 stderr=asyncio.subprocess.PIPE,
 )

Wrapping up

When we started to build our new application, we knew we would be waiting on the network and database. Since we also had work that could happen during the wait, we decided to use asyncio. Additionally we identified a critical path where we used Postgres to achieve concurrency control. To make critical path control easier we created a module and a series of tests. Once finished we realized this might be helpful to others looking for the same control, or as a reference for those learning to test with asyncio.

You can find the full implementation and Docker setup on Sourcehut.

Hackaday Connected World Follow Up

2026-08-24T23:19:53.977693+00:00

Recently Hackaday announced the results of the Connected World contest. It made my day when I read Sophi’s email telling me that ConnectedRoomba was one of the OSHPark certificate recipients. What may have seemed like a small announcement meant a lot to me. I’m still fairly new to this area of computing, and without formal training. Instead I spend a lot of time reading, listening and building to learn everything I can. Validation and success no matter how big or small help us all stay motivated to continue in our pursuits. Thank you to everybody at Hackaday for setting up a community and contest for us all to continue learning, sharing and hacking together.

Whats next

Everybody starts somewhere and the contest pushed me to get started on my first homebrew project. As part of this I found a lot of new areas to study up on. I’ve enrolled in the edX Embedded Systems course. If you’re taking that too reach out as I’d love to have a group to work with. Additionally I want to migrate the ngrok setup in my project to a route on my own domain, understand secure LoRa transmission and expand my electronics knowledge.

On the board front I found this interesting Feather PCB from @tannewt while debating what to do with the OSHPark certificate. I recently backed the FOMU and learned of FuPy so this seems like an interesting PCB to pick up, order some components and start learning electronics at a whole new level.

Congrats to everybody that participated in the Connected World contest. Have fun hacking on whatever comes next! Thank you Hackaday, DigiKey and OSHPark for kick starting this new learning path :).

Contact

If you want to chat feel free to follow up via email or on Sourcehut.

EdgeRouter X DDNS with Gandi

2026-08-24T23:19:53.977676+00:00

I recently setup a VPN for my home network. To make use of it from remote networks I need to be able to resolve the public IP of my router. Instead of hard coding the IP I setup an domain with Gandi and created an A Record that I update from my router.

Fetching and reporting your IP

This part was fairly easy. With a quick search I found that somebody else had already solved the problem of reporting the public IP from an Ubiquiti router to Gandi! Checkout their work here. Their README provides a nice easy walk through of the setup.

Scheduling it

With the above script updated and working on my router the next thing to do was schedule it.

*Quick note only specific directories persist between firmware updates on the EdgeRouter. Because of this I suggest putting the script above in config/scripts/ or config/user-data.*The EdgeRouter OS provides a helper utility called task-scheduler which wraps cron. The benefit of task-schedule is that is saves our commands to config so they persist through upgrades. ssh into your router:

ssh <user>@<router>
configure
set system task-scheduler task ddnsupdate
set system task-scheduler task ddnsupdate crontab-spec '0 5 * * 0'
set system task-scheduler task ddnsupdate executable path '/config/user-data/'
commit
save
cat /etc/cron.d/vyatta-crontab

EdgeRouter X PiHole Setup

2026-08-24T23:19:53.977662+00:00

I’ve seen a few post from people asking for help adding a PiHole to their network with an EdgeRouter. One solution I’ve seen is to use brittanics black-list. This is nice for those wanting to run software on their router, but I didn’t want the load, and I want the functionality that the PiHole provides. Hopefully this guide help those looking to add a PiHole in the future.

Setting up the PiHole

I’m going to assume you’ve already installed PiHole on your device. If not the docs are a great place to start. If you set this up on a Raspberry Pi I encourage you to disable autologin, add a new user, add the user to the sudo group and enable ssh. For more information checkout the RaspberryPi docs.

Configuring EdgeRouter to use the PiHole

I’m assuming your edgerouter is the DHCP server on your network.

With PiHole installed, connect the device to your network (preferably wired) and login to the Ubiquity web ui. Click on the Services tab.

On this tab you should see an action button on the right side of the screen across from your DHCP information. Click it, and select configure. In the pop up window select Leases, and you should see the device your PiHole is on. Click the Static MAC/IP Mapping tab and give this device a static IP.

While we are here click the details tab and add the IP as DNS 1.

Return to the main web ui Dashboard. At the bottom of the screen you should see a system tab with an arrow on the far right.

Click it and on the right side of the pop up add the IP you just assigned the PiHole as your Name Server.

With this in place login to your PiHole, navigate to network and you should see your router listed. The device should be highlighted green with a query count indicating that traffic is flowing through the PiHole as expected.

EdgeRouter X Home VPN Setup Pt 2

2026-08-24T23:19:53.977649+00:00

I am not a network or sysadmin by day. This is something I’m actively learning on and figuring out. If you see something wrong or have suggestions I would love to hear about it.

In part one we configured the network. Now we are ready to install Wireguard and create our interface. Before I jumped into doing this I referenced these post and docs.

To get started ssh into the EdgeRouter device.

ssh <user>@<edgerouterip>Once logged in we need to pull, install the Wireguard .deb.

cd /tmp*# Download the appropriate version, pay special attention here, if you are using the Ubiquity v2 firmware
# you will need the wireguard-v2-*
*curl -qLs https://github.com/Lochnair/vyatta-wireguard/releases/download/0.0.20190913-1/wireguard-v2.0-e50-0.0.20190913-1.debsudo dpkg -i wireguard.debAn important note from the source repo

Note that since Wireguard is not software bundled with the EdgeOS firmware, firmware upgrades necessitate re-installing the Wireguard debian package. Once the wireguard package is re-installed re-applying the existing Vyatta config file, or rebooting will restore your interfaces.

First things first we need to generate a private key for the router, and a public key to share with clients.

$ wg genkey | tee /dev/tty | wg pubkey
123ddgqeqe123123

This will output two lines. The first is your private key, the second is your public key. Keep these secure, but ready since you will need to provide the public key to all clients.

With our keys generated we can now configure the Wireguard interface. Ours will be wg0. In the terminal:

configureset interfaces wireguard wg0 address 192.168.55.1/24
set interfaces wireguard wg0 listen-port 51820
set interfaces wireguard wg0 route-allowed-ips true
set interfaces wireguard wg0 private-key <private-key-from above-output>commit
saveThis created a new wireguard network on 192.168.55.1/24; listening to port 51820 and will route all the traffic through wg0.

Now keeping our public key ready we can configure a client.

Configuring Wireguard on Ubuntu

If you’re using Ubuntu 19.10 wireguard should be available from apt by default:

sudo apt-get update
sudo apt-get install wireguardWith prior versions:

sudo add-apt-repository ppa:wireguard/wireguard
sudo apt-get update
sudo apt-get install wireguardOnce again we need to generate our keys, now on the client:

wg genkey | tee /dev/tty | wg pubkeyNow, create the wireguard interface, still on the client.

touch /etc/wireguard/wg0.conf
chown root:root /etc/wireguard/wg0.conf
chmod 600 /etc/wireguard/wg0.confsudo vim /etc/wireguard/wg0.conf<--------wg0.conf-------->

[Interface]
Address = 192.168.55.5/32
PrivateKey = <client-private-key>[Peer]
PublicKey = <router-public-key>
AllowedIPs = 192.168.55.0/24
Endpoint = publicipofrouter:51820

Peering the router and client

With the client configured and keeping the public key it generated, return to the router. ssh and run:

set interfaces wireguard wg0 peer allowed-ips 192.168.55.5/32 commit save

Starting your client VPN

With wg0 configured and ready bring up the VPN on our client.

sudo wg-quick up wg0

And verify connectivity by running sudo wg on the client, and router.

Next Steps

With VPN setup I’m now able to access and provide access to my device lab. This also keeps devices using this router that are not part of the lab separated.

Finally if you’re doing this for the first time some next steps you might want to take include:

Switch devices to only allowing ssh via keys.
Switch to a non default ssh port.
Setup fail2ban.
Pickup from here

EdgeRouter X Home VPN Setup Pt 1

2026-08-24T23:19:53.977637+00:00

Recently I got the itch to setup a VPN for my home network to access my device lab on the go, or share with others. My home setup isn’t too complicated, but it’s a bit different from other setups I found when I started down this path.

Network Components: Arris Surfboard SB6141, Ubiquiti EdgeRouter X, Ubiquiti AmplifiHD

**I am not a network or sysadmin by day. This is something I’m actively learning on and figuring out. If you see something wrong or have suggestions I would love to hear about it. **Reach out.

Preparing the network

As my starting point I had used the EdgeRouter wizard for initial setup way back when. The default places the network in the 192.168.1.0/24 range which should be changed to prevent a conflict for devices on remote networks. To add a new dhcp server handing out address in a new range we will use the ubiquiti ui.

To start login to the ubiquiti ui and navigate to the Services tab.

From here you can see + Add DHCP Server on the left side of the screen.

Select Add and configure a new DHCP server leasing addresses in a new range ( 192.168..0).

With this setup the next thing to do is test it works before removing the old DHCP server settings.

Return to your Dashboard, and locate the switch0 interface. To the far right you should see an actions button.

Click, select config, and add a manually configured IP for the dhcp server you just configured (192.168.x.1). With switch0 talking to our new network range return to the Services tab. Click actions on the original DHCP server, select disable, and then logout.

Now you can log back in on the new network range 192.168.x.1. Login, select switch0 from the Dashboardtab as we did earlier, and remove the original DHCP server. For any devices on your network that were active you will need to do a dhclient -r; dhclient to refresh your device (on *nix) ip and lease in the new range.

Next Steps

With the network configured we are now ready to install and setup wireguard Since this has already ran a bit long in the tooth part 2 can be found here.

Review EdX UT601 Embedded Systems

2026-08-24T23:19:53.977621+00:00

I recently completed my first EdX course Embedded Systems Shape the World and wanted to share a little bit about the experience.

For a while now I’ve been exploring various venues for continuing education. The longer I’m in my field the more I learn and then that leads to me realizing how much more I want to learn in new areas. That said I’ve never been great at taking courses that are not self paced partially because week to week my schedule can change dramatically between work and family. Because of this over time I’ve tried out multiple platforms of learning such as Pluralsight, Khan Academy, formal online masters programs etc. All of them have their pros and cons ranging of cost to quality to engaging content.

Last year I started learning more about SoC type hardware via Circuit Playground. This has lead me on an adventure to learn more and more about embedded systems, C and hardware. Most of this has been stitched together from various sources and ad hoc as the need arose in a personal project. Towards the end of summer I decided I wanted to formalize this learning and started to look around. There are online programs from universities like TESU, and individuals offering classes, but I stumbled across the UT 601 class on EdX and realized the setup would be a good fit for me. Additionally EdX offers verified courses with certificates which I thought might be nice in the future.

Signing up and getting verified with EdX was easy. I was able to use my laptop and phone to complete all the task in under 30 minutes. The layout of EdX is very similar to other online learning platforms that I’ve used.

UT 601

Once I started UT 601 I started to run into a few more barriers. The course requires the purchase of a Texas Instruments kit for use throughout, which makes sense this is an embedded systems course. What I wasn’t expecting was the use of Keil. To complete the course I needed to be able to install Keil 4.2, and a simulator DLL (which was pretty neat) on a Windows platform. A couple annoyances there. This is an online course with the goal of global education opportunities, but immediately I'm locked into a platform, and additionally Arm places Keil behind a personal information collection form. I was happy that Microsoft provides a Windows 10 ISO that I could use within a VM to work on the course. After downloading that though I found that VirtualBox didn't pass through the board USB connection so that I could make use of the Stellaris Debugging software/firmware that I would need. After some time fiddling with it I ended up switching to VMWare, and after switching the USB connection to pass through as 2.0 was able to get everything packaged up into a Windows VM with Keil, the Stellaris software, the simulator DLL and the appropriate Keil registry edits. In case it would ever help anybody my VMWare config file is here.

After spending a couple days getting the IDE, hardware and VM all setup and playing well together I dove into the course. Overall I enjoyed it. It exposed me to PIN programming and doing a lot of GPIO work that I haven’t done in the past. Additionally it was a good refresh on concepts at the beginning like pipelining. One thing I did notice is there was a big jump from lab 5 to 6. We went from editing template projects to writing most of the project from the ground. Each section provded a different amount of direction (not gradually declining, but instead seemingly random) on how to complete the lab. New concepts were quickly introduced and some lacking explination such as using the Keil Oscilliscope and Analyzer. Overall it was a good course, but I would suggest dedicating a couple weeks and doing it all at once due to how much it ramps up half way through. The accompanying book is made available in each section and I highly recommend reading it as the videos act more as highlights than covering the material at a level that prepares you for the labs.

The one thing that was a minor annoyance throughout was the reliance on Keil (IDE’s have a place but often hide what the compiler and tools are doing creating a gap in knowing how stuff works) and the problems experienced by taking this course in a VM. Other than that the course was interesting and challenging.

Wrapping up

Overall I’m glad I took and completed UT601. I learned a fair amount, and look forward to taking part 2 after the the new year. EdX is a platform I see myself continuing to use as it’s been super simple, has a range of interesting content, and the course facilitators are really responsive.

Cleaning Airflow Logs

2026-08-24T23:19:53.977606+00:00

At home and work I make use of Airflow to automate various batch/time based task. I’ve even setup a container based Airflow environment to make it easy to bring this up and down.

One of the things you quickly find with Airflow is that while it doesn’t need a lot of resources to run, it can quickly eat up whatever disk space you provide it with logs. When this happens the first knobs to look at turning are your log level and your schedulers dag bag refresh rate. While you may not be refreshing dags often your may want to keep your log level low to capture more data and use your log store to put a TTL on things at the INFO level. Unfortunately you can't completely turn off Airflows disk logging without building in some custom functionality today. To help manage this I wrote a small Python script that handles cleaning up the local logs on a given interval. Note if you're running Airflow in a setup other than LocalExecutor you will want to handle this with something like Cron instead of a dag since you need to clean logs up on the Scheduler, Worker and Webserver.

def truncateprocessmanagerlog(logbasepath):
    """
    The scheduler records all acitivty related to dag processing in the same file.
    This file can grow large fast, and is actively in use. Intead of unlinking the
    file and pulling it out from under the scheduler truncate.
    """
    dagprocessmanagerlog = os.path.join(
        logbasepath, "dagprocessormanager", "dagprocessormanager.log"
    )
    open(dagprocessmanagerlog, "w").close()


def traverseandunlink(fobject):
    """
    Traverse the log directory on the given airflow instance (webserver, scheduler,
    worker, etc) and remove any logs not modified in the last hour.
    """
    for entry in os.scandir(fobject):
        newfobject = os.path.join(fobject, entry)
        if os.path.isfile(newfobject):
            lastmodified = os.stat(newfobject).stmtime
            delta = datetime.utcnow().timestamp() - lastmodified
        if delta > HOURSINMILLISECONDS:
            print(
                f"{newfobject} has not been used in the last hour. \
   \nCleaning up."
            )
            os.unlink(newfobject)
        elif os.path.isdir(newfobject):
            traverseandunlink(newfobject)

The full script is available here.

Providing Context with Mocks

2026-08-24T23:19:53.977591+00:00

Day to day I spend a lot of time interacting with database systems. While this is super useful it can also create issues with testing that have been covered many many times in many other articles.

In some languages isolating database interactions comes from dependency injection and swapping out the interface while testing. I’ve seen this approach when working with C# for instance, but in Python I typically see code bases mocking out these interface points rather than passing in interfaces to functions and classes.

Mocking context

One of my favorite constructs in Python is the context manager. These are incredibly useful objects for defining what the creation and destruction of different interfaces should do viaenter and exit. (Append a for async context managers)

While useful they can be a bit tricky for mocking out in your test, and recently when I started doing just that I couldn’t find any good examples for accomplishing this. Below I’ve provided an example of mocking out a context manager in your test, showing when you are interacting with different parts of your mock and the mocked context manager API.

def updateentity(k, v):
 with pyodbc.connect(cnxnstr, autocommit=True): as cnxn:
  with cnxn.cursor() as crsr:
   crsr.execute()

from unittest import TestCase
from unittest.mock import patchclass InterfaceTest(TestCase):
 @patch("mod.pyodbc.connect")
 def testupdateentity(self, mockcnxn):
 # result of pyodbc.connect
 mockcnxncontextmanager = mockcnxn.returnvalue
 # object assigned to in with ... as con
 mockcm = mockcnxncontextmanager.enter.returnvalue
 # result of with ... as crsr, note the extra enter.returnvalue
 # from the context manager
 mockcrsr = mockcm.cursor.returnvalue.enter.returnvalue
 mockcrsr.fetchone.returnvalue = (1,)

Or if you want to test a sideeffect:

mockcrsr.fetchone.sideeffect**
*self.assertEqueal(....)

Good luck mocking, context is what you make it.

Additional Reading

If you want to know more about context managers in Python checkout:

Parsing Time with Python

2026-08-24T23:19:53.977579+00:00

I recently had the need to measure time and performance on an application that interacted with a lot of on disk files. Most of the time when talking about timing and measurement in Python we see the use of timeitand various built in timing techniques. For this work I wanted a little more information about how the application was interacting with the system, and what the performance looked like from outside the application. Getting a rough view of this is pretty easy on a nix using /usr/bin/time.

Parsing Time

To make use of time you simply call it with your application as an argument. You can find the time args with man time, but on useful one is the -v flag for more system information in the output, and an --output file path. Doing this you get a fair amount of page, time and system information in your output packaged up in a file that you can parse. In my script I'm also including some information in the file name so I can know what source file my application was parsing relating to that time information.

#!/bin/bash

SRCDIRPATH=/data
RESULTS=/profilefor file in $SRDDIRFILES; do
 filename=$(basename -- "$file")
 filebase="${filename%.*}"
 echo $filebase
 /usr/bin/time -v --output=$PROFILERESULTSDIR$filebase.txt cmd args
 echo "done"
donels -1sh $SRCDIRPATH &> profileddirectory.size

Once the application has ran you can see the output of time in your file, but you will also probably notice that it’s just a text blob not ready for aggregation. Overall parsing time is relatively straight forward with one gotcha. I use the below to translate the blob into rows and columns:

import os
from typing import Tuple, List, Any, Union

def formattimeprofileoutput(fpath, fobject) -> List[Any]:
 """
 Takes a directory of files containing the output of /usr/bin/time
 and transforms the time blob data to a series of rows and columns.
 """
 f = os.path.join(fpath, fobject)
 timemetrics = [fobject] with open(f, "r") as tfile:
 for line in tfile:
  if "Elapsed" not in line:
   cleanline = line.lstrip()
   metric, sep, value = cleanline.rpartition(":")
   timemetrics.append(value.strip())
  else:
   # Handling the special case of the Elapsed time
   # format using : in the time formatting.
   cleanline = line.lstrip()
   metric, sep, seconds = cleanline.rpartition(":")
   # we now have something like val = 43.45
   # metric = Elapsed (Wall Clock) time (H:MM:SS or M:ss) 1
   # partition again on metric, then combine back our time.
   metric, sep, minutes = metric.rpartition(":")
   # put time back into metrics
   value = f"{min}:{secs}"
   timemetrics.append(value.strip())
   # setup tool second metric for easier evaluation of
   # time metrics
   minutes = float(int(minutes) * 60)
   seconds = float(seconds)
   seconds += minutes
   timemetrics.append(seconds)
 return timemetrics

Notice the one edge case in Elapsed (wall clock) time...). All other rows end with \n and seperate the metric name from the value with :. Elapsed wall clock time however throws in a couple extra colons for fun. Overall not a big deal, but a little gotcha waiting in the details when going from a string to another object/format.

Using the script above you end up with a collection of rows and columns that you can then use to find out how your application performed for that run instance.

A quick bonus script, since my application was reading in and writing out new files I wanted to include the size of the input files so I could begin to understand the impact of the input file size on the applications time metrics.

import osdef formatsizeoutput(fpath, fobject) -> List[List[str]]:
 f = os.path.join(fpath, fobject)
 sizemetrics = [] with open(f, "r") as sfile:
 for line in sfile:
 metric, sep, filename = line.rpartition(" ")
 sizemetrics.append([metric.strip(), filename.strip()])
 return sizemetrics[1:]

With this and the above time parsing we have the input file size, name, application command, page and time information. More than enough to begin looking at what our application is doing from the outside.

Connected Roomba - Managing State

2026-08-24T23:19:53.977561+00:00

Last year I started work and completed the first prototype for managing a roomba via sms and radio. Overall the prototype was a successful, but over time highly unreliable in the face of failure. Most of this came down to state management for the API endpoint and the Roomba OI (Open Interface) code running on the Feather. This week I had the opportunity to sit down and fix some of that.

The latest version of the project can be found here.

Roomba

In previous version of the application that ran on the Feather listening for messages over radio I had managed the application state in this class:

class OpenInterface:
 def init(self, txpin, rxpin, brcpin, baudrate=115200):
 self.board = busio.UART(txpin, rxpin, baudrate=baudrate)
 self.txpin = txpin
 self.rxpin = rxpin
 self.brcpin = brcpin
 self.brcpin.direction = digitalio.Direction.OUTPUT
 self.baudrate = baudrate
 self.stopped = True

I had done this so that I couldn’t send the Roomba signals that were invalid for a given state based on the Open Interface documentation. The circuitroomba project were I originally implemented this actually did a lot more state management. Overall maybe this would be helpful during application development, but I found it made code on the board unreliable due to the size of the class object in memory and other work going on causing the board to eventually crash over an extended period of time.

The more I thought about this I also realized I had caused an even larger issue. The Roomba itself manages state internally. It has all of the logic laid out in the OI document impelmented internally keeping things “safe” and tracking if a given signal is valid or not. By adding my own state management layer on top of this I opened the door for all kinds of trouble. First if the internal Roomba logic differed from the OI documentation, or I implemented the OI logic incorrectly I would be sending the application developer down all kinds of paths trying to figure out why state transistion and command signals were not exhibiting the expected behavior. Why setup 2 FSMs when one will do, and only one ends up being the true dispatch? If we did this at the sms API layer we could have 3, all with the potential for bugs, unexpected behavior, logic mismatches, timing issues etc. It’s a combinatorial explosion of state management issues.

So stepping back, considering the separation of concerns I determined all the board needed to do was listen for a given signal flag and pass that on to the Roomba. From there the Roomba can determine if the signal should be acted on based on it’s internal state.

The new implementation discards the class object and instead just uses a super loop and signal functions.

while True:
 try:
 packet = rfm9x.receive(1) if packet is not None:
 packettxt = str(packet, "ascii")
 print(packettxt) if packettxt == "0":
 commandreceived(led)
 led.value = True
 stop(bot)
 led.value = False
 elif packettxt == "1":
 commandreceived(led)
 wakeup(brc)
 start(bot)
 led.value = True
 else:
 print("\nUnknown packet: {}\n".format(packettxt))
 except:
 pass

Additionally from time to time signals can have issues that previously caused hanging in the application. Now the logic inside the super loop is wrapped in a try/except to prevent corrupt date from completely crashing the application. Instead failures are ignored and we keep listening for the next signal. While this isn't always a viable solution in the case of signaling the Roomba the stakes are low and this is something I'm comfortable with.

Pi Zero

After fixing up the Feather board code I moved onto the Pi applications. Previously I had setup a Flask application to act as the SMS webhook for Twilio. This worked pretty well and was consistent over time, but there was the occasional hang running on the Zero that led me to look into managing the Python and Ngrok application with systemd. Converting from crontab was fairly easy. I created a few *.service files and placed them in /etc/systemd/system.

[Unit]
Description=sms listener
After=ngrok.service[Service]
Type=simple
User=pi
WorkingDirectory=/home/pi
ExecStart=/home/pi/.virtualenvs/lora-pi/bin/python /home/pi/projects/roombasupervisor/smslistener.py
Restart=on-failure[Install]
WantedBy=multi-user.target

Once the files were created I ran the following commands:

systemctl enable smslistener.service
systemctl start smslistener.service

And now all of the required applications (ngrok, sms listener, button listener) are managed by systemd. This controls their startup better than the previous crontab setup and has the added benefit of restarting the service if it fails our.

Wrapping up

By observing and understanding the ways in which the prototyped system failed I was able to identify areas where behavior and functionality could be simplified resulting in an overall more reliable system. If you have any other tips to share reach out and good luck hacking.

Kicking off Hardware Happy Hour 2020

2026-08-24T23:19:53.977539+00:00

Last year we kicked off the first Hardware Happy Hour in Louisville Kentucky, USA. We had a lot of fun sharing our projects with each other and hearing about all the things being built in our own back yard. If you’re building something you should try looking for an event in your area as more and more are popping up around the world.

Circuit Playground Workshop

This year we wanted to start things off by inviting everybody to build and learn together with the Circuit Playground Express. On January 29th we got together 15 makers and hackers to have some fun with Arduino and Circuit Python making LEDs blink and speakers buzz. To kick things off Auystn introduced the group to the Arduino IDE, getting it to recognize your board setup, and receiving feedback via the serial console. As with any workshop we had plenty of fun figuring out why this and that didn’t work on whatever OS, but in many ways I think that was exposure for those new to working with the boards and tools that unexpected behavior may occur, but we can find a solution.

Once everybody had a board up and working Austyn spent some time getting everybody comfortable with the Arduino syntax and constructs. That turned into showing how to make some noise followed by a quick on/off switch demo. With a couple more code demos and showing off the Arduino code library we decided to switch gears and look at Circuit Python before having some general open make time.

With Circuit Python we had the same demos with a different approach. Instead of using an IDE and editor we showed how you could put the board into bootloader mode and drag and drop the UF2 and code files directly on the board for loading. Along with that we demo’d the ability to use REPL driven development on the boards for quick prototyping and feedback.

Armed with Arduino and Circuit Python we decided it was time for us to step back and let people hack. Some had fun with accelerometer libraries while others scanned colors and lit up LEDs. By the end of the night I was rick rolled by a Circuit Playground.

More photos from the event here

Louisville Hardware Happy Hour 2020

As 2020 continues we have 3 more H3 events planned in Louisville. Similar to our 2019 event we are planning to have a Q2 and Q4 social. If you’re in the area we would love to see or hear about your project over some food and drink at Great Flood. In Q3 we are hoping to acquire some scopes to run a scope tutorial making use of the Circuit Playground boards and teaching attendees how they can see their programs in a new way.

Sponsorship Assistance

As I previously mentioned we are looking to run a scopes workshop this fall. If you or an organization you know is interested in sponsoring this event we are looking for help in acquiring digital scopes to provide attendees with. If you are interested in helping please reach out.

Where to follow

To keep up with future H3 Louisville events we have a group setup on gettogether.community and we are active in the #hardware channel for Louisville Slack.

Events will also be published to the Louisville Tech and H3 Louisville calendars.

You can find our code and presentations on Github.

The Structure and Interpretation of Computer Programs in 2020

2026-08-24T23:19:53.977521+00:00

Last week I had the opportunity to attend a course by David Beazley on SICP (The Structure and Interpretation of Computer Programs). SICP is a book that was first published in 1985 and has grown to have a bit of a reputation in various circles of software engineering. The book itself explores many areas of computer science with a language called Scheme ( a lisp). For the course we made use of Racket and Python to explore those same concepts working through the book with an eye to it’s impact on modern language and design.

A quick note. This course was unlike any other I have been in so far. David is really good at giving learners the time and space to think as he lays out really dense material and like a tour guide provides interesting insights about the landscape. Another great part of the class was the size and how Dave gives people time to engage each other. Each morning we shared ideas, experiences and other stories over breakfast before throwing ourselves into the material breaking for lunch to contemplate what we had just built or discovered rounded off with an afternoon coffee. Overall this was a fantastic educational experience and I hope I get the opportunity to repeat it with some of Dave’s other courses in the future.

Throughout the week we engaged with various problems in the book such as evaluation models, abstraction, symbolic data and more. Each time we approached a problem we were encouraged to think as language designers and implementors rather than language users. In doing so we put ourselves in a different state approaching problem solving by extending the features of our language. This led us to implementing object hierarchy and evaluation models, dispatching state handling and creating custom interpreters with domain specific features to solve problems.

One part of SICP book that stood out was the use of ‘wishful thinking’ as our programming model. We would look at a problem (for instance a constraints issue for assigning tenants to floors) and ask ourselves what a procedure or feature would look like for accepting the data and solving said problem. We would then implement this procedure and even mock calling other procedures that did not exist yet to model what we felt like an optimal interface might look like. From there we would build down implementing each new layer with wishful thinking. This came in contrast to a lot of my day to day experience approaching problems bottom up implementing low level details and data processing logic on our way to an isolated solution. In some ways it felt similar to TDD if you mock out all the functionality you don’t have yet, but again with the top down mindset of I want the langauge to do X for me.

Many other topics are covered throughout SICP. The book itself is very top down starting with the language and going all the way down to implementing a VM/register machine by the end of the book. Over time I may write some more about those topics. All in all the course was incredibly interesting. The conversations around lunch about the nature and philosophy of computing were a lot of fun and spawned by the material being covered as well as the various backgrounds we all had in our day to day work. You don’t leave the course with a new package or framework in your toolbelt. Instead you’ve examined of of the underlying fundamentals of languages, computing and software. This equips you with new models for thinking that will hopefully impact any future computing that you take part in. While the material is dense I think anybody that truly enjoys engaging with our craft would benefit from engaging with the SICP material in this setting.

If anybody has questions about the course I’d be more than happy to talk about the material. For those of you who don’t know David Beazley I would encourage you to visit his site or search his name on YouTube as he has a lot of interesting material that encourages the learner to engage the material.

(define zero(….))

2026-08-24T23:19:53.977504+00:00

A couple weeks ago I had the opportunity to attend the SICP course taught by David Beazley. I’ve written a short summary of my experience here (tldr; take the course if you get the chance). While the course as a whole was challenging and an interesting a couple of the exercises stood out to me, and I wanted to take a moment to share them here.

TRUE

At a deep level our computer is operating super fast on a state of ON/OFF with gates that define logic. Because of this it's an area of interest for me in how we express similar logic in our languages and the statement/operator capabilities we can build from that. Towards the beginning of day two we kicked things off by defining our own boolean logic in Racket. Our first step? Defining TRUE and FALSE.

(define (TRUE x y) x)
(define (FALSE x y) y)(define t (TRUE 0 1))
't
(define f (FALSE 0 1))
'fTake a minute and reread that block, because the first time I did it threw me for a loop. We just passed in the same arguments and got TRUE and FALSE. In Racket, and in this scenario we have defined the behavior of our basic TRUE and FALSE operators. The next challenge we were provided was to implement all boolean logic operators.

(define (NOT x) (x FALSE TRUE))(NOT TRUE)
'f
(NOT FALSE)
't

(define (AND x y) (x y x))
(AND TRUE FALSE)
't(define (OR x y) (x x y))
(OR FALSE FALSE)
'f
(OR TRUE FALSE)
't
(OR FALSE TRUE)
't*;;

Hint: x is not just TRUE/FLASE above. It is a procedure, it takes two arguments. And we now have our own set of truth tables.

Defining Zero

Before working on boolean logic we had been discussing the substitution model of evaluation and what you could express with it. After our truth searching exercise it seemed like looking at how numbers could work might be fun.

(define zero (lambda (f) (lambda (x) x)))
(define two (lambda (f) (lambda (x) (f (f x)))))
(define three (lambda (f) (lambda (x) (f (f (f x))))))

Defining numbers as symbols for the application of a function N times then let us implement addition:

(define (plus a b)
 (lambda (f) (lambda (x) ((a f) ((b f) x))))
 )(define five (plus two three))

Or to make it concrete

(define (inc x) (+ x 1))((five inc) 0)
'5

Numbers are weird and amazing. Ever since I realized that anything we can express in a program (that I’m writing in letters and symbols) can be boiled down to a series of 0s and 1s that were ultimately symbols that could be swapped out I’ve been captivated by the question of what numbers are. We did other interesting and exercises (mutation, building an interpreter, a register machine VM, and generic types), but something about the above left me considering the nature of logic and programming. It’s easy to get lost in the day to day problem solving, but when we get the chance to step back and look at the strangeness of what we are interacting with it can be a lot of fun. Here’s one last thought to have some fun with:

Always returns false, except for zero, because zero says don't do the function so we get back true

(define (zero? n) ((n (lambda (x) #f)) #t))

Create and Apply a Git Patch

2026-08-24T23:19:53.977491+00:00

I’ve been using Source Hut as my primary host for source control and builds for a few months. I really enjoy it, but one of the main things I had to learn up front was how to apply a patch in git. Unlike Github and many other git host Source Hut makes use of the git patch work flow instead of PRs. At first I found this to be a bit frustrating, but I’ve actually come to see the value in the email and patch workflow that is different from the IM and PR work flow that many of us are used to. Hopefully this helps somebody else that is learning to use patches in the future.

Build your feature or modify your code on a separate branch:

git checkout -b ...
git add ...
git commit
git push

Prepare a patchset:

git format-patch main

Alternative for a patch directory

git format-patch main -o patches

Or login and find the link to download the patch:

curl -O https://github.com/n0mn0m/circuitroomba/commit/ae635ce6533e33ff5277a0428a59c736a98649d6.patchls | grep ".patch"

Switch back to main:

git checkout main

Check the patchset changes

git apply --stat ae635ce6533e33ff5277a0428a59c736a98649d6.patch

Check for errors

git apply --check ae635ce6533e33ff5277a0428a59c736a98649d6.patch

Assuming the above command doesn’t generate any errors you are ready to apply a clean patch. The git amcommand below includes the --signoff flag for others to view.

git am --signoff < ae635ce6533e33ff5277a0428a59c736a98649d6.patch

And with that the patch has been applied to your main branch. Run your test again for sanity sake and push main.

Train All the Things - Planning

2026-08-24T23:19:53.977478+00:00

Earlier this year Hackaday announced the Train all the Things contest. I immediately knew I wanted to submit something, but figuring out what to build took me a little bit. For my side projects I like to make something that is useful to me, or somebody I know; while also learning something new. A few days after the contest was announced my daughter was in the basement playing outside my office/homelab when I remembered my wife had asked me if there was a way for her to know when I was working with somebody so that they could avoid coming down in the basement. I thought a voice driven display could be a fun solution.

Choosing Tools

After deciding on the project the next thing I wanted to figure out was what new boards I would need (if any) and how I would build my model. After doing some research I landed on Tensorflow as my path forward for deploying a model to a microcontroller. Having used Tensorflow the barrier for model creation is a bit lower, but I am really curious about Tensorflow Lite and the potential it provides. Additionally a relatively new book TinyML looks like a good resource to use along the way.

After settling on TF Lite the next thing was picking a board. Most of my embedded experience has been with CircuitPython and Rust. For this project I thought it would be fun to learn something new. The Espressif ESP-EYE caught my eye as an interesting board known to work with TF Lite. I’ve seen the ESP32 and 8266 in a lot of other projects, so learning the ESP toolchain seems valuable. Additionally a lot of the Espressif ecosystem seems to be built around FreeRTOS which provides a whole other avenue of learning and hacking.

Finally I will need a way to let somebody know when the model has picked up voice activity, to signal that I’m currently busy in the lab. The ESP32 has a WiFi chip providing the ability to send and receive signals via TCP if we want. The ESP-EYE has that built in, and I happend to have a PyPortal (with an ESP32) that could make a great display checking for a status using WiFi too. To signal from one to the other I decided to have some fun and use Cloudflare Workers K/V to set a bit from the ESP-EYE that would be read by the PyPortal at a given time interval to set the display.

Putting it all together the initial idea looks something like this:

Which allows me to have a small board in my homelab listening and the display above the stairwell where somebody can get a status update before they ever come down.

The code, docs, images etc for the project can be found here and I’ll be posting updates as I continue along to HackadayIO and this blog. If you have any questions or ideas reach out.

A Simple Status Page

2026-08-24T23:19:53.977464+00:00

I have a bad habit of creating side projects for my side projects.

A couple months ago I switched from running my blog with Pelican and Gitlab Pages to Zola and Cloudflare Workers. I didn’t do a write up on it, but if you’re interested there’s a good post by Steve Klabnik to get you started. It was a surprisingly easy switch, and gaps between writing haven’t been as difficult with the better tools. After getting that setup I read about Cloudflare Workers KV , thought it sounded really neat and started to think about what I might build.

On another project I need to signal between different systems a simple status. Naturally that lead to me building a status page. I setup a Cloudflare Worker that receives POST from N systems, stores the date of the last POST uses that to provide a status when asked.

const setCache = (key, data) => LOCALSTATUS.put(key, data);
const getCache = key => LOCALSTATUS.get(key);function sleep(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

function dateToStatus(dateTime) {
 var isoDateNow = Date.now();
 var dateDiff = (isoDateNow - dateTime);
 if (dateDiff < 180000) {
  return 1
 } else {
  return 0
 }
}

async function getStatuses() {
 const cacheKeys = await LOCALSTATUS.list();
 while (!(cacheKeys.listcomplete === true)) {
  sleep(5)
 }

 const numKeys = cacheKeys.keys.length;

 var statuses = []; for (var i = 0; i < numKeys; i++) {
  var c = cacheKeys.keys[i];
  var epcDate = await getCache(c.name);
  var data = {date: Number(epcDate), name: c.name};
  data.strDate = new Date(data.date).toISOString();
  data.status = dateToStatus(data.date);
  data.statusIndicator = getStatusIndicator(data.status);
  statuses.push(data);
 }

 const body = html(JSON.stringify(statuses || [])); return new Response(body, {
  headers: { 'Content-Type': 'text/html' },
 });
}

async function getStatus(cacheKey) {
  var cacheDate = await getCache(cacheKey); if (!cacheDate) {
    return new Response('invalid status key', { status: 500 });
  } else {
   var status = dateToStatus(cacheDate);
   return new Response(status, {status: 200});
  }
}

async function updateStatus(cacheKey) {
  try {
   var isoDate = Date.now();
   await setCache(cacheKey, isoDate);
   var strDate = new Date(isoDate).toISOString();
   return new Response((cacheKey + " set at " + strDate + "\n"), { status: 200 });
  } catch (err) {
   return new Response(err, { status: 500 });
  }
}

async function handleRequest(request) {
  let statusKey = new URL(request.url).searchParams.get('service');
  let queryType = new URL(request.url).searchParams.get('query');
  if (request.method === 'POST') {
   return updateStatus(statusKey);
  } else if (queryType === 'simple') {
   return getStatus(statusKey);
  } else {
   return getStatuses();
  }
}
addEventListener('fetch', event => {
 event.respondWith(handleRequest(event.request))
})

With that anything that can POST can "check in" with the endpoint. You can see it working here. I also went ahead and wrote a simple systemd service that I can drop on to different machines I want to have report in to the endpoint.

[Unit]
Description=Regular check in
Wants=check-in.timer[Service]
Type=oneshot
ExecStart=/usr/bin/curl -X POST https://status.slower.earth/?service=JETSON[Install]
WantedBy=multi-user.targetAnd a timer for the service.

[Unit]
Description=Run checkin every 2 minutes
Requires=check-in.service[Timer]
Unit=check-in.service
OnUnitInactiveSec=1m[Install]
WantedBy=timers.target

This was a fun “Serverless/FaaS” experiment that actually let me know my ISP was having an outage one morning before work. I’ve used other Functions as a service on other cloud platforms and while they all provide slightly different functionality (For instance Cloudflare being a CDN and the V8 isolate setup) Cloudflare Workers has been really easy to work with and a lot of fun to build experiments on. They even have a web playground that you can start with.

Two things I do wish were easier are interacting with K/V from Rust. This is probably partially related to how new I am to Rust, but working with K/V from JS is super easy, while this thread documents another experience with Workers and Rust in more detail. Another mild annoyance is working with different workers from the same machine and how API keys are handled. There are some suggestions for this, but non of them feel ergonomic at this time. Other than that my experience with Workers and K/V has been great and I’ve already got more ideas for future experiments.

The code, docs, etc for the project can be found here. If you have any questions or ideas reach out.

Train All the Things — Signaling

2026-08-24T23:19:53.977451+00:00

After figuring out what I was going to use for my project I started work with things I know. I already had some experience with Cloudflare workers building a home system status page, and Workers K/V makes storing and fetching data quick and easy. I ended up with a simple endpoint that I POST to set a bit after keyword detection, and the PyPortal retrieves that status to determine what to display:

const setCache = (key, data) => SIGNALS.put(key, data);
const getCache = key => SIGNALS.get(key);async function getStatus(cacheKey) {
 var serviceStat = await getCache(cacheKey); if (!serviceStat) {
  return new Response('invalid status key', { status: 500 });
 } else {
  return new Response(serviceStat, {status: 200});
 }
}

async function setStatus(cacheKey, cacheValue) {
 try {
  await setCache(cacheKey, cacheValue);
  return new Response((cacheKey + " set to " + cacheValue + "\n"), { status: 200 });
 } catch (err) {
  return new Response(err, { status: 500 });
 }
}

async function handleRequest(request) {
  var psk = await getCache("PSK")
  let presharedKey = new URL(request.url).searchParams.get('psk');
  let statusKey = new URL(request.url).searchParams.get('service');
  let statusValue = new URL(request.url).searchParams.get('status'); if (presharedKey === psk) {
  if (request.method === 'POST') {
   return setStatus(statusKey, statusValue);
  } else if (request.method === 'GET' && statusKey) {
   return getStatus(statusKey);
  } else {
   return new Response("\n", { status: 418 });
  }
  } else {
   return new Response("Hello")
  }
}

addEventListener('fetch', event => {
 event.respondWith(handleRequest(event.request))
})

Nothing tricky happening above, just checking the request, and calling the appropriate function to store or fetch the status bit. With the function deployed to my Cloudflare Worker and verified with some GET and POST calls I was ready to move on to the display.

The code, docs, images etc for the project can be found here and I’ll be posting updates as I continue along to HackadayIO and this blog. If you have any questions or ideas reach out.

Train All the Things — Display

2026-08-24T23:19:53.977436+00:00

Continuing my project with things I know the PyPortal display was up next. Last year I spent a few weeks playing with the portal to make a badge at Gen Con and had a lot of fun with it. Since that time CircuitPython 5 has been released and the portal now expects a few new modules which were easy enough to download and send to the board. The PyPortal makes it incredibly easy to point at an endpoint to fetch data:

import board
from adafruit import PyPortalpyportal

pyportal = PyPortal(
 url=<your url here>,
 default="green.bmp"
)

status = pyportal.fetch()
print(status)

With that small snippet we have our status, and all we need to do is put that in a loop to set the background depending on the bit returned.

import board
from time import sleep
from adafruitpyportal import PyPortaltry:
from secrets import secrets * # noqa

except ImportError:
 print("WiFi secrets are kept in secrets.py, please add them there!")

raisepyportal = PyPortal(
 url=secrets["signal"],
 defaultbg="green.bmp"
)

current = 0

while True:
 status = int(pyportal.fetch())
 if status == 0 and status == current:
     pass
 elif status == 0 and status != current:
     pyportal.setbackground("green.bmp")
 current = 0
     elif status == 1 and status != current:
 pyportal.setbackground("red.bmp")
     current = 1
 elif status == 1 and status == current:
     pass
 sleep(30)

Even though it’s a small snippet I want to point out a couple things. First I’m wrapping the return from fetch in a cast to int. If you use Python, but you are new to CircuitPython this may seem odd. If you don't do this and try to compare a string to an int you're probably not going to get the result you expect. Try it out in a repl and then follow up with CircuitPython Essentials . Also I'm only changing the background if the status we fetch is different than the current status. While repainting the screen is fast, it's noticeable and there's no reason to do it every 30 seconds if nothing is different.

That’s it. Now whenever the endpoint receives an update the portal will see that status change and update the display.

Thanks to Adafruit for publishing the case above. The logo on display is the Jolly Wrencher of Hackaday.

With the endpoint and display done I’m off into the unknown. I’ll be setting up the ESP-EYE to update the endpoint, training the voice model and finally running it all with FreeRTOS.

The code, docs, images etc for the project can be found here and I’ll be posting updates as I continue along to HackadayIO and this blog. If you have any questions or ideas reach out.

Train All the Things — Synthetic Generation

2026-08-24T23:19:53.977418+00:00

After getting the display and worker up and running I started down the path of training my model for keyword recognition. Right now I’ve settled on the wake words Hi Smalltalk. After the wake word is detected the model will then detect silence, on, off, or unknown.

My starting point for training the model was the microspeech and speechcommands tutorials that are part of the Tensorflow project. One of the first things I noticed while planning out this step was the lack of good wake words in the speech command dataset. There are many voice datasets available online, but many are unlabeled or conversational. Since digging didn't turn up much in the way of open labeled word datasets I decided to use on and off from the speech commands dataset since that gave me a baseline for comparison with my custom words. After recording myself saying hi and smalltalk less then ten times I knew I did not want to generate my own samples at the scale of the other labeled keywords.

Instead of giving up on my wake word combination I started digging around for options and found an interesting project where somebody had started down the path of generating labeled words with text to speech. After reading through the repo I ended up using espeak and sox to generate my labeled dataset.

The first step was to generate the phonemes for the wake words:

$ espeak -v en -X smalltalk

then stored the phoneme in a word file that will be used by generate.sh.

$ cat words
hi 001
busy 002
free 003
smalltalk 004

After modifying generate.sh from the spoken command repo (eliminating some extra commands and extending the loop to generating more samples) I had everything I needed to synthetically generate a new labeled word dataset.

#!/bin/bash
# For the various loops the variable stored in the index variable
# is used to attenuate the voices being created from espeak.*lastwordid=""cat words | while read word wordid phonemedo
echo $word
mkdir -p db/$word

if [[ $word != $lastword ]]; then
 versionid=0
 fi

lastword=$word

# Generate voices with various dialects
for i in english english-north en-scottish englishrp englishwmids english-us en-westindies
do
    # Loop changing the pitch in each iteration
    for k in $(seq 1 99); do
        # Change the speed of words per minute
        for j in 80 100 120 140 160; do
            echo $versionid "$phoneme" $i $j $k
            echo "$phoneme" | espeak -p $k -s $j -v $i -w db/$word/$versionid.wav
            # Set sox options for Tensorflow
            sox db/$word/$versionid.wav -b 16 --endian little db/$word/tf$versionid.wav rate 16k
            ((versionid++))
        done
     done
done

After the run I have samples and labels with a volume comparable to the other words provided by Google. The pitch, speed and tone of voice changes with each loop which will hopefully provide enough variety to make this dataset useful in training. Even if this doesn’t work out learning about espeak and sox was interesting. I've already got some future ideas on how to use those. If it does work the ability to generate training data on demand seems incredibly useful.

Next up, training the model and loading to the ESP-EYE. The code, docs, images etc for the project can be found here and I’ll be posting updates as I continue along to HackadayIO and this blog. If you have any questions or ideas reach out.

Train All the Things — Model Training

2026-08-24T23:19:53.977403+00:00

Recently I spent some time learning how to generate synthetic voices using espeak. After working with the tools to aligning with the Tensorflow keyword models expectations I was ready for training, and to see how well the synthetic data performed. TLDR: not well :)

I started by training using the keywords hi, smalltalk and on. This let me have a known working word while testing two synthetic words. Although training went well:

INFO:tensorflow:Saving to "/Users/n0mn0m/projects/on-air/voice-assistant/train/model/speechcommandstrain/tinyconv.ckpt-18000"
I0330 10:34:28.514455 4629171648 train.py:297] Saving to "/Users/n0mn0m/projects/on-air/voice-assistant/train/model/speechcommandstrain/tinyconv.ckpt-18000"
INFO:tensorflow:setsize=1445
I0330 10:34:28.570324 4629171648 train.py:301] setsize=1445
WARNING:tensorflow:Confusion Matrix:
 [[231 3 3 0 4]
 [ 2 178 6 29 26]
 [ 3 12 146 2 2]
 [ 4 17 2 352 21]
 [ 2 16 7 16 361]]
W0330 10:34:32.116044 4629171648 train.py:320] Confusion Matrix:
 [[231 3 3 0 4]
 [ 2 178 6 29 26]
 [ 3 12 146 2 2]
 [ 4 17 2 352 21]
 [ 2 16 7 16 361]]
WARNING:tensorflow:Final test accuracy = 87.8% (N=1445)
W0330 10:34:32.116887 4629171648 train.py:322] Final test accuracy = 87.8% (N=1445)The model didn’t respond well once it was loaded onto the ESP-EYE. I tried a couple more rounds with other keywords and spectrogram samples with similar results.

Because of the brute force nature that I used to generate audio the synthetic training data isn’t very representative of real human voices. While the experiment didn’t work out, I do think that generating data this way could be useful with the right amount of time and research. Instead of scaling parameters in a loop I think researching the characteristic of various human voices and using those to tune the data generated via espeak could actually work out well. That said it’s possible the model may pick up on characteristics of the espeak program too. Regardless, voice data that is ready for training is still a hard problem in need of more open solutions.

Along with the way I scaled the espeak parameters another monkey wrench is that the microspeech model makes use of a CNN and spectrogram of the input audio instead of full signal processing. This means it’s highly likely the model will work with voices around the comparison spectrogram well, but not generalize. This makes picking the right spectrogram relative to the user another key task.

Because of these results and bigger issues I ended up tweaking my approach and used visual as my wake word followed by on/off. All of these are available in the TF command words dataset, and visual seems like an ok wake word when controlling a display. For somebody working on a generic voice assistant you will want to work on audio segmentation since many datasets are sentences, or consider using something like Skainet. All of this was less fun than running my own model from synthtetic data, but I needed to continue forward. After a final round of training with all three words I followed the TF docs to represent the model as a C array and then flashed it onto the board with the rest of the program. Using idf monitor I was able to observe the model working as expected:

I (31) boot: ESP-IDF v4.1
I (31) boot: compile time 13:35:43
I (704) wifi: config NVS flash: enabled
I (734) WIFI STATION: Setting WiFi configuration SSID Hallow...
I (824) WIFI STATION: wifiinitsta finished.
I (1014) TFLITEAUDIOPROVIDER: Audio Recording started
Waking up
Recognized on
I (20434) HTTPSHANDLING: HTTPS Status = 200, contentlength = 1
I (20434) HTTPSHANDLING: HTTPEVENTDISCONNECTED
I (20444) HTTPSHANDLING: HTTPEVENTDISCONNECTED
Going back to sleep.
Waking up
Recognized off
I (45624) HTTPSHANDLING: HTTPS Status = 200, contentlength = 1
I (45624) HTTPSHANDLING: HTTPEVENTDISCONNECTED
I (45634) HTTPSHANDLING: HTTPEVENTDISCONNECTEDThis was an educational experiment. It helped me put some new tools in my belt while thinking further about the problem of voice and audio processing. I developed some [scripts](https://github.com/n0mn0m/on-air/tree/main/voice-assistant/train) to run through the full data generation, train and export cycle. Training will need to be done based on the architecture somebody is using, but hopefully it’s useful.

The code, docs, images etc for the project can be found here and I’ll be posting updates as I continue along to HackadayIO and this blog. If you have any questions or ideas reach out.

Train All the Things — Version 0.1

2026-08-24T23:19:53.977382+00:00

My first commit to on-air shows March 3, 2020. I know that the weeks leading up to that commit I spent some time reading through the TF Lite documentation, playing with Cloudflare Workers K/V and getting my first setup of esp-idf squared away. After that it was off to the races. I outlined my original goal in the planning post. I didn't quite get to that goal. The project currently doesn't have a VAD to handle the scenario where I forget to activate the display before starting a call or hangout. Additionally I wasn't able to train a custom keyword as highlighted in the custom model post. I was however able to get a functional implementation of the concept. I am able to hang the display up, and then in my lab with the ESP-EYEplugged in I can use the wake word visual followed by on/off to toggle the display status.

While it’s not quite what I had planned it’s a foundation. I’ve got a lot more tools and knowledge under my belt. Round 2 will probably involved Skainet just due to the limitations in voice data that’s readily available. Keep an eye out for a couple more post highlighting some bumps along the way and summary of lessons learned.

The code, docs, images etc for the project can be found here and I’ll be posting any further updates to HackadayIO. For anybody that might be interested in building this the instructions below provide a brief outline. Updated versions will be hosted in the repo. If you have any questions or ideas reach out.

Required Hardware:

ESP-EYE
Optional ESP-EYE case
PyPortal
Optional PyPortal case
Two 3.3v usb to outler adapters and two usb to usb mini cables

Two 3.3v micro usb wall outlet chargers

Build Steps:

Clone the on-air repo.

Cloudflare Worker:

Setup Cloudflare DNS records for your domain and endpoint, or setup a new domain with Cloudflare if you don’t have one to resolve the endpoint.
Setup a Cloudflare workers account with worker K/V.
Setup the Wrangler CLI tool.
cd into the on-air/sighandler directory.
Update toml
Run wrangler preview
wrangler publish
Update Makefile with your domain and test calling.

PyPortal:

Setup CircuitPython 5.x on the PyPortal.
If you’re new to CircuitPython you should read this first.
Go to the directory where you cloned on-air.
cd into display.
Update secrets.py` with your wifi information and status URL endpoint.
Copy code.py, secrets.py and the bitmap files in screens/ to the root of the PyPortal.
The display is now good to go.

ESP-EYE:

Setup esp-idf using the 4.1 release branch.
Install espeak and sox.
Setup a Python 3.7 virtual environment and install Tensorflow 1.15.
cd into on-air/voice-assistant/train
chmod +x orchestrate.sh and ./orchestrate.sh
Once training completes cd ../smalltalk
Activate the esp-idf tooling so that $IDFPATH is set correctly and all requirements are met.
idf.py menuconfig and set your wifi settings.
Update the URL in toggle\status.cc
This should match the host and endpoint you deployed the Cloudflare worker to above
idf.py build
idf.py --port \ flash monitor
You should see the device start, attach to WiFi and begin listening for the wake word “visual” followed by “on” or “off”.

Train All the Things — Wrapping Up

2026-08-24T23:19:53.977366+00:00

And now I’m at v0.1 of the on-air project. I was able to achieve what I was hoping to along the way. I learned more about model development, tensorflow and esp. While this version has some distinct differences from what I outlined for the logic flow (keywords, VAD) it achieves the functional goal. The code, docs, images etc for the project can be found in this repo, and the project details live on HackadayIO. When I get back to this project and work on v1.x I'll make updates available to each.

A couple thoughts having worked through this in the evening for a couple months:

I really should have outlined the states that the esp program was going to cycle through, and then mapped those into task on the FreeRTOS event loop. While the high level flow captures the external systems behavior the esp has the most moving parts at the applications level, and is where most of the state is influenced.
I want to spend some more time with C++ 14/17 understanding the gotchas of interfacing with C99. I ran into a few different struct init issues and found a few ways to solve them. I’m sure there is a good reason for different solutions, but it’s not something I’ve spent a lot of time dealing with so I need to learn.
While continuing to learn about esp-idf I want to look into some of the esp hal work too. I briefly explored esp-adf and skainet while working through on-air. Both focus on a couple boards but seems to have functionality that would be interesting for a variety of devices. Understanding the HAL and components better seems to be where to start.
Data, specifically structured data is going to continue to be a large barrier for open models and for anybody to be able to train a model for their own want/need. While sources like Kaggle, arvix, data.world and others have worked to help this there’s still a gulf between what I can get at home and what I can get at work. Additionally many open datasets are numeric or text datasets while video, audio and other sources are still lacking.
Document early, document often. Too many times I got so caught up in writing code, or just getting one more thing done that by the time I did that getting myself to do a thorough write up of issues I experienced, interesting findings, or even successful moments was difficult. I know that I put this off sometimes, and different parts of the project are not as well documented, or details have been lost to the days in between.
There’s a lot of fun stuff left to explore here I can see why I’ve heard a lot about esp and look forward to building more.

Train All the Things — Speed Bumps

2026-08-24T23:19:53.977351+00:00

As part of getting started on my project a couple months back I took a look at what boards were supported byTensorflow lite . Seeing an esp board I went that route since I’ve heard alot from the maker/hacker community and thought it would be a good opportunity to learn more. Additionally it’s been quite a while since I had a project that was primarily C/C++ so that was exciting. Like any good project I ran into multiple unexpected bumps, bugs and issues. Some were minor, others were frustrating. I'm capturing some of those here for anybody else that may be starting down the path of using Tensorflow Lite and an ESP32 board.

Tensorflow speed bumps

Getting started with TF Lite is easy enough, but something I noticed as I continued to work on the project is just how little things are designed specific to the platform. Instead the examples are setup with Arduino as a default, and then work is done to make that run on X target. In the case of the ESP-EYE this looks like packing everything into an Arduino compatible loop, and handling that in a single FreeRTOS task. I get the reason for this, but it's also a bit of a headache later on as it feels like an anti pattern when addin in new task and event handlers.

Another bump you are likely to notice is that the TF Lite examples rely on functionality present in the TF 1.xbranch for training, but require TF >= 2.2 for micro libs. Not the end of the world, but it means your going to manage multiple environts. If managing this using venv/virtualenv keep in mind you're going to need the esp-idf requirements in the 2.x environment, or just install in both as you may find yourself switching back and forth. In addition to python lib versions the examples note esp-idf 4.0, but you will want to use > =4.0with this commit or you will run into compiler failures. I ended up using 4.1 eventually, but something to note.

Finally interaction with the model feels flaky. It’s an example so this kind of makes sense, but I found that while the word detected was pretty accurate the newcommand and some of the attributes of the keyword being provided by the model weren't matching my expectation/use. I ended up using the score value and monitoring the model to setup the conditionals for responding to commands in my application.

Overall the examples are great to have, and walking you through the train, test and load cycle is really helpful. The main thing I wish I had known was that the TF Arduino path for ESP was pretty much the same as the ESP native path with regards to utility and functionality just using the esp-idf toolchain.

ESP speed bumps

From the ESP side of things the core idf tooling is nice. I like how open it is and how much I can understand the different pieces. This helped a few times when I ran into unexpected behavior. One thing to note is if you follow the documented path of cloning esp-idf you will want to consider how you manage the release branch you use and when you merge updates. Updates are not pushed into minor/bug fix branches instead they go into the release branch targeted on merge.

Being new to the esp platform something I didn’t know when I got started was that esp-idf 4.x released in February of 2020. Because of this alot of the documentation and examples such as ESP-WHO and esp-skainetare still based on 3.x which has a variety of differences and changes in things like the TCP/network stack. Because of this checking the version used in various docs, examples etc is (as usual) important. Since the TF examples reference version 4 that's where I started, but a lot of what's out there is based on v3.

One other bump somebody may run into is struct initialization in a modern toolchain when calling the underlying esp C libraries from C++. I spent some time digging around after transitioning the http request example into the TF C++ commandresponder code and the compiler told me I was missing uninitialized struct fields and their order made them required.

The example code:

esphttpclientconfigt config = {
 .url = "http://httpbin.org/get",
 .eventhandler = httpeventhandler,
 .userdata = localresponsebuffer,
};
esphttpclienthandlet client = esphttpclientinit(&config);
esperrt err = esphttpclientperform(client);

And how I had to do it in C++:

esphttpclientconfigt* config = (esphttpclientconfigt*)calloc(sizeof(esphttpclientconfigt), 1);
config->url = URL;
config->certpem = burningdaylightiorootcertpemstart;
config->eventhandler = httpeventhandler;esphttpclienthandlet client = esphttpclientinit(config);
esphttpclientsetmethod(client, HTTPMETHODPUT);
esperrt err = esphttpclientperform(client);

I had a similar issue with wifi and you can see the solution here.

I really enjoyed my lite trip into idf. It's an interesting set of components and followed a workflow that I use and appreciate. I wrote a couple aliases that somebody might find useful:

alias adf="export ADFPATH=$HOME/projects/esp-adf"
alias idf-refresh="rm -rf $HOME/projects/esp-idf && git clone --recursive git@github.com:espressif/esp-idf.git $HOME/projects/esp-idf && $HOME/projects/esp-idf/install.sh"
alias idf=". $HOME/projects/esp-idf/export.sh"
alias idf3="pushd $HOME/projects/esp-idf && git checkout release/v3.3 && popd && . $HOME/projects/esp-idf/export.sh"
alias idf4x="pushd $HOME/projects/esp-idf && git checkout release/v4.0 && popd && . $HOME/projects/esp-idf/export.sh"
alias idf4="pushd $HOME/projects/esp-idf && git checkout release/v4.1 && popd && . $HOME/projects/esp-idf/export.sh"
alias idf-test="idf.py --port /dev/cu.SLABUSBtoUART flash monitor"And I look forward to writing more about esp as I continue to use it in new projects.

Approaching the end of this project it’s been a larger undertaking than I expected, but I’ve learned a lot. It’s definitely generated a few new project ideas. The code, docs, images etc for the project can be found here and I’ll be posting updates as I continue along to HackadayIO and this blog. If you have any questions or ideas reach out.

GroupBy Fun with SQL and Python

2026-08-24T23:19:53.977336+00:00

A few months ago I had the opportunity to collaborate with some Data Scientist porting PySpark queries to raw Python. One of the primary areas of concern was aggregation statements. These were seen as functionality that would be particularly troublesome to write in Python. As an example I was provided a Spark SQL query similar to this:

text = ocrdata.filter(col('lvl')==5).filter(f.length(f.trim(col'txt'))) > 0).select(txtcols) \
 .withColumn('ctxt' casestd(col('txt'))) \
 .drop('txt') \
 .withColumn('tkpos', f.struct('wrdnm', 'ctxt')) \
 .groupBy(lncols) \
 .agg(f.sortarray(f.collectlist('tkpos')).alias('txtarray')) \
 .withColum('txtln', f.concatws(' ', col('txtarray.casestd'))) \
 .drop('txtarray')This query was transforming token data generated by Tesseract into lines. Beyond the aggregation operation there was also some concern that the operation may be ran against quite large datasets depending on how much [Tesseract](https://github.com/tesseract-ocr) output was being manipulated at once.

Outside of the raw functionality I was asked if the data could be structured to provide an interface with named columns in a style similar to SQL rather than having to reference positional data.

All of this seemed fairly straightforward. Provided with some sample data I pulled in the UDF that was already in Python and set out to apply the transformations first illustrating how we could interact with the data in a way similar to SQL with pipeline transformations and named references.

Porting Transformations

import itertoolsfrom csv import DictReader
from collections import namedtupledef casestd(x):
 if x.isupper():
 return x.title()
 elif not x.islower() and not x.isupper():
 return x.title()
 else:
 return xwith open("sampledata/export.csv") as sample:
 reader = DictReader(sample)
 data = [row for row in reader]a = filter(lambda x: int(x["level"]) == 5, data)
filtered = filter(lambda x: len(x["text"].strip()) > 0, a)
fixed = ({**row, 'text':casestd(row["text"])} for row in filtered)tkpos = namedtuple("tkpos", "wordnum, text")
result = (dict(row, **{"tkpos":tkpos(row["wrdnum"], row["text"])}) for row in fixed)To start I read the data in with a [DictReader](https://docs.python.org/3.7/library/csv.html#csv.DictReader) which allowed me to reference values by name like “level” and “text”. I then applied similar data transformations making use of [filter](https://docs.python.org/3.7/library/functions.html#filter), [comprehensions](https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions), and [unpacking](https://docs.python.org/3/reference/expressions.html) to try and keep a style similar to some PySpark operations.

Finally I put the rest of the transformations into a generator expression containing a dict of namedtuplevalues so that later operations could continue working on named values in a manner similar to SQL columns.

GROUPBY

With the transformation and named values part out of the way I moved onto the GROUPBY aggregations. Thinking about GROUPBY the goal is to apply an aggregation function to a unique value. That unique value can be represented multiple ways, but I wanted to show the idea behind what was happening to help with future port efforts. So on my first pass I wrote:

grouped = []
seen = set()*# Order is known because it represents data generated by tesseract
*for row in fixed:
 key = (row["pagenum"], row["blocknum"], row["parnum"], row["linenum"]) if key in seen:
 continue seen.add(key) line = [] for r in fixed:
 rkey = (r["pagenum', r["blocknum"], r["parnum"], r["linenum"])
 if key == rkey:
 line.append(r["ctxt"]) txt = " ".join(line)
 cleantxt = txt.strip() if cleantxt:
 grouped.append(
 {
 "pagenum": row["pagenum"],
 "blocknum": row["blocknum"],
 "parnum": row["parnum"],
 "lnnum": row["linenum"],
 "text": cleantxt,
 }
 )

Keeping in mind that this was to conceptualize what could be happening behind the scenes for the GROUPBYand AGG operation here we loop over our rows generating a hash from some values. Once we have this hash we check if we have seen it before by referencing a set. If this is a new value we find all values of the hash in our transformed data, append the tokens, handle empty tokens and finally add the data to our final dataset. At the end we have lines of text (instead of individual tokens) that can be referenced by page, block, paragraph and line number.

While this works it’s horribly inefficient. It stands out that we are reiterating our transformed data every time we find a new key. But the goal for this wasn’t to be efficient. It was to show the ideas expressed in SQL with Python. Specifically it was highlighting how to express a GROUPBY/AGG operation manually using hashes of values and tracking what we have and have not seen providing a final dataset that was the same as the output of the SQL statement.

itertools

Continuing on from that point one of my favorite Python modules is itertools. If you haven't spent much time with it I highly recommend taking some of your existing code and looking over it while scanning the itertools docs. I've used islice, chain and ziplongest innumberable times. Because of that I knew there was a handy groupby function stowed in there too:

Make an iterator that returns consecutive keys and groups from the iterable.\ The key is a function computing a key value for each element. If not specified\ or is None, key defaults to an identity function and returns the element\ unchanged.Generally, the iterable needs to already be sorted on the same key function.Replacing the block above:

final = []for key, group in itertools.groupby(
 req, key=lambda x: (x["pagenum"], x["blocknum"], x["parnum"], x["linenum"])
):
 line = "".join([row['text'] + " " for row in group])
 final.append({"pagenum": key[0],
 "blocknum": key[1],
 "parnum": key[2],
 "linenum": key[3],
 "text": line,
 })And with that change we have a clean, faster implementation. Additionally since this was a port of Spark SQL if the data was to get truly large it wouldn’t be much work to start iterating through all of the pipeline in batches since we can use generators all the way through.

Conclusion

So what was the point of sharing that here? Nothing specific. It was a fun exercise at the time, and it made me pause to consider how I would express GROUPBY on my own. The exercise also helped introduce some of my colleagues to the filter expression and in turn map and reduce. Using those they were able to express a lot of their pipeline concepts without a lot of the iteration structures they were used to having abstracted away. If you find yourself doing a lot of pipelining I recommend checking out itertools and functools. Both are built into the Python stdlib and provide a lot of helpful functionality.

Getting started with Resharper Global Tools

2026-08-24T23:19:53.977320+00:00

For a while now I’ve been interested in build tools, CI and code quality. I think I got a taste for it as a member of the PyMSSQL project and it has continued on from there. Recently I worked on the initial CI setup for a C# project. As part of the setup I took the time to look at what lint and analysis tools we wanted to integrate into our project.

For C# some of the more common tools appear to be:

Roslyn Analyzers
Sonarsource
NDepend
Jetbrains Resharper I won’t go into the full criteria for our choice of Resharper (I’ll update this post if I end up writing that up one day), instead I’ll summarize that Resharper provided:
easy cross platform setup
ide/editor and shell agnostic
works the same locally and in CI.
opinionated by default

Resharper Command Line Tools

From the docs

ReSharper Command Line Tools is a set of free cross-platform standalone tools\ that help you integrate automatic code quality procedures into your CI, version\ control, or any other server.You can also run coverage analysis from the command line.The Command Line Tools package includes the following tools:- InspectCode, which executes hundreds of ReSharper code inspections

dupFinder, which detects duplicated code in the whole solution or narrower\ scope
CleanupCode, which instantly eliminates code style violations and ensures a\ uniform code base### Install

To get started with Resharper tools (assuming you already have .NET Core installed) run

cd <project>
dotnet new tool-manifest
dotnet tool install JetBrains.ReSharper.GlobalTools --version 2020.2.4Which installs the [Resharper Global Tools](https://www.nuget.org/packages/JetBrains.ReSharper.GlobalTools/2020.2.4) at the project level. This then allows CI and other contributors to use dotnet tool restore in the future.

Configuration

Out of the box inspect, format, and dupefinder all have default configurations that work well. That said each team has their own needs and preferences you may want these tools to promote. While there are a few ways to configure these tools I found using editorconfig to be the most human readable approach.

For additional details on the editorconfig format see the docs and this property index.

Running

Running the tools from a shell is relatively easy:

jb cleanupcode --verbosity=ERROR --config=./.config/cleanup --settings=./.editorconfig --no-buildin-settings ./Project.sln
jb inspectcode --verbosity=ERROR Project.sln -o=./reports/resharperInspect.xml
jb dupfinder --verbosity=ERROR Project..sln -o=./reports/resharperDupFinder.xmlOne thing to note is that by default the autoformatting will attempt to enforce line endings. If you have a team working across multiple platforms and using git to automatically handle line endings these can come into conflict. It's up to you and your team to decide if you want to handle this by tweaking git behavior,editorconfig or another method.

CI

If your using Team City see this doc for details.

With everything running in our shell locally we can also set things up to run in our CI pipeline. Running the tools is easy as long as your CI platform has a shell like task/step/operator:

- script: |
  dotnet tool restore
  jb cleanupcode --verbosity=ERROR --config=./.config/cleanup --settings=./.editorconfig --no-buildin-settings ./Project.sln
  jb inspectcode --verbosity=ERROR Project.sln -o=./reports/resharperInspect.xml
  jb dupfinder --verbosity=ERROR Project..sln -o=./reports/resharperDupFinder.xml
  displayName: 'Resharper'

Of course you will probably break these up for easier maintenance and reporting.

Running the tools is easy. The trick is detecting when these tools find an issue. I’ll share what I did in case it’s helpful, but long term it would be great if Jetbrains had the tools exit with documented status codes for different issues. As it stands the tools only exit with an error if the tool fails, not when issues are reported.

CleanupCode

Since CleanupCode will format our file rewriting it on disk we can use git to detect the change.

formatted=$(git status --porcelain=v1 2>/dev/null | wc -l)
exit $formatted

dupFinder

dupFinder outputs an XML file highlighting any issues found. Powershell's built in XML support makes it easy enough to query this file and see if any issues exist.

$Result = Select-Xml -Path $(System.DefaultWorkingDirectory)/reports/resharperDupFinder.xml -XPath "/DuplicatesReport/Duplicates/*"
If ($Result -eq $null) { [Environment]::Exit(0) } Else { [Environment]::Exit(1) }### InspectCode

Similar to dupFinder InspectCode documents issues with an XML file, and once again we can use Powershell to detect if there are any issues to fix.

$Result = Select-Xml -Path $(System.DefaultWorkingDirectory)/reports/resharperInspect.xml -XPath "/Report/Issues/Project/*"
If ($Result -eq $null) { [Environment]::Exit(0) } Else { [Environment]::Exit(1) }And since dupFinder and InspectCode output XML it can be useful to save these as CI artifacts for review. In Azure Pipelines this looks like:

- task: PublishPipelineArtifact@1
  inputs:
  targetPath: '$(System.DefaultWorkingDirectory)/reports/'
  artifactName: 'measurements'
  condition: always()
  displayName: 'Save ReSharper Results For Review.'### Wrapping Up

We’ve been using the ReSharper tools for a few months now and I have to say they provided what I was looking for in the beginning. The tools have been easy to use, help us maintain our code and haven’t boxed us in or required a lot of extra time on configuration and unseen gotchas. The only criticism I have is cold start time is pretty slow for cleanupcode, and the return exit codes could be better. Both of these would also help with CI, and our git hook setup. Otherwise I think these will continue to serve us well and let us focus on our project delivery.

Roll your own git hook

2026-08-24T23:19:53.977304+00:00

As part of setting up tools to run in our CI pipeline I also setup a git pre-pushhook to run the same tools automatically in the local context. Git provides a variety of hooks as documented in the scm book, and they can be used to reliably automate different parts of your workflow.

In addition to the official docs this page has a nice summary

Code

The pre-push hook I built for our project looks like:

#!/bin/sh
PROJECTROOT=$(git rev-parse --show-toplevel)
echo "Running pre-push hook for${PROJECTROOT}"
dotnet restore $PROJECTROOTecho "Running resharper formatter"
dotnet jb cleanupcode --verbosity=ERROR --config=$PROJECTROOT/.config/cleanup --settings=$PROJECTROOT/.editorconfig --no-buildin-settings $PROJECTROOT/AMS.sln
formatted=$(git status --porcelain=v1 2>/dev/null | wc -l)

$formatted
echo "Running dotnet resharper inspector"
dotnet jb inspectcode --verbosity=ERROR AMS.sln -p=$PROJECTROOT/.editorconfig -o=$PROJECTROOT/reports/resharperInspect.xmlpwsh $PROJECTROOT/tools/CheckResharperInspection.ps1

if [[ $? -eq 0 ]]
then
 echo "Running resharper dupe finder"
else
 echo "Inspector Errors Found"
 exit $?
fi

dotnet jb dupfinder --verbosity=ERROR AMS.sln -o=$PROJECTROOT/reports/resharperDupFinder.xmlpwsh $PROJECTROOT/tools/CheckDupeFinder.ps1

if [[ $? -eq 0 ]]
then
 echo "Running dotnet test"
else
 echo "Dupe Errors Found"
 exit $?
fi

dotnet cake --target=docker-bg
dotnet cake --target=dotnet-test

if [[ $? -eq 0 ]]
then
 dotnet cake --target=docker-down
 echo "Go go go!"
else
 dotnet cake --target=docker-down
 echo "Test failed"
 exit 1
fi

The first thing that should stand out is that this is just a shell script. Git hooks are just that, making it easy to use shell, python, powershell or other tools with your hook. Write the script, link it to .git/hooks.

Script Breakdown

In this script the first thing I do is find the root of our project. This makes it easy to reference paths in a manner compatible with scripts and tools that are used throughout in other parts of our workflow.

Install

Since the hook above is just a shell script I like to keep it (and other hooks) in a tools subdirectory in the root project directory. Because git expects hooks to be under .git/hooks we can make it executable with a symlink.

ln -s -f ../../tools/pre-push.sh .git/hooks/pre-pushWith this in place we get feedback before each push so that we don’t have to correct linting issues later, and we have can be confident our commit(s) will run through CI successfully.

Wrapping Up

While you may have heard of projects like pre-commit or husky rolling your own hook is relatively straight forward. While wrappers may help with complex hook setups I personally like the low amount of indirection and abstraction that helps with debugging when rolling your own.

Self Hosting

2026-08-24T23:19:53.977288+00:00

Over the last few years I built up a sprawling list of dependencies for my home project and blog workflow. Earlier this year I decided it was time to cut down on that list and host my service dependencies locally where I could. While it took me a while I reached a point where I no longer tweak the setup week to week and decided it was time to write up the process.

A quick list of the tools I used for orchestration:

shell
DNS
Traefik2
docker/docker-compose
alpine linux

(D)DNS

The first thing I needed to do was make my services easy to reach local and remote. Since this is all running behind my home router that also means that my IP can change from time to time. To handle this I made use of Gandi’s DNS API, and setup a shell script to run with cron on my router to keep my DNS records up to date. With DNS ready I moved on to Traefik.

Traefik

Traefik is a really nice routing/proxy service that can inspect container labels and setup route forwarding while handling certificate management, traffic metrics and more. The main callout (other than what you will find in the docs ) is to keep an eye on what version you are using versus what others used in examples, and that non http based traffic (for instance ssh) requires a little more setup. Beyond that Traefik has been really nice to use and made adding/removing various services easy when coupled with docker.

docker-compose

While k8s is the current hot orchestration tool I wanted to keep things simple. I don’t have a need to cluster any of my home tools, and while distributed systems are interesting they also require a lot of work. I left those at my day job and use compose + duplicity for my home setup. This makes service management easy, the labels allow traefik to detect and handle traffic management while my duplicty ensures I won’t lose much work and can quickly restore my data and restart any services in a few minutes on any box with docker.

Services

A quick list of the services I’m hosting:

git
cgit
minio
teamcity
youtrack
rust home services API

The service management can be found here.

Wrap Up

I’ve started to self host a few times in the past and backed away. This time I think it’s here to stay. With my current setup I’m not worried about what happens when something crashes, certificate management is automated away and everything just works. I’ve linked to my orchestration code above, but if you have any questions, or suggestions send them my way. If you are starting out on your own self hosted setup, good luck, have fun it’s easier now than ever and I imagine it will continue to get better.

Use data structures for your business logic

2026-08-24T23:19:53.977267+00:00

A few months ago I was reviewing a PR that handled relationships between entities. As I was working through the code I started to notice a pattern that made me go back to the original feature ticket for a quick review of the acceptance criteria. As I suspected there was a list of around 10 “if this then that” scenarios detailed, all of which manifested as conditions in the code. Grabbing a pen and paper I started to draw out the criteria and as I suspected all the scenarios were captured by relationships and operations for a Tree.

Going back with this information I paired with the team on an update to the PR where we reduced the amount of conditions tied directly to the business domain, and refactored names so that future maintainers could interact with the code understanding a tree, but maybe not understanding all the business logic around the entities.

*in case it’s helpful the C5 project has some collections not found in the .NET Standard library for interacting with Trees. In general an interesting project I’m glad I learned about.*A similar opportunity emerged on the same project when we needed to make sure a value was unique over a series of operations. In this scenario while working on a collection of objects we were able to use a HashSetto exit if Add returned false instead of setting up a LINQ query. This resulted in less nesting, less code, and a simplified condition.

The Point

The reason I am writing this is that we should be using data structures to represent the business logic of our applications. This seems obvious, but too often I have seen implementations brute force conditions leaving data structures as an optimization, or a concern for “technical” projects. While we can use a series of conditions and predicates to meet requirements in a crude way, using data structures provides an abstraction that can elevate terse business logic to a construct future maintainers can derive extra meaning from.

Org templates and checklist

2026-08-24T23:19:53.976726+00:00

Last year I read the “The Checklist Manifesto” about the outsized impact a checklist can have on an individual and teams.

I also started using org-mode to keep a daily journal of what I'm working on, design notes, todos etc. Having spent a few weeks with org-journal I decided it was time to create my own templates. For my first template a checklist seemed like an easy target. I made a few like PR Review, and New Service which can be found in my dotfiles along with their bindings.

True to the goal, I’ve got a lot more consistent in my reviews and with capturing review artifacts. The service checklist has been turned into a dotnet template that saves even more time.

Yay checklist!

Reading highlights from Q4 2020

2026-08-24T23:19:53.976708+00:00

I read a lot of books this year, but rarely write anything up after finishing them. While I don’t think I have enough to say about any one book after my first read, I want to capture a sentence or two about them to look back on, and to share with others. If I mention a book here that you want to talk about send me a message.

Q4 2020 List

Zen and the Art of Motorcycle Maintenance

I’ve been really interested in the idea of quality and how we achieve it in what we build. Everybody uses a different definition of “good”, and I enjoyed the exploration of the topic in this book.

Blockchain Chicken Farm

Having grown up in rural kentucky and working in industrial chicken farming, then pivoting to writing software after high school this was a fantastic read. A lot to reflect on, the shared humanity, and the audacity of the software industry shined beginning to end.

Subprime Attention Crisis

A good exploration of the dangers in the ad based internet economy. While the ad industry has made a lot of tools available to those who couldn’t afford them outright, I think we can do better than the current ad economy. While shrinking sectors can hurt, I think we need to be careful not to accidentally lead individuals to believe we need to maintain the ad economy at the invasive scale we are at today.

Understanding Computation

As advertised. Computational theory using Ruby. I enjoyed the read and will probably revisit Part II sometime. It is a big book covering a wide range of topics and I don’t think I internalized Part II enough.

The Hardware Hacker

A collection of articles from “bunnie” relaying his experience and some philosophies while manufacturing in the open. The term shanzhai has popped up here and in other reading over the last few years.

Angular Development with Typescript, Second Edition

What it sounds like. Reading this for work, and it was helpful as I got going with Angular.

Programming Typescript

What it sounds like. A good read after going through the docs at typescriptlang.org.

Vader Down

Vader being Vader. Fun short read.

Shadowfall

This popped up as new at the library, and I was a few chapters in when I realized I had missed the first book in a series. So far I like it. It’s in the same vein as rogue squadron, and the author is working to write characters in a complicated environment. Curious to see how the series wraps up and if the character arcs land.

The joy of repair

2026-08-24T23:19:53.976697+00:00

A few weeks ago the Z Upper Right Assembly broke on my Mini 2. At first, I wasn’t quite sure what was wrong I only knew that the tool head couldn’t raise on the right-hand side. In an email to Adrian (at Lulzbot) we figured out the part had a hairline crack. In a year without covid I could have gone down to LVL1 to make a replacement part, but not this year. Luckily Lulzbot was easy to get a print from, and really fast too (kudos to them, and that’s nice to know for the future).

With this being my first repair I was a bit nervous. I had never taken my mini apart or tinkered too much instead opting for that with my prints. The good news is Lulzbot has amazing documentation for each printer that make repairs relatively straight forward. Beginning to end taking things apart and getting them back together per the doc took me an hour, and then it was go time. My first print was rocktopus which came out great. Seeing the right-hand axis work as expected was a huge relief, and gives me some confidence that I could do more for in the realm of mods and repairs in the future.

Why does this matter? Because if you can’t fix it, you don’t own it, and I was blown away how easy it was to fix my Lulzbot. On top of that it was fun. Outside of gardening I don’t have the opportunity to just fix stuff with my hands that much, and I miss that. Something to look into this year maybe?

Setting up a CI pipeline for Rust in Teamcity

2026-08-24T23:19:53.976682+00:00

Towards the end of last year I started working on a project in rust that would listen to a message queue and send an email. Additionally it used rocket to expose some diagnostic endpoints to check on the health of the service, change log levels, etc. When starting new projects I default to setting up a build pipeline for them to. For this project I setup pipelines in teamcity which was overall pretty easy, but sharing here for anybody else that may go down this path.

cargo-make

For new projects I like to capture the build, admin and CI steps in a way that makes it convenient for others to run on their local machine. Make and it’s derivatives (cmake, cake, etc) provide a useful task abstraction and Rust has the powerful cargo-make project that lets us capture task and mix together inline simple commands with scripts, dependencies etc.

For this project you can find my cargo make file here. I also experimented with using Powershell for my scripts/wrappers. I’ve been using this in my day job where our projects run on Win, macOS and Linux. Overall I’m pretty happy with the experience, but it is another tool to install and maintain along with various platforms missing support.

cargo test

Rust comes with a build tool and test runner built in via cargo. Running test is easy out of the box, but I needed to make use of a couple tools to get the cargo test output into a format that a CI tool parses. I ended getting test and coverage data in the junit and lcov formats that way various tools and platforms can be used across time and projects.

Teamcity

With those tools orchestrated via cargo make it's time to setup the build and test steps in Teamcity. Overall the process was pretty easy, but I ran into a couple bumps I'll highlight.

The cargo step doesn't support custom commands, so I don't use that by default
I wrote CI.ps1 as a wrapper to use in each step.
Enable the xml-report-plugin And with those two things the pipeline is ready to go. From there you may want to add your own environment variables, plugin, agent deps etc.

Next steps

With this pipeline up an running the next steps are:

Setup build caching with something like sccache
Work on local and CI build times
This has been written about a number of times I would need to make both of these better before taking the project further. As the project grows these would only get worse, and make the project unpleasant for others to work on.

Done

That’s it for now. I learned a lot along the way about Rust, cargo, and hooking it up with Teamcity. I’m not sure I’ll have a write up on artemis anytime soon. It was a good project, but I ultimately took another path. Hopefully this helps somebody, and as always feel free to reach out.

Finding good defaults

2026-08-24T23:19:53.976668+00:00

At heart many of us are tinkerers. We like to take things apart and see how they work. We enjoy spending hours customizing our tools, scripts and applications. But all of this adds up. It means that our tool works different from all the others. For somebody else coming behind us it may mean piecing together all the flags and tweaks no matter how well we documented or versioned tweaks in our repo, time will bring a divergence between the system and the repo.

A lot of this has set in for me over the years teaching programming, speaking with family about their devices, and onboarding to new code bases as I change teams.

I want to try out something new over the next few years, and that is finding good defaults. If I feel the need to tweak/change a tool out of the box, I’ll try other options. I have some theories around this.

Good defaults save us time
Don’t require extra conversations for team consensus
Help manage complexity
Build on expertise
Teach us to expect good defaults, and to build our software with good defaults
Good defaults help others join in

Against this list I also believe there is a counter culture around defaults wrapped up in programmer/hacker/software engineer reputation that drives us to distrust or question anything we haven’t pulled apart and customized. This creates an extra challenge.

I know that each domains “good defaults” will be different, but that doesn’t mean they don’t exist. My hypothesis is that they can save us time, help us build what matters, and help those who are join in, or come after. Lets find out.

Entropy

2026-08-24T23:19:53.976658+00:00

noun: entropy; plural noun: entropies; symbol: S

a thermodynamic quantity representing the unavailability of a system’s thermal energy for conversion into mechanical work, often interpreted as the degree of disorder or randomness in the system. “the second law of thermodynamics says that entropy always increases with time”
lack of order or predictability; gradual decline into disorder. “a marketplace where entropy reigns supreme”
(in information theory) a logarithmic measure of the rate of transfer of information in a particular message or language.

Everything we build and learn requires constant maintenance and labor.

Reading Highlights from Q1 2021

2026-08-24T23:19:53.976647+00:00

I read a lot of books throughout the year, but rarely write anything up after finishing them. While I don’t think I have enough to say about any one book after my first read, I want to capture a sentence or two about them to look back on, and to share with others. If I mention a book here that you want to talk about send me a message.

Q1 2021 List

A Philosophy of Software Design — a good read that will impact how you think about writing your code. Many of these ideas have permeated out into general software culture (I think, maybe I’m in a bubble), but I would recommend this.
Why we make things and why it matters — an interesting read on crafts and why we should embrace them.
Humankind — good, but long at many points
Kubernetes Getting Started — as advertised
The Unix Administrators Handbook — I had a former manager recommend this book to me. I’ve read it through twice now and I always learn something new.
Fluent Python 2nd Edition — A really good book on Python exploring areas that are not always tread, and Luciano does a great job of answering “why” as he goes.
Django Crash Course — as advertised, a good alternative or secondary first Django tutorial.
A Scoop of Django — A good read. I don’t feel like the answer of why is always explained to the level I would like. It’s hit and miss, sometimes it is, sometimes there is a link to a blog that I may get around to reading one day. A good book none the less.
How to Invent Everything — probably my favorite book so far this year. This is a lot of fun and I look forward to reading it with my daughter one day.
Think Julia — I’ve heard a bit about Julia and decided to check this out. A pretty good book revisiting a lot of programming first principles in the context of learning them with Julia.
Math for Programmers — I really enjoyed this. I try to keep math as a regular reading topic, and I like it when math is demonstrated via programming.

An Interlude

2026-08-24T23:19:53.976636+00:00

I started collecting notes here almost 3 years ago at my first PyCon. I had been working in Python for 3 years at that point and wanted to share some notes with anybody that might find them useful.

Since then I’ve veered into building a homelab, C#, rust, infrastructure, gardening and all kinds of other fun experiences.

More importantly my daughter was born, and is turning 3 this year. Over 2020 I became more and more aware of what I spent time on, and started to evaluate if I wanted to keep spending time on these activities. Many things have fallen to the cutting room floor, and writing up a post each month is one of those. I still do a lot of writing for work, and maybe if something from there manifest itself as a good topic I’ll share it, but I am not going to make myself sit down each month as I have in the past.

I do plan to have short journal snippets I capture each day that I will link to here eventually. Many of them will be software related as that’s what I spend so much time on each day, but all of them won’t be.

So yeah, hopefully the notes here have helped somebody. They will stay up, but I need to give myself permission to let this sit while I spend time elsewhere, maybe to return one day :)

As always if you read something interesting, or find a code snippet that could use expounding send me a message.

Till the next post surfaces 👋

Novel Interactions

2026-08-24T23:19:53.976625+00:00

I started my career as a software engineer working on a lot of data visualization projects. After a couple years I switched to working primarily on backend services, and then after doing that for almost 8 years I had the opportunity to start working on frontend projects again. Since switching back I pay more attention to my digital experiences and notices the ones that surprise me or stand out. I wanted to share some of those experiences from 2021.

Apple Pencil + Adobe Fresco + Watercolors

Toward the beginning of last year I started drawing again and came across Fresco on my iPad. While the whole app is impressive the water colors really surprised me. There was something about the first time I chose a water color brush and could see the colors diffusing on the canvas while I was moving my pencil that was really intriguing. Add to that the pressure sensitivity and I continue to find this really interesting.

PokPok

This is a game that my daughter likes to play on iPad from time to time and it really feels like a digital toy box. One of the play areas is a town map where there are tons of interesting interactions. You can move cars, people, animals and have little interactions with the mail, car wash etc. It feels really well made, and like it was made to encourage kids to use their imagination. There are no objectives or dark patterns, just play and it’s great.

GitHub CoPilot

I’ve spent a lot of my career building tools and services for ML projects which has lead me to be critical of many “AI” projects. This year when GitHub CoPilot launched I was able to get into the beta and found out it’s actually a pretty powerful tool. Yes you can find examples of how it can be wrong, and you can find plenty of tales of woe about the bugs it will introduce, but I’ve actually found it to be a really useful coding assistant to the point that I miss it (but can still do my work) when I don’t have it. While it does make some useful code recommendations it also does a great job of helping me write the docs which is worth a lot to me and future team members.

GarageBand on iPhone

I remember getting my first MacBook (white clam shell) and spending a lot of time playing with GarageBand. I was blown away that this was built in for free and could do so much coming from Windows and headless Linux. I had a lot of fun with it, but then slowly I was doing less with music and hadn’t touched GarageBand in years. This year I took a musicianship course with UC Berkeley (through Coursera) which required ear training throughout the week. I used a lot of tools for this, but one that I found to be really fun and interesting was GarageBand on iPhone. After installing it from the AppStore I was able to load up a keyboard that let me practice intervals with some nice sounding samples. From there I went on to explore the various digital instruments and track options. It’s pretty wild just how much is available for creating and interacting with tracks on my phone.

XCloud

Does anybody else remember MSN Zone? I do, and I also remember getting kicked off the dial up connection halfway through a match of Star Wars Battlegrounds because somebody would call the house. Fast forward 20 years and I was able to stream games on my iPad while Rachael and I were in Chicago. Game streaming has been a thing for a while but I really was amazed with the overall experience of the XCloud app. Logging in on the go and loading up Halo Wars 2 in HD with no issues for a couple matches was awesome. Add on being able to see a bit of what was going on with Redux when I did the same thing on my desktop browser and I continue to find XCloud to be really neat.

Spatial Audio

We switched to Apple Music from Spotify last year when they announced the addition of Dolby Atmos/Spatial audio. I remember listening to Hybrid Theory (Linkin Park) first and was blown away. This was an album I had listened to on repeat for years, first with my Sony Walkman disc player and then my RCA MP3 player after that. I felt like I was hearing sounds on tracks for the first time and I continue to go back and check it out week after week. Now I search around for albums I had heard in the past to see what new sounds I might hear while also enjoying what people are doing with spatial audio in games and other mediums.

Listen to Jonathan x Friends - Best of Dolby Atmos + Spatial Audio by Jonathan & Friends on Apple Music. So last year had quite a few new and interesting digital experiences. Hopefully this year has just as many for me to share.

I realized something similar a few years back.

2026-08-24T23:19:53.976614+00:00

I realized something similar a few years back. Discovery is arguably easier than ever, but it's also super easy to fall into just listening to what you know from established playlist.

I have found Bandcamp and some of their newsletters to be a good way to discover new artist and songs.

A few artist from the last 10ish years that you might give a listen to:

LEENALCHI
Super Future
Japanese Breakfast
The Weekend

Have fun, there's a lot of good stuff out there!

Productive tools can be observed

2026-08-24T23:19:53.976602+00:00

Last year I started working on a new React app. I was late to the React game, having only started using it with React 16 I had experience with some existing React projects, but this was an opportunity to start something new. Additionally, I was starting out small with one focused use case, but if that was successful [it was!] the app would grow to encompass quite a few screens and tools.

In the process of determining how state would be managed I wanted to test out using useContext and useReducer to see how far I could get before reaching for a state store. For the initial use case this worked great, and to this day I still make use of context for a few key parts of the app. Eventually though as the app grew, and as I wrote more custom reducers I decided it was time to explore my state store options. Along the way I looked at the latest versions of mobx-state-tree , redux (specifically RTK) , relay and others.

Ultimately I ended up choosing Redux Toolkit (for a variety of reasons), because it’s opinionated and easy to observe. The languages we use to solve problems today are not precise and encompassing enough to build systems that cover the universe of execution paths. Because of this it’s important that the tools you use help you understand the system when it behaves in an unexpected manner.

This is pretty trivial in Redux when using Redux Dev Tools. Add in using RTK Query and suddenly you might feel like you have superpowers for understanding the state patterns in your app. With the time travel and state sharing capabilities built right into Redux Dev Tools you can snapshot a session, share it and walk back through each dispatched action in a way that almost feels like magic compared to other debugging experiences I’ve had over the years [writing to the console, re-constructing state manually, browser debugger step by step].

And that’s one of the primary reasons I ended up using Redux in 2022. There are plenty of valid reasons to use other tools, but after all these years I accept that bugs and edge cases are going to happen. Knowing that I want tools that help me explain and understand them, so I can move on to the next step of determining a fix and shipping it.

Software Engineering Mythology

2026-08-24T23:19:53.976587+00:00

Over the last few years I’ve had the opportunity to see how a variety of teams\ and companies build software. Along with that (like many others in the field) I’ve kept up with the content coming through various aggregators. Among the many things that every team has to decide is how they are going to actually build and maintain a project(s). We make decisions like:

what language to use
how to manage dependencies
how to generate artifacts
how to manage artifacts
how to instrument our project
how to deliver our project
how to document our project

Some of these have grown community standards, some platforms make the decision for us, but often times these decisions are made off a mix of experience and a lot of personal preference. Those preferences I imagine are largely impacted by the industry of personality and marketing that has sprung up around:

software education
dogma articles
self proclaimed experts

Some of the problems with our judgement being so strongly influenced by these voices is that they often lack data for us to place them in our context and judge the merit of what is being offered. Rarely do we know what didn’t work, what came before, the team composition and history, measurements used to determine if something truly was better or more successful. Instead it’s anecdata, stories told through the lens of one reality, comments that evoke strong emotions etc.

This has made me more and more uncomfortable over time. So much time and effort is spent without good data, and we churn and churn and churn. Oddly there are resources published with data from academic sources like ACM, IEEE and industry sources like MS Research and Jetbrains (among many many others). These are not perfect, but they do capture a known context and pair that with data. Maybe the findings have distilled out to our industry over time, but I have a suspicion there is data there that raises questions and counterfactuals to what we extrapolate from ones individual anecdata that turns into the latest hot article/talk.

This isn’t meant to say that the advice we share and the experiences we live are not important. They are incredibly important. Instead I write about this as something I want to go to as a source of education and learning more often. I want to make sure I spend as much time with these sources as I do with articles in my RSS feed. I’m sure I’ll learn something along the way that I haven’t heard before. I hope to find some research spanning groups, companies and project scopes that provides a data analysis of the results on top of the stories of those followed.

ACM\ IEEE\ Microsoft Research\ Jetbrains Research\ ISO

Without Trust

2026-08-24T23:19:53.976576+00:00

There is a lof of talk these days about building systems that don’t require trust. On paper this has some appealing characteristics, and throughout history we have seen the issues that arise from unquestionable centralized authority, but right now I think we are also seeing the impacts of missing trust in the analog (physical) realm.

As our physical and digital existence converge and the lines become blurrier I think it’s pertinent to keep in mind the kind of systems and philosophies that we construct in each realm and how they can go on to influence the other.

I have a lot of thoughts and feelings on these subjects, and maybe I’ll go on to explore and write them down in the future, but for now I just simply want to say that I don’t want to live in a world without trust.

Interactions of Interest

2026-08-24T23:19:53.976564+00:00

Over at Hackaday there is an ongoing series of articles called Inputs of Interest. Last year I wrote my a post Novel Interactions that will end up being the first in an ongoing collection of articles called Interactions of Interest. I personally enjoy building and interacting with new experiences. For those that catch my interest and stay in my memory I’m going to collect and share them.

Sony Linkbuds — while I ultimately couldn’t find a cushion and ear fit that allowed me to listen for more than half an hour I thought the design around these was really creative. Being able to hear your environment while also listening to music, a podcast or book is really nice in a variety of situations. Additionally moving the earbud interactions from being on device to picking up movement from the area near your ear is something I think we might see more of from future devices.

Animoog Z — There are two synth related items on this list. I never used the original Animoog software, but I recently loaded Animoog Z, and it’s been a lot of fun. My daughter and I can sit down tweaking various parameters while launching new sounds into space just to see what we can create next.

Roland 50 — Roland is celebrating 50 years of making awesome devices and tools. In celebration of that they launched a synth web app that is mind-blowing. While this is a lot of fun to create with it is also inspiring to think that this is all running in the browser meaning they could launch it into the world and make it available across device, OS and app store environments with little friction.

Delta — A couple months ago I traveled for work. Along the way I found out I could head home earlier than planned, but I figured making that happen without it becoming a lot of trouble was unlikely. It’s pretty crazy that I was able to take care of bumping my flight up so I could be home with my family all in the Delta app in under 10 minutes without a change fee. Thinking back 10 years ago when I was traveling as a software consultant it’s wild how easy logistics interactions like this have come.

Apple Research App — For the last year I’ve opted in to a few different research studies available in the Apple Research app. Each week or month my phone reminds me I have some questions to answer, and ever so often even has me review the data that I’ve opted into sharing with these studies. What surprises me is how few studies there seem to be. Maybe this is a good thing. Maybe it means there is a lot of effort going into ensuring the quality of these studies, but the way the app works, how transparent it makes things and prompts you to keep in touch makes me wonder if it’s quality, or a lack of (legit) organizations knowing this is out there.

Tailored Experiences

2026-08-24T23:19:53.976553+00:00

A couple of months ago I got to spend a few days hacking on an idea involving client side Tensorflow.js. Having spent much of my career in what I call ML adjacent roles I was really surprised at the opportunities TF.js seem to open up.

One of the things that stood out to me is that the full training and deployment cycle can happen in real time client side. As I was using the tool I was able to load my models starting point and start feeding it data seeing the behavior and predictions from the model change immediately. I could even hop into dev tools to patch and tweak different settings of the model just to see what happened. I didn’t have to wait long cycles, I didn’t have to insert a new notebook cell and disrupt the flow. I was able to launch my web app, load in a checkpoint and go. Based on what I did the model was able to learn in real time (and make some admittedly hilarious mistakes along the way).

This makes me wonder what kind of experiences this is waiting to unlock. I don’t need to send data back to a centralized service for all of my data to be mixed in with the data of other individuals waiting for a new training cycle or model deployment my data was on my device, the model was on my device the update was immediate. Obviously there are risk here (filter bubble, convergence etc) to address, but the opportunity for useful new experiences appears strong.

While I haven’t had much time to explore TF.js since this initial interaction I hope to return to it sometime in the near future. I think that projects like automerge and tensorflow.js represent potential futures where our data is local to us. We can collaborate, share and build all without phoning home or opening the app store if we continue to push what is possible in our browser and devices.

Pokemon Legends Arceus: Exploring Hisui with my daughter

2026-08-24T23:19:53.976542+00:00

A couple of months ago I had a pile of games built up that were ready to trade in. When I took them to the shop I saw a new Pokemon game was out, and my daughter is now at the age were some games are becoming fun for her to play. Reading up on the open world nature of Arceus I thought it might be something fun for us to enjoy together. What I didn’t know was just how much fun we would end up having.

From the moment we got to pick our starter (Cyndaquil) we were having fun. My daughter loves all things monsters, loud and colorful so we were off to a good start. Once we powered through the open dialogue as fast as possible (who needs a story when there are monsters?) we were running through the Obsidian Fieldlands discovering all kinds of new Pokemon along the way. My daughter loved it, I was enjoying the nostalgia of seeing familiar faces from my childhood and we were spending time discovering a new world together.

By this point we’ve logged enough hours to explore the far corners of each map, track down alphas and space time distortions, address the rift and we are left tracking down a few lingering Pokedex entries so that we can meet Arceus. It’s been a fun journey around the world, but for me the best part has been seeing the happiness and discovery that my daughter has enjoyed along the way.

Along the way there have been a few things she discovered early on that have endured and become a constant part of our exploration. Anytime we see a shaking rock or tree we have to find out what’s in it. Often times this is met with “just geodude” or “another burmy”, but occasionally it’s an excited shout “WHAT IS THAT!” as Heracross or another unexpected Pokemon pops out.

Bubbles (space-time distortions) have become another ongoing source of entertainment. There is a sense of heightened risk as we are constantly being targeted by the Pokemon spawning in the bubble never knowing what will come next. It’s fun to race around picking up the random items while watching for any bubble denizens we might want to engage.

But most of all I love seeing the stories and interactions my daughter imagines with the non aggressive Pokemon interactions. Each map has a variety of Pokemon or locations were the Pokemon don’t become aggressive as you approach. Probably the most notable for us has been the Alabaster Icelands hot spring where there is often a Machop, Machoke and other Pokemon relaxing and enjoying the water. My daughter loves to run and jump into the water, share snacks (berries) and follow the other Pokemon as they hop and run around. The stories she makes in these moments are amazing and I can only imagine transcend what the designers were imagining as they opted to have less aggressive Pokemon scattered throughout the map. These moments have easily become my favorite part of the game. Maybe in a future Legends the Pokemon in your party will be able to join in these interactions or have non-combative interactions in the wild.

I hadn’t played many games in the last few years, but Legends has been a really fun way to spend time with my daughter. It’s had us sharing ideas, taking turns navigating the world and finding plenty of surprises on the journey to meet Arceus.

P.S. Outside of the fun these fun interactions I have found the game soundtrack to be great. I don’t have any other Pokemon tracks in my playlist, but I’ve added many from this game and I think I may have to go check out the music from previous entries that I haven’t played.

Interface Discovery

2026-08-24T23:19:53.976531+00:00

I spend a lot of my days building new tools that individuals use to power their day to day work. Because of this I always keep an eye out for behavior in the apps that I use which inspire me to consider how my apps work. One of the things that has stood out to me over the last year is how some apps introduce let you discover functionality.

Garageband Quick Help — a feature you can enable in Garageband and Logic Pro is quick help. Once enabled it introduces a short tooltip hover over. Considering the amount of options that these applications provide and the depth of the interface being able to simply hover over a button, knob or other part of the interface and immediately find out what it can do is incredibly useful.

Adobe stories — Stories seem like a nice way to share that new functionality is enabled while also creating an index that lets users return to the a story when they are ready to use a feature. Often times when I was using Fresco I would see a new story introducing a new tool, but it wasn’t something I was ready to use yet. I knew it was there though and often times I would return to the story in the next few weeks as I tried out something new. While plenty of apps offer changelogs the presentation and the fact that stories are always right there available in the app is a nice touch.

Emacs/Jetbrains/VS Code and probably your favorite editor — This one has been around for ages, but is still missing in many tools and web apps. Universal search related to functionality. Emacs has had M-x for who knows how long, Jetbrains added Shift + Shift and VS Code has the command palette (not quite the same, or as powerful but good). Being able to toggle into these universal search places and discover options, settings and commands is really valuable and I’ve found plenty of features I never knew existed just by searching a related term.

RTFM — nothing new here, but having picked up some new hobbies and devices over the last couple of years having some docs is super helpful. We often take this for granted, especially if we are building internal tools, but at the end of the day a user manual goes a long way to enabling somebody to read the instructions and be on their way. Bonus points if your docs are built in, indexed and up to date.

More Interactions of Interest

2026-08-24T23:19:53.976520+00:00

Following up on Interaction of Interest here are a few more things I’ve messed with in the last couple of months that I found to be pretty neat.

Using a Wacom Stylus in the browser — did you know that the HTML 5 pointer event has special properties for pens? I didn’t, but a few months back when I was exploring the benefits of using a wacom tablet with a web app I learned about them. Surprisingly ( to me) a pen just kind of works with many apps for simple events, but allow a lot of custom functionality. I think because of the ubiquity of the mouse and keyboard many apps are built without much thought given to what kind of input device might make the most sense. Of course the individual using the web app needs to possess the device, but I think many interactions can be enhanced through alternative input devices. Pens, knobs, touchpads etc.

Tensorflow.js — I wrote a short article on this one, but in general the short amount of time that I can to spend working with Tensorflow.js left me wanting more. I think that ML has a lot of benefits we are only beginning to discover. The prospect of enabling the use of ML in client web apps and pushing the learning cycle to the user device is really interesting to me.

Forecastle Cashless Wrist Band — Rachael and I recently went to Forecastle. The festival was a lot of fun, but a few weeks before we were set to attend we received wristbands for the event in the mail with registration instructions. Inside the band was an NFC chip that was your ticket into the event. You could also register a card with it to use as a form of payment anywhere in the festival. Having attended quite a few concerts, conventions and festivals this was new to me, but I thought it was awesome. For one it didn’t assume the attendee has a smart device to bring with them, and it also didn’t assume that their smart device would have a good connection to download or open an app, site etc at the event (a common source of issues). What I noticed was that lines to get in were way shorter. No shuffling for tickets, no getting your phone out to open an app, no waiting on face id or entering your passcode. You scan and your in. Payment seemed to have the same benefit of not shuffling for wallets, waiting for you phone/watch wallet to load etc. These bands really streamlined a lot of the wasted time at the end of the line. I hope to see more of these in the future.

Bluetooth Vinyl — This sounds like an oxymoron. I recently picked up a record player that was able to connect via bluetooth to my soundbar and…..it’s awesome. There is something enjoyable about placing the record on the plate, and changing the disk side to hear more as one side comes to a close. That said cable management isn’t fun, and worrying about preamps, speakers just for the player etc wasn’t something I wanted to get into. I’ve been surprisingly happy enjoying the ritual of vinyl with the ease of a bluetooth audio connection.

Pokemon Legends Arceus — another interaction I wrote more about in another article. This game reminded me how much fun exploring a game with somebody else is (I used to play primarily multi-player video games) while also driving home the creativity and care of Nintendo (and for this particular game GameFreak).

Notes from a year of working with Jack Audio Connection Kit

2026-08-24T23:19:53.976501+00:00

Last year I became the lead engineer on a new project. After joining the project I spent a lot of time helping users and engineers debug JACK audio behavior (although JACK typically wasn't the cause of the observed issue). While I had used Core Audio in the past I wasn't as familiar with JACK at the time, so I thought I would write down a few of the things I have learned to check first when somebody tells me they are having an issue with JACK. These are specific to running JACK on Linux, in my case Ubuntu.

flowchart LR A[Does JACK server start?] --> B{No} B --> C[What is your clock source, and is it valid?] C --> D{Does your device provide JACK a clock?} D --> E{"Does your device have an internal clock,\n or does it rely on an external clock\n (for instance a PCI MADI card setup)?"} C --> F{Are all of your cables connected?} F --> G{Are they in the right direction?} F --> H{Are they all showing healthy conditions on\n their LED or other connection health indicator?} B --> I[Does ALSA recognize your sound card/interface\n as a capture and/or playback device?] I --> J{No} J --> K{"How is the device connected (USB, PCI, etc),\n and do you need any additional drivers?"} J --> L{"Using the correct tools (lsusb, etc)\n does the kernel recognize the device is present?\n If not you're debugging hardware."} I --> M{Yes} M --> N{Does ALSA recognize that the device supports\n the sample rate you want to run the JACK server at?} M --> O{Do you need to customize any channel or other\n settings with alsa mixer to have the correct IO setup?} M --> P{Can you use alsa utilities to send or\n receive signal to the device?} P --> Q["If your clock source is valid, and ALSA\n recognizes the devices and can route signal in/out of the device\n (depending on playback and capture capabilities and requirements)\n then we probably have a JACK settings issue.\n If you don't verify the clock and ALSA\n settings first though you may be looking in the wrong spot for the issue."] C --> R{Clock is valid and ALSA can use the device} R --> S{Are you starting JACK in daemon mode or dbus?} S --> T{If daemon mode:} T --> U{Is there anything blocking the daemon from starting?\n For instance another jack server that is already\n running via pulseaudio or another process?} R --> V{Are you passing the right hardware identifier\n to JACK to select the correct playback/capture device\n that the process will use with the server?} R --> W{Are you starting the server with a sample rate,\n frame size and period supported by the device?} W --> X{Use ALSA tools or qjackctl to review device settings\n and configuration options.} R --> Y{What does JACK tell you if you start the process\n with the `--verbose` flag?}

That's a lot to check, and I employ various ALSA cli utilities along with JACK tools like qjackctl to help collect information as I iterate through the items above until I get to a state where the JACK server will start. Once I get the JACK server started if errors are reported it tends to fall into a couple categories (I'm sure there are more, this is just based on my experience so far):

JACK client issues
This is a pretty broad space since really a jack client can represent just about any application/algorithm that you want to wrap as a jack client and have audio or midi in/out port support. A few things I've seen:
1. The application runs at a fixed rate that isn't compatible with the rate the server is running at.
  1. You will either need to update the client, run the server at a different rate, or do sample rate conversion.
2. The application parses JACK server output information and doesn't behave correctly when the server runs in verbose mode.
  1. There is probably a better way to get the information the client needs without parsing jack server output.
3. The MIDI messages that the application receives are not parsed correctly.
  1. Use MIDI utilities to check the message format/structure and either fix the sending or receiving application code.
4. Resource errors - these tend to be addressable with standard tools (writing tests, profiling the application, static analysis, etc).
  1. Memory errors.
  2. Callback code doesn't execute within the allotted time.
  3. Improper port management.
  4. Logic bugs.
Unstable clock
This one is a little tricky. If the jack server starts, but loses the clock source you can check a few things.
1. Is the device still connected correctly, and if it's external did it lose power?
2. Check syslogs and dbus. Did the hardware disconnect and reconnect?
3. If the device clock is external does the device have any logs that you should review?
4. If none of the points above expose any useful information then you will probably need to do deeper monitoring, and possibly troubleshooting at the driver/device level.
  - Does the error happens on a set time interval
  - Can you replicate it on a second machine
  - What debug information is available at the driver level

Working with and building JACK applications has been a really interesting experience. Making sure everything aligns as expected from hardware through software has taught me a lot. Maybe some of this will help somebody else if they are debugging JACK workflows for the first time.

Healthy Systems

2026-08-24T23:19:53.976488+00:00

A friend once told me that a complicated system that works is made up of many simple systems that work. It's something that rings true to me and I think about every time I interact with a digital system.

To help build those simple system that we compose into complex systems I want to share a few items that help me measure how healthy a simple system is, or isn't.

Testing

This is my first step these days. Show me your test. If they are not there, then that tells me most of what I need to know about the state of the project. If they are there then I have a natural starting point for onboarding, and understanding. I have a way to execute and inspect at least parts of the system, and depending on the scope and levels of the test maybe the whole system.

It may be over stating it, but testing to me is one of the best skills a software engineer can develop, and the presence/scope or lack of automated test give me most of the information I need to know when considering the health of a project.

I think it's hard to over state how much you can learn from a good test suite, and how fast you can go with a robust reliable test suite. Over the years I've had many engineers ask me how they can know wether or not they are using the right abstractions and designs. I struggled to answer this for a long time (often recommending various books). These days I tell them to write the test. If it's hard to test, or the test sucks to write, then you're probably not on the right track with your design. Step back and reconsider your approach and what other options exist.

Publishing

What good is writing code if you can't ship it. Ultimately we are building these things to do something in the physical world right? Maybe there isn't a tangible change to the physical world, but somebody is interacting with or getting some form of utility from this thing we are building right? When we make changes we need a way to reliably ship those changes to their destination host system to run.

There are many ways to package up software and get it to a host system today (containers, debs, msi, app store artifacts, etc). Pick the one that makes the most sense for your project, use tools that help generate the target artifact (cpack, buildx, etc) and then build a publishing process (anything from a script that makes this a reproducible process to a fully automated pipeline) that makes this something that is easy to do (with appropriate permissions) once a change has been signed off on.

When publishing is hard I have found that it discourages work from getting finished. A lot of WIP builds up, or everything just takes longer. It's harder to keep up with what is or isn't done. The more you reduce the time between somebody deciding to implement a change, having it reviewed and then shipping it the better. This also helps make it easier to identify and fix bugs, because let's face it there will be bugs, there will be unknowns, the faster we can address them the better.

Docs

There are multiple forms of docs that a project might generate. There are two primary forms of documentation that I'm interested in while working on a project.

Decision Records

README and other developer guides are nice for helping me setup an environment, but if those don't exist hopefully I can get the information I need from pipelines, scripts and other files. What I will never be able to gain context on without docs (or word of mouth from project elders that have been around long enough to know, and have an accurate recollection) is why the project exist in it's current state. That's something Architecture Decision Records or similar documents can help with. Why were certain decisions made (what problem was being solved), what options were considered, and why did the team go with the solution that I'm looking at today.

User Docs

User docs help me understand what somebody using the system can expect. What have we communicated to our users that this tool/application/system can do. What guidance have we provided to them? How have we encouraged users to engage with our team and project if they have questions, need support, etc. The state of user docs conveys context about our system and how much we care about the people using our system.

Wrap up

I thought about including other items on this list. Abstractions, data structures, dependency management, build systems, etc but honestly I think that those roll up into the three points above. If you're able to publish your project reliably then you probably have build systems and dependency management covered. If you have a well orchestrated and reliable test harness then you likely have healthy(ish) abstractions, though maybe this doesn't extend to choosing the "best" data structures. And while you could have a healthy publishing and test system without docs I think that only works at a small scale. Ultimately we need to communicate across time, space and individuals and docs are a great way to do that, and there is not reason they can't be part of your publishing system so that your docs live with the project, and as the system evolves your docs evolve with it.

Surprising Assertions

2026-08-24T23:19:53.976472+00:00

Recently during code review (for a Python application). I asked somebody if they intentionally used the assert keyword rather than an if check outside of test, and if they were aware that calling the Python interpreter with the -O option would disable their assertions. This came as a surprise to the author (and rightfully so IMO, not many teams use the interpreter options and just take the default) who assumed they could safely take the pattern they used in test and apply it to some validation functions in the application.

A similar thing happens in C if NDEBUG is defined catching some engineers by surprise when their validation functions are no longer behaving as expected once their application moves from debug to release builds (or other optimized environments).

It's interesting (surprising?) behavior, and it makes me wonder what the history or context of the decision was around assert and how it can be removed from execution in both languages. Surprises like this are part of what makes engineering hard. If you expectations/mental model don't actually line up with the execution context and model your application can enter into surprising states that become harder to debug because you have to figure out that your mental model is disconnected from the actual state of the program.

Weighing feedback

2026-08-24T23:19:53.976456+00:00

For most of my career I've worked on software projects that could be considered internal tools. I love doing this because I'm close to the people that use the tools I build and support. This has a lot of advantages including fast feedback and being available to support individuals when they run into unexpected behavior with a tool. That said one thing you have to balance is the amount of feedback you receive about the tools you build. Sometimes your most vocal partners are also individuals that use the tools the most, other times those asking for the most features or telling you all the ways the tool could be better use it very little in comparison to the broader community of users. When considering some feedback last year my manager shared this quote with me that I've held on to:

The strength of ones opinion should be matched by the level of their involvement.

Building and Maintaining Teams

2026-08-24T23:19:53.976443+00:00

One of the biggest challenges I have seen teams consistently face is building teams, and once they have built a good (or even great) team maintaining it. Some of the most common challenges I have seen are:

An interview process that is disconnected from day to day work.
I'm not a fan of white board interviews, but if you're building a software engineering team you should have programming somewhere in your process. My preferred approach to this is some form of code review and possibly pair programming.
If your interview process doesn't stress the communication channels you expect people to participate in daily (for instance only having interviews that involve verbal speak and no writing component) you're setting everybody up to be surprised.
If your team regularly collaborates with other teams or stake holders they should be part of your process.
An interview that only provides feedback in one direction.
As much as some companies may not want to consider it, interviews provide feedback in both directions. The team is evaluating the candidate, but (a good candidate) is also evaluating the team to determine if this is somewhere they want to be, and where they would succeed.
Interview processes and candidate pipelines are only maintained when the team needs to hire somebody.
If you wait until you need to hire somebody to make sure your interview process is good to go you're already way behind. Steps will be missed, mistakes will be made. Create a consistent process, and iterate on it. Share it with the team. Do a dry run internally (remember how we used to prepare for public speaking speeches in school?).

Once you hire somebody the other hard part is maintaining teams. Life happens, you can't prevent that. What you can do is:

Make sure your onboarding process doesn't suck. Nothing sets a bad tone for a new team member like an onboarding process that shows the company doesn't care.
Keep a pulse on wether or not team members feel like their contributions make a difference. Good engineers care, people that care want to do something meaningful.
Push back on notions throughout the company that anybody is "plug and play". With the continued propagation of LLM and other tools and in an age where companies are looking for quick savings my experience is that every team member that is an integral part of the team provides a different lens or set of skills.
Ensure that team members take time off. A refresh mind and new perspective is good for everybody now and then.
Listen. People will tell you what's on their mind. Rarely have I seen people leave for reasons that were impossible to resolve, but plenty of times I've seen people leave because nobody was listening.

Culture Challenges + Software Solutions = Technical Debt

2026-08-24T23:19:53.976404+00:00

Over the years I've been on various teams that closely interact with engineers that are not software engineers, but need to embrace software practices or tools to some degree due to their project goals. Often this comes in the form of writing code, packaging and version control. I've also had conversations many times about the layers of abstraction we build on top of these tools or practices to help make it "easier" for non software engineers to use. For instance if we create a template test setup (test discovery configured, test directory setup, example test in place) everybody will write more test.

More often than not what I've seen is that the team ends up maintaining these templates and abstractions with very little engagement, but consistent maintenance to make sure these tools are aligned with the teams latest practices and in changes in the language ecosystem tool chain.

What seems to go unrecognized is that this isn't a technical problem to solve with software. The tools (and hopefully documentation) exists. This is a culture and education problem. Every time I've seen this happen the target audience for these abstracted tools and templates are very intelligent individuals fully capable of using the tools. However, that doesn't mean they want to use the tools. Solving skill gaps is done through education and hands on experience. However, if people don't want to use these tools, or do this kind of work ("I'm not a software engineer, I just write algorithms not test") that's a culture problem, how you handle that will differ per person and organization, but recognize trying to solve these cultural challenges with software will likely on lead to cost and maintenance for your team that is likely a net negative on resource commitment.

Work through cultural challenges through personal engagement and change mechanisms, not with software. Culture Challenges + Software Solution = Technical Debt.

Collecting quotes

2026-08-24T23:19:53.976381+00:00

Be curious. Read widely. Try new things. I think a lot of what people call intelligence boils down to curiosity. – Aaron Swartz

Before we start to write any piece of code, we should understand what that code is supposed to do. Understanding requires thinking, and thinking is hard. - Leslie Lamport

Writing is nature's way of letting you know how sloppy your thinking is. - Dick Guindon

What programming continuously teaches me is that asking the right questions is infinitely more important than writing the code. – Daniel Miessler

Make it work, then make it beautiful, then if you really, really have to, make it fast. 90 percent of the time, if you make it beautiful, it will already be fast. So really, just make it beautiful! - Joe Armstrong

Or put another way, the way to figure out the separation is by doing as much as you can without mutation, and then encapsulating the mutation separately. - Gary Bernhardt

A complex system that works has evolved from a simple system that worked. A complex system built from scratch won’t work. - John Gall (Gall's Law)

Everyone knows that debugging is twice as hard as writing a program in the first place. So if you’re as clever as you can be when you write it, how will you ever debug it? - Brian Kernighan (Kernighan's Law)

Oh really? There’s debate about open-source hardware? I’m going to keep shipping open-source hardware while you all argue about it. – Limor Fried

The best way to get the right answer on the internet is not to ask a question; it's to post the wrong answer. - Ward Cunningham (Cunningham's Law)

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure. - Melvin Conway

You haven’t mastered a tool until you understand when it should not be used. - Kelsey Hightower

When a measure becomes a target, it ceases to be a good measure. - Charles Goodhart (Goodhart's Law)

We must struggle to develop a suspicious nature as well as a lively imagination. - Herbert Leeds and Gerald M. Weinberg

Humans are allergic to change. They love to say, "We've always done it this way." I try to fight that. That's why I have a clock on my wall that runs counter-clockwise. - Rear Admiral Grace Hopper

Adding [human resources] to a late software project makes it later. - Fred Brooks

People are part of the system. The design should match the user's experience, expectations, and mental models. - Principle of Least Astonishment

To really understand a concept, you have to "invent" it yourself in some capacity. Understanding doesn't come from passive content consumption. It is always self-built. It is an active, high-agency, self-directed process of creating and debugging your own mental models. - François Chollet

Provide default values with care

2026-08-24T23:19:53.976359+00:00

Something that comes up from time to time in code review is the use of default arguments for function parameters. Typically the line of reasoning for their inclusion involves making it easier for users of the function signature by requiring them to pass less arguments. I'm not a fan of this line of reasoning for a few reasons:

If the function signature has become so long and convoluted that it's a burden to the caller then the function should probably be refactored. Not only is it a burden for the caller, it's probably difficult to test.
Default values are a leaky abstraction. Clearly the argument is important to the internal mechanics of our function. It also may be important to the caller. What if we decide the default value should change? Is that a major breaking change (yes)? What state is the caller in if we change the default. Does everything keep working for them with the new default, did they come to rely on our leaky abstraction beyond the call boundary and now their code breaks with the change?

Sometimes default argument values make sense. If you're writing an HTTP or ODBC library then a default timeout makes sense. When building applications or libraries that are specific to your company or project be careful. Context changes, people change, time moves forward and what once made sense as a default value will probably change. If it does it's hard to know the full impact.

Some notes and observations from using llms

2026-08-24T23:19:53.976324+00:00

I've been using some form of LLM code assist since GitHub Co-Pilot entered beta. Over the years that's included various GPT, Gemini and Claude versions along with trying different editor and CLI based assistants. How I integrate and use these tools is something that continues to change for personal and professional projects, but I thought it might be time to write down a few thoughts.

This one should be obvious, but in case it's not you need to try different models with different projects and teams to find which is the most efficient in that context. Right now I've have found Claude Sonnet 4 to be the model that provides the code I prefer to maintain, but it helps to have evaluation criteria that you use to check the results of each new model release, and for testing models across different vendors (Anthropic, OpenAI, Google, etc).
Fundamentally these tools can change how you work if you let them, but:
It's particularly important when using these tools to consider how different "modes" (agent, agent + mcp, ask, etc) trade off agency and how that impacts yourself, and your team.
For many of us code is only part of the project, and sometimes not even the hardest part, but consider that the LLM is more than happy to generate pages of code. That can be a burden to maintain, and if operating in agent mode I will wager when the agent runs off and implements a full feature that you don't know it the same way you would if you used the LLM to provide suggestions/ snippets and or you implemented it yourself. There is a cost there, short and long term.
There is a larger team and organization conversation going on right now that you need to engage in. These tools can boost and harm moral, and strong teams are foundational to success short and long term.
I go back and forth between how I use agents depending on the structure of my week. Some weeks I'm spending a lot of time pairing, meeting or doing task that don't provide hours at a time to be head down focused on the code in a flow state. When that happens I'm happy to have tools like LLM agents that I can prompt and check back on, but "with great power comes great responsibility". Don't check in that code without doing a thorough review on your end first (including a very detailed review of any generated tests). Your team will thank you for it, and you won't find people (like myself) leaving frustrated review comments about generated nonsensical code.
LLMs are very verbose, tame this with your prompts if you expect to thoroughly review and integrate their output.
Be careful when allowing LLMs to generate tests. I've heard so many people talk about how great it is they can use these tools to write test for code that doesn't have any, or to have it write test with the new feature they are implementing. That's not wrong, LLMs can generate test, but just like building maintainable systems is a skill writing good test is also a skill. I would wager that the code training datasets are not curated to projects such as AOSA. We would probably disappointed to see where most projects sit on a quality curve for training data sets. I can personally share that time and again LLMs will generate test that erroneously pass. They will remove assertions, write assertions that never fail, or write test that never call the function or DUT. Yes, you can have an LLM generate good test, but it takes work, and be cautious, our test are one of our strongest signal points in a project. Failing to keep their integrity strong will only lead to bugs, and likely create a code base that nobody wants to be responsible for.
If using these tools in a team setting engage with them as a team. Share prompts, agree on a set of minimal instructions. Make sure everybody understands context and default prompts. Set guidelines and expectations for how the team uses the tools and what quality markers are expected to be present and maintained.

There are hundreds of blog post out there about LLMs. If I had to sum up my thoughts: The latest LLM generation is impressive, and can transform how we work, but they can also change the level of agency that individuals exhibit. Use the tool, but don't turn off your brain, and recognize our bias to trust these tools.

Make it Count

2026-08-24T23:19:53.976295+00:00

Code review is part of daily life for many software engineers. It's a great way to learn and share what you know. Code review is the time for a team to take a breath, look at an implementation and navigate changes before the PR merges and becomes part of the teams public responsibility.

Every team does code review different. If you're new to code review, or want https://google.github.io/eng-practices/review/reviewer/ to see a well documented process to the practice maybe take a second to checkout Google's review practices.

While I think code review is critical to maintaining a healthy codebase, project and team, I've also started to notice the trend of "nits" in the last few years. Nits are small comments that don't really add value to the code review. They are largely subjective, and for the most part, don't really help the code.

For doc PRs these are typically grammar or spelling, which should be called out, and ideally corrected using existing tools pre review.

Instead they riddle the review with noise possibly masking more important comments, and typically indicate a lack of understanding the architecture or context of the PR. Alternatively they can be a sign that a reviewer feels like they have to say something, or find something that needs to be changed. While any given PR probably has something that could be improved (depending on the level of pedantry we want to apply) it's important to consider the goal of code review, if the comment truly improves the project/team, or if it's inconsequential. These inconsequential comments don't move the project forward, and in some ways impede the project by burdening the team, and causing unnecessary review loops. Instead as a reviewer consider the teams goals and the PR's goals, and focus on comments that help achieve those goals, benefit the project and grow the team. Make it count.

Agents and Effective Communication

2026-08-24T23:19:53.976262+00:00

Many questions remain around how to effectively produce and support software with AI. Time will help tell what does and doesn’t work.

One trend that I have seen stay consistent is that engineers who communicate well are orders of magnitude more effective with AI.

This makes sense to me. The interface to these AI is natural language, and if one struggled to share a concept with a peer, write a spec, or define an API boundary with another team the same challenges will exist with AI agents.

Effective communication and strong system thinking will define successful software engineering in the future.

Passing thoughts on Software Engineering with Agents

2026-08-24T23:19:53.976217+00:00

Since the end of last year using agents as part of the software development cycle has become a given for many teams. I’ve had the fortune to work adjacent to the ML/AI field for many years prior to my current role, and to use various approaches to automated software development as part of that. I do think that how we produce software has changed, and while I don’t think “code is cheap” (check your token bill), I do think the cost has shifted. I’m capturing a few thoughts about this here to look back on in another year or three.

Feedback loops are tighter.
- This is exposing areas of organizational dysfunction that had a release when the software development process took longer.
- This means everybody is being asked to review more code while still doing their work.
- Even if you trust the AI to code the team still needs the systems concepts and behavior in their head to move the project forward and perform support.
Your company probably doesn’t have the stomach for token spend like the highly public projects on the internet. Plan accordingly.
- I'm waiting for the on call reports of a system down fix looped blocked mid fixed by spend limits with no plan to increase the limit and it being "off hours".
These tools when used effectively can clear the chaff from your backlog.
Use agents to explore what you don’t know.
Lean into “boring” technology for more benefits.
- But also realize many “boring” technologies have unsafe defaults that make building in the agentic era prone to more security issues.
Setup agentic workflows and guidelines early. Structure work so agents have clear goals and project conventions.
- Setup work to keep the engineer in the loop. The machines don’t understand the broader organizational goals and horizons, but the team should.
- What works for one person may not work for another. These tools rely on effective communication.
Use the tools daily to know what does and doesn’t work. They are changing with each release, keep up, and dont get suck in transient practices that are now inefficient.

Parks

2026-08-24T23:19:53.976159+00:00

Since Maris was born I have spent more time in parks than the prior 28 years of my life. We visit the parks in Louisville all the time, and state parks have been destinations for family trips more than a few times, but on our last few trips we stopped by local city parks as stops between our destinations. Somewhere to stretch our legs, enjoy the air, and enjoy being outside sometimes alone and sometimes with other friendly park visitors. These small parks are a gift to their communities, and all of us. Sometimes over looked, but appreciated by many. All of them are different, and I’m thankful that parks exist in their many shapes and sizes from a little local park up to the national. I hope we always support and protect these spaces.